Skip to content

perf: more elegant solution to mem dump using unsafe and generics#501

Closed
gbotrel wants to merge 4 commits intorefactor/utils_side_effectfrom
perf/points_to_bytes
Closed

perf: more elegant solution to mem dump using unsafe and generics#501
gbotrel wants to merge 4 commits intorefactor/utils_side_effectfrom
perf/points_to_bytes

Conversation

@gbotrel
Copy link
Copy Markdown
Collaborator

@gbotrel gbotrel commented May 3, 2024

  • refactor: expose pedersen basis slices
  • feat: more elegant solution to mem dump using unsafe and generics

Motivation; enable other gnark objects (like groth16.ProvingKey) to be written and read very fast (though not portable). Limit code duplication between curves, fields, proof systems, etc.

  • Adds utils/unsafe with WriteSlice[S ~[]E, E any](w io.Writer, s S) error ... ;
  • kzg.UnsafeToBytes(...) []byte is now WriteDump(w io.Writer) error

🔥 Benchmarks for kzg SRS dump 🔥 --> (I mean using unsafe is a cheat, we just benchmark the non-unsafe method + real IO)

benchmark                                      old ns/op      new ns/op      delta
BenchmarkSerializeSRS/WriteTo-10               1227334792     1200767042     -2.16%
BenchmarkSerializeSRS/WriteRawTo-10            1291239625     1275224792     -1.24%
BenchmarkSerializeSRS/UnsafeToBytes-10         288414938      49002102       -83.01%
BenchmarkDeserializeSRS/UnsafeReadFrom-10      1589721583     1607936042     +1.15%
BenchmarkDeserializeSRS/UnsafeFromBytes-10     283540219      81233306       -71.35%

benchmark                                      old allocs     new allocs     delta
BenchmarkSerializeSRS/WriteTo-10               540            540            +0.00%
BenchmarkSerializeSRS/WriteRawTo-10            542            541            -0.18%
BenchmarkSerializeSRS/UnsafeToBytes-10         522            512            -1.92%
BenchmarkDeserializeSRS/UnsafeReadFrom-10      1046           1047           +0.10%
BenchmarkDeserializeSRS/UnsafeFromBytes-10     1029           1031           +0.19%

benchmark                                      old bytes      new bytes      delta
BenchmarkSerializeSRS/WriteTo-10               2147544696     2147544696     +0.00%
BenchmarkSerializeSRS/WriteRawTo-10            4295028640     4295028632     -0.00%
BenchmarkSerializeSRS/UnsafeToBytes-10         1610821246     67176202       -95.83%
BenchmarkDeserializeSRS/UnsafeReadFrom-10      1627598752     1627598848     +0.00%
BenchmarkDeserializeSRS/UnsafeFromBytes-10     1610820720     1610820744     +0.00%

@gbotrel gbotrel requested review from Tabaie and ivokub May 3, 2024 01:38
@github-actions
Copy link
Copy Markdown

github-actions bot commented May 3, 2024

📦 github.com/consensys/gnark-crypto/kzg

kzg/kzg.go:40:10: cannot use &kzg_bn254.SRS{} (value of type *"github.com/consensys/gnark-crypto/ecc/bn254/kzg".SRS) as SRS value in return statement: *"github.com/consensys/gnark-crypto/ecc/bn254/kzg".SRS does not implement SRS (missing method UnsafeFromBytes)
kzg/kzg.go:42:10: cannot use &kzg_bls12377.SRS{} (value of type *"github.com/consensys/gnark-crypto/ecc/bls12-377/kzg".SRS) as SRS value in return statement: *"github.com/consensys/gnark-crypto/ecc/bls12-377/kzg".SRS does not implement SRS (missing method UnsafeFromBytes)
kzg/kzg.go:44:10: cannot use &kzg_bls12378.SRS{} (value of type *"github.com/consensys/gnark-crypto/ecc/bls12-378/kzg".SRS) as SRS value in return statement: *"github.com/consensys/gnark-crypto/ecc/bls12-378/kzg".SRS does not implement SRS (missing method UnsafeFromBytes)
kzg/kzg.go:46:10: cannot use &kzg_bls12381.SRS{} (value of type *"github.com/consensys/gnark-crypto/ecc/bls12-381/kzg".SRS) as SRS value in return statement: *"github.com/consensys/gnark-crypto/ecc/bls12-381/kzg".SRS does not implement SRS (missing method UnsafeFromBytes)
kzg/kzg.go:48:10: cannot use &kzg_bls24315.SRS{} (value of type *"github.com/consensys/gnark-crypto/ecc/bls24-315/kzg".SRS) as SRS value in return statement: *"github.com/consensys/gnark-crypto/ecc/bls24-315/kzg".SRS does not implement SRS (missing method UnsafeFromBytes)
kzg/kzg.go:50:10: cannot use &kzg_bls24317.SRS{} (value of type *"github.com/consensys/gnark-crypto/ecc/bls24-317/kzg".SRS) as SRS value in return statement: *"github.com/consensys/gnark-crypto/ecc/bls24-317/kzg".SRS does not implement SRS (missing method UnsafeFromBytes)
kzg/kzg.go:52:10: cannot use &kzg_bw6761.SRS{} (value of type *"github.com/consensys/gnark-crypto/ecc/bw6-761/kzg".SRS) as SRS value in return statement: *"github.com/consensys/gnark-crypto/ecc/bw6-761/kzg".SRS does not implement SRS (missing method UnsafeFromBytes)
kzg/kzg.go:54:10: cannot use &kzg_bw6633.SRS{} (value of type *"github.com/consensys/gnark-crypto/ecc/bw6-633/kzg".SRS) as SRS value in return statement: *"github.com/consensys/gnark-crypto/ecc/bw6-633/kzg".SRS does not implement SRS (missing method UnsafeFromBytes)
kzg/kzg.go:56:10: cannot use &kzg_bw6756.SRS{} (value of type *"github.com/consensys/gnark-crypto/ecc/bw6-756/kzg".SRS) as SRS value in return statement: *"github.com/consensys/gnark-crypto/ecc/bw6-756/kzg".SRS does not implement SRS (missing method UnsafeFromBytes)

Copy link
Copy Markdown
Collaborator

@ivokub ivokub left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Imo we can add a simple guard in case the slice to write is empty to have better errors, but I think normally shouldn't happen.

return err
}

data := unsafe.Slice((*byte)(unsafe.Pointer(&s[0])), size*len(s))
Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Add guard in case len(s) == 0.

@gbotrel gbotrel deleted the branch refactor/utils_side_effect May 4, 2024 00:44
@gbotrel gbotrel closed this May 4, 2024
gbotrel added a commit that referenced this pull request May 4, 2024
* refactor: expose pedersen basis slices

* refactor: move test util package into isolated one

* feat: more elegant solution to mem dump using unsafe and generics

* fix: add BinaryDumper interface in kzg root package

* feat: add guard for len == 0
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants