chore: factor out StoreCore
#1240
Conversation
arthurpaulino
force-pushed
the
ap/store-core
branch
3 times, most recently
from
770422c to
6bf83ac
Compare
arthurpaulino
force-pushed
the
ap/store-core
branch
from
6bf83ac to
77c4506
Compare
`StoreCore` is a data structure that can encode data as DAGs of tagged pointers, which can be content-addressed with a custom hasher. The tag type as well as the digest type are generic. To accomplish this, all pointer types are now unified in a single type hierarchy.
arthurpaulino
force-pushed
the
ap/store-core
branch
from
77c4506 to
aed8503
Compare
|
!benchmark --features cuda |
Benchmark for bab22baClick to view benchmark
|
|
|
|
Hmm, that's bad. Hold on |
|
Nvm, it's okay. This is from https://github.com/lurk-lab/lurk-rs/actions/runs/8723674973/job/23932441551: LEM Fibonacci Prove - rc = 100/fib/num-100 (base): 1477.1±6.75ms LEM Fibonacci Prove - rc = 100/fib/num-200 (base): 2.8±0.01s |
| fn default() -> Self { | ||
| Self { | ||
| atom: Default::default(), | ||
| tuple2: Default::default(), | ||
| tuple3: Default::default(), | ||
| tuple4: Default::default(), | ||
| hasher: Default::default(), | ||
| comms: Default::default(), | ||
| dehydrated: Default::default(), | ||
| z_cache: Default::default(), | ||
| inverse_z_cache: Default::default(), | ||
| } | ||
| } |
There was a problem hiding this comment.
I don't think so because it requires further implementations for T and D
| fn hydrate_z_cache_with_ptr_vals(&self, ptrs: &[&IVal]) { | ||
| ptrs.chunks(256).for_each(|chunk| { | ||
| chunk.par_iter().for_each(|ptr| { | ||
| self.hash_ptr_val_unsafe(ptr); | ||
| }); | ||
| }); | ||
| } |
There was a problem hiding this comment.
I think it would be better to have some explicit topological sorting mechanism, like what happens in the current implementation of provenance hashing: https://github.com/lurk-lab/lurk-rs/blob/bab22ba2eb600c8f83c9d30b439084537a0f3b53/src/coroutine/memoset/mod.rs#L727-L746
There was a problem hiding this comment.
We also need to think about this, though: we want to avoid unnecessary hashing, and when we are no longer using all hashes for internal structure, that means there may well be pointers in the store that never need to be hashed. One strategy for proofs may involve ensuring that the store contains no superfluous hashes — so all those it does contain can be blindly transcribed into the trace.
I haven't thought this through completely, other than to note that the complete hydration method may somewhat contradict this approach.
There was a problem hiding this comment.
I think it would be better to have some explicit topological sorting mechanism
That's pretty much what hash_ptr_val does before calling hash_ptr_val_unsafe. So we can just call hash_ptr on specific pointers if we don't want to call hydrate_z_cache
| self.dehydrated.load().len() < 443 | ||
| } else { | ||
| // release mode | ||
| self.dehydrated.load().len() < 2497 |
StoreCoreis a data structure that can encode data as DAGs of tagged pointers, which can be content-addressed with a custom hasher. The tag type as well as the digest type are generic.To accomplish this, all pointer types are now unified in a single type hierarchy.