Refstats for internal Ref refactoring

[helins]

Some ref stats for inspiration with the internal refactoring mentioned at today's meeting.

Using a genesis state:

(def s (convex.cvm/state (convex.cvm/ctx)))
(def r (ACell/createPersisted s))
;; r:   {:direct 34839, :embedded 33858, :persisted 33406, :soft 11, :stored 33406, :total 34850}
(def r2 (ACell/createPersisted (.getValue r)))
;; r:   {:direct 33858, :embedded 33858, :persisted 1, :soft 992, :stored 1, :total 34850}
;; r2:  {:direct 33858, :embedded 33858, :persisted 992, :soft 992, :stored 992, :total 34850}

Using a large vector of Longs:

(def v (convex.cell/vector (map convex.cell/long (range 1000000))))
(def r (ACell/createPersisted v))
;; r:   {:direct 1066669, :embedded 1062501, :persisted 1066668, :soft 0, :stored 1066668, :total 1066669}
(def r2 (ACell/createPersisted (.getValue r)))
;; r:   {:direct 1062501, :embedded 1062501, :persisted 1, :soft 4168, :stored 1, :total 1066669}
;; r2:  {:direct 1062501, :embedded 1062501, :persisted 4168, :soft 4168, :stored 4168, :total 1066669}

Using a small vector:

(def v (convex.cell/* [:a :b :c]))
(def r (ACell/createPersisted v))
;; r:   {:direct 4, :embedded 4, :persisted 3, :soft 0, :stored 3, :total 4}
(def r2 (ACell/createPersisted (.getValue r)))
;; r:   {:direct 4, :embedded 4, :persisted 4, :soft 0, :stored 4, :total 4}
;; r2:  {:direct 4, :embedded 4, :persisted 4, :soft 0, :stored 4, :total 4}

[helins]

One consideration is that there are valid-ish reasons for holding to an old cell after it has been persisted. E.g. a cell X is kept and used for some work but at some point it is persisted deep in a collection Y. It's not intuitive that one would have to retrieve the new "version" of X from the new version of Y returned by ACell.createPersisted in order to keep using it. At least it would be cumbersome, especially in multithreaded situations.

But based on today's discussion it doesn't sound like a problem, does it? Using the new version and discarding the old one is supposed to be an optimization that allows soft refs to kick in while improving subsequent writes (requirement for a peer).

[mikera]

Yes, it should be fine to hold onto old refs, as long as you are careful about memory usage limits etc. Basically you only want to keep a (bounded+reasonable) number of direct refs around.

[mikera]

Believe all this is now fixes in recent network / storage refactoring for 0.7.10

[helins]

Behavior looks different but the core issue about orthogonal persistency is still there.
Now I get 0 soft refs with the following (whereas it produces 11 soft refs previously, which was still far from that 3% target we were discussing). Rewriting it again does not change anything in the ref stats:

(.cell.ref.stat (.db.write (.db.write *state*)))

{:embedded 50581,:persisted 49972,:direct 52236,:stored 49972,:soft 0,:total 52236}