experiment: modify candid tooling to preserve source occurrences of blob types when parsing and pretty-printing candid sources

[crusso]

Dfx selectively processes handwritten candid to inline imports and/or service arguments. It does so by first parsing, manipulating then pretty-printing the candid ast. During parsing, blob abbreviations are expanded and thus lost when pretty-printing. This is problematic for Motoko which has the dubious feature of using the candid source syntax to determine whether to import Candid blob/vec nat8 are Motoko Blob/[Nat8]. The former are equivalent Candid types, but inequivalent Motoko types.

This PR preserves the blob abbreviations when parsing allowing backends to selectively elide or retain them and Motoko translations to be consistent before and after simple candid AST manipulation.

This implementation assumes that blob is a keyword and can't be used as a Candid identifier (which is probably not entirely true since it can be accessed using a quoted identifier).

[github-actions]

Name	Max Mem (Kb)	Encode	Decode
blob	4_224	20_466_180 ($\textcolor{red}{0.01\%}$)	12_084_180 ($\textcolor{green}{-0.00\%}$)
btreemap	75_456	4_814_049_311 ($\textcolor{red}{0.00\%}$)	15_381_026_406 ($\textcolor{red}{0.00\%}$)
nns	192	2_331_960 ($\textcolor{red}{2.52\%}$)	14_355_820 ($\textcolor{red}{0.20\%}$)
option_list	576	7_313_201 ($\textcolor{red}{0.03\%}$)	33_889_575 ($\textcolor{red}{0.00\%}$)
text	6_336	28_849_631 ($\textcolor{red}{0.01\%}$)	17_839_969 ($\textcolor{red}{0.00\%}$)
variant_list	128	7_520_738 ($\textcolor{red}{0.02\%}$)	25_530_526 ($\textcolor{red}{0.00\%}$)
vec_int16	16_704	168_592_937 ($\textcolor{red}{0.00\%}$)	1_031_830_254 ($\textcolor{red}{0.00\%}$)

• Parser cost: 18_479_040 ($\textcolor{red}{0.70\%}$)
• Extra args: 3_353_189 ($\textcolor{green}{-0.39\%}$)

Click to see raw report

---------------------------------------------------

Benchmark: blob
  total:
    instructions: 32.55 M (0.01%) (change within noise threshold)
    heap_increase: 66 pages (no change)
    stable_memory_increase: 0 pages (no change)

  1. Encoding (scope):
    instructions: 20.47 M (0.01%) (change within noise threshold)
    heap_increase: 66 pages (no change)
    stable_memory_increase: 0 pages (no change)

  2. Decoding (scope):
    instructions: 12.08 M (-0.00%) (change within noise threshold)
    heap_increase: 0 pages (no change)
    stable_memory_increase: 0 pages (no change)

---------------------------------------------------

Benchmark: text
  total:
    instructions: 46.69 M (0.00%) (change within noise threshold)
    heap_increase: 99 pages (no change)
    stable_memory_increase: 0 pages (no change)

  1. Encoding (scope):
    instructions: 28.85 M (0.01%) (change within noise threshold)
    heap_increase: 99 pages (no change)
    stable_memory_increase: 0 pages (no change)

  2. Decoding (scope):
    instructions: 17.84 M (0.00%) (change within noise threshold)
    heap_increase: 0 pages (no change)
    stable_memory_increase: 0 pages (no change)

---------------------------------------------------

Benchmark: vec_int16
  total:
    instructions: 1.20 B (0.00%) (change within noise threshold)
    heap_increase: 261 pages (no change)
    stable_memory_increase: 0 pages (no change)

  1. Encoding (scope):
    instructions: 168.59 M (0.00%) (change within noise threshold)
    heap_increase: 261 pages (no change)
    stable_memory_increase: 0 pages (no change)

  2. Decoding (scope):
    instructions: 1.03 B (0.00%) (change within noise threshold)
    heap_increase: 0 pages (no change)
    stable_memory_increase: 0 pages (no change)

---------------------------------------------------

Benchmark: btreemap
  total:
    instructions: 20.20 B (0.00%) (change within noise threshold)
    heap_increase: 1179 pages (no change)
    stable_memory_increase: 0 pages (no change)

  1. Encoding (scope):
    instructions: 4.81 B (0.00%) (change within noise threshold)
    heap_increase: 257 pages (no change)
    stable_memory_increase: 0 pages (no change)

  2. Decoding (scope):
    instructions: 15.38 B (0.00%) (change within noise threshold)
    heap_increase: 922 pages (no change)
    stable_memory_increase: 0 pages (no change)

---------------------------------------------------

Benchmark: option_list
  total:
    instructions: 41.20 M (0.01%) (change within noise threshold)
    heap_increase: 9 pages (no change)
    stable_memory_increase: 0 pages (no change)

  1. Encoding (scope):
    instructions: 7.31 M (0.03%) (change within noise threshold)
    heap_increase: 0 pages (no change)
    stable_memory_increase: 0 pages (no change)

  2. Decoding (scope):
    instructions: 33.89 M (0.00%) (change within noise threshold)
    heap_increase: 9 pages (no change)
    stable_memory_increase: 0 pages (no change)

---------------------------------------------------

Benchmark: variant_list
  total:
    instructions: 33.05 M (0.01%) (change within noise threshold)
    heap_increase: 2 pages (no change)
    stable_memory_increase: 0 pages (no change)

  1. Encoding (scope):
    instructions: 7.52 M (0.02%) (change within noise threshold)
    heap_increase: 0 pages (no change)
    stable_memory_increase: 0 pages (no change)

  2. Decoding (scope):
    instructions: 25.53 M (0.00%) (change within noise threshold)
    heap_increase: 2 pages (no change)
    stable_memory_increase: 0 pages (no change)

---------------------------------------------------

Benchmark: nns
  total:
    instructions: 35.92 M (0.60%) (change within noise threshold)
    heap_increase: 3 pages (no change)
    stable_memory_increase: 0 pages (no change)

  0. Parsing (scope):
    instructions: 18.48 M (0.70%) (change within noise threshold)
    heap_increase: 3 pages (no change)
    stable_memory_increase: 0 pages (no change)

  1. Encoding (scope):
    instructions: 2.33 M (regressed by 2.52%)
    heap_increase: 0 pages (no change)
    stable_memory_increase: 0 pages (no change)

  2. Decoding (scope):
    instructions: 14.36 M (0.20%) (change within noise threshold)
    heap_increase: 0 pages (no change)
    stable_memory_increase: 0 pages (no change)

---------------------------------------------------

Benchmark: extra_args
  total:
    instructions: 3.35 M (-0.39%) (change within noise threshold)
    heap_increase: 0 pages (no change)
    stable_memory_increase: 0 pages (no change)

---------------------------------------------------
Successfully persisted results to canbench_results.yml

[successaje]

I’ve encountered a similar issue where there’s a mismatch between Blob and [Nat8] types when dealing with subaccounts in the ledgers. This mismatch seems to happen occasionally and can be quite tricky to debug. a consistent workaround or fix for this would be totally cool.