Lex refactor needs

[bnewbold]

Cross-reference to main upstream (atproto) branches: bluesky-social/atproto#658

Things that will require new lexgen feature support:

• type="cid-link", encodes as an object like { "$link": "<cid>" }. For binary CIDs, aka "Link" in IPLD data model)
• type="bytes", encodes as an object like { "$bytes": "<base64-encoded>" } for embedded bytes (which specific base64 config?)
• no image/video/audio, just type="blob" now. blob gains a size field (integer, size in bytes)
• type="number" changed to type="float" (though not yet used, and might change how that works)
• string types, which will work like type="string" format="at-uri"

The string types are not strictly needed to get lexgen to run, but we probably might as well wire up placeholder validation for functions in codegen code. The formats are at least: handle, nsid, at-identifier (handle or DID), uri (generic), datetime, at-uri, did, cid. I can port over the validation functions and their tests. I guess we haven't actually implemented the other validators yet, like maxLength. Maybe we punt on validation until a second pass (separate PR).

There are some semantics/benavioral changes, like idempotency of batch mutations and deleteRecord not failing if record already does not exist. I don't think the existing golang PDS is complete enough to care about those.

Once lexgen runs successfully, presumably there will be a bunch of broken stuff that we'll just need to get to compile.

cc: @whyrusleeping

[whyrusleeping]

can we close this now?

[bnewbold]

The string formatting stuff hasn't been touched yet, AFIAK. Might factor in to an iteration of lexgen? Also things like maxLength (both strings and arbitrary arrays), max sizes for blobs (in bytes), max grapheme lengths.

We could create a new issue to track that specific work to be done, and could be copied from the details here. Don't want to lose what is here without copying it out.

[bnewbold]

closing this old/stale issue; lex refactor was forever ago.

#449 is a PR with some partial/draft work on incorporating indigo/atproto/syntax package in to lexgen.

"grapheme length validation" will probably end up in indigo/atproto/data or indigo/atproto/lexicon.