Cool, well while I'm not really familiar with this code, or algorithm that is the only comments I had.
It appears to me that it should be isomorphic to the original implementation, so if it improves things...

That said I'm happy to give @ptersilie time to weigh in on things, if he feels inclined?

Just to check: this does the right thing on the grammars you care about? If so, you'll be an excellent reviewer and should feel free to merge (I've force pushed a squash).

I've run it on a few grammars I care about, and everything seems to work.
I'd like to do some more testing such as adding a nimbleparse option to dump the state table.
Then run that on my own as well as some of the "Used by" dependents listed on github here https://github.com/softdevteam/grmtools/network/dependents comparing the output with the current output.

Sadly a great portion of the grammars I have laying around don't produce a valid state table,
because they are testing various error conditions, with tool work usurping much of my usage.

So I'll try and do that tomorrow.

I think if it works on the ones you care about, we're probably good to go. I wouldn't object to a PR to add a -d (or whatever) option to nimbleparse to dump the statetable -- it should be straightforward as we already do that for the "bad" cases.

Well, running through things adding the -d flag hasn't quite helped as much as I'd hoped.
The issue is that while the actual state graph stidx's and the like seem to be deterministic,
they are printed from pp_core_states and the like in a non-deterministic order.

As such the below output part of a diff of the -d flag, from 2 runs of nimbeparse both before this patch.

 Stategraph:
 0:   [^ -> . File, {'$'}]
-     'val' -> 3
-     'annotated' -> 4
-     File -> 1
-     'term' -> 6
      D -> 2
+     File -> 1
+     'annotated' -> 4
+     'val' -> 3
      'proof' -> 5
+     'term' -> 6

If I look at that through diff -u st-old.txt st-old2.txt | grep ^[+-] | sort it seems like the state graph is probably unchanged, in that the + lines appear to align with the - ones besides the order of output. But that isn't a very satisfying or thorough test.

Edit: I'll have a short investigation into how hard it would be to make the pp_core_states output deterministic.

So I locally tested with fixed the pretty printing, checked the state tables of my projects and a bunch of projects from github with no hints of any unexpected differences.