Segfault during traversal of ETS table when compressed (OTP 27)

[zachallaun]

Describe the bug
I contribute to Lexical, an Elixir language server, and we've been experiencing crashes on OTP 27: lexical-lsp/lexical#788

We ultimately found that either a segmentation fault or a "no next heap size found" error was occurring when traversing an ETS table with ets:foldl/3, though I later determined that the crash occurs during any full traversal, e.g. ets:tab2list/1 or ets:tab2file/2.

The table was created with table options [named_table, ordered_set, compressed] and I found that removing compressed fixed the issue. Here's the PR: lexical-lsp/lexical#796

To Reproduce
Unfortunately, I haven't been able to create a minimal, out-of-project reproduction, nor one that doesn't require Elixir. Currently, assuming you have Elixir 1.17.2 and OTP 27 installed, here's the best reproduction I have:

1. Clone Lexical

   $ git clone https://github.com/lexical-lsp/lexical
   $ cd lexical
   $ mix deps.get

2. Save this as repro.exs:

   use Lexical.Server.IEx.Helpers
   
   entries =
     project()
     |> Search.Indexer.create_index()
     |> elem(1)
     |> Enum.flat_map(&Search.Store.Backends.Ets.Schemas.V3.to_rows/1)
   
   table = :ets.new(:table, [:compressed])
   
   :ets.insert(table, entries)
   
   :ets.tab2list(table)

3. Run the script:

   $ mix run repro.exs
   # ... various unrelated/expected warnings ...
   [1]    183109 segmentation fault  mix run repro.exs

There are a few other things I tried to simplify the reproduction, but none of them worked:

• Save entries to a file using :erlang.term_to_binary(entries, [:deterministic]) (and without deterministic) and read them in a clean iex. Inserting them this way and running tab2list did not trigger the fault.
• Iterate through entries, logging each entry and inserting them one at a time, trying :ets.tab2list(table) after each insert. This would allow me to see the last entry inserted prior to the segfault, but it ended up being a different entry every time.

If anyone has other suggestions, I'm happy to do whatever I can to make this easier to debug.

Expected behavior
No segfault. 🙂

Affected versions
27.0 and 27.0.1. Earlier versions appear unaffected.

Additional context
The vast majority of the time, the crash occurs as a segmentation fault, but occasionally crashed with a "no next heap size found" error and produced an erl_crash.dump. That dump can be found here.

[jhogberg]

Thanks for your report! There's a problem in term size calculation for compressed ETS tables, I'm trying to narrow it down. :-)

[zachallaun]

Thank you very much for looking into it!

[jhogberg]

I think I've fixed it with #8683, can you try it out on your end?

[zachallaun]

Awesome! I'll try to give it a go later tonight, but it may be tomorrow.

[zachallaun]

Got your branch built -- no crash! Thanks for getting to this so quickly.

[zachallaun]

This appears fixed in 27.1. Thanks again, @jhogberg!