Datatype deduplication 2: switch to re_arrow2

[teh-cmc]

Grunt work to switch to re_arrow2 and all the breaking changes that come with it.

• Fixes Fork arrow2 and get rid of polars #4789

---

Part of the tiny datatype deduplication PR series:

• Datatype deduplication 1: remove polars #4880
• Datatype deduplication 2: switch to re_arrow2 #4883

Checklist

• I have read and agree to Contributor Guide and the Code of Conduct
• I've included a screenshot or gif (if applicable)
• I have tested the web demo (if applicable):
  • Using newly built examples: app.rerun.io
  • Using examples from latest main build: app.rerun.io
  • Using full set of examples from nightly build: app.rerun.io
• The PR title and labels are set such as to maximize their usefulness for the next release's CHANGELOG

• PR Build Summary
• Docs preview
• Examples preview
• Recent benchmark results
• Wasm size tracking

[github-actions]

Size changes

Name	main	4883/merge	Change
plots.rrd	204.80 kiB	194.56 kiB	-5.00%

[emilk]

yay!

[teh-cmc]

This gives us a 20% memory usage improvement in the best case as is (instead of 50% back in the day, because we've optimized things since then, most notably we don't have any arrow extensions anymore).

We've basically reached the point where DataTypes have no heap overhead anymore, now the issue is that they have massive stack overhead (std::mem::size_of::<DataType>() = 48!) which turns into heap overhead everytime we slice data and get a dyn Array out.

Trying to remove the stack overhead from arrow2 is extremely painful, it breaks basically every line of code; and will be nullified when we switch to arrow-rs anyhow.
The only long-term viable solution I can think of is to implement our own slicing mechanism that isn't Box<dyn Array>, which has been the source of most of our compute and space problems since the beginning.

• We need a better data slicing mechanism than Box<dyn Array> #4884

[emilk]

Trying to remove the stack overhead from arrow2 is extremely painful, it breaks basically every line of code; and will be nullified when we switch to arrow-rs anyhow. The only long-term viable solution I can think of is to implement our own slicing mechanism that isn't Box<dyn Array>, which has been the source of most of our compute and space problems since the beginning.

Isn't the long-term solution to switch to arrow-rs?

[teh-cmc]

Trying to remove the stack overhead from arrow2 is extremely painful, it breaks basically every line of code; and will be nullified when we switch to arrow-rs anyhow. The only long-term viable solution I can think of is to implement our own slicing mechanism that isn't Box<dyn Array>, which has been the source of most of our compute and space problems since the beginning.

Isn't the long-term solution to switch to arrow-rs?

I'd say that switching to arrow-rs is the medium-term solution even, and should carry us a long-ish way.

Specifically, arrow-rs..:

• ..works with Arc<dyn Array> instead of Box<dyn Array>, so no costly virtual clones.
• ..makes sure to arc-ify DataType's internal fields to amortize heap costs (so same as what we did just now).
• ..reduced the stack size of DataType from 48 to 24 bytes.

But, there is still a lot of overhead when it comes to slicing data, e.g. a u32 slice is still std::mem::size_of::<PrimitiveArray<UInt32Type>>() = 96 bytes per cell...