[core]: support rename primary key columns and bucket key columns

[zhongyujiang]

Purpose

Support renaming primary keys and bucket keys that are not partition keys.

Tests

Added schema evolution tests and Spark e2e tests

API and Format

Not affected

Documentation

[zhongyujiang]

cc @JingsongLi @FangYongs Could you please review this? Thanks

[JingsongLi]

CC @tsreaper

[zhongyujiang]

Gentle ping @JingsongLi @tsreaper, can you help review this when you have time? This can be useful to us, thanks!

[JingsongLi]

Hi @zhongyujiang , can you rebase master?

Can this support when table is empty?

[zhongyujiang]

Hi @zhongyujiang , can you rebase master?

Rebased.

Can this support when table is empty?

I guess this is a typo, what you mean is 'non-empty'?
It appears that the bucket key and primary key are also projected based on the column name mapping, so renaming them does not affect the reading of old data. For this, I have added an end-to-end test testRenamePrimaryKey .

[zhongyujiang]

Can this support when table is empty?

I guess this is a typo, what you mean is 'non-empty'?
It appears that the bucket key and primary key are also projected based on the column name mapping, so renaming them does not affect the reading of old data. For this, I have added an end-to-end test testRenamePrimaryKey .

org.apache.paimon.schema.SchemaEvolutionUtil#createDataProjection

I have investigate more to confirm this.
Before reading the data files, the Reader maps the primary key (and other value columns) to the actual columns in the data files. Therefore, I believe that renaming the primary key should not affect the reading of historical data, as long as the partition columns are not renamed. This is because we rely on them when calculating the paths.
The relevant code can be found in org.apache.paimon.schema.SchemaEvolutionUtil#createDataProjection.

[zhongyujiang]

This refers to the actual column projection used when reading data files during the debugging process.

This refers to data written before rename, the projection columns are _KEY_a and a:

This refers to data written after rename, the projection columns are _KEY_a_ and a_:

[zhongyujiang]

The CI failure seems caused by flaky tests, unreleated to this, I've filed a issue on that: #3908

[JingsongLi]

Hi @zhongyujiang , can you rebase master?

Rebased.

Can this support when table is empty?

I guess this is a typo, what you mean is 'non-empty'? It appears that the bucket key and primary key are also projected based on the column name mapping, so renaming them does not affect the reading of old data. For this, I have added an end-to-end test testRenamePrimaryKey .

the bucket key and primary key are also projected based on the column name mapping.

It is true, but this is very dangerous and may lead to many bugs.

[zhongyujiang]

It is true, but this is very dangerous and may lead to many bugs.

Hi @JingsongLi Could you share your specific concerns? From what I've observed, renaming a primary key doesn't seem to differ from renaming other keys; it appears to be safe.

I believe renaming the primary key is a necessary part of full schema evolution, and we have such a requirement ourselves. So if you have any concerns, I would be more than happy to investigate further and try to address them. Thank you!

[JingsongLi]

re-open to trigger tests.

[JingsongLi]

+1