[core][bug] Fix the incorrect partition key format generated from the DATE type

[yuanoOo]

fix issue： #773

(Please specify the module before the PR name: [core] ... or [flink] ...)

Purpose

(What is the purpose of the change, or the associated issue)

Tests

(List UT and IT cases to verify this change)

API and Format

(Does this change affect API or storage format)

Documentation

(Does this change introduce a new feature)

[sonarqubecloud]

Kudos, SonarCloud Quality Gate passed!   

0 Bugs
0 Vulnerabilities
0 Security Hotspots
0 Code Smells

No Coverage information
0.0% Duplication

[JingsongLi]

also check Time Type

[JingsongLi]

I've thought about this fix, maybe we can make it more thorough. Like org.apache.paimon.utils.TypeUtils.castFromString, we can have a castToString. Then, the string will be SQL standard string.

But this will bring more serious issues, incompatibility, and we may need an option or an old class to fall back to the old path when we cannot find it. This is complicated.

Is this problem serious now? Can we keep it this way for now?

[yuanoOo]

I've thought about this fix, maybe we can make it more thorough. Like org.apache.paimon.utils.TypeUtils.castFromString, we can have a castToString. Then, the string will be SQL standard string.

But this will bring more serious issues, incompatibility, and we may need an option or an old class to fall back to the old path when we cannot find it. This is complicated.

Is this problem serious now? Can we keep it this way for now?

I think it's a very user experience affecting issue.
In a stream writing paimon scenario, it is very common to cast a field of timestamp type to DATE type in order to write to the corresponding partition.
However, in paimon, when the user uses a partition of type DATE, the partition path becomes a strange unreadable string of numbers, which may be unacceptable. If you want to get the correct partition string, but need to convert timestamp type to date string, which is counterintuitive and sql is tedious.

I think the current addition of support for DATE type partitioning is sufficient.

For the compatibility issue, it will be easy to solve if we can get the version information of paimon. For example, we can determine how to format DATE type partitions based on the version number. I think there are probably few users who use it for the strange performance of the current version of the DATE type partition.

[SteNicholas]

@yuanoOo, the partition key format is user-defined that the format like yyyyMMdd could convert from the date or timestamp functions via users. It doesn't need to use fixed format in Paimon. Meanwhile, solving the compatibility issue via version doesn't mark sense to me because it's cumbersome to maintain the compatibility solution.

[dailai]

I think this issue need to fix. For example, we develop a paimon connector of seatunnel which use java api of paimon also has the same problem.