Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[core] Refactor MetastoreClient methods to simplify catalog #4726

Merged
merged 4 commits into from
Dec 17, 2024

Conversation

JingsongLi
Copy link
Contributor

Purpose

New MetastoreClient API:

/**
 * A metastore client related to a table. All methods of this interface operate on the same specific
 * table.
 */
public interface MetastoreClient extends AutoCloseable {

    void addPartition(LinkedHashMap<String, String> partition) throws Exception;

    void addPartitions(List<LinkedHashMap<String, String>> partitions) throws Exception;

    void dropPartition(LinkedHashMap<String, String> partition) throws Exception;

    void dropPartitions(List<LinkedHashMap<String, String>> partitions) throws Exception;

    void markPartitionDone(LinkedHashMap<String, String> partition) throws Exception;

    default void alterPartition(
            LinkedHashMap<String, String> partition, PartitionStats partitionStats)
            throws Exception {
        throw new UnsupportedOperationException();
    }

    /** Factory to create {@link MetastoreClient}. */
    interface Factory extends Serializable {

        MetastoreClient create();
    }
}

Tests

API and Format

Documentation

});
if (metastoreClient != null && partitions.size() > 0) {
try {
metastoreClient.addPartitions(
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

addPartitions?not dropPartitions?

@@ -64,7 +57,7 @@ public PartitionStatisticsReporter(FileStoreTable table, MetastoreClient client)
this.snapshotManager = table.snapshotManager();
}

public void report(String partition, long modifyTime) throws Exception {
public void report(String partition, long modifyTimeMillis) throws Exception {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

modifyTimeInMillis

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

See System.currentTimeMillis

Partition hivePartition;
List<String> partitionValues = new ArrayList<>(partition.values());

Map<String, String> statistic = new HashMap<>();
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Map<String, String> statistic = new HashMap<>(5);

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For these small collection, we don't need to specific the init size.

Copy link
Contributor

@leaves12138 leaves12138 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1

@JingsongLi JingsongLi merged commit 659cf39 into apache:master Dec 17, 2024
12 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants