[Core]Support async lookup

[neuyilan]

Purpose

Currently, lookup join does not really support asynchronous, and the purpose of this PR is to support asynchronous lookup join.

Tests

IT in LookupJoinITCase.java

API and Format

no

Documentation

no doc

[neuyilan]

@JingsongLi Could you please riview this PR? Thanks

[JingsongLi]

Hi @neuyilan , thanks for your contribution!

But I think there's a very big risk here, can we change it a little bit one by one to make everything in there thread-safe?

[JingsongLi]

Can we just create a lock in LookupFile?

[neuyilan]

Can we just create a lock in LookupFile?

I think is ok, create one lock in LookupFile, the same DataFileMeta file use the same lock to ensure thread safe, is that what you mean?

[JingsongLi]

Yes

[neuyilan]

But I think there's a very big risk here, can we change it a little bit one by one to make everything in there thread-safe?
Hi, @JingsongLi What do you mean change it a little bit one by one?

In the current design, FileStoreLookupFunction is non thread safe. This class is the entry point for lookup, so it is necessary to ensure the thread safety of FileStoreLookupFunction.
The current design mainly focuses on the following points:

1. Ensure thread safety when initializing FileStoreLookupFunction;
2. The implementation of two lookup caches, namely in LookupLevels and RocksDBState, ensures thread safety.
3. Since both lookup serialization and deserialization use the same object, this PR also ensures thread safety when serializing and deserializing lookup keys and values.

[JingsongLi]

But I think there's a very big risk here, can we change it a little bit one by one to make everything in there thread-safe?
Hi, @JingsongLi What do you mean change it a little bit one by one?

In the current design, FileStoreLookupFunction is non thread safe. This class is the entry point for lookup, so it is necessary to ensure the thread safety of FileStoreLookupFunction. The current design mainly focuses on the following points:

1. Ensure thread safety when initializing FileStoreLookupFunction;
2. The implementation of two lookup caches, namely in LookupLevels and RocksDBState, ensures thread safety.
3. Since both lookup serialization and deserialization use the same object, this PR also ensures thread safety when serializing and deserializing lookup keys and values.

Thanks for your reply, can you split the PR to multiple PRs? Ensure that every change is thoroughly discussed, otherwise there are too many things to discuss in a large PR, making it difficult to review.

[neuyilan]

But I think there's a very big risk here, can we change it a little bit one by one to make everything in there thread-safe?
Hi, @JingsongLi What do you mean change it a little bit one by one?

In the current design, FileStoreLookupFunction is non thread safe. This class is the entry point for lookup, so it is necessary to ensure the thread safety of FileStoreLookupFunction. The current design mainly focuses on the following points:

1. Ensure thread safety when initializing FileStoreLookupFunction;
2. The implementation of two lookup caches, namely in LookupLevels and RocksDBState, ensures thread safety.
3. Since both lookup serialization and deserialization use the same object, this PR also ensures thread safety when serializing and deserializing lookup keys and values.

Thanks for your reply, can you split the PR to multiple PRs? Ensure that every change is thoroughly discussed, otherwise there are too many things to discuss in a large PR, making it difficult to review.

Got it, I will do this.

[neuyilan]

I will close this PR, and submit new PRs
step1 PR: #4423 support async lookup in hash cache store.