[flink] Support to set the table option by dynamic global job configu… #2104
Conversation
| * @param context The table factory context. | ||
| * @return The dynamic options of this target table. | ||
| */ | ||
| static Map<String, String> getDynamicTableConfigOptions(DynamicTableFactory.Context context) { |
There was a problem hiding this comment.
Can you also document this in engine/flink page?
| conf.keySet() | ||
| .forEach( | ||
| (key) -> { | ||
| Matcher matcher = pattern.matcher(key); |
There was a problem hiding this comment.
pattern.matcher is very slow, can we just check starts with paimon. first?
There was a problem hiding this comment.
It's only happened in client planner phase and the pattern is not complex
There was a problem hiding this comment.
You can do a little benchmark for these two methods, and I think there may be hundreds options need to be check.
There was a problem hiding this comment.
OpenJDK 64-Bit Server VM 1.8.0_292-b10 on Mac OS X 10.16
Apple M1 Pro
test-pattern: Best/Avg Time(ms) Row Rate(M/s) Per Row(ns) Relative
OPERATORTEST_test-pattern_pattern 12530 / 12779 0.0 41767.0 1.0X
OPERATORTEST_test-pattern_startswith 500 / 538 0.6 1666.2 25.1X
Per row will run for 100 key. so the max cost for 100 key match is 41767 ns. The startsWith is much faster, but I think both of the cost is acceptable, and the startWith check is not flexible when we only want to set for the specific table
| SELECT * FROM T; | ||
|
|
||
| -- set scan.timestamp-millis=1697018249000 for the table default.T in any catalog | ||
| SET 'paimon.*.default.T.scan.timestamp-millis' = '1697018249000'; |
There was a problem hiding this comment.
Can we just support paimon.{key} for all tables?
There was a problem hiding this comment.
for all table may cause unexpected behavior, if we use two table but only want to set option to one of them
There was a problem hiding this comment.
Yes, there is unexpected behavior, but there is also user case to define options for all tables.
There was a problem hiding this comment.
Maybe not scan.timestamp-millis, but some options about execution.
There was a problem hiding this comment.
Yes, there is unexpected behavior, but there is also user case to define options for all tables.
if user want to define for all tables they can set paimon.*.*.*.config_key=value. This means all tables will take this config_key. if user want to set options for table in a database, they can set paimon.*.db.*.config_key=value. They just have to keep in mind that these three part controls the catalog, database, and table scope.
|
Updated, Please take a look again @JingsongLi |
apache#2104) (cherry picked from commit 5c001b4)
…ration
Purpose
Linked issue: close #2037
Tests
API and Format
Documentation