Unity table registration docs: table metadata #7531
Conversation
Jonathan-Rosenberg
added
exclude-changelog
|
😭 Deploy PR Preview a274046 failed. Build logs
🤖 By surge-preview |
E2E Test Results - DynamoDB Local - Local Block Adapter |
| @@ -741,7 +743,7 @@ Parameters: | |||
|
|
|||
| - `action(table)`: The global action table | |||
| - `table_descriptors_path(string)`: The path under which the table descriptors of the provided `table_paths` reside. | |||
| - `delta_table_paths(table)`: Table names to physical paths mapping (e.g. `{ table1 = "s3://mybucket/mytable1", table2 = "s3://mybucket/mytable2" }`) | |||
| - `delta_table_details(table)`: Table names to physical paths mapping and table metadata (e.g. `{table1 = {path = "s3://mybucket/mytable1", metadata = {id = "table_1_id", name = "table1", ...}}, table2 = {path = "s3://mybucket/mytable2", metadata = {id = "table_2_id", name = "table2", ...}}}`.) | |||
There was a problem hiding this comment.
I hope I don't understand this: right now it looks like we broke backwards compatibility of Lua scripts earlier, and now we're documenting it?
There was a problem hiding this comment.
You understand correctly.
The change was made at #7527.
What would you suggest on doing?
There was a problem hiding this comment.
In a minor version bump, I would expect the older form (string that is a path) to remain valid, maybe deprecated, and the new form also to be allowed. Once that happened documentation would pretty much solve itself, something like:
| - `delta_table_details(table)`: Table names to physical paths mapping and table metadata (e.g. `{table1 = {path = "s3://mybucket/mytable1", metadata = {id = "table_1_id", name = "table1", ...}}, table2 = {path = "s3://mybucket/mytable2", metadata = {id = "table_2_id", name = "table2", ...}}}`.) | |
| - `delta_table_details(table)`: Map of table names to physical paths as a string, or to physical paths and table metadata as a JSON object (e.g. `{table1 = {path = "s3://mybucket/mytable1", metadata = {id = "table_1_id", name = "table1", ...}}, table2 = "s3://mybucket/mytable2"}}`.) |
It might be water under the bridge, particularly if this feature is new enough and has not had many users.
There was a problem hiding this comment.
It might be water under the bridge, particularly if this feature is new enough and has not had many users.
I'm very positive that this is the case. If anything, the usage of this script is mainly as part of the Unity Catalog registration flow, and it won't be necessary to change the existing code (not breaking the flow of using the output of the delta exporter)
Jonathan-Rosenberg
added
include-changelog
No description provided.