-
Notifications
You must be signed in to change notification settings - Fork 360
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
* delta client impl * delta exporter * Use the earliest available version to build the Delta log * pass listen address to hooks * add stat_object api call to the lakefs lua client * add get_repo to eventually fetch its storage namespace * delta lake exporter example * add error return. remove redundant lines. add godocs for some funcitons * extract get_storage_namespace to the internal lua file. change response of delta_exporter, remove prints, remove old impl * remove comment * go mod tidy * pass aws properties directly to fetchS3Table * json encoding refactor * fix load test * fix tests * fix tests * goimports * add indentation to lua test * remove storage_utils.lua * Rename `ListeningAddress` (and `listeningAddress`) to `ServerAddress`. Lua refactoring Remove unneeded `listeningAddress` member from `DeltaClient` * fix test * lint * pass a delta client to the export_delta function lua-require packages * upgrade aws-sdk-v2 * go mod tidy * use get_storage_prefix from the internal package. Remove it from docs * remove get_storage_namespace and the `get_repo` method from lakeFS client since the storage namespace can be fetched from the global action object * use get_storage_prefix from the internal package * use correct error when returning * comment the delta_log_entry_key_generator function and return a complete key name (including ".json" suffix) * use a writer_client function instead of a storage client when passed to the export_delta_log function * updated version of delta go * severAddress -> lakeFSAddr + comment * change delta client builder API and drop storage type (since always using the S3 gateway). Use regex to validate lakeFS address * rename the lakeFS address field and add clarifying comment
- Loading branch information
1 parent
6c1448c
commit bb38ba0
Showing
26 changed files
with
714 additions
and
237 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,25 @@ | ||
| --[[ | ||
| action_args: | ||
| - repo | ||
| - commit_id | ||
| export_delta_args: | ||
| - table_paths: ["path/to/table1", "path/to/table2", ...] | ||
| - lakefs_key | ||
| - lakefs_secret | ||
| - region | ||
| storage_client: | ||
| - put_object: function(bucket, key, data) | ||
| ]] | ||
| local aws = require("aws") | ||
| local formats = require("formats") | ||
| local delta_export = require("lakefs/catalogexport/delta_exporter") | ||
|
|
||
| local sc = aws.s3_client(args.aws.aws_access_key_id, args.aws.aws_secret_access_key, args.aws.aws_region) | ||
|
|
||
| local delta_client = formats.delta_client(args.lakefs.access_key_id, args.lakefs.secret_access_key, args.aws.aws_region) | ||
| local delta_table_locations = delta_export.export_delta_log(action, args.table_paths, sc.put_object, delta_client) | ||
| for t, loc in pairs(delta_table_locations) do | ||
| print("Delta Lake exported table \"" .. t .. "\"'s location: " .. loc .. "\n") | ||
| end |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.