Delta Lake Catalog Exporter

[Jonathan-Rosenberg]

Closes #6965

Changes

1. Change the encoding/json lua library to accept an optional options table. This is in order to pass an indentation if needed.
2. Implement a delta client for lua which can get a delta lake table.
3. Pass listeningAddress variable to the actions service. This is to allow hooks to communicate back to the running lakeFS server if they run on the same host, e.g. lua hooks.
4. Implement the delta_exporter lua function that allows configuring a delta table exporting hook.
5. Extend the lakeFS lua client with stat_object and get_repo operations.

NOTICE 1

This PR depends on the merging of treeverse/delta-go#2.

NOTICE 2

Documentation will be added in a different PR.

How was this tested?

Manually tested on the following Delta Lake log constructs:

1. A delta log with no checkpoints (i.e. starts with entry 000000000000000000.json).
2. A delta log with a checkpoint and past logs (e.g. 000000000000000004.json and all other json entries up to and including checkpoint 00000000000000000010.checkpoint.parquet, and entry 00000000000000000010.json)
3. A delta log with a checkpoint and no past logs (e.g. starting from 00000000000000000010.checkpoint.parquet, followed by a number of log entries).

[Isan-Rivkin]

Not fully in the context yet but I can't help not wonder already.
Is there a reason to depend on passing lakeFS credentials here?

Until now we worked with the policy that lua lakeFS client is using the context of the logged in user in the context during runtime to authenticate with lakeFS api.

[Jonathan-Rosenberg]

As mentioned F2F, this is due to the need to supply the delta-go package's client AWS key and secret to communicate with the S3 gateway.
On top of that, the administrator might configure a user that has permission to read the Delta table from its location such that every commit will export it even if the user that committed it doesn't have the right permissions.

[Isan-Rivkin]

100% 👍
Should document (in general I know not in this PR, what lakeFS permissions are required)
The technical problem reusing same technique as lakeFS lua client is because of the creds provider that expects credentials. 😢

CredsProvider: aws.CredentialsProviderFunc(func(ctx context.Context) (aws.Credentials, error) {

[Isan-Rivkin]

Why is the sort function needed here, don't see its being used? If it's not needed let's remove it.
If you prefer to keep it then no need to check for nil:

Suggested change

	local function sortedKeys(query, sortFunction)
	local keys, len = {}, 0
	for k,_ in pairs(query) do
	len = len + 1
	keys[len] = k
	end
	if sortFunction ~= nil then
	table.sort(keys, sortFunction)
	else
	table.sort(keys)
	end
	return keys
	end
	local function sortedKeys(query, sortFunction)
	local sort = sortFunction or
	local keys, len = {}, 0
	for k,_ in pairs(query) do
	len = len + 1
	keys[len] = k
	end
	table.sort(keys, sortFunction)
	return keys
	end

[Jonathan-Rosenberg]

It's used in delta_exporter.
Not sure I understand the suggested change, but I do agree that the nil check is redundant. 🙏

[Isan-Rivkin]

I meant specifically the sortFunction argument in the function, I don't see it being used anywhere?
If you prefer to leave it regardless, please add a comment on the sortFunction expected signature otherwise it'll be hard to use it without going into the implementation of sortedKeys

[Jonathan-Rosenberg]

sortFunction is passed to table.sort to apply the sorting logic (otherwise it uses default lexicographic sort). I don't need to pass, but since it's a util function, the API should suggest a way to sort keys.
I'll add a comment.

[Isan-Rivkin]

Review WIP - if you prefer all at once LMK 🙏

[Isan-Rivkin]

General review comment - what is the plan about tests?

1. Reference to glue and symlink exporter test integration in esti - can we add a test here?

2. Lua package unit testing - see reference hive_partition_pager.lua test

[github-actions]

♻️ PR Preview 2b9f8a2 has been successfully destroyed since this PR has been closed.

_{🤖 By surge-preview}

[Jonathan-Rosenberg]

@Isan-Rivkin, tests will be added on a different PR.
Thank you 🙏

[Isan-Rivkin]

LGTM!
I left 2 comments un-resolved as reminders incase it's missed - feel free resolve.
Thank you for taking the time and addressing the comments!
Can't wait to export to Delta 🕺