Docs: Glue Exporter and related #6823

Isan-Rivkin · 2023-10-19T15:06:45Z

Will merge when I have 2 human approvers ✅ ✅

What is covered in the docs

Lua Library Spec
Data Catalogs export high level
AWS Glue integration with Catalog Exports guide

lakeFS User perspective POV

A User should read the AWS Glue integration to achieve their goal.
From time to time they should jump to the General explanation and Library Spec to get deeper understand to some open questions they might have.

github-actions · 2023-10-19T15:07:53Z

♻️ PR Preview a0c0cc9 has been successfully destroyed since this PR has been closed.

_{🤖 By surge-preview}

Co-authored-by: Yoni <[email protected]>

arielshaqed

Thanks!

This is really neat. I found the use of the passive voice somewhat difficult to read, and I suggest using an active voice in several places. You are of course free to disregard this, or alternatively to find a few other places that continue to use the passive voice.

docs/howto/catalog_exports.md

arielshaqed · 2023-10-24T15:13:09Z

docs/howto/catalog_exports.md

+
+#### Hive tables
+
+Hive metadata server tables are essentially just a set of objects that share a prefix, with no table metadata stored on the object store.  You need to configure prefix, partitions, and schema.


IIRC we already define "hive tables" in our docs elsewhere. Can we unify these definitions? (Fine to open an issue to do this in a future PR)

Issue: #6863

You're right but the current definition is so slim 1 sentence and an example so I prefer not to unify this currently since it's really a per-case definition.
I'll open an issue because I prefer to have a more mature and organized definition when doing that and not ad-hoc in this PR.

arielshaqed · 2023-10-24T15:21:06Z

docs/howto/hooks/lua.md

+
+### `aws/glue.get_table(database, table [, catalog_id)`
+
+Get Table from Glue Catalog.


Capitalization seems different than elsewhere.

arielshaqed · 2023-10-24T15:21:15Z

docs/howto/hooks/lua.md

@@ -142,6 +142,55 @@ or:

 Deletes all objects under the given prefix

+### `aws/glue`
+
+AWS Glue Client.


Capitalization seems different than elsewhere.

docs/integrations/glue_metastore.md

Co-authored-by: Ariel Shaqed (Scolnicov) <[email protected]>

johnnyaug

I think the two new pages should be merged to one, under the "How To" category.
It should be a guide teaching the user how to use lakeFS together with Glue Data Catalog.
My reasoning is that it's not really an integration but more of a cookbook.

(we can add a page in the integration section that will only contain a link to the guide).

I'm leaving my comments on the "glue_hive_metastore.md" file, since I think it's the one we should keep (and like I said, IMO should move to be under How-To).

docs/integrations/glue_metastore.md

johnnyaug · 2023-10-25T12:48:56Z

docs/integrations/glue_metastore.md

+## About Glue Metastore
+
+[AWS Glue](https://docs.aws.amazon.com/glue/latest/dg/tables-described.html) has a metastore that can store metadata related to Hive and other services (such as Spark and Trino). It has metadata such as the location of the table, information about columns, partitions and much more.


Not needed, IMO

johnnyaug · 2023-10-25T13:20:41Z

docs/integrations/glue_metastore.md

+## Support in lakeFS
+
+The integration between Glue and lakeFS is based on [Data Catalog Exports]({% link howto/catalog_exports.md %}).
+
+### What is supported 
+
+- Creating a unique table in Glue Catalog per lakeFS repository / ref / commit. 
+- No data copying is required, the table location is a path to a symlinks structure in S3 based on Hive's [SymlinkTextInputFormat](https://svn.apache.org/repos/infra/websites/production/hive/content/javadocs/r2.1.1/api/org/apache/hadoop/hive/ql/io/SymlinkTextInputFormat.html) and the [table partitions](https://docs.aws.amazon.com/glue/latest/dg/tables-described.html#tables-partition) are maintained.
+- Tables are described via [Hive format in `_lakefs_tables/<my_table>.yaml`]({% link howto/catalog_exports.md %}#hive-tables).
+- Currently the data query in Glue metastore is Read-Only operation and mutating data requires writting to lakeFS and letting the export hook run.
+
+### How it works 
+
+Based on event lakeFS events such as `post-commit` an Action will run a script that will create Symlink structures in S3 and then will register a table in Glue.
+The Table data location will be the generated Symlinks root path.
+
+There are 4 key pieces:
+
+1. Table description at `_lakefs_tables/<your-table>.yaml`
+2. Lua script that will do the export using [symlink_exporter]({% link howto/hooks/lua.md %}#lakefscatalogexportsymlink_exporter) and [glue_exporter]({% link howto/hooks/lua.md %}#lakefscatalogexportglue_exporter) packages.
+3. [Action Lua Hook]({% link howto/catalog_exports.md %}#running-an-exporter) to execute the lua hook.
+4. Write some lakeFS table data ([Spark]({% link integrations/spark.md %}), CSV, etc)
+
+To learn more check [Data Catalog Exports]({% link howto/catalog_exports.md %}).


Suggested change

## Support in lakeFS

The integration between Glue and lakeFS is based on [Data Catalog Exports]({% link howto/catalog_exports.md %}).

### What is supported

- Creating a unique table in Glue Catalog per lakeFS repository / ref / commit.

- No data copying is required, the table location is a path to a symlinks structure in S3 based on Hive's [SymlinkTextInputFormat](https://svn.apache.org/repos/infra/websites/production/hive/content/javadocs/r2.1.1/api/org/apache/hadoop/hive/ql/io/SymlinkTextInputFormat.html) and the [table partitions](https://docs.aws.amazon.com/glue/latest/dg/tables-described.html#tables-partition) are maintained.

- Tables are described via [Hive format in `_lakefs_tables/<my_table>.yaml`]({% link howto/catalog_exports.md %}#hive-tables).

- Currently the data query in Glue metastore is Read-Only operation and mutating data requires writting to lakeFS and letting the export hook run.

### How it works

Based on event lakeFS events such as `post-commit` an Action will run a script that will create Symlink structures in S3 and then will register a table in Glue.

The Table data location will be the generated Symlinks root path.

There are 4 key pieces:

1. Table description at `_lakefs_tables/<your-table>.yaml`

2. Lua script that will do the export using [symlink_exporter]({% link howto/hooks/lua.md %}#lakefscatalogexportsymlink_exporter) and [glue_exporter]({% link howto/hooks/lua.md %}#lakefscatalogexportglue_exporter) packages.

3. [Action Lua Hook]({% link howto/catalog_exports.md %}#running-an-exporter) to execute the lua hook.

4. Write some lakeFS table data ([Spark]({% link integrations/spark.md %}), CSV, etc)

To learn more check [Data Catalog Exports]({% link howto/catalog_exports.md %}).

This guide will show you how to use lakeFS with the Glue Data Catalog.

You'll be able to query your lakeFS data by specifying the branch, ref, or commit in your SQL query.

Currently, only read operations are supported on the tables.

You will set up the automation required to work with lakeFS on top of the Glue Data Catalog, including:

1. Create a table descriptor under `_lakefs_tables/<your-table>.yaml`. This will represent your table schema.

2. Write an exporter script that will:

* Mirror your branch's state into symlink files readable by Athena

* Export the table descriptors from your branch to the Glue Catalog.

3. Set up lakeFS hooks that will run the above script when specific events occur.

Thanks! That's an awesome suggestion, did that with minor changes + added links.

docs/integrations/glue_metastore.md

johnnyaug · 2023-10-25T13:30:56Z

docs/integrations/glue_metastore.md

+
+### Write some table data
+
+Add some table data. 


I suggest to keep this section short, even a one-liner if that's possible. It's not in the scope of the article, it's more like a means to an end.
It can be something like "Insert data under the table path, using your preferred method. For example, here is how to write data using Spark: ..."

The commit bit doesn't need to be in the code, you can simply say "then perform a commit in lakeFS".

@itaiad200 @arielshaqed opinions?

@johnnyaug is right that this doesn't belong in "integrations". Unfortunately if you remove this text, it means you've just been volunteered to write the blog post / tech note.

@arielshaqed so setting volunteering aside, you agree that this step by step tutorial on how to write the data itself should be removed to a general guidance? :)

@johnnyaug I took your advice and removed 99% of the content from there!

johnnyaug · 2023-10-25T13:35:50Z

docs/integrations/glue_metastore.md

+### Add Glue Exporter
+
+The current step requires 2 things: 
+1. Lua script.
+2. Action to trigger the script.
+
+#### Lua packages (Exporters)
+
+- [symlink_exporter]({% link howto/hooks/lua.md %}#lakefscatalogexportsymlink_exporter)
+- [glue_exporter]({% link howto/hooks/lua.md %}#lakefscatalogexportglue_exporter)
+
+#### Exporter Script 
+
+##### 1. Create Lua script:
+
+For the simple strategy of creating a glue table per repo / branch / commit we can simply copy-paste the following script and re-use it.
+Upload the script to `scripts/animals_exporter.lua` (could be any path).


Suggested change

### Add Glue Exporter

The current step requires 2 things:

1. Lua script.

2. Action to trigger the script.

#### Lua packages (Exporters)

- [symlink_exporter]({% link howto/hooks/lua.md %}#lakefscatalogexportsymlink_exporter)

- [glue_exporter]({% link howto/hooks/lua.md %}#lakefscatalogexportglue_exporter)

#### Exporter Script

##### 1. Create Lua script:

For the simple strategy of creating a glue table per repo / branch / commit we can simply copy-paste the following script and re-use it.

Upload the script to `scripts/animals_exporter.lua` (could be any path).

### The exporter script

Upload the following script to your main branch under `scripts/animals_exporter.lua` (or a path of your choice).

like it! but with a little twist I kept the libraries reference.

johnnyaug · 2023-10-25T13:37:03Z

docs/integrations/glue_metastore.md

+local res = glue_exporter.export_glue(glue, db, table_path, table_input, action, {debug=true})
+```
+
+##### 2. Configure Action Hooks:


Suggested change

##### 2. Configure Action Hooks:

### Configure Action Hooks

Co-authored-by: Yoni <[email protected]>

arielshaqed

Yoni is right, as always.

StillLGTM: I liked it before, and I am sure I will like it even more with all of Yoni's comments!

arielshaqed · 2023-10-26T08:10:59Z

docs/integrations/glue_metastore.md

+
+### Write some table data
+
+Add some table data. 


@johnnyaug is right that this doesn't belong in "integrations". Unfortunately if you remove this text, it means you've just been volunteered to write the blog post / tech note.

Isan-Rivkin · 2023-10-26T09:23:30Z

@johnnyaug thanks for the great comments I took all of them!
One thing I wouldn't want to change is the merge of the 2 documents.
The idea is that Catalog Exports is a high level concept and we are going to add more exporters, there's no reason for it to live in 1 document that's even more distracting.

johnnyaug

Awesome, thank you for considering my comments!

glue client lua docs

c1cc898

Isan-Rivkin self-assigned this Oct 19, 2023

Isan-Rivkin added export-hooks include-changelog PR description should be included in next release changelog docs Improvements or additions to documentation labels Oct 19, 2023

Isan-Rivkin changed the title ~~[Docs]: Glue Exporter and related~~ Docs: Glue Exporter and related Oct 19, 2023

Isan-Rivkin added 23 commits October 19, 2023 18:40

symlink exporter docs

15b7545

added export hooks full docs

ffb4f3d

fixes

82be2e9

fixes

02875d3

fixes

94f8225

tab fixes

2b401b9

indent

d0098c8

update glue exporter

f5bef53

initial tutorial

c91bd6b

Added docs

cad6acc

fix link

8351cae

update docs

3d5cb91

fix broken anchors

3776994

fix redirects

4d6b01b

fix docs

0e0bc71

WIP

d09808d

pending

ad41921

added table

3a8638b

indent

2d65f56

fix link

bc062be

Add imag

296f31f

docs

34b6699

update location

1399999

Isan-Rivkin and others added 4 commits October 24, 2023 09:59

Update docs/howto/catalog_exports.md

deab3f2

Co-authored-by: Yoni <[email protected]>

Update docs/howto/catalog_exports.md

0c896f4

Co-authored-by: Yoni <[email protected]>

Update docs/howto/catalog_exports.md

232b781

Co-authored-by: Yoni <[email protected]>

fix partial comments

ac70832

Isan-Rivkin requested a review from johnnyaug October 24, 2023 07:07

arielshaqed approved these changes Oct 24, 2023

View reviewed changes

Isan-Rivkin and others added 9 commits October 25, 2023 10:05

Update docs/integrations/glue_metastore.md

d6f0e31

Co-authored-by: Ariel Shaqed (Scolnicov) <[email protected]>

Update docs/integrations/glue_metastore.md

a38480a

Co-authored-by: Ariel Shaqed (Scolnicov) <[email protected]>

Update docs/integrations/glue_metastore.md

3a5788c

Co-authored-by: Ariel Shaqed (Scolnicov) <[email protected]>

Update docs/howto/catalog_exports.md

0ae1d76

Co-authored-by: Ariel Shaqed (Scolnicov) <[email protected]>

Update docs/howto/catalog_exports.md

d000484

Co-authored-by: Ariel Shaqed (Scolnicov) <[email protected]>

Update docs/howto/catalog_exports.md

f5f4406

Co-authored-by: Ariel Shaqed (Scolnicov) <[email protected]>

Update docs/howto/catalog_exports.md

2bb6fc6

Co-authored-by: Ariel Shaqed (Scolnicov) <[email protected]>

Update docs/howto/catalog_exports.md

ac45af7

Co-authored-by: Ariel Shaqed (Scolnicov) <[email protected]>

comments review fixes

51b14a0

Isan-Rivkin mentioned this pull request Oct 25, 2023

[Docs]: Unify Hive Table definition in lakeFS docs #6863

Closed

johnnyaug reviewed Oct 25, 2023

View reviewed changes

Isan-Rivkin and others added 4 commits October 25, 2023 18:36

update fixes

6559100

Update docs/integrations/glue_metastore.md

6719ba5

Co-authored-by: Yoni <[email protected]>

Update docs/integrations/glue_metastore.md

f8c337d

Co-authored-by: Yoni <[email protected]>

Update docs/integrations/glue_metastore.md

6d4a022

Co-authored-by: Yoni <[email protected]>

arielshaqed approved these changes Oct 26, 2023

View reviewed changes

Isan-Rivkin added 3 commits October 26, 2023 11:43

fixes

9200b8a

fix

0265cb9

remove example

a0c0cc9

Isan-Rivkin requested a review from johnnyaug October 26, 2023 09:21

johnnyaug approved these changes Oct 26, 2023

View reviewed changes

Isan-Rivkin merged commit 3c59c82 into master Oct 26, 2023
29 of 30 checks passed

Isan-Rivkin deleted the 6685-export-hooks-docs branch October 26, 2023 12:13

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Docs: Glue Exporter and related #6823

Docs: Glue Exporter and related #6823

Isan-Rivkin commented Oct 19, 2023 •

edited

Loading

github-actions bot commented Oct 19, 2023 •

edited

Loading

arielshaqed left a comment

arielshaqed Oct 24, 2023

Isan-Rivkin Oct 25, 2023 •

edited

Loading

arielshaqed Oct 24, 2023

arielshaqed Oct 24, 2023

johnnyaug left a comment

johnnyaug Oct 25, 2023

johnnyaug Oct 25, 2023

Isan-Rivkin Oct 26, 2023

johnnyaug Oct 25, 2023

Isan-Rivkin Oct 25, 2023

arielshaqed Oct 26, 2023

Isan-Rivkin Oct 26, 2023 •

edited

Loading

Isan-Rivkin Oct 26, 2023

johnnyaug Oct 25, 2023

Isan-Rivkin Oct 25, 2023

johnnyaug Oct 25, 2023

arielshaqed left a comment

arielshaqed Oct 26, 2023

Isan-Rivkin commented Oct 26, 2023 •

edited

Loading

johnnyaug left a comment


		#### Hive tables

		Hive metadata server tables are essentially just a set of objects that share a prefix, with no table metadata stored on the object store. You need to configure prefix, partitions, and schema.


		### `aws/glue.get_table(database, table [, catalog_id)`

		Get Table from Glue Catalog.

		## About Glue Metastore

		[AWS Glue](https://docs.aws.amazon.com/glue/latest/dg/tables-described.html) has a metastore that can store metadata related to Hive and other services (such as Spark and Trino). It has metadata such as the location of the table, information about columns, partitions and much more.

Docs: Glue Exporter and related #6823

Docs: Glue Exporter and related #6823

Conversation

Isan-Rivkin commented Oct 19, 2023 • edited Loading

Will merge when I have 2 human approvers ✅ ✅

What is covered in the docs

lakeFS User perspective POV

github-actions bot commented Oct 19, 2023 • edited Loading

arielshaqed left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Isan-Rivkin Oct 25, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

johnnyaug left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Isan-Rivkin Oct 26, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

arielshaqed left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Isan-Rivkin commented Oct 26, 2023 • edited Loading

johnnyaug left a comment

Choose a reason for hiding this comment

Isan-Rivkin commented Oct 19, 2023 •

edited

Loading

github-actions bot commented Oct 19, 2023 •

edited

Loading

Isan-Rivkin Oct 25, 2023 •

edited

Loading

Isan-Rivkin Oct 26, 2023 •

edited

Loading

Isan-Rivkin commented Oct 26, 2023 •

edited

Loading