Skip to content

Commit

Permalink
fix broken anchors
Browse files Browse the repository at this point in the history
  • Loading branch information
Isan-Rivkin committed Oct 22, 2023
1 parent 3d5cb91 commit 3776994
Show file tree
Hide file tree
Showing 3 changed files with 14 additions and 3 deletions.
2 changes: 1 addition & 1 deletion docs/howto/hooks/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ redirect_from:
- /hooks/index.html
- /hooks/
- /setup/hooks.html
- integrations/catalog_exports.html
- /integrations/catalog_exports.html
---

# Actions and Hooks in lakeFS
Expand Down
3 changes: 2 additions & 1 deletion docs/integrations/catalog_exports.md
Original file line number Diff line number Diff line change
Expand Up @@ -90,7 +90,8 @@ Note: Check the [lua Library reference]({% link howto/hooks/lua.md %}#lua-librar
#### Running an Exporter

Exporters are meant to run as [Lua hooks]({% link howto/hooks/lua.md %}).
Actions trigger can be configured with [events and branches]({% link howto/hooks/index.md#action-file-schema %}), incase custom filtering logic is required it can be achieved in the Lua script itself.

Actions trigger can be configured with [events and branches]({% link howto/hooks/index.md %}#action-file-schema), incase custom filtering logic is required it can be achieved in the Lua script itself.
The default table name when exported is `${repository_id}_${_lakefs_tables/TABLE.md(name field)}_${ref_name}_${short_commit}`.

Example of an action that will be triggered when a `post-commit` event happens in the `export_table` branch.
Expand Down
12 changes: 11 additions & 1 deletion docs/integrations/glue_metastore.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,16 @@ This part explains about how Glue Metastore work with lakeFS.
## Support in lakeFS

The integration between Glue and lakeFS is based on [Data Catalog Exports](({% link integrations/catalog_exports.md %})).
It allows defining a table

### What is supported

- Creating a uniqueue table in Glue Catalog per lakeFS repository / ref / commit.
- No data copying is required, the table location is a path to a symlinks structure in S3 based on Hive's [SymlinkTextInputFormat](https://svn.apache.org/repos/infra/websites/production/hive/content/javadocs/r2.1.1/api/org/apache/hadoop/hive/ql/io/SymlinkTextInputFormat.html) and the [table partitions](https://docs.aws.amazon.com/glue/latest/dg/tables-described.html#tables-partition) are maintained.
- Tables are described via [Hive format in _lakefs_tables/<my_table>.yaml](({% link integrations/catalog_exports.md %}#hive-tables))
- Currently the data query in Glue metastore is Read-Only operation and mutating data requires writting to lakeFS and letting the export hook run.

### Pre-requisite

1. Write some lakeFS table data (i.e [Spark]({% link integrations/spark.md %}) or some CSV)
2. AWS Credentials with permission to manage Glue, Athena Query and S3 access.
3. lakeFS [Actions]({% link howto/hooks/index.md %}) enabled.

0 comments on commit 3776994

Please sign in to comment.