Skip to content

Commit

Permalink
update docs
Browse files Browse the repository at this point in the history
  • Loading branch information
Isan-Rivkin committed Oct 22, 2023
1 parent 8351cae commit 3d5cb91
Show file tree
Hide file tree
Showing 3 changed files with 16 additions and 1 deletion.
1 change: 1 addition & 0 deletions docs/howto/hooks/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,7 @@ redirect_from:
- /hooks/index.html
- /hooks/
- /setup/hooks.html
- integrations/catalog_exports.html
---

# Actions and Hooks in lakeFS
Expand Down
12 changes: 12 additions & 0 deletions docs/integrations/catalog_exports.md
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,18 @@ While different systems support reading directly from lakefs (i.e Gateway endpoi

With Data Catalog Exports, one can leverage the versioning capabilities of lakeFS in external data warehouses and manage tables with branches and commits.

Once hooks are set up, querying lakeFS data from e.g. Athena, Trino and other catalog-dependant tools looks like this:

```sql
use main;
use my_branch; -- any branch
use v101; -- or tag

SELECT * FROM users
INNER JOIN events
ON users.id = events.user_id; -- SQL stays the same, branch or tag exist as schema
```

## How it works

There are several well known formats that exist today that could help export existing tables in lakeFS into a "native" object store representation
Expand Down
4 changes: 3 additions & 1 deletion docs/integrations/glue_metastore.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,8 +17,10 @@ This part explains about how Glue Metastore work with lakeFS.

[AWS Glue](https://docs.aws.amazon.com/glue/latest/dg/tables-described.html) has a metastore that can store metadata related to Hive and other services (such as Spark and Trino). It has metadata such as the location of the table, information about columns, partitions and many more.


## Support in lakeFS

The integration between Glue and lakeFS is based on [Data Catalog Exports](({% link integrations/catalog_exports.md %})).
On a high level
It allows defining a table


0 comments on commit 3d5cb91

Please sign in to comment.