Skip to content

Commit

Permalink
Update docs with deprecate feature warning
Browse files Browse the repository at this point in the history
  • Loading branch information
nopcoder committed Sep 10, 2023
1 parent 8372d32 commit f1a7d7c
Show file tree
Hide file tree
Showing 4 changed files with 23 additions and 9 deletions.
23 changes: 14 additions & 9 deletions docs/integrations/athena.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,10 @@ redirect_from: /using/athena.html
---

# Using lakeFS with Amazon Athena

{: .warning }
**Deprecated Feature:** This feature is being phased out. A superior replacement will be introduced soon.

[Amazon Athena](https://aws.amazon.com/athena/) is an interactive query service that makes it easy to analyze data in Amazon S3 using standard SQL.
{:.pb-5 }

Expand Down Expand Up @@ -47,28 +51,29 @@ To query that table with Athena, you need to use the `create-symlink` command as

```shell
lakectl metastore create-symlink \
--repo example \
--branch main \
--path my_table \
--from-client-type hive \
--from-schema default \
--from-table my_table \
--to-schema default \
--to-table my_table
--repo example \
--branch main \
--path my_table \
--from-client-type hive \
--from-schema default \
--from-table my_table \
--to-schema default \
--to-table my_table
```

The command will generate two notable outputs:

1. For each partition, the command will create a symlink file:

```shell
aws s3 ls s3://my-bucket/my-repo-prefix/my_table/ --recursive
aws s3 ls s3://my-bucket/my-repo-prefix/my_table/ --recursive
2021-11-23 17:46:29 0 my-repo-prefix/my_table/symlinks/example/main/my_table/year=2021/month=11/symlink.txt
2021-11-23 17:46:29 60 my-repo-prefix/my_table/symlinks/example/main/my_table/year=2021/month=12/symlink.txt
2021-11-23 17:46:30 60 my-repo-prefix/my_table/symlinks/example/main/my_table/year=2022/month=1/symlink.txt
```

An example content of a symlink file, where each line represents a single object of the specific partition:

```text
s3://my-bucket/my-repo-prefix/5bdc62da516944b49889770d98274227
s3://my-bucket/my-repo-prefix/64262fbf3d6347a79ead641d2b2baee6
Expand Down
3 changes: 3 additions & 0 deletions docs/integrations/dbt.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,9 @@ redirect_from: /using/dbt.html

# Integrating dbt and lakeFS

{: .warning }
**Deprecated Feature:** This feature is being phased out. A superior replacement will be introduced soon.

[dbt](https://www.getdbt.com/) can run on lakeFS with a Spark adapter or Presto/Trino adapter.
Both Spark and Presto use Hive metastore or Glue to manage tables and views.
When creating a branch in lakeFS, you receive a logical copy of the data that can be accessed by `s3://my-repo/branch/...`
Expand Down
3 changes: 3 additions & 0 deletions docs/integrations/glue_hive_metastore.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,9 @@ redirect_from: /using/glue_hive_metastore.html

# Using lakeFS with the Glue/Hive Metastore

{: .warning }
**Deprecated Feature:** This feature is being phased out. A superior replacement will be introduced soon.

{% include toc_2-3.html %}

## About Glue / Hive Metastore
Expand Down
3 changes: 3 additions & 0 deletions docs/integrations/presto_trino.md
Original file line number Diff line number Diff line change
Expand Up @@ -116,6 +116,9 @@ WITH (

### Example of copying a table with [metastore tools](glue_hive_metastore.md):

{: .warning }
**Deprecated Feature:** This feature is being phased out. A superior replacement will be introduced soon.

Copy the created table `page_views` on schema `main` to schema `example_branch` with location `s3a://example/example_branch/page_views/`
```shell
lakectl metastore copy --from-schema main --from-table page_views --to-branch example_branch
Expand Down

0 comments on commit f1a7d7c

Please sign in to comment.