Skip to content

Commit

Permalink
Update readme
Browse files Browse the repository at this point in the history
  • Loading branch information
elic-eon committed Oct 3, 2024
1 parent e31e77c commit 85eb05a
Show file tree
Hide file tree
Showing 2 changed files with 68 additions and 0 deletions.
1 change: 1 addition & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -48,6 +48,7 @@ Each connector is placed under its own directory under [metaphor](./metaphor) an

| Connector Name | Metadata |
|-------------------------------------------------------------------|------------------------------------------|
| [athena](metaphor/athena/) | Schema, description, queries |
| [azure_data_factory](metaphor/azure_data_factory/) | Lineage, Pipeline |
| [bigquery](metaphor/bigquery/) | Schema, description, statistics, queries |
| [bigquery.lineage](metaphor/bigquery/lineage/) | Lineage |
Expand Down
67 changes: 67 additions & 0 deletions metaphor/athena/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,67 @@
# Athena Connector

This connector extracts technical metadata and query history from AWS Athena using the [boto3](https://boto3.amazonaws.com/v1/documentation/api/latest/index.html) library.

## Setup

We recommend creating a dedicated AWS IAM user for the crawler with limited permissions based on the following IAM policy:

``` json
{
"Version": "2012-10-17",
"Statement":
[
{
"Effect": "Allow",
"Action":
[
"athena:ListDataCatalogs",
"athena:ListDatabases",
"athena:ListTableMetadata",
"athena:ListQueryExecutions",
"athena:BatchGetQueryExecution",
"glue:GetDatabases",
"glue:GetTables"
],
"Resource":
[
"*"
]
}
]
}
```

## Config File

Create a YAML config file based on the following template.

### Required Configurations

You must specify an AWS user credential to access QuickSight API. You can also specify a role ARN and let the connector assume the role before accessing AWS APIs.

```yaml
aws:
access_key_id: <aws_access_key_id>
secret_access_key: <aws_secret_access_key>
region_name: <aws_region_name>
assume_role_arn: <aws_role_arn> # If using IAM role
```
### Optional Configurations
#### Output Destination
See [Output Config](../common/docs/output.md) for more information.
## Testing
Follow the [Installation](../../README.md) instructions to install `metaphor-connectors` in your environment (or virtualenv).

Run the following command to test the connector locally:

```shell
metaphor athena <config_file>
```

Manually verify the output after the run finishes.

0 comments on commit 85eb05a

Please sign in to comment.