From 85eb05aec1f36c24527a48781e4760c81c9a09ee Mon Sep 17 00:00:00 2001 From: Scott Huang Date: Thu, 3 Oct 2024 15:50:29 +0800 Subject: [PATCH] Update readme --- README.md | 1 + metaphor/athena/README.md | 67 +++++++++++++++++++++++++++++++++++++++ 2 files changed, 68 insertions(+) diff --git a/README.md b/README.md index 6dca611c..02e0a346 100644 --- a/README.md +++ b/README.md @@ -48,6 +48,7 @@ Each connector is placed under its own directory under [metaphor](./metaphor) an | Connector Name | Metadata | |-------------------------------------------------------------------|------------------------------------------| +| [athena](metaphor/athena/) | Schema, description, queries | | [azure_data_factory](metaphor/azure_data_factory/) | Lineage, Pipeline | | [bigquery](metaphor/bigquery/) | Schema, description, statistics, queries | | [bigquery.lineage](metaphor/bigquery/lineage/) | Lineage | diff --git a/metaphor/athena/README.md b/metaphor/athena/README.md index e69de29b..1e0caeb4 100644 --- a/metaphor/athena/README.md +++ b/metaphor/athena/README.md @@ -0,0 +1,67 @@ +# Athena Connector + +This connector extracts technical metadata and query history from AWS Athena using the [boto3](https://boto3.amazonaws.com/v1/documentation/api/latest/index.html) library. + +## Setup + +We recommend creating a dedicated AWS IAM user for the crawler with limited permissions based on the following IAM policy: + +``` json +{ + "Version": "2012-10-17", + "Statement": + [ + { + "Effect": "Allow", + "Action": + [ + "athena:ListDataCatalogs", + "athena:ListDatabases", + "athena:ListTableMetadata", + "athena:ListQueryExecutions", + "athena:BatchGetQueryExecution", + "glue:GetDatabases", + "glue:GetTables" + ], + "Resource": + [ + "*" + ] + } + ] +} +``` + +## Config File + +Create a YAML config file based on the following template. + +### Required Configurations + +You must specify an AWS user credential to access QuickSight API. You can also specify a role ARN and let the connector assume the role before accessing AWS APIs. + +```yaml +aws: + access_key_id: + secret_access_key: + region_name: + assume_role_arn: # If using IAM role +``` + +### Optional Configurations + +#### Output Destination + +See [Output Config](../common/docs/output.md) for more information. + +## Testing + +Follow the [Installation](../../README.md) instructions to install `metaphor-connectors` in your environment (or virtualenv). + +Run the following command to test the connector locally: + +```shell +metaphor athena +``` + +Manually verify the output after the run finishes.