diff --git a/metaphor/alation/README.md b/metaphor/alation/README.md index 693d2665..1e6d0812 100644 --- a/metaphor/alation/README.md +++ b/metaphor/alation/README.md @@ -15,15 +15,14 @@ Create a YAML config file based on the following template. ```yaml url: token: -output: - file: - directory: ``` -See [Output Config](../common/docs/output.md) for more information on `output`. - ### Optional Configurations +#### Output Destination + +See [Output Config](../common/docs/output.md) for more information. + #### Snowflake, MSSQL and Synapse Account If there are data sources from Snowflake, MSSQL or Synapse, please provide their accounts as follows, @@ -48,7 +47,7 @@ If not provided, each dataset's first owner will be considered as the author. If Follow the [Installation](../../README.md) instructions to install `metaphor-connectors` in your environment (or virtualenv). Make sure to include either `all` or `datahub` extra. -To test the connector locally, change the config file to output to a local path and run the following command +Run the following command to test the connector locally: ```shell metaphor alation diff --git a/metaphor/azure_data_factory/readme.md b/metaphor/azure_data_factory/readme.md index dbf8f6db..732d99cb 100644 --- a/metaphor/azure_data_factory/readme.md +++ b/metaphor/azure_data_factory/readme.md @@ -25,17 +25,19 @@ client_secret: # The client secret value (not secret ID) subscription_id: # Azure subscription id -output: - file: - directory: ``` +### Optional Configurations + +#### Output Destination + +See [Output Config](../common/docs/output.md) for more information. ## Testing Follow the [Installation](../../README.md) instructions to install `metaphor-connectors` in your environment (or virtualenv). Make sure to include either `all` or `datafactory` extra. -To test the connector locally, change the config file to output to a local path and run the following command +Run the following command to test the connector locally: ```shell metaphor azure_data_factory diff --git a/metaphor/bigquery/README.md b/metaphor/bigquery/README.md index f147769f..61ee1633 100644 --- a/metaphor/bigquery/README.md +++ b/metaphor/bigquery/README.md @@ -75,13 +75,8 @@ Create a YAML config file based on the following template. ```yaml project_ids: - -output: - file: - directory: ``` -See [Output Config](../common/docs/output.md) for more information on `output`. - To connect to BigQuery, either the keyfile path or credentials from the JSON keyfile must be set in the config as following: ```yaml @@ -112,6 +107,10 @@ credentials: ### Optional Configurations +#### Output Destination + +See [Output Config](../common/docs/output.md) for more information. + #### Filtering See [Filter Config](../common/docs/filter.md) for more information on the optional `filter` config. @@ -182,9 +181,9 @@ filter: Follow the [Installation](../../README.md) instructions to install `metaphor-connectors` in your environment (or virtualenv). Make sure to include either `all` or `bigquery` extra. -To test the connector locally, change the config file to output to a local path and run the following command +Run the following command to test the connector locally: -``` +```shell metaphor bigquery ``` diff --git a/metaphor/bigquery/lineage/README.md b/metaphor/bigquery/lineage/README.md index 10234c0c..912da377 100644 --- a/metaphor/bigquery/lineage/README.md +++ b/metaphor/bigquery/lineage/README.md @@ -33,7 +33,7 @@ batch_size: Follow the [Installation](../../../README.md) instructions to install `metaphor-connectors` in your environment (or virtualenv). Make sure to include either `all` or `bigquery` extra. -To test the connector locally, change the config file to output to a local path and run the following command +Run the following command to test the connector locally: ```bash metaphor bigquery.lineage diff --git a/metaphor/bigquery/profile/README.md b/metaphor/bigquery/profile/README.md index abcfc8a5..37db1722 100644 --- a/metaphor/bigquery/profile/README.md +++ b/metaphor/bigquery/profile/README.md @@ -24,7 +24,7 @@ See [Sampling Config](../../common/docs/sampling.md) for details. Follow the [Installation](../../README.md) instructions to install `metaphor-connectors` in your environment (or virtualenv). Make sure to include either `all` or `bigquery` extra. -To test the connector locally, change the config file to output to a local path and run the following command +Run the following command to test the connector locally: ```bash metaphor bigquery.profile diff --git a/metaphor/custom/data_quality/README.md b/metaphor/custom/data_quality/README.md index 1c75eb51..8c22b126 100644 --- a/metaphor/custom/data_quality/README.md +++ b/metaphor/custom/data_quality/README.md @@ -34,14 +34,15 @@ datasets: ... ... ... -output: - file: - directory: ``` > Note: You only need to specify `account` if the platform is `SNOWFLAKE`. -See [Output Config](../../common/docs/output.md) for more information on `output`. +### Optional Configurations + +#### Output Destination + +See [Output Config](../common/docs/output.md) for more information. ### Examples @@ -64,16 +65,13 @@ datasets: last_run: 2022-10-16T07:00:40 targets: - column: col2 -output: - file: - directory: /output ``` ## Testing Follow the [Installation](../../README.md) instructions to install `metaphor-connectors` in your environment (or virtualenv). -To test the connector locally, change the config file to output to a local path and run the following command +Run the following command to test the connector locally: ```shell metaphor custom.data_quality diff --git a/metaphor/custom/governance/README.md b/metaphor/custom/governance/README.md index 87cd5ebf..6e085eae 100644 --- a/metaphor/custom/governance/README.md +++ b/metaphor/custom/governance/README.md @@ -45,14 +45,15 @@ datasets: ... ... ... -output: - file: - directory: ``` > Note: You only need to specify `account` if the platform is `SNOWFLAKE`. -See [Output Config](../../common/docs/output.md) for more information on `output`. +### Optional Configurations + +#### Output Destination + +See [Output Config](../common/docs/output.md) for more information. ### Examples @@ -68,9 +69,6 @@ datasets: email: bob@test.com - type: email: alice@test.com -output: - file: - directory: /output ``` Here's another example showing how to tag a Snowflake table as `golden`, and the `email` column as `pii`. @@ -87,9 +85,6 @@ datasets: - column: email tags: - pii -output: - file: - directory: /output ``` The following example shows how to add a description to a Redshift table. @@ -102,16 +97,13 @@ datasets: descriptions: - description: A fancy description for the table email: charlie@test.com -output: - file: - directory: /output ``` ## Testing Follow the [Installation](../../README.md) instructions to install `metaphor-connectors` in your environment (or virtualenv). -To test the connector locally, change the config file to output to a local path and run the following command +Run the following command to test the connector locally: ```shell metaphor custom.governance diff --git a/metaphor/custom/lineage/README.md b/metaphor/custom/lineage/README.md index 0c649559..292c8b27 100644 --- a/metaphor/custom/lineage/README.md +++ b/metaphor/custom/lineage/README.md @@ -23,14 +23,15 @@ lineages: account: # only for Snowflake ... ... -output: - file: - directory: ``` > Note: You only need to specify `account` if the platform is `SNOWFLAKE`. -See [Output Config](../../common/docs/output.md) for more information on `output`. +### Optional Configurations + +#### Output Destination + +See [Output Config](../common/docs/output.md) for more information. ### Examples @@ -47,16 +48,13 @@ lineages: - platform: SNOWFLAKE name: db.schema.table3 account: snowflake_account -output: - file: - directory: /output ``` ## Testing Follow the [Installation](../../README.md) instructions to install `metaphor-connectors` in your environment (or virtualenv). -To test the connector locally, change the config file to output to a local path and run the following command +Run the following command to test the connector locally: ```shell metaphor custom.lineage diff --git a/metaphor/custom/metadata/README.md b/metaphor/custom/metadata/README.md index 7148b7cd..c1a1be4c 100644 --- a/metaphor/custom/metadata/README.md +++ b/metaphor/custom/metadata/README.md @@ -22,14 +22,15 @@ datasets: : ... ... -output: - file: - directory: ``` > Note: You only need to specify `account` if the platform is `SNOWFLAKE`. -See [Output Config](../../common/docs/output.md) for more information on `output`. +### Optional Configurations + +#### Output Destination + +See [Output Config](../common/docs/output.md) for more information. ### Examples @@ -49,16 +50,13 @@ datasets: key4: f1: value1, f2: value2 -output: - file: - directory: /output ``` ## Testing Follow the [Installation](../../README.md) instructions to install `metaphor-connectors` in your environment (or virtualenv). -To test the connector locally, change the config file to output to a local path and run the following command +Run the following command to test the connector locally: ```shell metaphor manual.metadata diff --git a/metaphor/custom/query_attributions/README.md b/metaphor/custom/query_attributions/README.md index 22e9ed91..4c74d326 100644 --- a/metaphor/custom/query_attributions/README.md +++ b/metaphor/custom/query_attributions/README.md @@ -19,12 +19,13 @@ attributions: : ... ... -output: - file: - directory: ``` -See [Output Config](../../common/docs/output.md) for more information on `output`. +### Optional Configurations + +#### Output Destination + +See [Output Config](../common/docs/output.md) for more information. ### Examples @@ -36,16 +37,13 @@ attributions: queries: query_id_1: joe@test.com query_id_2: jane@test.com -output: - file: - directory: /output ``` ## Testing Follow the [Installation](../../README.md) instructions to install `metaphor-connectors` in your environment (or virtualenv). -To test the connector locally, change the config file to output to a local path and run the following command +Run the following command to test the connector locally: ```shell metaphor custom.query_attributions diff --git a/metaphor/datahub/README.md b/metaphor/datahub/README.md index 6ea7378d..1eecafc6 100644 --- a/metaphor/datahub/README.md +++ b/metaphor/datahub/README.md @@ -26,15 +26,14 @@ Create a YAML config file based on the following template. host: port: token: # This is the personal access token. -output: - file: - directory: ``` -See [Output Config](../common/docs/output.md) for more information on `output`. - ### Optional Configurations +#### Output Destination + +See [Output Config](../common/docs/output.md) for more information. + #### Snowflake, MSSQL and Synapse Account If there are data sources from Snowflake, MSSQL or Synapse, please provide their accounts as follows, @@ -59,7 +58,7 @@ If not provided, each dataset's first owner will be considered as the author. If Follow the [Installation](../../README.md) instructions to install `metaphor-connectors` in your environment (or virtualenv). Make sure to include either `all` or `datahub` extra. -To test the connector locally, change the config file to output to a local path and run the following command +Run the following command to test the connector locally: ```shell metaphor datahub diff --git a/metaphor/dbt/README.md b/metaphor/dbt/README.md index 83fc0f98..c7c82e46 100644 --- a/metaphor/dbt/README.md +++ b/metaphor/dbt/README.md @@ -17,15 +17,14 @@ Create a YAML config file based the following template. ```yaml manifest: run_results: -output: - file: - directory: ``` -See [Output Config](../common/docs/output.md) for more information on `output`. - ### Optional Configurations +#### Output Destination + +See [Output Config](../common/docs/output.md) for more information. + #### Snowflake Account If the dbt project is using Snowflake, please provide the Snowflake account as follows, @@ -135,9 +134,9 @@ models: Follow the [Installation](../../README.md) instructions to install `metaphor-connectors` in your environment (or virtualenv). Make sure to include either `all` or `dbt` extra. -To test the connector locally, change the config file to output to a local path and run the following command +Run the following command to test the connector locally: -``` +```shell metaphor dbt ``` diff --git a/metaphor/dbt/cloud/README.md b/metaphor/dbt/cloud/README.md index eedcf543..a91fcec0 100644 --- a/metaphor/dbt/cloud/README.md +++ b/metaphor/dbt/cloud/README.md @@ -21,19 +21,18 @@ job_ids: project_ids: - - -output: - file: - directory: ``` You can extract `account_id` & `job_ids` from particular dbt job URLs, which have the format `https://cloud.getdbt.com/#/accounts//projects//jobs//`. It is also possible to specify the IDs for the projects to extract. The connector will extract the last successful run of each of the project's jobs. -See [Output Config](../common/docs/output.md) for more information on `output`. - ### Optional Configurations +#### Output Destination + +See [Output Config](../common/docs/output.md) for more information. + #### Base URL If you're using dbt [Single Tenancy](https://docs.getdbt.com/docs/cloud/about-cloud/tenancy#single-tenant), you'll also need to specify a different base URL: @@ -56,9 +55,9 @@ If `environment_ids` are specified, only jobs run within those environments are Follow the [Installation](../../README.md) instructions to install `metaphor-connectors` in your environment (or virtualenv). Make sure to include either `all` or `dbt` extra. -To test the connector locally, change the config file to output to a local path and run the following command +Run the following command to test the connector locally: -``` +```shell metaphor dbt.cloud ``` diff --git a/metaphor/fivetran/README.md b/metaphor/fivetran/README.md index b8e3d682..777347a8 100644 --- a/metaphor/fivetran/README.md +++ b/metaphor/fivetran/README.md @@ -16,17 +16,19 @@ Create a YAML config file based on the following template. ```yaml api_key: api_secret: - -output: - file: - directory: ``` +### Optional Configurations + +#### Output Destination + +See [Output Config](../common/docs/output.md) for more information. + ## Testing Follow the [Installation](../../README.md) instructions to install `metaphor-connectors` in your environment (or virtualenv). -To test the connector locally, change the config file to output to a local path and run the following command +Run the following command to test the connector locally: ```shell metaphor fivetran diff --git a/metaphor/glue/README.md b/metaphor/glue/README.md index c3084f91..23e180c5 100644 --- a/metaphor/glue/README.md +++ b/metaphor/glue/README.md @@ -43,18 +43,19 @@ aws: secret_access_key: region_name: assume_role_arn: # If using IAM role -output: - file: - directory: ``` -See [Output Config](../common/docs/output.md) for more information on `output`. +### Optional Configurations + +#### Output Destination + +See [Output Config](../common/docs/output.md) for more information. ## Testing Follow the [Installation](../../README.md) instructions to install `metaphor-connectors` in your environment (or virtualenv). -To test the connector locally, change the config file to output to a local path and run the following command +Run the following command to test the connector locally: ```shell metaphor glue diff --git a/metaphor/hive/README.md b/metaphor/hive/README.md index 05448aaa..6c959852 100644 --- a/metaphor/hive/README.md +++ b/metaphor/hive/README.md @@ -17,18 +17,16 @@ host: port: auth_user: password: - -output: - file: - directory: ``` For testing environments there could be no authentication. In that case, do not set `auth_user` and `password` -See [Output Config](../common/docs/output.md) for more information on `output`. - ### Optional Configurations +#### Output Destination + +See [Output Config](../common/docs/output.md) for more information. + #### Column Statistics See [Column Statistics](../../common/docs/column_statistics.md) for details. @@ -37,7 +35,7 @@ See [Column Statistics](../../common/docs/column_statistics.md) for details. Follow the [Installation](../../README.md) instructions to install `metaphor-connectors` in your environment (or virtualenv). -To test the connector locally, change the config file to output to a local path and run the following command +Run the following command to test the connector locally: ```shell metaphor hive diff --git a/metaphor/kafka/README.md b/metaphor/kafka/README.md index 09c31b00..773885a3 100644 --- a/metaphor/kafka/README.md +++ b/metaphor/kafka/README.md @@ -19,17 +19,16 @@ bootstrap_servers: - host: port: schema_registry_url: -output: - file: - directory: ``` To use HTTP basic authentication for the schema registry, specify the credentials in `schema_regitry_url` using the format `https://:@host:port`. -See [Output Config](../common/docs/output.md) for more information on `output`. - ### Optional Configurations +#### Output Destination + +See [Output Config](../common/docs/output.md) for more information. + #### SASL Authentication You can optionally authenticate against the brokers by adding the following SASL configurations: @@ -148,7 +147,7 @@ The results in the following schemas Follow the [Installation](../../README.md) instructions to install `metaphor-connectors` in your environment (or virtualenv). Make sure to include either `all` or `kafka` extra. -To test the connector locally, change the config file to output to a local path and run the following command +Run the following command to test the connector locally: ```shell metaphor kafka diff --git a/metaphor/looker/README.md b/metaphor/looker/README.md index 8e0706fe..2a6d9c10 100644 --- a/metaphor/looker/README.md +++ b/metaphor/looker/README.md @@ -61,15 +61,10 @@ connections: default_schema: account: platform: SNOWFLAKE -output: - file: - directory: ``` Note that `connections` is a mapping of database connection names to connection settings. You can find these settings under `Admin` > `Connections`. For now, the only platform supported is `SNOWFLAKE` with `account` set to the matching [Snowflake Account Identifier](https://docs.snowflake.com/en/user-guide/admin-account-identifier.html). -See [Output Config](../common/docs/output.md) for more information on `output`. - The connector also needs to parse the LookML project to extract additional metadata. There are two ways to provide the project source code. If the source code is in local environment, we can set the path to the project root as following: ```yaml @@ -102,11 +97,15 @@ verify_ssl: false timeout: 30 # default 120 seconds ``` +#### Output Destination + +See [Output Config](../common/docs/output.md) for more information on the optional `output` config. + ## Testing Follow the [Installation](../../README.md) instructions to install `metaphor-connectors` in your environment (or virtualenv). Make sure to include either `all` or `looker` extra. -To test the connector locally, change the config file to output to a local path and run the following command +Run the following command to test the connector locally: ```shell metaphor looker diff --git a/metaphor/metabase/README.md b/metaphor/metabase/README.md index e2bbd489..e4de5ad5 100644 --- a/metaphor/metabase/README.md +++ b/metaphor/metabase/README.md @@ -21,16 +21,19 @@ Create a YAML config file based on the following template. server_url: // e.g. "https://xxx.metabaseapp.com" for Metabase Cloud username: password: -output: - file: - directory: ``` +### Optional Configurations + +#### Output Destination + +See [Output Config](../common/docs/output.md) for more information. + ## Testing Follow the [Installation](../../README.md) instructions to install `metaphor-connectors` in your environment (or virtualenv). Make sure to include either `all` or `metabase` extra. -To test the connector locally, change the config file to output to a local path and run the following command +Run the following command to test the connector locally: ```shell metaphor metabase diff --git a/metaphor/monte_carlo/README.md b/metaphor/monte_carlo/README.md index 3b6ed23b..0232ae91 100644 --- a/metaphor/monte_carlo/README.md +++ b/metaphor/monte_carlo/README.md @@ -16,13 +16,14 @@ Create a YAML config file based on the following template. api_key_id: api_key_secret: data_platform: # SNOWFLAKE, BIGQUERY, REDSHIFT, etc. -output: - file: - directory: ``` ### Optional Configurations +#### Output Destination + +See [Output Config](../common/docs/output.md) for more information. + #### Snowflake Account If some of the monitored data assets are Snowflake datasets, please provide the Snowflake account as follows, @@ -35,7 +36,7 @@ snowflake_account: Follow the [Installation](../../README.md) instructions to install `metaphor-connectors` in your environment (or virtualenv). Make sure to include either `all` or `monte_carlo` extra. -To test the connector locally, change the config file to output to a local path and run the following command. +Run the following command to test the connector locally: ```shell metaphor monte_carlo diff --git a/metaphor/mssql/README.md b/metaphor/mssql/README.md index 68c19d93..11dfcf4d 100644 --- a/metaphor/mssql/README.md +++ b/metaphor/mssql/README.md @@ -28,14 +28,8 @@ username: # The MSSQL server login username password: # The MSSQL server login password endpoint: # The MSSQL server endpoint - -output: - file: - directory: # the output result directory ``` -See [Output Config](../common/docs/output.md) for more information on `output`. - ### Optional Configurations You could specfic `server_name` or `tenant_id` to connector. @@ -48,11 +42,15 @@ tenant_id: # The azure directory (tenant) id See [Filter Configurations](../common/docs/filter.md) for more information on the optional `filter` config. +#### Output Destination + +See [Output Config](../common/docs/output.md) for more information on the optional `output` config. + ## Testing Follow the [Installation](../../README.md) instructions to install `metaphor-connectors` in your environment (or virtualenv). Make sure to include either `all` or `mssql` extra. -To test the connector locally, change the config file to output to a local path and run the following command. +Run the following command to test the connector locally: ```shell metaphor mssql diff --git a/metaphor/mysql/README.md b/metaphor/mysql/README.md index 399898aa..69107cdd 100644 --- a/metaphor/mysql/README.md +++ b/metaphor/mysql/README.md @@ -23,13 +23,8 @@ host: user: password: database: -output: - file: - directory: ``` -See [Output Config](../common/docs/output.md) for more information on `output`. - ### Optional Configurations By default, the connector will connect using the default MySQL port 3306. You can change it using the following config: @@ -40,11 +35,15 @@ port: See [Filter Configurations](../common/docs/filter.md) for more information on the optional `filter` config. +#### Output Destination + +See [Output Config](../common/docs/output.md) for more information on the optional `output` config. + ## Testing Follow the [Installation](../../README.md) instructions to install `metaphor-connectors` in your environment (or virtualenv). Make sure to include either `all` or `mysql` extra. -To test the connector locally, change the config file to output to a local path and run the following command +Run the following command to test the connector locally: ```shell metaphor mysql diff --git a/metaphor/notion/README.md b/metaphor/notion/README.md index e4444e1e..0ba8fb77 100644 --- a/metaphor/notion/README.md +++ b/metaphor/notion/README.md @@ -20,10 +20,6 @@ Create a YAML config file based on the following template. notion_api_token: azure_openAI_key: - -output: - file: - directory: ``` ### Optional Configurations @@ -42,11 +38,15 @@ notion_api_version: # "2022-06-08" include_text: # False ``` +#### Output Destination + +See [Output Config](../common/docs/output.md) for more information on the optional `output` config. + ## Testing -Follow the [installation](../../README.md) instructions to install `metaphor-connectors` in your environment (or virtualenv). Make sure to include the `notion` or `all` extra. +Follow the [installation](../../README.md) instructions to install `metaphor-connectors` in your environment (or virtualenv). Make sure to include the `notion` or `all` extra. -To test the connector locally, change the config file to output to a local path and run the following command +Run the following command to test the connector locally: ```shell metaphor notion diff --git a/metaphor/postgresql/README.md b/metaphor/postgresql/README.md index 08601e6b..97b169ac 100644 --- a/metaphor/postgresql/README.md +++ b/metaphor/postgresql/README.md @@ -31,13 +31,8 @@ host: user: password: database: -output: - file: - directory: ``` -See [Output Config](../common/docs/output.md) for more information on `output`. - ### Optional Configurations By default, the connector will connect using the default PostgreSQL port 5432. You can change it using the following config: @@ -48,11 +43,15 @@ port: See [Filter Configurations](../common/docs/filter.md) for more information on the optional `filter` config. +#### Output Destination + +See [Output Config](../common/docs/output.md) for more information on the optional `output` config. + ## Testing Follow the [Installation](../../README.md) instructions to install `metaphor-connectors` in your environment (or virtualenv). Make sure to include either `all` or `postgresql` extra. -To test the connector locally, change the config file to output to a local path and run the following command +Run the following command to test the connector locally: ```shell metaphor postgresql diff --git a/metaphor/postgresql/profile/README.md b/metaphor/postgresql/profile/README.md index 1cfc83cb..d856bbbf 100644 --- a/metaphor/postgresql/profile/README.md +++ b/metaphor/postgresql/profile/README.md @@ -26,7 +26,7 @@ See [Sampling Config](../../common/docs/sampling.md) for details. Follow the [Installation](../../README.md) instructions to install `metaphor-connectors` in your environment (or virtualenv). Make sure to include either `all` or `postgresql` extra. -To test the connector locally, change the config file to output to a local path and run the following command +Run the following command to test the connector locally: ```shell metaphor postgresql.profile diff --git a/metaphor/postgresql/usage/README.md b/metaphor/postgresql/usage/README.md index bd25bc49..50497a7e 100644 --- a/metaphor/postgresql/usage/README.md +++ b/metaphor/postgresql/usage/README.md @@ -10,10 +10,10 @@ The config file inherits all the required and optional fields from the general P Follow the [Installation](../../README.md) instructions to install `metaphor-connectors` in your environment (or virtualenv). Make sure to include either `all` or `postgresql` extra. -To test the connector locally, change the config file to output to a local path and run the following command +Run the following command to test the connector locally: ```shell metaphor postgresql.usage ``` -Manually verify the output after the run finishes. \ No newline at end of file +Manually verify the output after the run finishes. diff --git a/metaphor/power_bi/README.md b/metaphor/power_bi/README.md index 8dd43366..a85c6b1e 100644 --- a/metaphor/power_bi/README.md +++ b/metaphor/power_bi/README.md @@ -45,10 +45,6 @@ tenant_id: # The Power BI tenant ID client_id: # The Azure Application client id secret: # The client secret value (not secret ID) - -output: - file: - directory: ``` ### Optional Configurations @@ -67,12 +63,15 @@ When extracting lineages, the connector parses the snowflake account directly fr snowflake_account: ``` +#### Output Destination + +See [Output Config](../common/docs/output.md) for more information on the optional `output` config. ## Testing Follow the [Installation](../../README.md) instructions to install `metaphor-connectors` in your environment (or virtualenv). Make sure to include either `all` or `power_bi` extra. -To test the connector locally, change the config file to output to a local path and run the following command +Run the following command to test the connector locally: ```shell metaphor power_bi diff --git a/metaphor/redshift/README.md b/metaphor/redshift/README.md index c1bef61e..440746c6 100644 --- a/metaphor/redshift/README.md +++ b/metaphor/redshift/README.md @@ -42,15 +42,14 @@ host: user: password: database: -output: - file: - directory: ``` -See [Output Config](../common/docs/output.md) for more information on `output`. - ### Optional Configurations +#### Output Destination + +See [Output Config](../common/docs/output.md) for more information. + #### Port By default, the connector will connect using the default Redshift port 5439. You can change it using the following config: @@ -87,7 +86,7 @@ query_log: Follow the [Installation](../../README.md) instructions to install `metaphor-connectors` in your environment (or virtualenv). Make sure to include either `all` or `redshift` extra. -To test the connector locally, change the config file to output to a local path and run the following command +Run the following command to test the connector locally: ```shell metaphor redshift diff --git a/metaphor/redshift/lineage/README.md b/metaphor/redshift/lineage/README.md index c31e4f85..5e7969fd 100644 --- a/metaphor/redshift/lineage/README.md +++ b/metaphor/redshift/lineage/README.md @@ -33,7 +33,7 @@ include_self_lineage: Follow the [Installation](../../README.md) instructions to install `metaphor-connectors` in your environment (or virtualenv). Make sure to include either `all` or `redshift` extra. -To test the connector locally, change the config file to output to a local path and run the following command +Run the following command to test the connector locally: ```bash metaphor redshift.lineage diff --git a/metaphor/redshift/profile/README.md b/metaphor/redshift/profile/README.md index 6501c471..035d2886 100644 --- a/metaphor/redshift/profile/README.md +++ b/metaphor/redshift/profile/README.md @@ -26,7 +26,7 @@ See [Sampling Config](../../common/docs/sampling.md) for details. Follow the [Installation](../../README.md) instructions to install `metaphor-connectors` in your environment (or virtualenv). Make sure to include either `all` or `postgresql` extra. -To test the connector locally, change the config file to output to a local path and run the following command +Run the following command to test the connector locally: ```shell metaphor redshift.profile diff --git a/metaphor/s3/README.md b/metaphor/s3/README.md index 8041c1bc..e275e79e 100644 --- a/metaphor/s3/README.md +++ b/metaphor/s3/README.md @@ -32,14 +32,8 @@ verify_ssl: # Whether or not to verify SSL certificates. By default SSL certificates are verified. You can provide the following values: # * False - do not validate SSL certificates. SSL will still be used, but SSL certificates will not be verified. # * path/to/cert/bundle.pem - A filename of the CA cert bundle to use. You can specify this argument if you want to use a different CA cert bundle than the one used by botocore. - -output: - file: - directory: ``` -See [Output Config](../common/docs/output.md) for more information on `output`. - #### Path specifications This specifies the files / directories to be parse as datasets. Each `path_spec` should follow the below format: @@ -137,11 +131,17 @@ All other file types are automatically ignored. If not provided, all these file The excluded URIs do not support labels. +### Optional Configurations + +#### Output Destination + +See [Output Config](../common/docs/output.md) for more information. + ## Testing Follow the [Installation](../../README.md) instructions to install `metaphor-connectors` in your environment (or virtualenv). Make sure to include either `all` or `s3` extra. -To test the connector locally, change the config file to output to a local path and run the following command. +Run the following command to test the connector locally: ```shell metaphor s3 diff --git a/metaphor/snowflake/README.md b/metaphor/snowflake/README.md index f1679eec..4a8707ee 100644 --- a/metaphor/snowflake/README.md +++ b/metaphor/snowflake/README.md @@ -77,9 +77,6 @@ user: password: role: # Optional. Will use default role if not specified. default_database: -output: - file: - directory: ``` If using key pair authentication: @@ -92,9 +89,6 @@ private_key: passphrase: role: # Optional. Will use default role if not specified. default_database: -output: - file: - directory: ``` The `private_key.passphrase` is only needed if using encrypted version of the private key. Otherwise, it can be omitted from the config. @@ -109,10 +103,12 @@ private_key: -----END ENCRYPTED PRIVATE KEY----- ``` -See [Output Config](../common/docs/output.md) for more information on `output`. - ### Optional Configurations +#### Output Destination + +See [Output Config](../common/docs/output.md) for more information. + #### Filtering See [Filter Config](../common/docs/filter.md) for more information on the optional `filter` config. @@ -160,7 +156,7 @@ query_tag: # Default to 'MetaphorData' Follow the [Installation](../../README.md) instructions to install `metaphor-connectors` in your environment (or virtualenv). Make sure to include either `all` or `snowflake` extra. -To test the connector locally, change the config file to output to a local path and run the following command +Run the following command to test the connector locally: ```shell metaphor snowflake diff --git a/metaphor/snowflake/lineage/README.md b/metaphor/snowflake/lineage/README.md index 83e8d7bf..0614d181 100644 --- a/metaphor/snowflake/lineage/README.md +++ b/metaphor/snowflake/lineage/README.md @@ -31,9 +31,9 @@ batch_size: Follow the [Installation](../../README.md) instructions to install `metaphor-connectors` in your environment (or virtualenv). Make sure to include either `all` or `snowflake` extra. -To test the connector locally, change the config file to output to a local path and run the following command +Run the following command to test the connector locally: -``` +```shell metaphor snowflake.linage ``` diff --git a/metaphor/snowflake/profile/README.md b/metaphor/snowflake/profile/README.md index 95ff6431..19446636 100644 --- a/metaphor/snowflake/profile/README.md +++ b/metaphor/snowflake/profile/README.md @@ -46,9 +46,9 @@ See [Sampling Config](../../common/docs/sampling.md) for details. Follow the [Installation](../../README.md) instructions to install `metaphor-connectors` in your environment (or virtualenv). Make sure to include either `all` or `snowflake` extra. -To test the connector locally, change the config file to output to a local path and run the following command +Run the following command to test the connector locally: -``` +```shell metaphor snowflake.profile ``` diff --git a/metaphor/synapse/README.md b/metaphor/synapse/README.md index a67f1fc6..8ded007a 100644 --- a/metaphor/synapse/README.md +++ b/metaphor/synapse/README.md @@ -12,7 +12,7 @@ Create and grant read-only permissions to a dedicated user in [Synapse workspace GRANT CONNECT ANY DATABASE TO ; GRANT VIEW ANY DEFINITION TO ; ``` - + - For [dedicated pool](https://learn.microsoft.com/en-us/azure/synapse-analytics/sql-data-warehouse/sql-data-warehouse-overview-what-is): ```sql USE master; @@ -25,7 +25,7 @@ Create and grant read-only permissions to a dedicated user in [Synapse workspace GRANT VIEW DEFINITION TO ; GRANT VIEW DATABASE STATE TO ; ``` - + ## Config File Create a YAML config file based on the following template. @@ -37,14 +37,8 @@ tenant_id: # Azure tenant ID server_name: # Synapse workspace name username: password: - -output: - file: - directory: # the output result directory ``` -See [Output Config](../common/docs/output.md) for more information on `output`. - ### Optional Configurations By default, the connector will not crawl the query log unless you specify `lookback_days`. @@ -56,11 +50,15 @@ query_log: See [Filter Configurations](../common/docs/filter.md) for more information on the optional `filter` config. +#### Output Destination + +See [Output Config](../common/docs/output.md) for more information on the optional `output` config. + ## Testing Follow the [Installation](../../README.md) instructions to install `metaphor-connectors` in your environment (or virtualenv). Make sure to include either `all` or `synapse` extra. -To test the connector locally, change the config file to output to a local path and run the following command. +Run the following command to test the connector locally: ```shell metaphor synapse diff --git a/metaphor/tableau/README.md b/metaphor/tableau/README.md index 0c64c758..71feb743 100644 --- a/metaphor/tableau/README.md +++ b/metaphor/tableau/README.md @@ -35,9 +35,6 @@ site_name: // The Tableau Server site you are authenticating with access_token: token_name: token_value: -output: - file: - directory: ``` If authenticate via user password: @@ -48,9 +45,6 @@ site_name: // The Tableau Server site you are authenticating with user_password: username: password: -output: - file: - directory: ``` > When connecting to the [Default Site](https://help.tableau.com/current/server/en-us/sites_intro.htm#the-default-site) of a Tableau Server, set `site_name` to an empty string, i.e. `site_name: ''`. @@ -59,6 +53,10 @@ output: ### Optional Configurations +#### Output Destination + +See [Output Config](../common/docs/output.md) for more information. + #### Alternative Server URL If the tableau users use a different URL to view content on Tableau than the server URL for fetching metadata, please provide the alternative base URL so the crawler can generate direct links to the assets: @@ -115,7 +113,7 @@ include_personal_space: True Follow the [Installation](../../README.md) instructions to install `metaphor-connectors` in your environment (or virtualenv). Make sure to include either `all` or `tableau` extra. -To test the connector locally, change the config file to output to a local path and run the following command +Run the following command to test the connector locally: ```shell metaphor tableau diff --git a/metaphor/thought_spot/README.md b/metaphor/thought_spot/README.md index 8ee5d2ca..fc4f17a1 100644 --- a/metaphor/thought_spot/README.md +++ b/metaphor/thought_spot/README.md @@ -28,17 +28,19 @@ secret_key: # If using password to authenticate password: - -output: - file: - directory: ``` +### Optional Configurations + +#### Output Destination + +See [Output Config](../common/docs/output.md) for more information. + ## Testing Follow the [Installation](../../README.md) instructions to install `metaphor-connectors` in your environment (or virtualenv). Make sure to include either `all` or `thought_spot` extra. -To test the connector locally, change the config file to output to a local path and run the following command +Run the following command to test the connector locally: ```shell metaphor thought_spot diff --git a/metaphor/trino/README.md b/metaphor/trino/README.md index c376659b..27198a43 100644 --- a/metaphor/trino/README.md +++ b/metaphor/trino/README.md @@ -5,6 +5,7 @@ This connector extracts metadata from Trino using [Trino Python Client](https:// ## Setup To run a Trino db instance locally, create the docker container with: + ```shell docker run --name trino -d -p 8080:8080 trinodb/trino ``` @@ -24,19 +25,19 @@ Create a YAML config file based on the following template. ### Required Configurations You must specify the host, port, and the user in order to connect to Trino. + ```yaml host: port: username: -output: - file: - directory: ``` -See [Output Config](../common/docs/output.md) for more information on `output`. - ### Optional Configurations +#### Output Destination + +See [Output Config](../common/docs/output.md) for more information. + #### Authentication Currently password and JWT token based authentication methods are supported. @@ -75,7 +76,7 @@ By default the following are excluded: Follow the [Installation](../../README.md) instructions to install `metaphor-connectors` in your environment (or virtualenv). Make sure to include either `all` or `trino` extra. -To test the connector locally, change the config file to output to a local path and run the following command +Run the following command to test the connector locally: ```shell metaphor trino diff --git a/metaphor/unity_catalog/README.md b/metaphor/unity_catalog/README.md index 017e0b12..e69a49b9 100644 --- a/metaphor/unity_catalog/README.md +++ b/metaphor/unity_catalog/README.md @@ -19,15 +19,14 @@ Create a YAML config file based on the following template. ```yaml host: token: -output: - file: - directory: ``` -See [Output Config](../common/docs/output.md) for more information on `output`. - ### Optional Configurations +#### Output Destination + +See [Output Config](../common/docs/output.md) for more information. + #### Filtering See [Filter Configurations](../common/docs/filter.md) for more information on the optional `filter` config. @@ -74,7 +73,7 @@ query_log: Follow the [Installation](../../README.md) instructions to install `metaphor-connectors` in your environment (or virtualenv). Make sure to include either `all` or `unity_catalog` extra. -To test the connector locally, change the config file to output to a local path and run the following command +Run the following command to test the connector locally: ```shell metaphor unity_catalog diff --git a/metaphor/unity_catalog/profile/README.md b/metaphor/unity_catalog/profile/README.md index b2e4199d..78175cd0 100644 --- a/metaphor/unity_catalog/profile/README.md +++ b/metaphor/unity_catalog/profile/README.md @@ -19,22 +19,21 @@ Create a YAML config file based on the following template. ```yaml host: token: -output: - file: - directory: ``` -See [Output Config](../common/docs/output.md) for more information on `output`. - ### Optional Configurations See [Filter Configurations](../common/docs/filter.md) for more information on the optional `filter` config. +#### Output Destination + +See [Output Config](../common/docs/output.md) for more information on the optional `output` config. + ## Testing Follow the [Installation](../../README.md) instructions to install `metaphor-connectors` in your environment (or virtualenv). Make sure to include either `all` or `unity_catalog` extra. -To test the connector locally, change the config file to output to a local path and run the following command +Run the following command to test the connector locally: ```shell metaphor unity_catalog.profile