Skip to content

Commit

Permalink
Update README (#126)
Browse files Browse the repository at this point in the history 
Reorganizing README to group instructions by interface
(Streamlit/CLI/Python).
  • Loading branch information
@sfc-gh-cnivera
sfc-gh-cnivera authored Aug 2, 2024
1 parent 5dec794 commit 2d63258
Showing 1 changed file with 75 additions and 79 deletions.
154 changes: 75 additions & 79 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -2,15 +2,10 @@

The `Semantic Model Generator` is used to generate a semantic model for use in your Snowflake account.

Your workflow should be:
You can generate semantic models through our Streamlit app, the command line, or directly in your Python code.
Please complete the instructions in [setup](#setup), then proceed to the instructions for your desired approach.

1. [Setup](#setup) to set credentials.
2. [Usage](#usage) to create a model either through Python or command line.
3. [Post Generation](#post-generation) to fill out the rest of your semantic model.
4. [Validating Your Final Semantic Model](#validating-yaml-updates) to ensure any changes you've made are valid.
5. [Admin App](#admin-app) spin up the admin app for easier generation of your semantic model.

Or, if you want to see what a semantic model looks like, skip to [Examples](#examples).
If you want to see what a semantic model looks like, skip to [Examples](#examples).

## Setup

Expand All @@ -32,7 +27,7 @@ account, [follow these instructions](https://docs.snowflake.com/en/user-guide/or
your Snowflake deployment. We would recommend setting it regardless.

We recommend setting these environment variables by creating a `.env` file in the root directory of this repo. See the
examples in [`.env.example`](.env.example) for reference.
examples in [`.env.example`](.env.example) for reference and proper syntax for `.env` files.

However, if you would like to set these variables directly in your shell/Python environment,

Expand Down Expand Up @@ -155,35 +150,74 @@ set SNOWFLAKE_AUTHENTICATOR=externalbrowser
os.environ['SNOWFLAKE_AUTHENTICATOR'] = 'externalbrowser'
```

## Usage
## Streamlit App

We offer a convenient Streamlit app that supports creating semantic models from scratch as well as iterating on existing ones uploaded to a Snowflake stage.

To install dependencies for the Streamlit app, run

```bash
make setup_admin_app
```

Once installed, you can run the app using the provided Makefile target, or with your current version of Python manually specified:

```bash
# Make target
make run_admin_app

# directly
python3.11 -m streamlit run admin_apps/app.py
```

## CLI Tool

You may also generate a semantic model directly from the CLI. To do this, first install the CLI tool dependencies, which differ from the Streamlit app's dependencies.

Unlike the Streamlit route above, using the CLI assumes that you will manage your environment using `poetry` and `pyenv` for Python versions.
This has only been tested on MacOS/Linux.

1. If you need brew, run `make install-homebrew`.
2. If you need pyenv, `make install-pyenv` and `make install-python-3.8`.
3. Run `make setup` to install all external dependencies into your Poetry environment. This will also install `poetry` if needed.
4. Spawn a shell in the virtual environment using `poetry shell`. This will activate your virtual environment.


### Generation
You are now ready to generate semantic models via the CLI! The generation command uses the following syntax:

```bash
python -m semantic_model_generator.generate_model \
--base_tables "['<your-database-name-1>.<your-schema-name-1>.<your-base-table-or-view-name-1>','<your-database-name-2>.<your-schema-name-2>.<your-base-table-or-view-name-2>']" \
--semantic_model_name "<a-meaningful-semantic-model-name>" \
--snowflake_account="<your-snowflake-account>"
```

You may generate a semantic model for a given list of fully qualified tables following the `{database}.{schema}.{table}`
format. Each table in this list should be a physical table or a view present in your database.

If your snowflake tables and comments do not have comments, we currently
leverages [cortex LLM function](https://docs.snowflake.com/en/user-guide/snowflake-cortex/llm-functions) to
auto-generate description suggestions. Those generation are suffixed with '__' and additional comment to remind you to
confirm/modity the descriptions.

All generated semantic models by default are saved either under `semantic_model_generator/output_models` if running from
the root of this project or the current directory you're in.

### Generation - Python

1. Ensure you have installed the Python package. Note, the version below should be the latest version under the `dist/`
directory.
### Validation
You may also use the CLI tool to validate one of your semantic models. From inside your Poetry shell, run

```bash
pip install dist/*.whl
python -m semantic_model_generator.validate_model \
--yaml_path="/path/to/your/model_yaml.yaml \
--snowflake_account="<your-account-name>"
```
2. Activate Python shell
## Python
You may also create/validate your semantic models from directly within your Python code. First, ensure that you have installed the Python package. Note, the version below should be the latest version under the `dist/` directory.
```bash
python
pip install dist/*.whl
```
3. Generate a semantic model.
### Generation
```python
from semantic_model_generator.generate_model import generate_base_semantic_model_from_snowflake
Expand All @@ -200,31 +234,32 @@ generate_base_semantic_model_from_snowflake(
)
```
### Generation - CLI
### Validation
Unlike the Python route above, using the CLI assumes that you will manage your environment using `poetry` and `pyenv`
for Python versions.
This has only been tested on Mas OS/Linux.

1. If you need brew, `make install-homebrew`.
2. If you need pyenv, `make install-pyenv` and `make install-python-3.8`.
3. `make setup` Make setup will install poetry if needed.
```python
from semantic_model_generator.validate_model import validate_from_local_path
This is the script version run on the command line.
YAML_PATH = "/path/to/your/model_yaml.yaml"
SNOWFLAKE_ACCOUNT = "<your-snowflake-account>"
1. `poetry shell` . This will activate your virtual environment.
2. Run on your command line.
validate_from_local_path(
yaml_path=YAML_PATH,
snowflake_account=SNOWFLAKE_ACCOUNT
)
```bash
python -m semantic_model_generator.generate_model \
--base_tables "['<your-database-name-1>.<your-schema-name-1>.<your-base-table-or-view-name-1>','<your-database-name-2>.<your-schema-name-2>.<your-base-table-or-view-name-2>']" \
--semantic_model_name "<a-meaningful-semantic-model-name>" \
--snowflake_account="<your-snowflake-account>"
```
### Post-Generation
#### Additional Fields to Fill Out
## Usage
### Auto-Generated Descriptions
If your snowflake tables and comments do not have comments, we currently
leverages [cortex LLM function](https://docs.snowflake.com/en/user-guide/snowflake-cortex/llm-functions) to
auto-generate description suggestions. Those generation are suffixed with '__' and additional comment to remind you to
confirm/modity the descriptions.
### Additional Fields to Fill Out
**IMPORTANT**: After generation, your YAML files will have a series of lines with `# <FILL-OUT>`. Please take the time
to fill these out with your business context, or else subsequent validation of your model will fail.
Expand All @@ -239,45 +274,6 @@ In addition, consider adding the following elements to your semantic model:
2. Synonyms. Any additional synonyms for column names.
3. Filters. Additional filters with their relevant `expr`.
#### Validating Yaml Updates

After you've edited your semantic model, you can validate this file before uploading.

1. Using Python. Ensure you've installed the package.

```python
from semantic_model_generator.validate_model import validate_from_local_path

YAML_PATH = "/path/to/your/model_yaml.yaml"
SNOWFLAKE_ACCOUNT = "<your-snowflake-account>"

validate_from_local_path(
yaml_path=YAML_PATH,
snowflake_account=SNOWFLAKE_ACCOUNT
)

```

2. Using the command line. Ensure `poetry shell` is activated.

```bash
python -m semantic_model_generator.validate_model \
--yaml_path="/path/to/your/model_yaml.yaml \
--snowflake_account="<your-account-name>"
```
## admin-app
To install the admin app dependencies, please run
```bash
make setup_admin_app
```
from the root repo directory. This will install dependencies from the base wheel as well as admin-app specific
dependencies.
See `admin_apps/README.md` for an overview of the feature as well as how to run the app.
## Examples
Expand Down

0 comments on commit 2d63258

Please sign in to comment.