This repository contains the data-ingestion job to propagate Pub/Sub data to the database.
It relies on Dialect map IO, which provides PubSub / API connection capabilities, and Dialect map schemas, which provides data (de)serialization capabilities, in order to send validated records to the Dialect map private API.
Python dependencies are specified on the multiple files within the reqs directory.
In order to install all the development packages, as well as the defined commit hooks:
make install-devAll Python files are formatted using Black, and the custom properties defined
in the pyproject.toml file.
make checkProject testing is performed using Pytest. In order to run the tests:
make testThe project contains a main.py module exposing a CLI with several commands:
python3 src/main.py [OPTIONS] [COMMAND] [ARGS]...This command starts a Google Pub/Sub subscription reading job, that dispatches data-diff message records coming from the Dialect map data repository, to the Dialect map private API.
| ARGUMENT | ENV VARIABLE | REQUIRED | DESCRIPTION |
|---|---|---|---|
| --gcp-project | - | Yes | GCP project name |
| --gcp-key-path | - | Yes | GCP Service account key path |
| --subscription | - | Yes | GCP Pub/Sub subscription name |
| --api-url | - | Yes | Private API base URL |