InGen is a command line tool written on top of pandas and great_expectations to perform small scale data transformations and validations without writing code. It is designed for developers and analysts to quickly transform data by specifying their requirements in a simple YAML file.
Note: This project has been tested on python version 3.9.6 on MacOS Ventura with apple chip.
To install the project locally follow the steps:
- Make sure you have Python 3.9.6 installed on your system.
- To be able to build the project locally, you will need to install the
buildpackagepip install build - Clone the repository
git clone [email protected]:blackrock/ingen.git - Build the project
cd ingen python -m build - Install the wheel
pip install dist/ingen-*.whl - Run the project
python -m ingen <metadata file path>
Checkout the sample metadata files in the examples directory to see how InterfaceGenerator can be used to solve
common data problems.
| Examples |
|---|
| Merge two CSV files |
| Database as source |
| Convert XML to CSV/Excel |
For a complete list of configuration options, see the metadata reference documentation here.
All contributions are welcome, please see open issues or create a new issue to discuss your ideas. Please see our contributing guidelines for more information.