This repository contains lambda function and ECS task code for the Connected Vehicle (CV) Pilot data ingestion pipeline (from the private ingestion buckets to the public ITS Sandbox). For more information on ITS Sandbox data, please refer to the ITS Sandbox README page.
The utilities for working with the ITS Sandbox data and Socrata have been moved to the Sandbox Exporter package. Utilities that have migrated include: Sandbox Exporter, Data Flattener, and Socrata Connector. The pipeline for ingestion from the ITS Sandbox into Socrata has been moved to the cv-lake-to-socrata repository.
These instructions will get you a copy of the project up and running on your local machine for development and testing purposes. See deployment for notes on how to deploy the project on a live system.
If you plan to deploy the script on AWS Lambda, you need access to an AWS account and be able to assign role(s) to a lambda function. There needs to be a role that is able to invoke lambda functions and perform list/read/write actions to relevant buckets in S3.
If you plan to deploy the script on your local machine, you need the following:
- Have access to Python 3.6+. You can check your python version by entering
python --versionandpython3 --versionin command line. - Have access to the command line of a machine. If you're using a Mac, the command line can be accessed via the Terminal, which comes with Mac OS. If you're using a Windows PC, the command line can be accessed via the Command Prompt, which comes with Windows, or via Cygwin64, a suite of open source tools that allow you to run something similar to Linux on Windows.
- Have your own Free Amazon Web Services account.
- Create one at http://aws.amazon.com
- Obtain Access Keys:
- On your Amazon account, go to your profile (at the top right)
- My Security Credentials > Access Keys > Create New Access Key
- Record the Access Key ID and Secret Access Key ID (you will need them in step 4)
- Save your AWS credentials in your local machine, using one of the following method:
- shared credentials file: instructions at https://boto3.amazonaws.com/v1/documentation/api/latest/guide/configuration.html#shared-credentials-file.
- environmental variables: instructions at https://boto3.amazonaws.com/v1/documentation/api/latest/guide/configuration.html#environment-variables
- Download the script by cloning the git repository at https://github.com/usdot-its-jpo-data-portal/cv-ingest-to-lake. You can do so by running the following in command line.
git clone https://github.com/usdot-its-jpo-data-portal/cv-ingest-to-lake.git. If unfamiliar with how to clone a repository, follow the guide at https://help.github.com/en/articles/cloning-a-repository. - Navigate into the repository folder by entering
cd cv-ingest-to-lakein command line. - Create a virtualenv folder by running
virtualenv --python=python3 temp_env/. If you do not have virtualenv installed, you may install it by following the virtualenv installation instruction. - Activate the virtualenv by running
source temp_env/bin/activate. - Install the required packages in your virtualenv by running
pip install -r src/requirements.txt.
To build the Docker image, run:
docker build -t cv-ingest-to-lake .
To build the image without cache, run:
docker build -t cv-ingest-to-lake . --no-cache
If this is your first time running tests, run the following line in your command line to install the required packages:
pip install coverage "pytest<5"
Run the test by entering the following in command line:
coverage run -m pytest
Run coverage report by entering the following in command line:
coverage report -m
- To prepare the code package for deployment to AWS Lambda, run
sh package.shto build the packages. This will create two zipped files in the repo's root folder:ingest_to_lake.zip. - For each of the lambdas, create a lambda function in your AWS account "from scratch" with the following settings:
- Runtime: Python 3.8
- Permissions: Use an existing role (choose existing role with full lambda access (e.g. policy AWSLambdaFullAccess), list/read/write permission to your destination s3 bucket, and full ECS access)
- In the configuration view of your lambda function, set the following:
- For the
cv-ingest-to-lakefunction:- In "Function code" section, select "Upload a .zip file" and upload the
cv-ingest-to-lake.zipfile as your "Function Package." - In "Environment variables" section, set the following:
TARGET_BUCKET: the destination s3 bucket (sandbox bucket).- default set as: usdot-its-cvpilot-public-data
ECS_TASK_JSON: Stringified json containing the following information, replacing<values>with your own. {"cluster": , "launchType": "FARGATE", "taskDefinition": , "count": 1, "platformVersion": "LATEST", "networkConfiguration": {"awsvpcConfiguration": {"subnets": , "securityGroups": , "assignPublicIp": "DISABLED"}}, "overrides": {"containerOverrides": []}}
- In "Basics settings" section, set adequate Memory and Timeout values. Memory of 1664 MB and Timeout value of 10 minutes should be plenty.
- In "Triggers" section, set the S3 bucket(s) where data provider(s) is depositing data files to trigger on Object Creation.
- In "Function code" section, select "Upload a .zip file" and upload the
- For the
- Make sure to save all of your changes.
Deploy built Docker image to AWS Elastic Container Registry (ECR). Steps 2-5 are also available via AWS Console when you select your ECR repository and click on "View push commands". Replace all reference of $AWS_ACCOUNT_NUMBER with your own AWS account number.
-
If repository does not exist yet, create repository in AWS Console or using AWS Command Line Tool
-
Retrieve an authentication token and authenticate your Docker client to your registry.
aws ecr get-login-password --region us-east-1 | docker login --username AWS --password-stdin $AWS_ACCOUNT_NUMBER.dkr.ecr.us-east-1.amazonaws.com -
If you've already built your container, tag your image so you can push the image to this repository:
docker tag cv-ingest-to-lake:latest $AWS_ACCOUNT_NUMBER.dkr.ecr.us-east-1.amazonaws.com/cv-ingest-to-lake:latest -
Run the following command to push this image to your newly created AWS repository:
docker push $AWS_ACCOUNT_NUMBER.dkr.ecr.us-east-1.amazonaws.com/cv-ingest-to-lake:latest
Note that in the task definition, this container should have the default environment variable TARGET_BUCKET, which is set to the same value as in the lambda function.
The lambda function is expected to be invoked code.
The ECS task is expected to be invoked by the cv-ingest-to-lake lambda function.
In our deployment, the cv-ingest-to-lake is invoked by deposit of data into private S3 ingestion buckets.
Status: This project is in the release phase.
Release Frequency: This project is updated irregularly.
Retention: This project will remain publicly accessible for a minimum of five years (until at least 08/19/2026).
- 2.0.0
- Moved sandbox UI html page to sandbox repository.
- Moved code for ingesting sandbox data to Socrata to cv-lake-to-socrata.
- Allows the same ingestion code to be deployed and run as an ECS Task and have the lambda function trigger an ECS Task with the triggering event information when data comes from NYCDOT.
- Rename repostory to cv-ingest-to-lake.
- 1.0.0
- Refactored to use our sandbox_exporter package to reduce duplicative code.
- 0.1.0
- Initial version.
This project is licensed under the Apache 2.0 License. See LICENSE for more information.
- Fork it (https://github.com/usdot-its-jpo-data-portal/cv-ingest-to-lake/fork)
- Create your feature branch (git checkout -b feature/fooBar)
- Commit your changes (git commit -am 'Add some fooBar')
- Push to the branch (git push origin feature/fooBar)
- Create a new Pull Request
Contact Name: ITS JPO
Contact Information: [email protected]
When you copy or adapt from this code, please include the original URL you copied the source code from and date of retrieval as a comment in your code.
Thank you to the Department of Transportation for funding to develop this project.