This is a Python project designed for us as an Amazon Web Services (AWS) Lambda function. It is set up to poll SQS for validation result messages, aggregate the errors into a packaged message, and send it to a Slack channel.
Designed for use in conjunction with the DataHub Canary Lambda function.
Since this code is designed to run in AWS, you will need an AWS account. Additionally you will need an SQS queue to which results are pushed (see the DataHub Canary Lambda function), and lastly a Slack webhook to which this function can send messages.
- Python 3.7 (virtualenv recommended when running locally)
- pip3
- SQS Queue with messages matching correct schema (see details below)
- (If using larger messages) S3 storage bucket for hosting large SQS messages
- Slack channel webhook
The following table lists the environment variables used for configuration.
| Property | Description |
|---|---|
| SQS_RESULT_QUEUE | SQS queue name (not URL or ARN, just the name) from which validation results will arrive |
| SLACK_WEBHOOK | Slack channel webhook to which messages will be sent |
| VERBOSE_OUTPUT | Set to TRUE to activate, all other falues will set this to false |
| RECIPIENTS_DICT | Stringified JSON with data provider names as the keys and list of emails as the values |
| SENDER | Email address from which to send the aggregated count and validation report |
| CC | Comma delimited email addresses that the email report will be Cc'ed to |
While the function works best as a Lambda function, you may also test it locally. You will still need to set up the aformentioned SQS queue and Slack webhook.
We recommend you use virtualenv.
Note that you will also need to have an AWS profile set up, see instructions here.
To install and run the function, follow the following steps:
pip install -r requirements.txtpython main.py
To deploy this as an AWS Lambda function:
- Create a new Lambda function with runtime
Python 3.7 - Give the lambda function appropriate permissions
s3:Get*for your S3/SQS storage buckets3:List*for your S3/SQS storage bucketsqs:*for your SQS queueses:*for permission to send email through AWS SES.
- Run
package.shlocally which will createresults.zip - Upload the
results.zipas the Lambda function code - Add a CloudWatch CRON event to trigger the function periodically
- Set the environment variables
Please use GitHub pull requests and issues to contribute to this project and report problems.
The ITS DataHub development team.
This project is licensed under the Apache License - see the LICENSE file for details.
SQS Extended Library Code sourced from https://github.com/timothymugayi/boto3-sqs-extended-client-lib. Markdown library sourced from https://python-markdown.github.io/.
Agency: DOT
Short Description: A Python AWS Lambda function designed to poll SQS for validation result messages, aggregate the errors into a packaged message, and send it to a Slack channel.
Status: Released
Tags: connected, vehicles, transportation, python, aws, lambda
Labor hours: 0
Contact Name: Brian Brotsos
Contact Phone: (202) 366-3000