Add indexes to tables to improve performance

v0.1.26-beta

Merge pull request #77 from pageuppeople-opensource/execution_model-i…

Allow values to be preserved after row deletion

v0.1.25-beta

Merge pull request #76 from pageuppeople-opensource/preserve-after-de…

Increase read-timeout from 60 to 200 when using AWSLambdaSource

v0.1.24-beta

Merge pull request #75 from pageuppeople-opensource/aws-source-increa…

Assume Role Support

Changes

• Add CONNECTION_DATA_ROLE_KEY constant
• Add CONNECTION_DATA_FUNCTION_KEY constant
• Add CONNECTION_DATA_TENANT_KEY constant
• Add AWS_SERVICE_LAMBDA constant
• Add AWS_SERVICE_S3 constant
• Add sts client to AWSLambdaDataSource
• Add __assume_role(...) method
  • Returns credentials from response
• Add __get_aws_client(...) method
• Add __refresh_aws_clients_if_expired(...) method
• Reformat code

Notes

Assume Role

The following clients will now be instantiated with AWS credentials:

• Lambda
• S3

If the clients are nearing their expiration date (~5 minutes), we will re-assume the role for a fresh set of credentials and update the clients accordingly.

AWS Lambda Connection String

Now expects role=arn:aws:iam:{accountId}:role/{roleName}

Add retry delay to lambda calls

There are some scenarios where if a lambda fails for a transient issue, an immediate retry is not sufficient.

Update boto3 to 1.9.224

Merge pull request #72 from pageuppeople-opensource/task/update-boto-3

Update boto3 to 1.9.224

Add retry policy when invoking lambda

When using, AWSLambdaDataSource, we now try at least 3 times before giving up in case of any errors received from the lambda.

Strip postgres-unsupported utf8 null characters

• Strip postgres-unsupported utf8 null characters
• Send column metadata when using AWSLambdaDataSource

Log all available details when logging execution and execution-model

• Log all available details when logging execution and execution-model
• Record execution-time as at least 1 second per model / execution in case of sub-second executions
• Log DataSource to be used for execution

Persist count of batches processed per model and aggregate per execution

v0.1.17-beta

Merge pull request #60 from pageuppeople-opensource/feature/SP-149-pe…