refactor!: Consistent Environment Variable and Application Names (#141)

* refactor!(envar): Standardize Env Var

* refactor!(cmd): Standardize Cmd Main Files

* docs(terraform): Update Terraform Docs

* docs(cmd): Update Autoscale Env Var

CONTRIBUTING.md

@@ -55,7 +55,7 @@ We rely on contributors to test changes before they are submitted as pull reques
 
 #### Environment Variables
 
-Applications may implement runtime settings that are managed by environment variables. For example, the [AWS Lambda application](/cmd/aws/lambda/substation/) uses `SUBSTATION_HANDLER` to manage [invocation settings](https://docs.aws.amazon.com/lambda/latest/dg/lambda-invocation.html).
+Applications may implement runtime settings that are managed by environment variables. For example, the [AWS Lambda application](/cmd/aws/lambda/substation/) uses `SUBSTATION_LAMBDA_HANDLER` to manage [invocation settings](https://docs.aws.amazon.com/lambda/latest/dg/lambda-invocation.html). These should reference the application by name, if possible.
 
 #### Configurations
 

README.md

@@ -444,10 +444,10 @@ module "node" {
 
     env = {
       "SUBSTATION_CONFIG" : "https://localhost:2772/applications/substation/environments/example/configurations/node"
+      "SUBSTATION_DEBUG" : true
       # This Substation node will ingest data from API Gateway. More nodes can be 
       # deployed to ingest data from other sources, such as Kinesis or SQS.
-      "SUBSTATION_HANDLER" : "AWS_API_GATEWAY"
-      "SUBSTATION_DEBUG" : true
+      "SUBSTATION_LAMBDA_HANDLER" : "AWS_API_GATEWAY"
     }
   }
 

build/terraform/aws/README.md

@@ -1,112 +1,28 @@
 # terraform
 
-This directory contains Terraform configuration files for deploying Substation to AWS.
+This directory contains Terraform modules for deploying Substation to AWS.
 
 ## Using Terraform
 
-An overview on how Terraform is used and the write, plan, apply workflow is available [here](https://www.terraform.io/intro).
-
-Please use the official [Terraform documentation](https://registry.terraform.io/providers/hashicorp/aws/latest/docs) for further details on using Terraform to deploy AWS applications or for boiler plate examples.
+An overview on how Terraform is used and the write, plan, apply workflow is available [here](https://www.terraform.io/intro). Refer to the [AWS provider documentation](https://registry.terraform.io/providers/hashicorp/aws/latest/docs) for more information on using Terraform with AWS.
 
 ## Modules
 
-Due to the potentially endless number of deployment configurations, Substation comes with Terraform modules that act as component templates for different parts of the system. These modules are designed to be flexible enough to support every known deployment configuration. For an example deployment, see `example.tf` .
-
-### API Gateway
-
-This module is used as a template for deploying new API Gateway endpoints. 
-
-Read more about API Gateway [here](https://aws.amazon.com/api-gateway/).
-
-#### Kinesis
-
-This module creates an API Gateway that sends a record to a Kinesis Data Stream. To prevent hot shards, the partition key is the request ID of the HTTP request.
-
-#### Lambda
-
-This module creates an API Gateway that invokes and sends a record to a Lambda function.
-
-### CloudWatch
-
-#### Destination
-
-This module creates a CloudWatch Logs destination that can be used to receive logs from any AWS account or region and send them to a destination.
-
-#### Subscription
-
-This module creates a CloudWatch Logs subscription filter that can be used to send logs from a CloudWatch Logs group to a destination. Use this with the `Destination` module to send logs from any AWS account or region to a single destination.
-
-### DynamoDB
-
-This module is used as a template for deploying new DynamoDB tables with autoscaling enabled. These tables have [time to live](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/TTL.html) enabled and users can optionally use it by writing values to the `ttl` column.
-
-Read more about DynamoDB [here](https://aws.amazon.com/dynamodb/).
-
-### ECR
-
-This module is used as a template for deploying new image repositories to the AWS Elastic Container Registry (ECR) service.
-
-There are two things to be aware of when deploying new image repositories:
-
-* Terraform does not manage the build and deployment of container images; after an image repository is created, then container build and upload should happen externally via Docker (see `build/docker` for more information)
-* By default all Lambda deployments use containers, but a Lambda deployment will fail if the image repository is empty; prevent deployment failures by building and uploading images before deploying Lambda
-
-Read more about ECR [here](https://aws.amazon.com/ecr/).
-
-### Event Bridge
-
-#### Lambda
-
-This module is used to create Event Bridge rules that trigger a Lambda.
-
-Read more about Event Bridge [here](https://aws.amazon.com/eventbridge/).
-
-### Kinesis
-
-This module is used to create new Kinesis Data Streams (KDS) and accompanying CloudWatch alarms. The streams created by this module are intended to be used with Substation's autoscaling application -- this feature provides stream autoscaling at a significantly reduced cost compared to Kinesis Firehose.
-
-For best practices, we recommend users deploy data pipelines that use two Kinesis streams: a stream for pre-processed (i.e., raw) data and a stream for post-processed (i.e., processed) data. This deployment strategy is useful for supporting concurrent access to data across many applications.
-
-Read more about Kinesis Data Streams [here](https://aws.amazon.com/kinesis/data-streams/).
-
-### KMS
-
-This module is used to create encryption keys used across a deployment. Any number of keys can be made, but by default we create one key that is shared by all encrypted resources and data.
-
-Read more about the Key Management Service [here](https://aws.amazon.com/kms/).
-
-### Lambda
-
-This module is used to create and manage Lambda, which is the recommended service for data processing. 
-
-This module is flexible enough to deploy supporting apps (such as `cmd/aws/lambda/kinesis_autoscaling`) and custom apps (such as apps that provide data enrichment functionality). When new Lambda are created with this module, an accompanying AppConfig configuration profile is created under the `substation` application.
-
-Read more about AWS Lambda [here](https://aws.amazon.com/lambda/).
-
-### S3
-
-This module is used as a template for deploying new S3 buckets. These buckets are private and objects are encrypted.
-
-Read more about S3 [here](https://aws.amazon.com/s3/).
-
-#### WORM
-
-This module creates a write once, read many (WORM) S3 bucket using [Object Lock](https://docs.aws.amazon.com/AmazonS3/latest/userguide/object-lock-overview.html) and applies the [Compliance](https://docs.aws.amazon.com/AmazonS3/latest/userguide/object-lock-overview.html#object-lock-retention-modes) retention mode setting. Objects written to these buckets _*cannot be deleted*_.
-
-### SNS
-
-This module is used for creating SNS topics. Topics can be standard or first-in-first-out (FIFO).
-
-Read more about SNS [here](https://aws.amazon.com/sns/).
-
-### SQS
-
-This module is used for creating SQS queues. Queues can be standard or first-in-first-out (FIFO).
-
-Read more about SQS [here](https://aws.amazon.com/sqs/).
-
-### Networking
-
-This module can be used to create a custom VPC with outbound connectivity via a NAT gateway to a public subnet that contains an IGW. This allows for connectivity with VPC only services.
-
-Read more about VPCs [here](https://aws.amazon.com/vpc/).
+Due to the potentially endless number of deployment configurations, Substation includes Terraform modules for these AWS services:
+
+* API Gateway
+* AppConfig
+* CloudWatch
+* DynamoDB
+* Elastic Container Registry (ECR)
+* Event Bridge
+* Kinesis Data Streams
+* Key Management Service (KMS)
+* Lambda
+* S3
+* Secrets Manager
+* SNS
+* SQS
+* VPC Networking
+
+Refer to each module's README for more information. Several examples of how to use these modules are available [here](/examples/terraform/aws/).

build/terraform/aws/appconfig/README.md

@@ -23,12 +23,14 @@ No modules.
 | [aws_appconfig_application.app](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/appconfig_application) | resource |
 | [aws_appconfig_deployment_strategy.instant](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/appconfig_deployment_strategy) | resource |
 | [aws_appconfig_environment.env](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/appconfig_environment) | resource |
+| [aws_lambda_permission.allow_appconfig](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/lambda_permission) | resource |
 
 ## Inputs
 
 | Name | Description | Type | Default | Required |
 |------|-------------|------|---------|:--------:|
 | <a name="input_config"></a> [config](#input\_config) | Configuration for the AppConfig application:<br><br>    * name: The name of the AppConfig application.<br>    * environments: A list of environments to create for the AppConfig application. | <pre>object({<br>    name = string<br>    environments = list(object({<br>      name = string<br>    }))<br>  })</pre> | n/a | yes |
+| <a name="input_lambda"></a> [lambda](#input\_lambda) | Lambda function used to validate configuration profiles. | <pre>object({<br>    name = string<br>    arn  = string<br>    role = object({<br>      name = string<br>      arn  = string<br>    })<br>  })</pre> | `null` | no |
 | <a name="input_tags"></a> [tags](#input\_tags) | n/a | `map(any)` | `{}` | no |
 
 ## Outputs
@@ -37,4 +39,5 @@ No modules.
 |------|-------------|
 | <a name="output_arn"></a> [arn](#output\_arn) | The ARN of the AppConfig application. |
 | <a name="output_id"></a> [id](#output\_id) | The ID of the AppConfig application. |
+| <a name="output_lambda"></a> [lambda](#output\_lambda) | The validator Lambda function passed to the AppConfig application. |
 <!-- END_TF_DOCS -->

build/terraform/aws/lambda/README.md

@@ -41,7 +41,7 @@ No modules.
 | Name | Description | Type | Default | Required |
 |------|-------------|------|---------|:--------:|
 | <a name="input_access"></a> [access](#input\_access) | List of IAM ARNs that are granted access to the resource. | `list(string)` | `[]` | no |
-| <a name="input_appconfig"></a> [appconfig](#input\_appconfig) | AppConfig application used for configuring the function. If not provided, then no AppConfig configuration will be created for the function. | <pre>object({<br>    arn = string<br>    id  = string<br>  })</pre> | `null` | no |
+| <a name="input_appconfig"></a> [appconfig](#input\_appconfig) | AppConfig application used for configuring the function. If not provided, then no AppConfig configuration will be created for the function. | <pre>object({<br>    arn = string<br>    id  = string<br>    lambda = optional(object({<br>      name = string<br>      arn  = string<br>      role = object({<br>        name = string<br>        arn  = string<br>      })<br>    }))<br>  })</pre> | `null` | no |
 | <a name="input_config"></a> [config](#input\_config) | Configuration for the Lambda function:<br><br>    * name: The name of the Lambda function.<br>    * description: The description of the Lambda function.<br>    * image\_uri: The URI of the container image that contains the function code.<br>    * image\_arm: Determines whether the image is an ARM64 image.<br>    * timeout: The amount of time that Lambda allows a function to run before stopping it. The default is 300 seconds.<br>    * memory: The amount of memory that your function has access to. The default is 1024 MB.<br>    * env: A map that defines environment variables for the function.<br>    * vpc\_config: A map that defines the VPC configuration for the function.<br>    * iam\_statements: A list of custom IAM policy statements to attach to the function's role. | <pre>object({<br>    name        = string<br>    description = string<br>    image_uri   = string<br>    image_arm   = bool<br>    timeout     = optional(number, 300)<br>    memory      = optional(number, 1024)<br>    env         = optional(map(any), null)<br>    vpc_config = optional(object({<br>      subnet_ids         = list(string)<br>      security_group_ids = list(string)<br>      }), {<br>      subnet_ids         = []<br>      security_group_ids = []<br>    })<br>    iam_statements = optional(list(object({<br>      sid       = string<br>      actions   = list(string)<br>      resources = list(string)<br>    })), [])<br>  })</pre> | n/a | yes |
 | <a name="input_kms"></a> [kms](#input\_kms) | Customer managed KMS key used to encrypt the function's environment variables. If not provided, then an AWS managed key is used. See https://docs.aws.amazon.com/lambda/latest/dg/security-dataprotection.html#security-privacy-atrest for more information. | <pre>object({<br>    arn = string<br>    id  = string<br>  })</pre> | `null` | no |
 | <a name="input_tags"></a> [tags](#input\_tags) | Tags to apply to all resources. | `map(any)` | `{}` | no |

cmd/aws/lambda/README.md

@@ -22,7 +22,7 @@ This app handles Kinesis Data Stream autoscaling through SNS notifications and C
 
 Stream utilization is based on volume (i.e., 60, 000 events per minute) and size (i.e., 10GB data per minute); these values are converted to a percentage (0.0 to 1.0) and the maximum of either is considered the stream's current utilization.
 
-By default, streams must be above the upper threshold for all 5 minutes to scale up and below the lower threshold for at least 57 minutes to scale down. These values can be overriden by the environment variables SUBSTATION_AUTOSCALING_UPSCALE_DATAPOINTS (cannot exceed 5 minutes) and SUBSTATION_AUTOSCALING_DOWNSCALE_DATAPOINTS (cannot exceed 60 minutes).
+By default, streams must be above the upper threshold for all 5 minutes to scale up and below the lower threshold for at least 57 minutes to scale down. These values can be overriden by the environment variables AUTOSCALE_KINESIS_UPSCALE_DATAPOINTS (cannot exceed 5 minutes) and AUTOSCALE_KINESIS_DOWNSCALE_DATAPOINTS (cannot exceed 60 minutes).
 
 For example:
 

cmd/aws/lambda/substation/main.go

@@ -16,10 +16,10 @@ var (
 	handler string
 
 	// errLambdaMissingHandler is returned when the Lambda is deployed without a configured handler.
-	errLambdaMissingHandler = fmt.Errorf("SUBSTATION_HANDLER environment variable is missing")
+	errLambdaMissingHandler = fmt.Errorf("SUBSTATION_LAMBDA_HANDLER environment variable is missing")
 
 	// errLambdaInvalidHandler is returned when the Lambda is deployed with an unsupported handler.
-	errLambdaInvalidHandler = fmt.Errorf("SUBSTATION_HANDLER environment variable is invalid")
+	errLambdaInvalidHandler = fmt.Errorf("SUBSTATION_LAMBDA_HANDLER environment variable is invalid")
 
 	// errLambdaInvalidJSON is returned when the Lambda is deployed with a transform that produces invalid JSON.
 	errLambdaInvalidJSON = fmt.Errorf("transformed data is invalid JSON and cannot be returned")
@@ -87,7 +87,7 @@ func main() {
 
 func init() {
 	var ok bool
-	handler, ok = os.LookupEnv("SUBSTATION_HANDLER")
+	handler, ok = os.LookupEnv("SUBSTATION_LAMBDA_HANDLER")
 	if !ok {
 		panic(fmt.Errorf("init handler %s: %v", handler, errLambdaMissingHandler))
 	}

examples/terraform/aws/cloudwatch_logs/cross_account_cross_region/terraform/consumer.tf

@@ -10,7 +10,7 @@ module "lambda_consumer" {
 
     env = {
       "SUBSTATION_CONFIG" : "http://localhost:2772/applications/substation/environments/example/configurations/consumer"
-      "SUBSTATION_HANDLER" : "AWS_KINESIS_DATA_STREAM"
+      "SUBSTATION_LAMBDA_HANDLER" : "AWS_KINESIS_DATA_STREAM"
       "SUBSTATION_DEBUG" : true
     }
   }

examples/terraform/aws/cloudwatch_logs/to_lambda/terraform/consumer.tf

@@ -13,7 +13,7 @@ module "lambda_consumer" {
 
     env = {
       "SUBSTATION_CONFIG" : "http://localhost:2772/applications/substation/environments/example/configurations/consumer"
-      "SUBSTATION_HANDLER" : "AWS_LAMBDA"
+      "SUBSTATION_LAMBDA_HANDLER" : "AWS_LAMBDA"
       "SUBSTATION_DEBUG" : true
     }
   }

examples/terraform/aws/dynamodb/cdc/terraform/node.tf

@@ -9,7 +9,7 @@ module "node" {
     image_arm   = true
     env = {
       "SUBSTATION_CONFIG" : "http://localhost:2772/applications/substation/environments/example/configurations/node"
-      "SUBSTATION_HANDLER" : "AWS_DYNAMODB_STREAM"
+      "SUBSTATION_LAMBDA_HANDLER" : "AWS_DYNAMODB_STREAM"
       "SUBSTATION_DEBUG" : true
     }
   }

examples/terraform/aws/kinesis/firehose/terraform/_resources.tf

@@ -179,7 +179,7 @@ module "processor" {
     timeout = 60
     env = {
       "SUBSTATION_CONFIG" : "http://localhost:2772/applications/substation/environments/example/configurations/processor"
-      "SUBSTATION_HANDLER" : "AWS_KINESIS_DATA_FIREHOSE"
+      "SUBSTATION_LAMBDA_HANDLER" : "AWS_KINESIS_DATA_FIREHOSE"
       "SUBSTATION_DEBUG" : true
     }
   }

examples/terraform/aws/kinesis/multistream/terraform/publisher.tf

@@ -11,7 +11,7 @@ module "lambda_publisher" {
 
     env = {
       "SUBSTATION_CONFIG" : "http://localhost:2772/applications/substation/environments/example/configurations/publisher"
-      "SUBSTATION_HANDLER" : "AWS_KINESIS_DATA_STREAM"
+      "SUBSTATION_LAMBDA_HANDLER" : "AWS_KINESIS_DATA_STREAM"
       "SUBSTATION_DEBUG" : true
     }
   }

examples/terraform/aws/kinesis/multistream/terraform/subscriber.tf

@@ -11,7 +11,7 @@ module "lambda_subscriber" {
 
     env = {
       "SUBSTATION_CONFIG" : "http://localhost:2772/applications/substation/environments/example/configurations/subscriber"
-      "SUBSTATION_HANDLER" : "AWS_KINESIS_DATA_STREAM"
+      "SUBSTATION_LAMBDA_HANDLER" : "AWS_KINESIS_DATA_STREAM"
       "SUBSTATION_DEBUG" : true
     }
   }

examples/terraform/aws/kinesis/time_travel/terraform/enrichment.tf

@@ -10,7 +10,7 @@ module "lambda_enrichment" {
 
     env = {
       "SUBSTATION_CONFIG" : "http://localhost:2772/applications/substation/environments/example/configurations/enrichment"
-      "SUBSTATION_HANDLER" : "AWS_KINESIS_DATA_STREAM"
+      "SUBSTATION_LAMBDA_HANDLER" : "AWS_KINESIS_DATA_STREAM"
       "SUBSTATION_DEBUG" : true
     }
   }

examples/terraform/aws/kinesis/time_travel/terraform/subscriber.tf

@@ -10,7 +10,7 @@ module "lambda_subscriber" {
 
     env = {
       "SUBSTATION_CONFIG" : "http://localhost:2772/applications/substation/environments/example/configurations/subscriber"
-      "SUBSTATION_HANDLER" : "AWS_KINESIS_DATA_STREAM"
+      "SUBSTATION_LAMBDA_HANDLER" : "AWS_KINESIS_DATA_STREAM"
       "SUBSTATION_DEBUG" : true
     }
   }

examples/terraform/aws/lambda/appconfig/terraform/node.tf

@@ -14,7 +14,7 @@ module "node" {
     timeout = 10
     env = {
       "SUBSTATION_CONFIG" : "http://localhost:2772/applications/substation/environments/example/configurations/node"
-      "SUBSTATION_HANDLER" : "AWS_LAMBDA"
+      "SUBSTATION_LAMBDA_HANDLER" : "AWS_LAMBDA"
       "SUBSTATION_DEBUG" : true
     }
   }

examples/terraform/aws/lambda/microservice/terraform/microservice.tf

@@ -12,7 +12,7 @@ module "microservice" {
     timeout = 10
     env = {
       "SUBSTATION_CONFIG" : "http://localhost:2772/applications/substation/environments/example/configurations/microservice"
-      "SUBSTATION_HANDLER" : "AWS_LAMBDA"
+      "SUBSTATION_LAMBDA_HANDLER" : "AWS_LAMBDA"
       "SUBSTATION_DEBUG" : true
     }
   }

examples/terraform/aws/lambda/vpc/terraform/whatismyip.tf

@@ -18,7 +18,7 @@ module "whatismyip" {
 
     env = {
       "SUBSTATION_CONFIG" : "http://localhost:2772/applications/substation/environments/example/configurations/whatismyip"
-      "SUBSTATION_HANDLER" : "AWS_LAMBDA"
+      "SUBSTATION_LAMBDA_HANDLER" : "AWS_LAMBDA"
       "SUBSTATION_DEBUG" : true
     }
   }

examples/terraform/aws/s3/data_lake/terraform/node.tf

@@ -23,7 +23,7 @@ module "lambda_node" {
 
     env = {
       "SUBSTATION_CONFIG" : "http://localhost:2772/applications/substation/environments/example/configurations/node"
-      "SUBSTATION_HANDLER" : "AWS_API_GATEWAY"
+      "SUBSTATION_LAMBDA_HANDLER" : "AWS_API_GATEWAY"
       "SUBSTATION_DEBUG" : true
     }
   }

examples/terraform/aws/s3/sns/terraform/node.tf

@@ -11,7 +11,7 @@ module "lambda_node" {
 
     env = {
       "SUBSTATION_CONFIG" : "http://localhost:2772/applications/substation/environments/example/configurations/node"
-      "SUBSTATION_HANDLER" : "AWS_S3_SNS"
+      "SUBSTATION_LAMBDA_HANDLER" : "AWS_S3_SNS"
       "SUBSTATION_DEBUG" : true
     }
   }

examples/terraform/aws/s3/xdr/terraform/node.tf

@@ -10,7 +10,7 @@ module "lambda_node" {
 
     env = {
       "SUBSTATION_CONFIG" : "http://localhost:2772/applications/substation/environments/example/configurations/node"
-      "SUBSTATION_HANDLER" : "AWS_S3"
+      "SUBSTATION_LAMBDA_HANDLER" : "AWS_S3"
       "SUBSTATION_DEBUG" : true
     }
   }