Merge pull request #26 from ExpediaDotCom/fix_rate_limit_issue6

Fix "Rate exceeded" issues for large clusters

Dockerfile

@@ -1,6 +1,16 @@
 # https://hub.docker.com/r/library/node/
 
-FROM node:4.1
+FROM node:9.11.1-alpine
+
+# >> FIX:
+# Fixes error Ubuntu: "gyp ERR! stack Error: Can't find Python executable "python", you can set the PYTHON env variable"
+# REF: https://gist.github.com/vidhill/0a85dc1848feee4171944dc4d7757895
+# REF: https://github.com/nodejs/node-gyp/issues/1105
+
+# build base includes g++ and gcc and Make
+RUN apk update && apk add python build-base
+
+# << END FIX
 
 # Bundle app source
 COPY . /

README.md

@@ -1,6 +1,6 @@
 # c3vis - Cloud Container Cluster Visualizer
 
-Helps visualize the resource utilisation of Amazon ECS clusters.
+Helps visualize the resource reservation of Amazon ECS clusters.
 
 Deploying software as “containers” promises to solve many problems with regards to interoperability of environments, speed to deploy, and cost reduction.
 But understanding where our software lives now becomes more difficult both for development and operations teams.
@@ -16,118 +16,94 @@ Each unique Task Definition is represented as a different color, with the legend
 Each Task will contain one or more containers, the task box shows accumulated reserved memory or CPU for all containers in the Task. ECS Services are not currently represented.
 
 
-## Configure
+## Configuration
 
-Displaying live ECS data requires server-side AWS credentials.
+See [CONFIGURATION](docs/CONFIGURATION.md) for details on server-side configurable options that affect cache entry TTL and AWS API call throttling.
 
-### Region
+## Configuring AWS SDK
 
-Before running, add a file named ```aws_config.json``` to this project root directory.  At a minimum set the default region:
+See [AWS_SDK_CONFIGURATION](docs/AWS_SDK_CONFIGURATION.md) for instructions 
+on configuring the AWS SDK for server-side AWS connectivity.
 
-```
-{
-  "region": "<default-region>"
-}
-```
+## Requirements
+
+Node >= 0.12
 
-Alternatively, set the environment variable "AWS_REGION" before starting the server.
+## Building and Running
 
-### Credentials
+The c3vis server is based on ExpressJS. The client predominantly uses D3.js, 
+jQuery and Bootstrap.
 
-AWS credentials properties "accessKeyId" and "secretAccessKey" can be added to the aws_config.json file as per https://docs.aws.amazon.com/AWSJavaScriptSDK/guide/node-configuring.html.
+Run the following to build and run the server ("package.json" contains instructions to pre-install required node modules):
 
-Otherwise, the credentials will be loaded from the Shared Credentials File or Environment Variables or IAM roles if deployed to an AWS instance.
+```
+npm start
+```
 
-When using an IAM role, ensure the role has the following access:
+Now browse to the app at `http://localhost:3000`.
 
-* ecs:listContainerInstances
-* ecs:describeContainerInstances
-* ecs:listTasks
-* ecs:describeTasks
-* ecs:describeTaskDefinition
-* ecs:listClusters
-* ec2:describeInstance
+## Testing
 
-Sample IAM Inline Policy:
+To run the server-side unit test suite with mocha and chai:
+
 ```
-{
-    "Version": "2012-10-17",
-    "Statement": [
-        {
-            "Effect": "Allow",
-            "Action": [
-                "ecs:listContainerInstances",
-                "ecs:describeContainerInstances",
-                "ecs:listTasks",
-                "ecs:describeTasks",
-                "ecs:describeTaskDefinition",
-                "ecs:listClusters",
-                "ec2:describeInstances"
-            ],
-            "Resource": [
-                "*"
-            ]
-        }
-    ]
-}
+npm run test
 ```
 
-**WARNING:** c3vis makes ECS data from the above API calls (including environment variables in task definitions) available to clients/browsers.
-Ensure the c3vis server is available only to users that should have access to this information.
+## Usage
 
+### Approach
 
-## Requirements
+When a client browser first connects to the c3vis server, the Cluster dropdown will be populated with ECS cluster names for the configured region.
 
-Node >= 0.12
+Select from the dropdown to view the resources allocated to that cluster. If no cluster names appear in the dropdown, check the server logs and ensure the correct region is configured (see below).
 
-## Building and Running
+The list of clusters and the user's current selection are stored in cookies. Use the ```[refresh list]``` dropdown entry to refresh the list of clusters.
 
-Server is based on ExpressJS. Client uses D3.js.
+The Y axis shows total memory or CPU available for the instances. Memory is the default resource type represented. Use the "resourceType" query parameter to toggle between "memory" and "cpu".  E.g. ```localhost:3000/?resourceType=cpu```
 
-Run the following to build and run the server ("package.json" contains instructions to pre-install required node modules):
+The X axis displays the Private IP Address for each EC2 instance. Right-clicking the IP address shows the context menu with links to browse the instance in the ECS and EC2 consoles.
 
-```
-npm start
-```
+### AWS API Call Throttling
 
-This will run ```npm install``` and ```node --harmony ./bin/www```
-(NOTE: ```"--harmony"``` is required for ES6 functionality such as Array.find())
+In order to prevent AWS API Rate limiting issues for large clusters, the server:
 
-Now browse to the app at `http://localhost:3000`.
+* Introduces a delay between API calls (configurable via `aws.apiDelay` setting)
+* Limits the number of items retrieved per page in `list` and `describe` API calls (configurable via `aws.*PageSize`)
+* Limits the number of asynchronous API calls it makes at a time (configurable via `aws.maxSimultaneous*Calls`)
 
-### Usage
+You can increase or decrease each of these settings to suit each environment c3vis is deployed to.
 
-When a client browser first connects to the c3vis server the Cluster dropdown will be populated with ECS cluster names for the configured region.
+### Short Polling, Server-Side Caching and Fetch Status
 
-Select from the dropdown to view the resources allocated to that cluster. If no cluster names appear in the dropdown, check the server logs and ensure the correct region is configured (see below).
+For each cluster requested, the server caches cluster data in-memory while the client polls the server until the cache is populated.
 
-The list of clusters and the user's current selection are stored in cookies. Use the ```[refresh list]``` dropdown entry to refresh the list of clusters.
+For an explanation on how the client polls the server for cluster data and the applicable fetch statuses, see [SHORT_POLLING_FETCH_STATUS](docs/SHORT_POLLING_FETCH_STATUS.md).
 
-The Y axis shows total memory or CPU available for the instances. Memory is the default resource type represented. Use the "resourceType" query parameter to toggle between "memory" and "cpu".  E.g. ```localhost:3000/?resourceType=cpu```
 
-The X axis displays the Private IP Address for each EC2 instance. Right-clicking the IP address shows the context menu with links to browse the instance in the ECS and EC2 consoles.
+## Debugging
 
 ### Sample Clusters for Testing
 
 From the browser, use a ```"?static=true"``` query parameter to have the server return static test data. Useful for testing when server is unable to connect to AWS.
 
 Browse to `http://localhost:3000/?static=true`.
 
-### Debugging
+### Server Debug Logging
 
-Add the following line to server-side Javascript code to add a breakpoint:
+To see all debug log entries:
 
 ```
-debugger;
+DEBUG=* npm start
 ```
 
-then run the debugger with:
+To see just API debug log entries:
 
 ```
-node debug --harmony ./bin/www
+DEBUG=api npm start
 ```
 
-### Running with Docker
+## Running with Docker
 
 Build and tag the image:
 
@@ -141,8 +117,14 @@ Run the container: (can remove ```AWS_ACCESS_KEY_ID``` and ```AWS_SECRET_ACCESS_
 docker run -e "AWS_REGION=<region>" -e "AWS_ACCESS_KEY_ID=<accesskey>" -e "AWS_SECRET_ACCESS_KEY=<secretkey>" -p 3000:3000 c3vis
 ```
 
+E.g. To run with `prod` target environment configuration:
+
+```
+docker run -e "TARGET_ENV=prod" -p 3000:3000 c3vis
+```
+
 
-Browse to `<docker localhost>:3000` (e.g. [http://192.168.99.100:3000](http://192.168.99.100:3000))
+Browse to [http://localhost:3000](http://localhost:3000)
 
 
 # Credits
@@ -154,4 +136,4 @@ Created by [Matt Callanan](https://github.com/mattcallanan) with contributions f
 
 This project is available under the [Apache 2.0 License](http://www.apache.org/licenses/LICENSE-2.0.html).
 
-Copyright 2015 Expedia Inc.
+Copyright 2018 Expedia Inc.

TODO.md

@@ -1,19 +1,24 @@
+## UI
 * Menubar with:
   * Selectable Region
   * Selectable AWS account – requires ability for mutliple config files server side
 * Show clusters as tabbed view with one cluster per tab
-* Add toggle button to switch between memory vs CPU resourceType
 * Show an exploded view of task with more details when hovering over tasks:
   * Show containers within tasks
   * Show memory breakdown across containers
 * Sliding timebar to see historical data for comparison (like google street view)
 * Show container actual memory utilisation vs reserved memory utilisation
+* Provide access to more troubleshooting information (such as docker logs, ECS logs)
+* Add footer with fetched/expiry timestamp, #instances/services/tasks, Average CPU/Memory Reservation
+
+## Server
 * Write a plugin system that lets adopters plugin their own statistics from favourite monitoring tool
 * Pluggable backend system that could support other public or private cloud providers
-* Provide access to more troubleshooting information (such as docker logs, ECS logs)
-* Cache responses server-side to reduce AWS API calls
-* Make the data transfer between client and server more efficient - Separate requests for task and instance data and populate graph asynchronously
+* Return instances with FETCHED_INSTANCES FetchStatus to allow client to draw instances outline until tasks retrieved asynchronously
 * Arrow functions: https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Functions/Arrow_functions
-* Testing
+
+
+
+## Testing
   * Capture ECS JSON responses for testing and replay with mock AWS ECS server
   * https://fbflex.wordpress.com/2013/11/18/mocking-out-amazon-aws-sdk-with-the-betamax-recording-proxy-for-testing/

bin/www

@@ -7,12 +7,13 @@
 var app = require('../app');
 var debug = require('debug')('c3vis:server');
 var http = require('http');
+var config = require('../config/config');
 
 /**
  * Get port from environment and store in Express.
  */
 
-var port = normalizePort(process.env.PORT || '3000');
+var port = normalizePort(config.port);
 app.set('port', port);
 
 /**

config/config.js

@@ -0,0 +1,23 @@
+const lodash = require('lodash');
+const TARGET_ENV = process.env.TARGET_ENV || 'dev';
+
+function _loadDefaultConfig() {
+  return require('./defaults.js');
+}
+
+function _loadOverrideConfig(targetEnvironment) {
+  try {
+    // Extend configuration with environment-specific configuration
+    console.debug(`Overriding default configuration with '${targetEnvironment}' environment configuration from ${_overrideConfigFilename(targetEnvironment)} (TARGET_ENV=${process.env.TARGET_ENV}, NODE_ENV=${process.env.NODE_ENV})`);
+    return require(_overrideConfigFilename(targetEnvironment));
+  } catch (err) {
+    console.error(`ERROR: Could not load configuration file for target environment '${targetEnvironment}'. Skipping. (${err})`);
+    return {}
+  }
+}
+
+function _overrideConfigFilename(targetEnvironment) {
+  return `./env/${targetEnvironment}.js`;
+}
+
+module.exports = lodash.merge(_loadDefaultConfig(), _loadOverrideConfig(TARGET_ENV));

config/defaults.js

@@ -0,0 +1,15 @@
+module.exports = {
+  environmentName: undefined,
+  port: process.env.PORT || 3000,
+  clusterStateCacheTtl: 30 * 60 * 1000,  // Invalidate clusters in cache after 30 minutes
+  aws: {
+    configFile: './aws_config.json',
+    apiDelay: 100, // milliseconds to pause between AWS API calls to prevent API rate limiting
+    listInstancesPageSize: 100,          // max 100
+    describeInstancesPageSize: 100,      // max 100
+    listTasksPageSize: 100,              // max 100
+    describeTasksPageSize: 100,          // max 100
+    maxSimultaneousDescribeTasksCalls: 2,
+    maxSimultaneousDescribeTaskDefinitionCalls: 1,
+  }
+};

config/env/dev.js

@@ -0,0 +1,4 @@
+module.exports = {
+  // Add dev environment config overrides here and enable at startup with TARGET_ENV=dev environment variable
+  environmentName: "Development"
+};

config/env/prod.js

@@ -0,0 +1,4 @@
+module.exports = {
+  // Add prod environment config overrides here and enable at startup with TARGET_ENV=prod environment variable
+  environmentName: "Production"
+};

config/env/test.js

@@ -0,0 +1,4 @@
+module.exports = {
+  // Add test environment config overrides here and enable at startup with TARGET_ENV=test environment variable
+  environmentName: "Test"
+};

docs/AWS_SDK_CONFIGURATION.md

@@ -0,0 +1,85 @@
+# Configuring AWS SDK
+
+The c3vis server uses the AWS JavaScript SDK to connect to AWS APIs.
+
+As per [Configuring the SDK for JavaScript](https://docs.aws.amazon.com/sdk-for-javascript/v2/developer-guide/configuring-the-jssdk.html), the AWS JavaScript SDK will get its configuration from the server's environment.
+
+## Provide Explicit AWS SDK Configuration with `aws_config.json` Configuration File
+
+AWS SDK configuration can be overridden by providing an `aws_config.json` file (this file location is overridable with `aws.configFile` option, see [CONFIGURATION.md](CONFIGURATION.md)).
+
+E.g. to set the region used by c3vis server to `us-east-1`, create an `aws_config.json` file in the root directory with the following:
+
+```
+{
+  "region": "us-east-1"
+}
+```
+
+The contents of this file override all other sources of AWS SDK configuration.  
+The settings are applied to the AWS Global Configuration Object using `AWS.config.update()` as per [Using the Global Configuration Object](https://docs.aws.amazon.com/sdk-for-javascript/v2/developer-guide/global-config-object.html)
+
+## AWS Region
+
+As per above section, AWS Region can be provided in local `aws_config.json` file.
+
+Otherwise the Region will be configured as per [Setting the AWS Region](https://docs.aws.amazon.com/sdk-for-javascript/v2/developer-guide/setting-region.html).
+
+## AWS Credentials
+
+If using `aws_config.json` file as per above section, you can add AWS credentials properties `accessKeyId` and `secretAccessKey` to the `aws_config.json` 
+See [https://docs.aws.amazon.com/sdk-for-javascript/v2/developer-guide/loading-node-credentials-json-file.html](https://docs.aws.amazon.com/sdk-for-javascript/v2/developer-guide/loading-node-credentials-json-file.html).
+
+*NOTE: Storing credentials in plaintext file is not recommended, especially if there is a risk this file could be committed to version control.*
+
+Otherwise, the credentials will be loaded as per priority listed [here](https://docs.aws.amazon.com/sdk-for-javascript/v2/developer-guide/setting-credentials-node.html).
+
+## IAM Role Permissions
+
+### EC2 IAM Role Permissions
+
+When running c3vis on EC2 instances using an IAM role, ensure the role has the 
+following permissions:
+
+* `ecs:listContainerInstances`
+* `ecs:describeContainerInstances`
+* `ecs:listTasks`
+* `ecs:describeTasks`
+* `ecs:describeTaskDefinition`
+* `ecs:listClusters`
+* `ec2:describeInstance`
+
+Sample IAM Inline Policy:
+```
+{
+    "Version": "2012-10-17",
+    "Statement": [
+        {
+            "Effect": "Allow",
+            "Action": [
+                "ecs:listContainerInstances",
+                "ecs:describeContainerInstances",
+                "ecs:listTasks",
+                "ecs:describeTasks",
+                "ecs:describeTaskDefinition",
+                "ecs:listClusters",
+                "ec2:describeInstances"
+            ],
+            "Resource": [
+                "*"
+            ]
+        }
+    ]
+}
+```
+
+### ECS IAM Task Role
+
+When running c3vis on an ECS cluster, you can use an ECS Task IAM Role, which
+ can be created using the process documented [here](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/task-iam-roles.html#create_task_iam_policy_and_role).
+Ensure the IAM Policy has the permissions listed above.
+
+## Security Warning 
+
+**WARNING:** c3vis makes ECS data from the above API calls (including environment variables in task definitions) available to clients/browsers.
+Ensure the c3vis server is available only to users that should have access to this information.