Release 1.6.0

Release 1.6.0

CHANGELOG.rst

@@ -2,6 +2,14 @@
 CHANGELOG
 =========
 
+1.6.0
+=====
+* Refactor scaling up to take into account the number of pending/requested jobs/slots and instance slots.
+* Refactor scaling down to scale down faster and take advantage of per-second billing.
+* Add `scaledown_idletime` parameter as part of scale-down refactoring
+* Lock hosts before termination to ensure removal of dead compute nodes from host list
+* Fix HTTP proxy support
+
 1.5.4
 =====
 * Add option to disable ganglia `extra_json = { "cfncluster" : { "ganglia_enabled" : "no" } }`

README.rst

@@ -1,36 +1,117 @@
-CfnCluster
-==========
+==============================
+CfnCluster - HPC for the Cloud
+==============================
 
-.. image:: https://travis-ci.org/awslabs/cfncluster.png?branch=develop
-   :target: https://travis-ci.org/awslabs/cfncluster
+|Build Status| |Version|
+
+.. |Build Status| image:: https://travis-ci.org/awslabs/cfncluster.png?branch=develop
+   :target: https://travis-ci.org/awslabs/cfncluster/
    :alt: Build Status
+.. |Version| image:: https://badge.fury.io/py/cfncluster.png
+    :target: https://badge.fury.io/py/cfncluster
 
 CfnCluster ("cloud formation cluster") is a framework that deploys and
 maintains high performance computing clusters on Amazon Web Services
 (AWS). Developed by AWS, CfnCluster facilitates both quick start proof
 of concepts (POCs) and production deployments. CfnCluster supports
 many different types of clustered applications and can easily be
 extended to support different frameworks. The CLI is stateless,
-everything is done using CloudFormation or resources within AWS.
+everything is done using CloudFormation or resources within AWS.\
+
+Quick Start
+-----------
+First, install the library:
+
+.. code-block:: sh
+
+    $ pip install cfncluster
+
+Next, configure your aws credentials and default region:
+
+.. code-block:: sh
+
+    $ aws configure
+    AWS Access Key ID [None]: YOUR_KEY
+    AWS Secret Access Key [None]: YOUR_SECRET
+    Default region name [us-east-1]:
+    Default output format [None]:
+
+Then, run cfncluster configure:
+
+.. code-block:: ini
+
+  $ cfncluster configure
+  Cluster Template [default]:
+  AWS Access Key ID []:
+  AWS Secret Access Key ID []:
+  Acceptable Values for AWS Region ID:
+      ap-south-1
+      ...
+      us-west-2
+  AWS Region ID [us-east-1]:
+  VPC Name [myvpc]:
+  Acceptable Values for Key Name:
+    keypair1
+    keypair-test
+    production-key
+  Key Name []:
+  Acceptable Values for VPC ID:
+    vpc-1kd24879
+    vpc-blk4982d
+  VPC ID []:
+  Acceptable Values for Master Subnet ID:
+    subnet-9k284a6f
+    subnet-1k01g357
+    subnet-b921nv04
+  Master Subnet ID []:
+
+Now you can create your first cluster;
+
+.. code-block:: sh
+
+  $ cfncluster create myfirstcluster
+
+
+After the cluster finishes creating, log in:
+
+.. code-block:: sh
+
+  $ cfncluster ssh myfirstcluster
+
+You can view the running compute hosts:
+
+.. code-block:: sh
+
+  $ qhost
+
+For more information on any of these steps see the `Getting Started Guide`_.
+
+.. _`Getting Started Guide`: https://cfncluster.readthedocs.io/en/latest/getting_started.html
 
 Documentation
-=============
+-------------
 
 Documentation is part of the project and is published to -
 https://cfncluster.readthedocs.io/. Of most interest to new users is
 the Getting Started Guide -
 https://cfncluster.readthedocs.io/en/latest/getting_started.html.
 
 Issues
-======
+------
 
 Please open a GitHub issue for any feedback or issues:
 https://github.com/awslabs/cfncluster.  There is also an active AWS
 HPC forum which may be helpful:https://forums.aws.amazon.com/forum.jspa?forumID=192.
 
+Changes
+-------
+
 CfnCluster 1.6 IAM Change
 =========================
-Between CfnCluster 1.5.3 and 1.6.0 we made a change to the CfnClusterInstancePolicy that adds “s3:GetObject” permissions on objects in <REGION>-cfncluster bucket.
+Between CfnCluster 1.5.3 and 1.6.0 we made a change to the CfnClusterInstancePolicy that adds “s3:GetObject” permissions
+on objects in <REGION>-cfncluster bucket, "autoscaling:SetDesiredCapacity", "autoscaling:DescribeTags" permissions and
+"cloudformation:DescribeStacks" permissions on <REGION>:<ACCOUNT_NAME>:<STACK_NAME>.
+
 If you’re using a custom policy (e.g. you specify "ec2_iam_role" in your config) be sure it includes this new permission. See https://cfncluster.readthedocs.io/en/latest/iam.html
 
 CfnCluster 1.5 IAM Change

amis.txt

@@ -1,88 +1,88 @@
 # alinux
-ap-northeast-1: ami-00047986f877e6760
-ap-northeast-2: ami-0ca5afbc9d0b20f3d
-ap-northeast-3: ami-0b3af1c2fe35b1ae4
-ap-south-1: ami-0250a8315b099785d
-ap-southeast-1: ami-038312aa862774599
-ap-southeast-2: ami-0dce078cd2c343a5c
-ca-central-1: ami-04e27bd54e171fbad
-eu-central-1: ami-075bf1c36cf6b855f
-eu-west-1: ami-08d507a759b9ee0e4
-eu-west-2: ami-03ddfb1462e2c92fd
-eu-west-3: ami-0e72015cf2b21159f
-sa-east-1: ami-02bd755bfe8704cef
-us-east-1: ami-07f61c3e18538c70c
-us-east-2: ami-096b7925b1f9c8cbb
-us-gov-west-1: ami-cfd34dae
-us-west-1: ami-04794da830cb9dc86
-us-west-2: ami-03f86a2920360dfbe
+ap-northeast-1: ami-0775ab7ad13c0e74f
+ap-northeast-2: ami-09fdf91b4a93730f0
+ap-northeast-3: ami-03a0e4ad407dd7f55
+ap-south-1: ami-008d194619d177765
+ap-southeast-1: ami-08eef0c7d23753d49
+ap-southeast-2: ami-08d6ef8690d843ceb
+ca-central-1: ami-00f519fdeb5feb438
+eu-central-1: ami-07fa07aba1f6fc571
+eu-west-1: ami-09cbb5230da5b6cc2
+eu-west-2: ami-006cd2956238a6dd7
+eu-west-3: ami-0fce6e16d9970d71b
+sa-east-1: ami-0165d34de1afd1618
+us-east-1: ami-0fd8b1245d1435bf3
+us-east-2: ami-0cb3d04ae4b4ca8db
+us-gov-west-1: ami-e0fd6781
+us-west-1: ami-02f24f766b5b7502f
+us-west-2: ami-01e7f12fc989e7004
 # centos6
-ap-northeast-1: ami-00c9ad360d255b462
-ap-northeast-2: ami-08e84ffe9aead7873
-ap-northeast-3: ami-056531169587ffc23
-ap-south-1: ami-03a57d339b58b2f23
-ap-southeast-1: ami-00063e5635939c9d1
-ap-southeast-2: ami-015e730525001bf53
-ca-central-1: ami-0fea713db092ac77f
-eu-central-1: ami-0f6b91d011f44aaf6
-eu-west-1: ami-029accac823a2a844
-eu-west-2: ami-032b6b895dab544c6
-eu-west-3: ami-039be4bd46c878e2c
-sa-east-1: ami-0e507bdec09b8a091
-us-east-1: ami-091bbc83b27f92634
-us-east-2: ami-0fa89e68b0222f50b
-us-west-1: ami-06f3909be633646b9
-us-west-2: ami-087d4f29315fd19d2
+ap-northeast-1: ami-0f676dc2bebea5b1a
+ap-northeast-2: ami-0adf39100d009ad31
+ap-northeast-3: ami-0da12c87d2aee0ef5
+ap-south-1: ami-05a8a28e5c8b74642
+ap-southeast-1: ami-02ce4e46ca873697e
+ap-southeast-2: ami-013201a7fda637e83
+ca-central-1: ami-0ee6c95ad927b2da4
+eu-central-1: ami-0de3eb16b173ea6b3
+eu-west-1: ami-0c8eca7deb52169cc
+eu-west-2: ami-013d7f40fc791f845
+eu-west-3: ami-0cac2d11252d18c27
+sa-east-1: ami-074dbf929dc0c8de0
+us-east-1: ami-044fd430f1fd3b163
+us-east-2: ami-03c508534d6f54859
+us-west-1: ami-05ee27a886e16ba39
+us-west-2: ami-08675896633a1d9fb
 # centos7
-ap-northeast-1: ami-09dc3b175ed2905f8
-ap-northeast-2: ami-03f494fed9f0f9c98
-ap-northeast-3: ami-04094dd3b4013b75f
-ap-south-1: ami-0c20297a2c00b0528
-ap-southeast-1: ami-0f610508513f2012f
-ap-southeast-2: ami-09d0b89439c0c1b65
-ca-central-1: ami-0362a3ef26ab5dfce
-eu-central-1: ami-03d5519f5f4e7619f
-eu-west-1: ami-0575304add4986551
-eu-west-2: ami-08263362b21b98916
-eu-west-3: ami-0befb803ea7ce1b83
-sa-east-1: ami-0860c11596f954bbc
-us-east-1: ami-06c93be112f880acd
-us-east-2: ami-0493261a1b2adfd27
-us-west-1: ami-0af95249ebb8de7a5
-us-west-2: ami-0a259cf8220fabdcb
+ap-northeast-1: ami-0b13eb5fa94691483
+ap-northeast-2: ami-015bf65572df2d94d
+ap-northeast-3: ami-0479721fa287d3cf7
+ap-south-1: ami-087bda89547e089dc
+ap-southeast-1: ami-0d339c926f9d35ba3
+ap-southeast-2: ami-0ac15e60210a10310
+ca-central-1: ami-068d6a6fc5e7987f3
+eu-central-1: ami-0f73d5d5bb22f7609
+eu-west-1: ami-05479870b31fa535e
+eu-west-2: ami-08c5c6f990f8253e1
+eu-west-3: ami-0d6c4b2dde29e341f
+sa-east-1: ami-00cfa447d16329380
+us-east-1: ami-0fb2dca992af734f2
+us-east-2: ami-03421fa006b964767
+us-west-1: ami-012c5239be2beb7f1
+us-west-2: ami-0053bef6747ca58bf
 # ubuntu1404
-ap-northeast-1: ami-09dc9e9da730c0248
-ap-northeast-2: ami-06346bcc8b825a458
-ap-northeast-3: ami-0ae6e02016436404a
-ap-south-1: ami-0e4294a5b9d5de425
-ap-southeast-1: ami-01e6359021e3bac23
-ap-southeast-2: ami-0cfebd51a3470f1d4
-ca-central-1: ami-0097472502a04776d
-eu-central-1: ami-0a82ac2872821deef
-eu-west-1: ami-0d6c90f092266b787
-eu-west-2: ami-0f23dffc4db122b53
-eu-west-3: ami-02b6ca8fd14f58265
-sa-east-1: ami-04e943b561cd32659
-us-east-1: ami-0920fa86d5a85a07a
-us-east-2: ami-0d63a602f26709e70
-us-gov-west-1: ami-c2d34da3 
-us-west-1: ami-06fe4f837d8715dbb
-us-west-2: ami-01170828a21d86a72
+ap-northeast-1: ami-05b67b05deba61e8c
+ap-northeast-2: ami-06aa825c33309fdfd
+ap-northeast-3: ami-0c03a23da1606fa3a
+ap-south-1: ami-096a418446cd1eeb1
+ap-southeast-1: ami-0d413135984f7bd97
+ap-southeast-2: ami-079737fc9c635c09a
+ca-central-1: ami-00cec9c492f236616
+eu-central-1: ami-0d6242ced07309e7b
+eu-west-1: ami-06581a7f68c539440
+eu-west-2: ami-00621b5f273783bce
+eu-west-3: ami-07da72e5a66a93c71
+sa-east-1: ami-0f4a298763a012d8e
+us-east-1: ami-07df96c1d8e62161d
+us-east-2: ami-0f7c7f69478093925
+us-gov-west-1: ami-38f36959
+us-west-1: ami-077e3b1efa9443476
+us-west-2: ami-0581577b81f865ae9
 # ubuntu1604
-ap-northeast-1: ami-0b0bf645cde38fe52
-ap-northeast-2: ami-0b2c6c9e988689d7c
-ap-northeast-3: ami-0dae0e941e3bce10b
-ap-south-1: ami-011d655005fe854e9
-ap-southeast-1: ami-03428422e2cb20c9f
-ap-southeast-2: ami-0bbef9e2b1170757f
-ca-central-1: ami-0916119fde83c29b7
-eu-central-1: ami-003273c4072efbe43
-eu-west-1: ami-009af0d819973e1d6
-eu-west-2: ami-029f01bbfec52a10e
-eu-west-3: ami-0ae405adcb3d1477c
-sa-east-1: ami-0b0057ecd5c465d09
-us-east-1: ami-0dfb1662fcc845c55
-us-east-2: ami-01bd1361428c384c3
-us-gov-west-1: ami-0fd6486e
-us-west-1: ami-0e144dd24634766a2
-us-west-2: ami-02087bf0ff99cc127
+ap-northeast-1: ami-0232ea8f3f82712a2
+ap-northeast-2: ami-08d64fa4a89f85427
+ap-northeast-3: ami-0cfdcf1603db9a672
+ap-south-1: ami-009f971485befe66a
+ap-southeast-1: ami-02ab2d97bea65a826
+ap-southeast-2: ami-0fc63c095a0c31f0a
+ca-central-1: ami-0705c0b15666e7d60
+eu-central-1: ami-08ab664e40e331729
+eu-west-1: ami-096183a7e31fc4e35
+eu-west-2: ami-0a4fbb1dd660013b5
+eu-west-3: ami-0fcf3de0f1913a190
+sa-east-1: ami-06fa4a172ebe11a10
+us-east-1: ami-0b9750a5856891754
+us-east-2: ami-0a2c6075e91d8bbc3
+us-gov-west-1: ami-3bf3695a
+us-west-1: ami-00b0c33ebc9576bf8
+us-west-2: ami-00e5e4cc71cf05d6e