Skip to content

Latest commit

 

History

History
204 lines (145 loc) · 8.04 KB

README.md

File metadata and controls

204 lines (145 loc) · 8.04 KB
Raw

Micro Army

This is a tool to quickly turn on some number of AWS micro instances and have them slam a webserver simultaneously. The micro's are effectively Siege cannons.

Siege is a flexible load testing tool. You can configure different payloads and frequencies and all kinds of good stuff. So the trick for microarmy is to get Siege on a bunch of computers quickly and coordinate the micro instances to work in parallel.

The micro instances are controlled via SSH in parallel thanks to Eventlet + Paramiko.

The micros report the statistics of their run back to the controlling script, which then aggregates this data into a CSV.

After running the test you can shut down all of your micros and quit micro army.

Slides

I gave a talk on Micro Army back in May of 2011. The slides are available here.

100 boxes in parallel

I recently tested deployment of 100 ec2 micros. On average I found I was able to turn configure 2 micros in about 58 seconds. I tried configuring 100 micros in parallel and found this only took 106 seconds. Slightly more, but negligibly so.

Example Use:

This is what it looks like to use microarmy.

$ ./command_center.py

microarmy> long_help

  long_help:    This.
  status:       Get info about current cannons
  deploy:       Deploys N cannons
  setup:        Runs the setup functions on each host
  config:       Allows a user to specify existing cannons
  config_siege: Create siege config from specified dictionary
  siege_urls:   Specify list of URLS to test against
  single_url:   Only hit one url when firing off your next test
  all_urls:     Revert to using configured urls (turns off single_url)
  fire:         Asks for a url and then fires the cannons
  mfire:        Runs `fire` multiple times and aggregates totals
  term:         Terminate cannons
  quit:         Exit command center

microarmy> deploy
Deploying cannons...  Done!
Hosts config: [(u'i-4c4ff03c', u'ec2-107-21-75-120.compute-1.amazonaws.com'), (u'i-4e4ff03e', u'ec2-50-42-133-31.compute-1.amazonaws.com')]

microarmy> setup
  Setting up cannons - time: 1317352247.23
  Loading cannons...  Done!
  Siege config written, deploying to cannons
  Configuring siege...  Done!
  Siege urls written, deploying to cannons
  Configuring urls...  Done!
  Finished setup - time: 1317352305.56
  Sending reboot message to cannons

microarmy> fire
  target: http://brubeck.io
Results ]------------------
Num_Trans,Elapsed,Tran_Rate
3424,9.15,374.21
3424,9.17,373.39

microarmy> term

microarmy> quit

Configure siege dynamically

Typically you'd want to configure the siege config in your local_settings.py like so...

siege_config = {
    'connection': 'close',
    'concurrency': 5,
    'internet': 'true'
}

If you wish to configure it dynamically...

microarmy> config_siege
  Siege config detected in settings and will be automatically deployed with "setup"
  Continue? (y/n) y
  Enter siege config data: {'connection': 'close', 'concurrency': 5, 'benchmark': 'true'}
  Siege config written, deploying to cannons
  Configuring siege...  Done!

All of the above will write a ~/.siegerc config on the cannon machines like so:

connection = close
benchmark = true
concurrency = 5

Configure siege urls

Typically you'd want to configure the urls for siege to hit in your local_settings.py like so:

siege_urls = [
   'http://localhost/',
   'http://localhost/test'
]

If you wish to configure them dynamically...

microarmy> siege_urls
  Urls detected in settings and will be automatically deployed with "setup"
  Continue? (y/n) y
  Enter urls: ['http://localhost/', 'http://localhost/test/']
  Urls written, deploying to cannons
  Configuring urls...  Done!

All of the above will write ~/urls.txt on the cannon machines like so:

http://localhost/
http://localhost/test/

Hit a single url target instead of the configured list

You might want to hit one single url when firing off a test, to do so, don't configure any urls in your local_settings.py or...

microarmy> single_url
  Bypassing configured urls
microarmy> fire
  target: http://localhost/test_one/

To switch back to your configured urls...

microarmy> all_urls
  Using configured urls

Find and reuse previously deployed cannons

Sometimes you might forget that you deployed a whole mess of cannons already. In that case, run the following...

microarmy> find_cannons
Deployed cannons: [(u'i-1a6d127a', u'ec2-50-17-80-241.compute-1.amazonaws.com'), (u'i-1c6d127c', u'ec2-184-73-117-126.compute-1.amazonaws.com'), (u'i-e06d1280', u'ec2-50-16-106-209.compute-1.amazonaws.com'), (u'i-e26d1282', u'ec2-50-17-26-28.compute-1.amazonaws.com'), (u'i-e46d1284', u'ec2-50-16-169-72.compute-1.amazonaws.com'), (u'i-e66d1286', u'ec2-184-73-114-245.compute-1.amazonaws.com'), (u'i-e86d1288', u'ec2-184-72-92-234.compute-1.amazonaws.com'), (u'i-ea6d128a', u'ec2-184-73-148-253.compute-1.amazonaws.com'), (u'i-ec6d128c', u'ec2-107-20-112-149.compute-1.amazonaws.com'), (u'i-ee6d128e', u'ec2-50-16-24-210.compute-1.amazonaws.com'), (u'i-f06d1290', u'ec2-204-236-251-120.compute-1.amazonaws.com'), (u'i-f46d1294', u'ec2-50-19-24-63.compute-1.amazonaws.com'), (u'i-f66d1296', u'ec2-107-20-95-203.compute-1.amazonaws.com'), (u'i-f86d1298', u'ec2-174-129-76-108.compute-1.amazonaws.com'), (u'i-fa6d129a', u'ec2-50-16-64-128.compute-1.amazonaws.com')]
Would you like to import these cannons now? (y/n) y

Now you have a whole new battery of cannons to fire away. To be safe, you may want to run setup again.

Terminate all cannons we know about

Thanks to tagging instances in EC2, we can get an inventory of all the cannons we've deployed over time. That's good because you don't want to leave those little beasts running.

If you're all done and want to be sure your instances are terminated, do the following...

microarmy> find_cannons
Deployed cannons: [(u'i-1a6d127a', u'ec2-50-17-80-241.compute-1.amazonaws.com'), (u'i-1c6d127c', u'ec2-184-73-117-126.compute-1.amazonaws.com'), (u'i-e06d1280', u'ec2-50-16-106-209.compute-1.amazonaws.com'), (u'i-e26d1282', u'ec2-50-17-26-28.compute-1.amazonaws.com'), (u'i-e46d1284', u'ec2-50-16-169-72.compute-1.amazonaws.com'), (u'i-e66d1286', u'ec2-184-73-114-245.compute-1.amazonaws.com'), (u'i-e86d1288', u'ec2-184-72-92-234.compute-1.amazonaws.com'), (u'i-ea6d128a', u'ec2-184-73-148-253.compute-1.amazonaws.com'), (u'i-ec6d128c', u'ec2-107-20-112-149.compute-1.amazonaws.com'), (u'i-ee6d128e', u'ec2-50-16-24-210.compute-1.amazonaws.com'), (u'i-f06d1290', u'ec2-204-236-251-120.compute-1.amazonaws.com'), (u'i-f46d1294', u'ec2-50-19-24-63.compute-1.amazonaws.com'), (u'i-f66d1296', u'ec2-107-20-95-203.compute-1.amazonaws.com'), (u'i-f86d1298', u'ec2-174-129-76-108.compute-1.amazonaws.com'), (u'i-fa6d129a', u'ec2-50-16-64-128.compute-1.amazonaws.com')]
Would you like to import these cannons now? (y/n) y
microarmy> cleanup
Deployed cannons destroyed

Requirements

There are only a few requirements. Everything required for the micros is installed on the micros, after all.

$ pip install eventlet paramiko boto pyyaml

Config

You should create a local_settings.py inside the repo and fill in the following keys. Look at settings.py for more information.

  • aws_access_key
  • aws_secret_key
  • security_groups
  • key_pair_name
  • num_cannons
  • ec2_ssh_key

Here is an example:

aws_access_key = 'ABCDEFGHIJKLMNOPQRST'
aws_secret_key = 'abcdefghij/KLMNOPQRSTUVWXY/zabcdefghijkl'
security_groups = ['MicroArmy'] # must support incoming SSH + outgoing HTTP
key_pair_name = 'micros'
ec2_ssh_key = '/Users/jd/.ec2/micros.pem'
num_cannons = 2
siege_config = {
    'connection': 'close',
    'concurrency': 5,
    'internet': 'true'
}

siege_urls = [
   'http://localhost/',
   'http://localhost/test'
]