Skip to content

Debugging

Jump to bottom
Scott Sievert edited this page Feb 17, 2017 · 1 revision

Note: we describe the steps graphically at the bottom of the page. This will likely be useful for beginners.

Log in to machine

  • Logging into a machine:
cd NEXT/ec2
python next_ec2.py --key-pair=$KEY_PAIR --identity-file=$KEY_FILE docker_login amazon_machine_name

# amazon machine name from `python next_ec2.py ... launch amazon_machine_name`
# also viewable (with other stuff) on the AWS console

By default, the below will assume you are docker_login'd unless explicitly mentioned otherwise.

Sync your changes

  • To sync any changes you make locally on your machine to the remote ec2 machine, run the command python next_ec2.py ... rsync amazon_machine_name.
  • We have a method to have your changes propagate instantly using the command line environment variable CELERY_ON. To use this, on the remote machine run export CELERY_ON=False (capitalization important) before running docker-compose up.
  • You can also allow your changes to propagate using a slower method by running
# remote machine
docker-compose stop # typically via cntrl-c
# docker-compose rm # pro: don't rerun experiment_*.script. con: not as safe.
docker-compose up

This allows the changes to propagate with all current experiments in place. You don't have to rerun your experiment script.

View the logs

Run docker-compose logs -f to view the running logs. When you docker-compose up, these logs will run automatically. This is where utils.debug_print (and print) show up.

  • When docker_login'd, NEXT prints out a lot of stuff, meaning it's hard to see debugging printing information. We have developed a utils function just for this and the usage is given below:
import next.utils as utils
utils.debug_print("This will be bold and yellow!")

Important tips

  • Disable CELERY_ON. When docker_login'd to the machine, edit the file docker-compose.yml and change the line CELERY_ON: True to CELERY_ON: False before running docker-compose up. This will allow you to view your changes by rsyncing up and allow them to propagate. Every time you rsync, docker-compose.yml changes and takes effect when you rerun docker-compose up.

Graphical representation

A graphical representation of the debug workflow.

Clone this wiki locally