Running an Experiment

Goal

To launch an experiment. After we launch, we will be able to see the dashboard/query page/experiment list.

Prereqs

• a NEXT machine must be launched (on EC2). More documentation can be found at EC2-launch.
• the proper environment variables must be exported. More documentation can be found at AWS-Account-Quickstart.

Launching an experiment

Before we run the examples, we have to tell our computer where NEXT is running. To do this, we run the command

export NEXT_BACKEND_GLOBAL_HOST=[public dns]  # found at AWS > EC2 > [click on instance] > Public DNS

Once this is done, we can run the examples in NEXT/examples. For example, we can run

cd NEXT/examples/strange_fruit_triplet
python experiment_triplet.py

This will launch an experiment an experiment for us (and also print out an experiment ID, etc). We can visit the list of experiments at http://[public-dns]:8000/dashboard/experiment_list, which lists the experiment IDs and date created.

The experiment list:

The question that a participant in your experiment will see:

Note: many examples of launching experiments are available at NEXT/examples, for example NEXT/examples/cartoon_dueling/init.yaml. Running these files will launch an experiment for you.

Notes for algorithm designer

Launching other experiments can be adapted from the examples. Given that you want to run under a specific app, the experiment_<app>.py shouldn't change too much -- values should be changed. To add your algorithm to the algorithms this experiment chooses from, add your algorithm in alg_list by editing the list that includes algorithm titles.

## for strange_fruit_triplet/experiment_triplet.py
# old
alg_ids = ['RandomSampling', 'CrowdKernel']
# new, with your algorithm added
alg_ids = ['RandomSampling', 'CrowdKernel', 'yourAlg']

More documentation on developing an algorithm can be found in Basic-Algorithms.