Required Inputs

###initExp

Dictionary for initialization of experiment.

####Arguments:

• args = Dictionary with the following keys:

  • participant_to_algorithm_management:"one-to-one" | "one-to-many"
    How particpants are routed to algorithms.

  • alg_list: List, all of whose elements are as follows:

    • Dictionary with the following keys:
      • alg_label:string
        A name for this algorithm of your choosing.
      • alg_id:"CrowdKernel" | "UncertaintySampling" | "STE" | "RandomSampling"
        A supported algorithm type in the system.
      • test_alg_label:string
        Which algorithm to test against.

    A list of algorithms you plan to use in this experiment.

  • algorithm_management_settings:Dictionary with the following keys:

    • params:Dictionary with the following keys:
      • proportions:List, all of whose elements are as follows:
        • Dictionary with the following keys:
          • alg_label:string
            An alg label, matching one of the ones declared in alg_list.
          • proportion:num
            Proportion of queries to give to this algorithm. List of proportions per app type.
    • mode:"fixed_proportions"
• app_id = [App ID]

####Example

{"args":{
   "participant_to_algorithm_management":"one_to_one",
   "alg_list":[
      {"alg_label":"test","alg_id":"RandomSampling","test_alg_label":"test"},
      {"alg_label":"foo","alg_id":"UncertaintySampling","test_alg_label":"test"},
      {"alg_label":"bar","alg_id":"CleverSampling","test_alg_label":"test"}
   ],
   "algorithm_management_settings":{
      "params":{
        "proportions":[
          {"alg_label":"test","proportion":0.2},
          {"alg_label":"foo","proportion":0.4},
          {"alg_label":"bar","proportion":0.4}
        ]
      }
   }
 }
}

Note that mode is omitted because it has a default value

###getQuery

A request to determine which query to pose to the user

####Arguments:

• exp_uid = string
• args = Dictionary with the following keys:
  • participant_uid:string
    unique identifier of session for a participant answering questions

####Example

{"exp_uid":"12345",
 "args":{
   "participant_uid":"123456abcdef"
 }
}

###processAnswer(exp_uid, exp_key, args)

Report back the reward of pulling the arm suggested by getQuery

####Arguments:

• exp_uid = string
• exp_key = string
• args = Dictionary with the following keys:
  • query_uid:string
    Unique identifier of query

####Example

{"exp_uid":"12345",
 "args":{
   "query_uid":"fedcba654321"
 }
}

###getModel

Get the current state of the internal models from each algorithm

####Arguments:

• exp_uid = string
• args = Dictionary with the following keys:
  • alg_list = List, all of whose elements are as follows:

####Example

{"exp_uid":"12345",
 "args":{
   "alg_list":["AlgorithmName1","AlgorithmName3"]
 }
}