Home

The Evaluation Toolkit (ETK)

Welcome to the Evaluation Toolkit Lab. We have created a toolkit designed to help you conduct image-based perceptual studies on your own visualizations. Using this toolkit, you can quickly construct and conduct studies that provide feedback on your visualizations, so that you can evaluate, understand and improve them. If you have a visualization, and want to know how effective it is, our toolkit can help.

The ETK consists of a set of modules that leverage the JavaScript API within Qualtrics Survey software (www.qualtrics.com) to easily implement and run different types of standard psychophysical experiments with images as the experimental stimuli. Each module consists of JavaScript, HTML and CSS scripts that are inserted into a Qualtrics question/survey. The researcher needs to make only minimal changes (e.g., update the list of images to use in a study) in order to get a study question set up.

The ETK may not be able to design a user study for you. But once you have an experimental design, the ETK makes it a snap to implement it within Qualtrics.

Currently, there are seven modules:

• Two-alternative Forced Choice – a two alternative forced choice method
  

• Method of Adjustment – a method of adjustment method that cycles through a carousel of image to find a threshold point
  

• Sliding Method of Adjustment — a variant on the method of adjustment, it cycles through the carousel of images showing two at a time to find the threshold point
  

• Round Robin Comparison – a round robin comparison method
  

• Compare Two Arrays – a method that compares each image in an array to its counterpart in a second array; could be used as a 2AFC variant or to hard-code specific comparisons
  

• Click Counting – a simple script to count the number of times an image is clicked.
  

• KeyTask – a script that displays a stimuli image along with a set of answer keys; keys can be coded by color or by name
  

Citation

If you use ETK for an evaluation study, we ask that you please cite our paper: @inproceedings {Turton2017etk, booktitle = {EuroVis 2017 – Short Papers}, editor = {Barbora Kozlikova and Tobias Schreck and Thomas Wischgoll}, title = {{ETK: An Evaluation Toolkit for Visualization User Studies}}, author = {Turton, Terece L. and Berres, Anne S. and Rogers, David H. and Ahrens, James}, year = {2017}, publisher = {The Eurographics Association}, ISBN = {978-3-03868-043-7}, DOI = {10.2312/eurovisshort.20171131}}

Acknowledgements

Funding for this project created under a grant from the US Department of Energy, Office of Science, Advanced Scientific Research Computing (Dr. Lucy Nowell, Program Manager).

The icon for this project was downloaded from The Noun Project (www.nounproject.comr), and was created by Delwar Hossain.

This wiki replaces the deprecated etklab.org website.