The Chicago-crime-vis corpus is a multimodal corpus containing data from the study of interactions of 16 subjects with a visualization expert. The participants explored visualizations on a large screen related to Chicago crime data to determine the deployment of police officers based on crimes in different areas of the city, time of the day, season, etc. For the interaction they used speech and hand gesture and the Visualization Expert was a human generating visualization from a different room. The corpus has 3179 transcribed utterances covering 1,879 word types and a vocabulary of 38,105 word tokens.
The corpus has been referenced in the following publications:
- A. Kumar, J. Aurisano, B. Di Eugenio, and A. Johnson. (2020). Intelligent Assistant for Exploring Data Visualizations. In Proceedings of the 33rd International FLAIRS Conference, Miami, FL, May 17-20.
- A. Kumar, B. Di Eugenio, J. Aurisano, and A. Johnson. (2020). Augmenting Small Data to Classify Contextualized Dialogue Acts for Exploratory Visualization. In Proceedings of the 12th Language Resources and Evaluation Conference (LREC). Marseille, France, May 11-16.
- A. Kumar, B. Di Eugenio, J. Aurisano, A. Johnson, A. Alsaiari, N. Flowers, A. Gonzalez and J. Leigh. Multimodal Coreference Resolution for Exploratory Data Visualization Dialogue: Context-Based Annotation and Gesture Identification.SemDial 2017, The 21st Workshop on the Semantics and Pragmatics of Dialogue, August 15-17, Saarbrücken, Germany.
This repository contains three folders -
This folder has annotated data in anvil files for the 16 subjects (subject5 through subject20). The data in the anvil files are in xml format. Annotation is done for dialogue acts, contextualized actionable requests, gestures and referring expressions. Out of the 3179 utterances 449 were identified as Actionable Requests and annotated with one the 8 possible types of Dialogue Acts. The Dialogue Acts are:
- CREATE-VISUALIZATION
- CLARIFICATION
- WINDOW-MANAGEMENT
- FACT-BASED
- PREFERENCE-BASED
- MODIFY-VISUALIZATION
- APPEARANCE
- HIGH-LEVEL-QUERY
For gesture annotation, out of 536 identified gestures, 380 were annotated as pointing gestures. Out of these pointing gestures 159 pointed to visualizations on the screen and the rest were pointing to entities within the visualizations. For referring expression annotation, a total of 294 references were labeled in the corpus (176 text references and 118 gesture references). For slot annotation, 680 slot fillers were tagged with slots from our knowledge ontology.
The augmented_data folder contains the Chicago-crime-vis corpus with 160 subjects based on paraphrasing techniques to increase the original corpus size by a factor of 10 i.e., 16 subjects enlarged to 160 subjects leading to an increase to 15K utterances.
16 videos of approximately 1 hour in length each were transcribed into 16 text files capturing the visualization dialogue with each subject.
These are the actual visualizations produced by the data analysis expert during the visualization dialogue. The file name corresponds to the actual visualization identifier that the subject was shown on the screen. The annotations in the anvil files also use the same identifiers, e.g., the referring expression annotations identify the referent visualization targets by their identifiers.
We regret that we cannot release the videos or the speech channel due to human subject protections.
This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.
@inproceedings{kumar2020augmenting,
title={Augmenting small data to classify contextualized dialogue acts for exploratory visualization},
author={Kumar, Abhinav and Di Eugenio, Barbara and Aurisano, Jillian and Johnson, Andrew},
booktitle={Proceedings of the 12th Language Resources and Evaluation Conference},
pages={590--599},
year={2020}
}This work was supported, initially, by NSF award IIS 1445751; and currently, by NSF award CNS 1625941, and by a UIC University Scholar award to Barbara Di Eugenio.