Merge pull request #33 from stanford-oval/paper_results

push first paper results

docs/paper_results.md

@@ -1 +1,5 @@
-Coming next as we finish wrapping up this repo as a package!
+# Restaurants dataset
+
+We based our experiments on a total of 1828 restaurants from Yelp.com across 4 cities. The list of restaurants' ids used in our study can be found at `paper_data/restaurants_list.txt`. For instance, for `NX8VYnWFQ2ZY-H0HwvfPTw`, the corresponding restaurant is https://www.yelp.com/biz/NX8VYnWFQ2ZY-H0HwvfPTw. For each restaurant, the top 20 reviews and top 20 popular dishes were used for our experiments.
+
+We collected 100 single-turn questions via Prolific. The results of the SUQL system can be found at `paper_data/restaurants_single_turn_SUQL.csv`. Here is a breakdown of what the columns mean: **Unique ID** (a unique row identifier), **Utterance** (the collected user question), **Predicted SUQL** (the predicted SUQL with gpt-3.5-turbo-0613), **Agent response** (the SUQL system's response), **1st entity**, **2nd entity**, **3rd entity** (the up-to-three returned entities), **Entity Annotation**,**Whether Wrong Parse** (whether this is a wrong SUQL parse), **Prolific ID** (prolific ID associated with the submission), **Structural Unstructural annotation** (whether the question requires only structured data or a combination, used for Table 4 in the paper). The false positives from the SUQL system (based on annotation) can be found in `paper_data/restaurants_single_turn_SUQL_fp.txt`