Update to use the public defog_data library

[wongjingping]

Update our evaluation code to use the metadata and supplementary data from the public defog_data library

[wongjingping]

Tested and works with the openai runner (which calls prune_metadata_str which gets the embeddings and supplementary column info from defog-data) :

$ python main.py \
  -q data/questions_gen.csv \
  -o results/my_query_generator.csv \
  -g oa \
  -f prompts/prompt.md \
  -m gpt-3.5-turbo-0613 \
  -n 3 \
  -p 3
script_dir: /Users/jp/workspace/miniconda3/lib/python3.11/site-packages/defog_data
/Users/jp/workspace/miniconda3/lib/python3.11/site-packages/bitsandbytes/cextension.py:34: UserWarning: The installed version of bitsandbytes was compiled without GPU support. 8-bit optimizers, 8-bit multiplication, and GPU quantization are unavailable.
  warn("The installed version of bitsandbytes was compiled without GPU support. "
'NoneType' object has no attribute 'cadam32bit_grad_fp32'
preparing questions...
Embeddings file /Users/jp/workspace/sql-eval/data/embeddings.pkl does not exist.
Embeddings file /Users/jp/workspace/sql-eval/data/embeddings.pkl does not exist.
Embeddings file /Users/jp/workspace/sql-eval/data/embeddings.pkl does not exist.
  0%|                                                              | 0/3 [00:00<?, ?it/s]Finished embedding academic 42 columns
Finished embedding academic 42 columns
Finished embedding advising 111 columns
Finished embedding advising 111 columns
Finished embedding academic 42 columns
Finished embedding advising 111 columns
Finished embedding atis 127 columns
Finished embedding atis 127 columns
Finished embedding geography 29 columns
Finished embedding geography 29 columns
Finished embedding restaurants 12 columns
Finished embedding restaurants 12 columns
Finished embedding scholar 28 columns
Finished embedding scholar 28 columns
Finished embedding atis 127 columns
Finished embedding yelp 37 columns
Saved embeddings to file /Users/jp/workspace/sql-eval/data/embeddings.pkl
Finished embedding yelp 37 columns
Saved embeddings to file /Users/jp/workspace/sql-eval/data/embeddings.pkl
Finished embedding geography 29 columns
Finished embedding restaurants 12 columns
Finished embedding scholar 28 columns
Finished embedding yelp 37 columns
Saved embeddings to file /Users/jp/workspace/sql-eval/data/embeddings.pkl
Correct so far: 3/3 (100.00%): 100%|███████████████████████| 3/3 [00:06<00:00,  2.11s/it]
                exact_match  correct
query_category
date_functions          1.0      1.0
Average correct rate: 1.00

The creation of the embedding 3x here is due to the parallel factor of 3 and where None of the parallel threads have the embeddings initially but then independently create a new embedding pickle.

[rishsriv]

Tested and verified that this works. Thank you for the changes – makes it so much easier to work on this across machines!