GENOME: GenerativE Neuro-symbOlic visual reasoning by growing and reusing ModulEs

[ Project Page | Arxiv Paper ]

This repository contains the official code for GNSVR: Generative Neuro-Symbolic Visual Reasoning by Growing and Reusing Modules

Install Dependencies

Firstly, install dependencies of ViperGPT.

Then:

pip install -r requirements.txt

OpenAI key

To run the OpenAI models, you will need to configure an OpenAI key. This can be done by signing up for an account e.g. here, and then creating a key in account/api-keys. Create a file api.key and store the key in it.

Running the code

GQA

Inference

scripts/gqa/gqa_stage3.sh

python main.py \
    --model text-davinci-003 \
    --dataset imgedit \
    --test_num 50 \
    --stage 3 \
    --inference_prompt_path prompts/imgedit/imgedit_stage3.prompt \
    --dataset_dir dataset/imgedit \
    --ann_path imgedit_ann.json \
    --image_path imgedit_images/ \
    --save_output \
    --output_dir save/results/imgedit/imgedit_stage3/ \
    --use_new_module \
    --threshold 0.1 \
    --module_save_dir transfer/transfer_gqa_modules/ \
        

RefCOCO

Inference

scripts/refcoco/refcoco_stage3.sh

python main.py \
    --model gpt-3.5-turbo-instruct \
    --dataset refcoco \
    --coco_dir /path/to/coco/ \
    --test_num 100 \
	--stage 3 \
    --inference_prompt_path prompts/refcoco/refcoco_stage3.prompt \
    --dataset_dir /path/to/refcoco/ \
    --save_output \
    --output_dir save/results/refcoco/refcoco_stage3/
        

Knowledge-based Object Tagging

Inference

scripts/okdet/okdet_stage3.sh

python main.py \
    --model text-davinci-003 \
    --dataset okdet \
    --test_num 50 \
    --stage 3 \
    --inference_prompt_path prompts/okdet/okdet_stage3.prompt \
    --dataset_dir dataset/okdet \
    --ann_path okdet_ann.json \
    --image_path okdet_images/ \
    --save_output \
    --output_dir save/results/okdet/okdet_stage3/ \
    --use_new_module \
    --threshold 0.1 \
    --module_save_dir transfer/transfer_gqa_modules/ \
        

Image Editing

Inference

scripts/imgedit/imgedit_stage3.sh

python main.py \
    --model text-davinci-003 \
    --dataset imgedit \
    --test_num 50 \
    --stage 3 \
    --inference_prompt_path prompts/imgedit/imgedit_stage3.prompt \
    --dataset_dir dataset/imgedit \
    --ann_path imgedit_ann.json \
    --image_path imgedit_images/ \
    --save_output \
    --output_dir save/results/imgedit/imgedit_stage3/ \
    --use_new_module \
    --threshold 0.1 \
    --module_save_dir transfer/transfer_gqa_modules/ \
        

More tasks will be added soon.

Citation

If you use this work or find it helpful, please consider citing: (bibtex)

@inproceedings{gnsvr2023,
         author = {Chen, Zhenfang and Sun, Rui and Liu, Wenjun and Hong, Yining and Gan, Chuang},
         title = {GENOME: Generative Neuro-Symbolic Visual Reasoning by Growing and Reusing Modules},
         journal = {arXiv},
         year = {2023},
        }