diff --git a/freebase/ComplexQuestions.md b/freebase/ComplexQuestions.md index 9837f551..8b265d08 100644 --- a/freebase/ComplexQuestions.md +++ b/freebase/ComplexQuestions.md @@ -5,6 +5,7 @@ | Model / System | Year | Average F1 | Reported by | |:-------------------:|:---------:|:----------:|:----------------------------------------------------------------------------:| +| TFS-KBQA | 2024 | 44.0 | [Shouhui Wang and Biao Qin](https://aclanthology.org/2024.lrec-main.1074.pdf)| | Lan and Jiang [1] | 2020 | 43.3 | [Yonghui Jia and Wenliang Chen](https://arxiv.org/pdf/2204.12808.pdf) | | Reranking | 2022 | 42.9 | [Yonghui Jia and Wenliang Chen](https://arxiv.org/pdf/2204.12808.pdf) | | Luo et al. [2] | 2020 | 42.8 | [Yonghui Jia and Wenliang Chen](https://arxiv.org/pdf/2204.12808.pdf) | diff --git a/freebase/ComplexWebQuestions.md b/freebase/ComplexWebQuestions.md index 2ffe0a84..d53144e8 100644 --- a/freebase/ComplexWebQuestions.md +++ b/freebase/ComplexWebQuestions.md @@ -14,6 +14,7 @@ | FLAN-T5 | 2023 | - | - | 46.69 | EN | [Tan et al.](https://arxiv.org/pdf/2303.07992.pdf) | | BART-large | 2022 | 68.2 | - | - | EN | [Hu et al.](https://aclanthology.org/2022.coling-1.145.pdf) | | FiDeLiS + gpt-4-turbo | 2024 | 64.32| 71.47 | - | EN | [Sui et al.](https://arxiv.org/pdf/2405.13873) | +| TFS-KBQA | 2024 | 63.6 | - | - | EN | [Shouhui Wang and Biao Qin](https://aclanthology.org/2024.lrec-main.1074.pdf) | | DECAF (BM25 + FiD-3B) | 2022 | - | 70.4 | - | EN | [Yu et al.](https://arxiv.org/pdf/2210.00063.pdf) | | CBR-KBQA | 2022 | - | 70.4 | - | EN | [Yu et al.](https://arxiv.org/pdf/2210.00063.pdf) | | DECAF (BM25 + FiD-large) | 2022 | - | 68.1 ± 0.5 | - | EN | [Yu et al.](https://arxiv.org/pdf/2210.00063.pdf) | @@ -28,6 +29,7 @@ | ProgramTransfer-pa | 2022 | 54.5 | 54.3 | - | EN | [Cao et al.](https://aclanthology.org/2022.acl-long.559.pdf) | | KD-CoT | 2024 | - | 53.92 | - | EN | [Sui et al.](https://arxiv.org/pdf/2405.13873) | | NSM+h | 2022 | - | 53.9 | - | EN | [Yu et al.](https://arxiv.org/pdf/2210.00063.pdf) | +| FC-KBQA | 2024 | 54.1 | - | - | EN | [Shouhui Wang and Biao Qin](https://aclanthology.org/2024.lrec-main.1074.pdf) | | REAREV | 2022 | - | 52.9 | - | EN | [Costas Mavromatis and George Karypis](https://arxiv.org/pdf/2210.13650.pdf) | | QNRKGQA+h | 2022 | - | 51.5 | - | EN | [Ma et al.](https://link.springer.com/chapter/10.1007/978-3-031-10983-6_11) | | SR+NSM | 2022 | - | 50.2 | - | EN | [Yu et al.](https://arxiv.org/pdf/2210.00063.pdf) | @@ -45,6 +47,7 @@ | TERP | 2022 | - | 49.2 | - | EN | [Qiao et al.](https://aclanthology.org/2022.coling-1.156.pdf) | | TeacherNet | 2022 | 44.0 | 48.8 | - | EN | [Cao et al.](https://aclanthology.org/2022.acl-long.559.pdf) | | NSM | 2022 | 44.0 | 48.8 | - | EN | [Du et al.](https://arxiv.org/pdf/2209.03005.pdf) | +| RnG-KBQA | 2024 | 42.3 | - | - | EN | [Shouhui Wang and Biao Qin](https://aclanthology.org/2024.lrec-main.1074.pdf) | | NSM+h | 2022 | - | 48.8 | - | EN | [Ma et al.](https://link.springer.com/chapter/10.1007/978-3-031-10983-6_11) | | TransferNet | 2022 | - | 48.6 | - | EN | [Du et al.](https://arxiv.org/pdf/2209.03005.pdf) | | NSM+p | 2022 | - | 48.3 | - | EN | [Ma et al.](https://link.springer.com/chapter/10.1007/978-3-031-10983-6_11) | diff --git a/freebase/WebQuestionsSP.md b/freebase/WebQuestionsSP.md index f623842e..7e781a96 100644 --- a/freebase/WebQuestionsSP.md +++ b/freebase/WebQuestionsSP.md @@ -6,11 +6,14 @@ | Model / System | Year | F1 | Hits@1 | Accuracy | Language | Reported by | | :---------------------------------: | :--: | :--------: | :--------: | :------: | :------: | :-----------------------------------------------------------------------------------: | | chatGPT | 2023 | - | - | 83.70 | EN | [Tan et al.](https://arxiv.org/pdf/2303.07992.pdf) | +| TFS-KBQA | 2024 | 79.9 | 79.8 | - | EN | [Shouhui Wang and Biao Qin](https://aclanthology.org/2024.lrec-main.1074.pdf) | +| Pangu | 2024 | 79.6 | - | - | EN | [Shouhui Wang and Biao Qin](https://aclanthology.org/2024.lrec-main.1074.pdf) | | TIARA | 2022 | 78.9 | 75.2 | - | EN | [Shu et. al.](https://aclanthology.org/2022.emnlp-main.555.pdf) | | DECAF (DPR + FiD-3B) | 2022 | 78.8 | 82.1 | - | EN | [Yu et al.](https://arxiv.org/pdf/2210.00063.pdf) | | FiDeLiS + gpt-4-turbo | 2024 | 78.32 | 84.39 | - | EN | [Sui et al.](https://arxiv.org/pdf/2405.13873) | | GPT-3.5v3 | 2023 | - | - | 79.60 | EN | [Tan et al.](https://arxiv.org/pdf/2303.07992.pdf) | | DECAF (DPR + FiD-large) | 2022 | 77.1 ± 0.2 | 80.7 ± 0.2 | - | EN | [Yu et al.](https://arxiv.org/pdf/2210.00063.pdf) | +| FC-KBQA | 2024 | 76.9 | - | - | EN | [Shouhui Wang and Biao Qin](https://aclanthology.org/2024.lrec-main.1074.pdf) | | FiDeLiS + gpt-3.5-turbo | 2024 | 76.78 | 79.32 | - | EN | [Sui et al.](https://arxiv.org/pdf/2405.13873) | | UniK-QA | 2022 | - | 79.1 | - | EN | [Yu et al.](https://arxiv.org/pdf/2210.00063.pdf) | | Sun et al. | 2024 | - | 77.8 | - | EN | [Sun et al.](https://aclanthology.org/2024.lrec-main.496.pdf) | diff --git a/systems.md b/systems.md index 9ef0d5c9..d4236a4a 100644 --- a/systems.md +++ b/systems.md @@ -151,3 +151,11 @@ | Think-on-Graph (ToG) | Sui et al. | [Link](https://arxiv.org/pdf/2405.13873) | yes | [Link](https://github.com/IDEA-FinAI/ToG) | [Link](https://arxiv.org/pdf/2307.07697) | This paper proposes a new LLM-KG integrating paradigm “LLM ⊗ KG ” which treats the LLM as an agent to interactively explore related entities and relations on KGs and perform reasoning based on the retrieved knowledge. The authors further implement this paradigm by introducing a new approach called Think-on-Graph (ToG), in which the LLM agent iteratively executes beam search on KG, discovers the most promising reasoning paths, and returns the most likely reasoning results. | Sun et al. | | KD-CoT | Sui et al. | [Link](https://arxiv.org/pdf/2405.13873) | yes | [Link](https://github.com/AdelWang/KD-CoT/tree/main) | [Link](https://arxiv.org/pdf/2308.13259) | This paper proposes a framework called Knowledge-Driven Chain-of-Thought (KD-CoT) to verify and modify reasoning traces in CoT via interaction with external knowledge, and thus overcome the hallucinations and error propagation. Concretely, the authors formulate the CoT rationale process of LLMs into a structured multi-round QA format. In each round, LLMs interact with a QA system that retrieves external knowledge and produce faithful reasoning traces based on retrieved precise answers. The structured CoT reasoning of LLMs is facilitated by our developed KBQA CoT collection, which serves as in-context learning demonstrations and can also be utilized as feedback augmentation to train a robust retriever. | Wang et al. | | FiDeLiS | Sui et al. | [Link](https://arxiv.org/pdf/2405.13873) | no | - | [Link](https://arxiv.org/pdf/2405.13873) | This paper proposes a retrieval-exploration interactive method, FiDeLiS to handle intermediate steps of reasoning grounded by KGs. Specifically, the authors propose Path-RAG module for retrieving useful intermediate knowledge from KG for LLM reasoning. FiDeLiS incorporates the logic and common-sense reasoning of LLMs and the topological connectivity of KGs into the knowledge retrieval process, which provides more accurate retrieving performance. Furthermore, FiDeLiS leverages deductive reasoning capabilities of LLMs as a better criterion to automatically guide the reasoningprocess in a stepwise and generalizable manner. Deductive verification serves as a precise indicator for when to cease further reasoning, thus avoiding misleading the chains of reasoning and unnecessary computation. | Sui et al. | +| Subgraph Retrieval (SR) | Shouhui Wang and Biao Qin | [Link](https://aclanthology.org/2024.lrec-main.1074.pdf) | yes | [Link](https://github.com/RUCKBReasoning/SubgraphRetrievalKBQA) | [Link](https://aclanthology.org/2022.acl-long.396v2.pdf) | This paper introduces a trainable subgraph retriever (SR) for multi-hop Knowledge Base Question Answering (KBQA) that is decoupled from the reasoning process, allowing for better scalability and adaptability across various reasoning models. The SR model achieves significant improvements over existing methods by allowing a more targeted retrieval of subgraphs from large knowledge bases, reducing the reasoning space and bias. It employs dual-encoder architecture for efficient subgraph expansion, supports weakly supervised pre-training, and allows end-to-end fine-tuning with feedback from the reasoning model, enhancing overall QA performance. Extensive tests on benchmarks like WebQSP and CWQ show that SR not only improves retrieval effectiveness but also sets new performance standards for embedding-based KBQA systems. | Zhang et al. | +| ReTraCk | Shouhui Wang and Biao Qin | [Link](https://aclanthology.org/2024.lrec-main.1074.pdf) | no | - | [Link](https://aclanthology.org/2021.acl-demo.39/) | This paper presents Retriever-Transducer-Checker (ReTraCk), a neural semantic parsing framework for large scale knowledge base question answering (KBQA). ReTraCk is designed as a modular framework to maintain high flexibility. It includes a retriever to retrieve relevant KB items efficiently, a transducer to generate logical form with syntax correctness guarantees and a checker to improve the transduction procedure. | Chen et al. | +| RnG-KBQA | Shouhui Wang and Biao Qin | [Link](https://aclanthology.org/2024.lrec-main.1074.pdf) | yes | [Link](https://github.com/salesforce/rng-kbqa) | [Link](https://aclanthology.org/2022.acl-long.417.pdf) | This paper presents RnG-KBQA, a Rank-and-Generate approach for KBQA, which remedies the coverage issue with a generation model while preserving a strong generalization capability. The approach first uses a contrastive ranker to rank a set of candidate logical forms obtained by searching over the knowledge graph. It then introduces a tailored generation model conditioned on the question and the top-ranked candidates to compose the final logical form. | Ye et al. | +| ArcaneQA | Shouhui Wang and Biao Qin | [Link](https://aclanthology.org/2024.lrec-main.1074.pdf) | yes | [Link](https://github.com/dki-lab/ArcaneQA) | [Link](https://arxiv.org/pdf/2204.08109) | This paper presents ArcaneQA, a novel generation-based model that addresses both the large search space and the schema linking challenges in a unified framework with two mutually boosting ingredients: dynamic program induction for tackling the large search space and dynamic contextualized encoding for schema linking. Experimental results on multiple popular KBQA datasets demonstrate the highly competitive performance of ArcaneQA in both effectiveness and efficiency. | Gu and Su | +| Program Transfer | Shouhui Wang and Biao Qin | [Link](https://aclanthology.org/2024.lrec-main.1074.pdf) | yes | [Link](https://github.com/THU-KEG/ProgramTransfer) | [Link](https://aclanthology.org/2022.acl-long.559/) | This paper proposes the approach of program transfer, which aims to leverage the valuable program annotations on the rich-resourced KBs as external supervision signals to aid program induction for the low-resourced KBs that lack program annotations. The authors design a novel two-stage parsing framework with an efficient ontology-guided pruning strategy. First, a sketch parser translates the question into a high-level program sketch, which is the composition of functions. Second, given the question and sketch, an argument parser searches the detailed arguments from the KB for functions. During the searching, we incorporate the KB ontology to prune the search space. | Cao et al. | +| Pangu | Shouhui Wang and Biao Qin | [Link](https://aclanthology.org/2024.lrec-main.1074.pdf) | yes | [Link](https://github.com/dki-lab/Pangu) | [Link](https://aclanthology.org/2023.acl-long.270.pdf) | This paper proposes Pangu, a generic framework for grounded language understanding that capitalizes on the discriminative ability of LMs instead of their generative ability. Pangu consists of a symbolic agent and a neural LM working in a concerted fashion: The agent explores the environment to incrementally construct valid plans, and the LM evaluates the plausibility of the candidate plans to guide the search process. | Gu et al. | +| FC-KBQA | Shouhui Wang and Biao Qin | [Link](https://aclanthology.org/2024.lrec-main.1074.pdf) | no | - | [Link](https://aclanthology.org/2023.acl-long.57.pdf) | This paper proposes a Fine-to-Coarse Composition framework for KBQA (FC-KBQA) to both ensure the generalization ability and executability of the logical expression. The main idea of FC-KBQA is to extract relevant finegrained knowledge components from KB and reformulate them into middle-grained knowledge pairs for generating the final logical expressions. | Zhang et al. | +| TFS-KBQA | Shouhui Wang and Biao Qin | [Link](https://aclanthology.org/2024.lrec-main.1074.pdf) | yes | [Link](https://github.com/shouh/TFS-KBQA) | [Link](https://aclanthology.org/2024.lrec-main.1074.pdf) | This study adopts the LLMs, such as Large Language Model Meta AI (LLaMA), as a channel to connect natural language questions with structured knowledge representations and proposes a Three-step Fine-tune Strategy based on large language model to implement the KBQA system (TFS-KBQA). This method achieves direct conversion from natural language questions to structured knowledge representations, thereby overcoming the limitations of existing KBQA methods, such as addressing large search and reasoning spaces and ranking massive candidates. | Shouhui Wang and Biao Qin |