diff --git a/README.md b/README.md index b0ec2f55..b17f4991 100644 --- a/README.md +++ b/README.md @@ -18,7 +18,7 @@ Using a global and open resource, trusting evaluation results will be easier. In There are multiple ways to add a new result to the leaderboard: -1. You can directly edit the leaderboard for your dataset of interest. You can add columns, new rows or even change individual cells. This is the easiest way to add a new result. If a leaderboard for a dataset is not yet available, you can simply add it by pasting a markdown table, similar to the other ones in the repository. Look at [this example]() for a template. +1. You can directly edit the leaderboard for your dataset of interest. You can add columns, new rows or even change individual cells. This is the easiest way to add a new result. If a leaderboard for a dataset is not yet available, you can simply add it by pasting a markdown table, similar to the other ones in the repository. Please also add an entry of the system to systems_updated.md if none exists already. Look at [this example](https://github.com/KGQA/leaderboard/blob/v2.0/template_leaderboard.md?plain=1) for a template. 1. You can also simply open an issue and we will add the result for you. If you have an entire new dataset, which you want to add to the leaderboard, you'll have to create a pull request. This is how we expect you to add a new dataset. (P.S.: You can also simply create an issue and we will add the dataset for you.) @@ -27,7 +27,7 @@ If you have an entire new dataset, which you want to add to the leaderboard, you - Find the appropriate knowledge graph folder, e.g. "freebase" - Create these markdown files: - ^(datasetName).md - This is the header of your dataset in plain markdown. - - (datasetName).md - This is the main leaderboard table. You can copy the template from [here]() + - (datasetName).md - This is the main leaderboard table. You can copy the template from [here](https://github.com/KGQA/leaderboard/blob/v2.0/template_leaderboard.md?plain=1) - $(datasetName).md - This is the footer of your dataste, again in plain markdown. - Add correspnding route in the file frontend/static/entries.txt - Create a pull request diff --git a/systems.md b/systems.md index 73f5dacc..d87567fe 100644 --- a/systems.md +++ b/systems.md @@ -139,9 +139,10 @@ | TIARA | Shu et al. | [Link](https://aclanthology.org/2022.emnlp-main.555.pdf) | yes | [Link](https://github.com/microsoft/KC/tree/main/papers/TIARA) | [Link](https://aclanthology.org/2022.emnlp-main.555.pdf) | In this paper, we present a new KBQA model, TIARA, which addresses those issues by applying multi-grained retrieval to help the PLM focus on the most relevant KB contexts, viz., entities, exemplary logical forms, and schema items. Moreover, constrained decoding is used to control the output space and reduce generation errors. | Shu et al. | | MACRE | Xu et al. | [Link](https://link.springer.com/chapter/10.1007/978-3-031-30672-3_40) | no | - | [Link](https://link.springer.com/chapter/10.1007/978-3-031-30672-3_40) | MACRE is a novel approach for multi-hop question answering over KGs via contrastive relation embedding (MACRE) powered by contrastive relation embedding and context-aware relation ranking. | Xu et al. | | KGQAcl/rr | Hu et al. | [Link](https://arxiv.org/pdf/2303.10368.pdf) | yes | [Link](https://github.com/HuuuNan/PLMs-in-Practical-KBQA) | [Link](https://arxiv.org/pdf/2303.10368.pdf) | KGQA-CL and KGQA-RR are tow frameworks proposed to evaluate the PLM's performance in comparison to their efficiency. Both architectures are composed of mention detection, entity disambiguiation, relation detection and answer query building. The difference lies on the relation detection module. KGQA-CL aims to map question intent to KG relations. While KGQA-RR ranks the related relations to retrieve the answer entity. Both frameworks are tested on common PLM, distilled PLMs and knowledge-enhanced PLMs and achieve high performance on three benchmarks. | Hu et al. | +| AE-TQ | Long et al. | [Link](https://link.springer.com/chapter/10.1007/978-3-031-20891-1_6) | no | | [Link](https://link.springer.com/chapter/10.1007/978-3-031-20891-1_6) | AE-TQ leverages abstract meaning representation (AMR) for understanding complex questions, and produces question-oriented semantic information for explicit and effective temporal reasoning. A question is parsed into a semantic information structure (SISs), and for each SIS with unknown elements, the TKG is queried to obtain candidate answers, which are then reasoned together with the temporal constraints in pursuit of the final answer. | Long et al. | | W. Han et al. | Han et al. | [Link](https://link.springer.com/chapter/10.1007/978-3-031-30672-3_39) | no | - | [Link](https://link.springer.com/chapter/10.1007/978-3-031-30672-3_39) | This model is based on machine reading comprehension. To transform a subgraph of the KG centered on the topic entity into text, the subgraph is sketched through a carefully designed schema tree, which facilitates the retrieval of multiple semantically-equivalent answer entities. Then, the promising paragraphs containing answers are picked by a contrastive learning module. Finally, the answer entities are delivered based on the answer span that is detected by the machine reading comprehension module. | Han et al. | | GAIN | Shu et al. | [Link](https://arxiv.org/pdf/2309.08345.pdf) | no | - | [Link](https://arxiv.org/pdf/2309.08345.pdf) | GAIN is not a KGQA system, but a data augmentation method named Graph seArch and questIon generatioN (GAIN). GAIN applies to KBQA corresponding to logical forms or triples, and scales data volume and distribution through four steps: 1) Graph search: Sampling logical forms or triples from arbitrary domains in the KB, without being restricted to any particular KBQA dataset. 2) Training question generator on existing KBQA datasets, i.e., learning to convert logical forms or triples into natural language questions. 3) Verbalization: Using the question generator from step 2 to verbalize sampled logical forms or triples from step 1, thus creating synthetic questions. 4) Training data expansion: Before fine-tuning any neural models on KBQA datasets, GAIN-synthetic data can be used to train these models or to expand the corpus of in-context samples for LLMs. That is, as a data augmentation method, GAIN is not a KBQA model, but it is used to augment a base KBQA model. | Shu et al. | -| JarvisQALcs | Jaradeh et al. | [Link](https://arxiv.org/pdf/2006.01527) | no | | | same as reporting paper | JarvisQA a BERT based system to answer questions on tabular views of scholarly knowledge graphs. | Jaradeh et al. | +| JarvisQALcs | Jaradeh et al. | [Link](https://arxiv.org/pdf/2006.01527) | no | - | [Link](https://arxiv.org/pdf/2006.01527) | JarvisQA is a BERT based system to answer questions on tabular views of scholarly knowledge graphs. | Jaradeh et al. | | DGEN | Lv et al. | [Link](https://www.researchsquare.com/article/rs-4292912/v1) | no | - | [Link](https://www.sciencedirect.com/science/article/pii/S0925231222012905?ref=pdf_download&fr=RR-2&rr=892a7e715f3c62f5) | The Dynamic Graph Expansion Network (DGEN) is a network that forms a joint subgraph between the question and entities for reasoning, where the whole question is regarded as a node of the subgraph. At each step we believe that there is reasoning related information in the context of the question which will affect the auxiliary entity and intermediate path selection. To this end, we design a dynamic subgraph module and a joint reasoning module. The former is intended to dynamically adjust the subgraph in light of the question context. Built on top of that, the latter is intended to combine the interaction between the dynamic subgraph and intermediate supervision signal for joint reasoning. Through this idea of dynamic subgraph joint reasoning, we are able to distill the information that is the most relevant to reasoning the answer on both sides of the question context and knowledge base (KB). | Wu et al. | | RE-KBQA | Lv et al. | [Link](https://www.researchsquare.com/article/rs-4292912/v1) | yes | [Link](https://github.com/yongcaoplus/RE-KBQA) | [Link](https://arxiv.org/pdf/2305.02118) | RE-KBQA, utilizes relations in the knowledge base to enhance entity representation and introduce additional supervision. We explore guidance from relations in three aspects, including (1) distinguishing similar entities by employing a variational graph auto-encoder to learn relation importance; (2) exploring extra supervision by predicting relation distributions as soft labels with a multi-task scheme; (3) designing a relation-guided re-ranking algorithm for post-processing. | Cao et al. | | ReifKB | Lv et al. | [Link](https://www.researchsquare.com/article/rs-4292912/v1) | yes | [Link](https://github.com/google-research/language) | [Link](https://arxiv.org/pdf/2002.06115) | We describe a novel way of representing a symbolic knowledge base (KB) called a sparse-matrix reified KB. This representation enables neural KB inference modules that are fully differentiable, faithful to the original semantics of the KB, expressive enough to model multi-hop inferences, and scalable enough to use with realistically large KBs. The sparse-matrix reified KB can be distributed across multiple GPUs, can scale to tens of millions of entities and facts, and is orders of magnitude faster than naive sparse-matrix implementations. | Cohen et al. | diff --git a/systems_updated.md b/systems_updated.md index d91f7cc8..d87567fe 100644 --- a/systems_updated.md +++ b/systems_updated.md @@ -140,3 +140,25 @@ | MACRE | Xu et al. | [Link](https://link.springer.com/chapter/10.1007/978-3-031-30672-3_40) | no | - | [Link](https://link.springer.com/chapter/10.1007/978-3-031-30672-3_40) | MACRE is a novel approach for multi-hop question answering over KGs via contrastive relation embedding (MACRE) powered by contrastive relation embedding and context-aware relation ranking. | Xu et al. | | KGQAcl/rr | Hu et al. | [Link](https://arxiv.org/pdf/2303.10368.pdf) | yes | [Link](https://github.com/HuuuNan/PLMs-in-Practical-KBQA) | [Link](https://arxiv.org/pdf/2303.10368.pdf) | KGQA-CL and KGQA-RR are tow frameworks proposed to evaluate the PLM's performance in comparison to their efficiency. Both architectures are composed of mention detection, entity disambiguiation, relation detection and answer query building. The difference lies on the relation detection module. KGQA-CL aims to map question intent to KG relations. While KGQA-RR ranks the related relations to retrieve the answer entity. Both frameworks are tested on common PLM, distilled PLMs and knowledge-enhanced PLMs and achieve high performance on three benchmarks. | Hu et al. | | AE-TQ | Long et al. | [Link](https://link.springer.com/chapter/10.1007/978-3-031-20891-1_6) | no | | [Link](https://link.springer.com/chapter/10.1007/978-3-031-20891-1_6) | AE-TQ leverages abstract meaning representation (AMR) for understanding complex questions, and produces question-oriented semantic information for explicit and effective temporal reasoning. A question is parsed into a semantic information structure (SISs), and for each SIS with unknown elements, the TKG is queried to obtain candidate answers, which are then reasoned together with the temporal constraints in pursuit of the final answer. | Long et al. | +| W. Han et al. | Han et al. | [Link](https://link.springer.com/chapter/10.1007/978-3-031-30672-3_39) | no | - | [Link](https://link.springer.com/chapter/10.1007/978-3-031-30672-3_39) | This model is based on machine reading comprehension. To transform a subgraph of the KG centered on the topic entity into text, the subgraph is sketched through a carefully designed schema tree, which facilitates the retrieval of multiple semantically-equivalent answer entities. Then, the promising paragraphs containing answers are picked by a contrastive learning module. Finally, the answer entities are delivered based on the answer span that is detected by the machine reading comprehension module. | Han et al. | +| GAIN | Shu et al. | [Link](https://arxiv.org/pdf/2309.08345.pdf) | no | - | [Link](https://arxiv.org/pdf/2309.08345.pdf) | GAIN is not a KGQA system, but a data augmentation method named Graph seArch and questIon generatioN (GAIN). GAIN applies to KBQA corresponding to logical forms or triples, and scales data volume and distribution through four steps: 1) Graph search: Sampling logical forms or triples from arbitrary domains in the KB, without being restricted to any particular KBQA dataset. 2) Training question generator on existing KBQA datasets, i.e., learning to convert logical forms or triples into natural language questions. 3) Verbalization: Using the question generator from step 2 to verbalize sampled logical forms or triples from step 1, thus creating synthetic questions. 4) Training data expansion: Before fine-tuning any neural models on KBQA datasets, GAIN-synthetic data can be used to train these models or to expand the corpus of in-context samples for LLMs. That is, as a data augmentation method, GAIN is not a KBQA model, but it is used to augment a base KBQA model. | Shu et al. | +| JarvisQALcs | Jaradeh et al. | [Link](https://arxiv.org/pdf/2006.01527) | no | - | [Link](https://arxiv.org/pdf/2006.01527) | JarvisQA is a BERT based system to answer questions on tabular views of scholarly knowledge graphs. | Jaradeh et al. | +| DGEN | Lv et al. | [Link](https://www.researchsquare.com/article/rs-4292912/v1) | no | - | [Link](https://www.sciencedirect.com/science/article/pii/S0925231222012905?ref=pdf_download&fr=RR-2&rr=892a7e715f3c62f5) | The Dynamic Graph Expansion Network (DGEN) is a network that forms a joint subgraph between the question and entities for reasoning, where the whole question is regarded as a node of the subgraph. At each step we believe that there is reasoning related information in the context of the question which will affect the auxiliary entity and intermediate path selection. To this end, we design a dynamic subgraph module and a joint reasoning module. The former is intended to dynamically adjust the subgraph in light of the question context. Built on top of that, the latter is intended to combine the interaction between the dynamic subgraph and intermediate supervision signal for joint reasoning. Through this idea of dynamic subgraph joint reasoning, we are able to distill the information that is the most relevant to reasoning the answer on both sides of the question context and knowledge base (KB). | Wu et al. | +| RE-KBQA | Lv et al. | [Link](https://www.researchsquare.com/article/rs-4292912/v1) | yes | [Link](https://github.com/yongcaoplus/RE-KBQA) | [Link](https://arxiv.org/pdf/2305.02118) | RE-KBQA, utilizes relations in the knowledge base to enhance entity representation and introduce additional supervision. We explore guidance from relations in three aspects, including (1) distinguishing similar entities by employing a variational graph auto-encoder to learn relation importance; (2) exploring extra supervision by predicting relation distributions as soft labels with a multi-task scheme; (3) designing a relation-guided re-ranking algorithm for post-processing. | Cao et al. | +| ReifKB | Lv et al. | [Link](https://www.researchsquare.com/article/rs-4292912/v1) | yes | [Link](https://github.com/google-research/language) | [Link](https://arxiv.org/pdf/2002.06115) | We describe a novel way of representing a symbolic knowledge base (KB) called a sparse-matrix reified KB. This representation enables neural KB inference modules that are fully differentiable, faithful to the original semantics of the KB, expressive enough to model multi-hop inferences, and scalable enough to use with realistically large KBs. The sparse-matrix reified KB can be distributed across multiple GPUs, can scale to tens of millions of entities and facts, and is orders of magnitude faster than naive sparse-matrix implementations. | Cohen et al. | +| RCAANet | Lv et al. | [Link](https://www.researchsquare.com/article/rs-4292912/v1) | no | - | [Link](https://www.researchsquare.com/article/rs-4292912/v1) | This paper proposes a new multi-hop model, called Relation Crossfusion Attention Assistance Networks(RCAANet), which has the following advantages: 1) Accuracy. It can exploit the interactive information of question representation and relation embedding to help calculate the accurate relation distribution of each hop. 2) Effectiveness. RCAANet outperforms previous models, improving multi-hop question answering accuracy on two widely used complex question answering datasets. 3) Transparency. Similar to TransferNet, it is completely attention-based, so its intermediate steps are easily visualized and understood by humans. | Lv et al. | +| Sun et al. | Sun et al. | [Link](https://aclanthology.org/2024.lrec-main.496.pdf) | no | - | [Link](https://aclanthology.org/2024.lrec-main.496.pdf) | Our pipeline (i) retrieves over documents of triples grouped by entity, (ii) re-ranks triples from these documents with context: triples in the 1-hop neighborhood of the documents’ subject entity, and (iii) generates an answer from highly relevant re-ranked triples. | Sun et al. | +| Reasoning on Graphs (RoG) | Sui et al. | [Link](https://arxiv.org/pdf/2405.13873) | no | - | [Link](https://arxiv.org/pdf/2310.01061) | This paper proposes a novel method called reasoning on graphs (RoG) that synergizes LLMs with KGs to enable faithful and interpretable reasoning. Specifically, it presents a planning-retrieval-reasoning framework, where RoG first generates relation paths grounded by KGs as faithful plans. These plans are then used to retrieve valid reasoning paths from the KGs for LLMs to conduct faithful reasoning. Furthermore, RoG not only distills knowledge from KGs to improve the reasoning ability of LLMs through training but also allows seamless integration with any arbitrary LLMs during inference. | Luo et al. | +| Think-on-Graph (ToG) | Sui et al. | [Link](https://arxiv.org/pdf/2405.13873) | yes | [Link](https://github.com/IDEA-FinAI/ToG) | [Link](https://arxiv.org/pdf/2307.07697) | This paper proposes a new LLM-KG integrating paradigm “LLM ⊗ KG ” which treats the LLM as an agent to interactively explore related entities and relations on KGs and perform reasoning based on the retrieved knowledge. The authors further implement this paradigm by introducing a new approach called Think-on-Graph (ToG), in which the LLM agent iteratively executes beam search on KG, discovers the most promising reasoning paths, and returns the most likely reasoning results. | Sun et al. | +| KD-CoT | Sui et al. | [Link](https://arxiv.org/pdf/2405.13873) | yes | [Link](https://github.com/AdelWang/KD-CoT/tree/main) | [Link](https://arxiv.org/pdf/2308.13259) | This paper proposes a framework called Knowledge-Driven Chain-of-Thought (KD-CoT) to verify and modify reasoning traces in CoT via interaction with external knowledge, and thus overcome the hallucinations and error propagation. Concretely, the authors formulate the CoT rationale process of LLMs into a structured multi-round QA format. In each round, LLMs interact with a QA system that retrieves external knowledge and produce faithful reasoning traces based on retrieved precise answers. The structured CoT reasoning of LLMs is facilitated by our developed KBQA CoT collection, which serves as in-context learning demonstrations and can also be utilized as feedback augmentation to train a robust retriever. | Wang et al. | +| FiDeLiS | Sui et al. | [Link](https://arxiv.org/pdf/2405.13873) | no | - | [Link](https://arxiv.org/pdf/2405.13873) | This paper proposes a retrieval-exploration interactive method, FiDeLiS to handle intermediate steps of reasoning grounded by KGs. Specifically, the authors propose Path-RAG module for retrieving useful intermediate knowledge from KG for LLM reasoning. FiDeLiS incorporates the logic and common-sense reasoning of LLMs and the topological connectivity of KGs into the knowledge retrieval process, which provides more accurate retrieving performance. Furthermore, FiDeLiS leverages deductive reasoning capabilities of LLMs as a better criterion to automatically guide the reasoningprocess in a stepwise and generalizable manner. Deductive verification serves as a precise indicator for when to cease further reasoning, thus avoiding misleading the chains of reasoning and unnecessary computation. | Sui et al. | +| Subgraph Retrieval (SR) | Shouhui Wang and Biao Qin | [Link](https://aclanthology.org/2024.lrec-main.1074.pdf) | yes | [Link](https://github.com/RUCKBReasoning/SubgraphRetrievalKBQA) | [Link](https://aclanthology.org/2022.acl-long.396v2.pdf) | This paper introduces a trainable subgraph retriever (SR) for multi-hop Knowledge Base Question Answering (KBQA) that is decoupled from the reasoning process, allowing for better scalability and adaptability across various reasoning models. The SR model achieves significant improvements over existing methods by allowing a more targeted retrieval of subgraphs from large knowledge bases, reducing the reasoning space and bias. It employs dual-encoder architecture for efficient subgraph expansion, supports weakly supervised pre-training, and allows end-to-end fine-tuning with feedback from the reasoning model, enhancing overall QA performance. Extensive tests on benchmarks like WebQSP and CWQ show that SR not only improves retrieval effectiveness but also sets new performance standards for embedding-based KBQA systems. | Zhang et al. | +| ReTraCk | Shouhui Wang and Biao Qin | [Link](https://aclanthology.org/2024.lrec-main.1074.pdf) | no | - | [Link](https://aclanthology.org/2021.acl-demo.39/) | This paper presents Retriever-Transducer-Checker (ReTraCk), a neural semantic parsing framework for large scale knowledge base question answering (KBQA). ReTraCk is designed as a modular framework to maintain high flexibility. It includes a retriever to retrieve relevant KB items efficiently, a transducer to generate logical form with syntax correctness guarantees and a checker to improve the transduction procedure. | Chen et al. | +| RnG-KBQA | Shouhui Wang and Biao Qin | [Link](https://aclanthology.org/2024.lrec-main.1074.pdf) | yes | [Link](https://github.com/salesforce/rng-kbqa) | [Link](https://aclanthology.org/2022.acl-long.417.pdf) | This paper presents RnG-KBQA, a Rank-and-Generate approach for KBQA, which remedies the coverage issue with a generation model while preserving a strong generalization capability. The approach first uses a contrastive ranker to rank a set of candidate logical forms obtained by searching over the knowledge graph. It then introduces a tailored generation model conditioned on the question and the top-ranked candidates to compose the final logical form. | Ye et al. | +| ArcaneQA | Shouhui Wang and Biao Qin | [Link](https://aclanthology.org/2024.lrec-main.1074.pdf) | yes | [Link](https://github.com/dki-lab/ArcaneQA) | [Link](https://arxiv.org/pdf/2204.08109) | This paper presents ArcaneQA, a novel generation-based model that addresses both the large search space and the schema linking challenges in a unified framework with two mutually boosting ingredients: dynamic program induction for tackling the large search space and dynamic contextualized encoding for schema linking. Experimental results on multiple popular KBQA datasets demonstrate the highly competitive performance of ArcaneQA in both effectiveness and efficiency. | Gu and Su | +| Program Transfer | Shouhui Wang and Biao Qin | [Link](https://aclanthology.org/2024.lrec-main.1074.pdf) | yes | [Link](https://github.com/THU-KEG/ProgramTransfer) | [Link](https://aclanthology.org/2022.acl-long.559/) | This paper proposes the approach of program transfer, which aims to leverage the valuable program annotations on the rich-resourced KBs as external supervision signals to aid program induction for the low-resourced KBs that lack program annotations. The authors design a novel two-stage parsing framework with an efficient ontology-guided pruning strategy. First, a sketch parser translates the question into a high-level program sketch, which is the composition of functions. Second, given the question and sketch, an argument parser searches the detailed arguments from the KB for functions. During the searching, we incorporate the KB ontology to prune the search space. | Cao et al. | +| Pangu | Shouhui Wang and Biao Qin | [Link](https://aclanthology.org/2024.lrec-main.1074.pdf) | yes | [Link](https://github.com/dki-lab/Pangu) | [Link](https://aclanthology.org/2023.acl-long.270.pdf) | This paper proposes Pangu, a generic framework for grounded language understanding that capitalizes on the discriminative ability of LMs instead of their generative ability. Pangu consists of a symbolic agent and a neural LM working in a concerted fashion: The agent explores the environment to incrementally construct valid plans, and the LM evaluates the plausibility of the candidate plans to guide the search process. | Gu et al. | +| FC-KBQA | Shouhui Wang and Biao Qin | [Link](https://aclanthology.org/2024.lrec-main.1074.pdf) | no | - | [Link](https://aclanthology.org/2023.acl-long.57.pdf) | This paper proposes a Fine-to-Coarse Composition framework for KBQA (FC-KBQA) to both ensure the generalization ability and executability of the logical expression. The main idea of FC-KBQA is to extract relevant finegrained knowledge components from KB and reformulate them into middle-grained knowledge pairs for generating the final logical expressions. | Zhang et al. | +| TFS-KBQA | Shouhui Wang and Biao Qin | [Link](https://aclanthology.org/2024.lrec-main.1074.pdf) | yes | [Link](https://github.com/shouh/TFS-KBQA) | [Link](https://aclanthology.org/2024.lrec-main.1074.pdf) | This study adopts the LLMs, such as Large Language Model Meta AI (LLaMA), as a channel to connect natural language questions with structured knowledge representations and proposes a Three-step Fine-tune Strategy based on large language model to implement the KBQA system (TFS-KBQA). This method achieves direct conversion from natural language questions to structured knowledge representations, thereby overcoming the limitations of existing KBQA methods, such as addressing large search and reasoning spaces and ranking massive candidates. | Shouhui Wang and Biao Qin | +| TSQA | Gao et al. | [Link](https://arxiv.org/pdf/2402.16568) | no | - | [Link](https://arxiv.org/pdf/2203.00255) | In this paper, we propose a timesensitive question answering (TSQA) framework to tackle these problems. TSQA features a timestamp estimation module to infer the unwritten timestamp from the question. We also employ a time-sensitive KG encoder to inject ordering information into the temporal KG embeddings that TSQA is based on. | Shang et al. | +| GenTKGQA | Gao et al. | [Link](https://arxiv.org/pdf/2402.16568) | no | - | [Link](https://arxiv.org/pdf/2402.16568) | This paper first proposes a novel generative temporal knowledge graph question answering framework, GenTKGQA, which guides LLMs to answer temporal questions through two phases: Subgraph Retrieval and Answer Generation. First, we exploit LLM’s intrinsic knowledge to mine temporal constraints and structural links in the questions without extra training, thus narrowing down the subgraph search space in both temporal and structural dimensions. Next, we design virtual knowledge indicators to fuse the graph neural network signals of the subgraph and the text representations of the LLM in a non-shallow way, which helps the open-source LLM deeply understand the temporal order and structural dependencies among the retrieved facts through instruction tuning. | Gao et al. |