diff --git a/freebase/WebQuestionsSP.md b/freebase/WebQuestionsSP.md index 0b99b378..60583bdf 100644 --- a/freebase/WebQuestionsSP.md +++ b/freebase/WebQuestionsSP.md @@ -3,99 +3,98 @@ datasetUrl: https://www.microsoft.com/en-us/download/details.aspx?id=52763 --- -| Model / System | Year | F1 | Hits@1 | Accuracy | Language | Reported by | -| :---------------------------------: | :--: | :--------: | :--------: | :------: | :------: | :-----------------------------------------------------------------------------------: | -| chatGPT | 2023 | - | - | 83.70 | EN | [Tan et al.](https://arxiv.org/pdf/2303.07992.pdf) | -| TIARA | 2022 | 78.9 | 75.2 | - | EN | [Shu et. al.](https://aclanthology.org/2022.emnlp-main.555.pdf) | -| DECAF (DPR + FiD-3B) | 2022 | 78.8 | 82.1 | - | EN | [Yu et al.](https://arxiv.org/pdf/2210.00063.pdf) | -| GPT-3.5v3 | 2023 | - | - | 79.60 | EN | [Tan et al.](https://arxiv.org/pdf/2303.07992.pdf) | -| DECAF (DPR + FiD-large) | 2022 | 77.1 ± 0.2 | 80.7 ± 0.2 | - | EN | [Yu et al.](https://arxiv.org/pdf/2210.00063.pdf) | -| UniK-QA | 2022 | - | 79.1 | - | EN | [Yu et al.](https://arxiv.org/pdf/2210.00063.pdf) | -| TERP | 2022 | - | 76.8 | - | EN | [Qiao et al.](https://aclanthology.org/2022.coling-1.156.pdf) | -| SQALER+GNN | 2022 | - | 76.1 | - | EN | [Costas Mavromatis and George Karypis](https://arxiv.org/pdf/2210.13650.pdf) | -| EmQL | 2020 | - | 75.5 | - | EN | [Yu et al.](https://arxiv.org/pdf/2210.00063.pdf) | -| GMT-KBQA | 2022 | 76.6 | - | 73.1 | EN | [Hu et al.](https://aclanthology.org/2022.coling-1.145.pdf) | -| GPT-3.5v2 | 2023 | - | - | 72.34 | EN | [Tan et al.](https://arxiv.org/pdf/2303.07992.pdf) | -| Program Transfer | 2022 | 76.5 | 74.6 | - | EN | [Yu et al.](https://arxiv.org/pdf/2210.00063.pdf) | -| RnG-KBQA (T5-large) | 2022 | 76.2 ± 0.2 | 80.7 ± 0.2 | - | EN | [Yu et al.](https://arxiv.org/pdf/2210.00063.pdf) | -| RnG-KBQA | 2022 | 75.6 | - | 71.1 | EN | [Hu et al.](https://aclanthology.org/2022.coling-1.145.pdf) | -| ArcaneQA | 2022 | 75.3 | - | - | EN | [Yu et al.](https://arxiv.org/pdf/2210.00063.pdf) | -| QNRKGQA+h | 2022 | - | 75.7 | - | EN | [Ma et al.](https://link.springer.com/chapter/10.1007/978-3-031-10983-6_11) | -| DECAF (BM25 + FiD-large) | 2022 | 74.9 ± 0.3 | 79.0 ± 0.4 | - | EN | [Yu et al.](https://arxiv.org/pdf/2210.00063.pdf) | -| MRP-QA-marginal_prob | 2022 | 74.9 | - | - | EN | [Wang et al.](https://aclanthology.org/2022.naacl-main.294.pdf) | -| QNRKGQA | 2022 | - | 74.9 | - | EN | [Ma et al.](https://link.springer.com/chapter/10.1007/978-3-031-10983-6_11) | -| ReTrack | 2022 | 74.7 | - | - | EN | [Hu et al.](https://aclanthology.org/2022.coling-1.145.pdf) | -| ReTrack | 2021 | 74.6 | 74.7 | - | EN | [Yu et al.](https://arxiv.org/pdf/2210.00063.pdf) | -| BART-large | 2022 | 74.6 | - | - | EN | [Hu et al.](https://aclanthology.org/2022.coling-1.145.pdf) | -| Subgraph Retrieval | 2022 | 74.5 | 83.2 | - | EN | [Shu et. al.](https://aclanthology.org/2022.emnlp-main.555.pdf) | -| QGG | 2022 | 74.0 | - | - | EN | [Yu et al.](https://arxiv.org/pdf/2210.00063.pdf) | -| CBR-KBQA | 2021 | 72.8 | - | 69.9 | EN | [Yu et al.](https://arxiv.org/pdf/2210.00063.pdf) | -| GPT-3 | 2023 | - | - | 67.78 | EN | [Tan et al.](https://arxiv.org/pdf/2303.07992.pdf) | -| KGQA-RR(Roberta) | 2023 | - | - | 64.59 | EN | [Hu et al.](https://arxiv.org/pdf/2303.10368.pdf) | -| KGQA-RR(Luke) | 2023 | - | - | 64.52 | EN | [Hu et al.](https://arxiv.org/pdf/2303.10368.pdf) | -| KGQA-RR(Kepler) | 2023 | - | - | 64.46 | EN | [Hu et al.](https://arxiv.org/pdf/2303.10368.pdf) | -| KGQA-RR(Bert) | 2023 | - | - | 64.11 | EN | [Hu et al.](https://arxiv.org/pdf/2303.10368.pdf) | -| KGQA-RR(Albert) | 2023 | - | - | 63.89 | EN | [Hu et al.](https://arxiv.org/pdf/2303.10368.pdf) | -| KGQA-RR(XLnet) | 2023 | - | - | 63.87 | EN | [Hu et al.](https://arxiv.org/pdf/2303.10368.pdf) | -| KGQA-RR(DistilBert) | 2023 | - | - | 63.59 | EN | [Hu et al.](https://arxiv.org/pdf/2303.10368.pdf) | -| KGQA-RR(DistilRoberta) | 2023 | - | - | 62.57 | EN | [Hu et al.](https://arxiv.org/pdf/2303.10368.pdf) | -| KGQA-CL(Roberta) | 2023 | - | - | 62.32 | EN | [Hu et al.](https://arxiv.org/pdf/2303.10368.pdf) | -| KGQA-CL(Luke) | 2023 | - | - | 62.31 | EN | [Hu et al.](https://arxiv.org/pdf/2303.10368.pdf) | -| KGQA-CL(Kepler) | 2023 | - | - | 62.02 | EN | [Hu et al.](https://arxiv.org/pdf/2303.10368.pdf) | -| KGQA-CL(Bert) | 2023 | - | - | 61.76 | EN | [Hu et al.](https://arxiv.org/pdf/2303.10368.pdf) | -| KGQA-CL(DistilBert) | 2023 | - | - | 61.49 | EN | [Hu et al.](https://arxiv.org/pdf/2303.10368.pdf) | -| KGQA-CL(Albert) | 2023 | - | - | 61.47 | EN | [Hu et al.](https://arxiv.org/pdf/2303.10368.pdf) | -| KGQA-CL(XLnet) | 2023 | - | - | 61.46 | EN | [Hu et al.](https://arxiv.org/pdf/2303.10368.pdf) | -| KGQA-CL(DistilRoberta) | 2023 | - | - | 61.05 | EN | [Hu et al.](https://arxiv.org/pdf/2303.10368.pdf) | -| KGQA-CL(GPT2) | 2023 | - | - | 60.49 | EN | [Hu et al.](https://arxiv.org/pdf/2303.10368.pdf) | -| W. Han et al. | 2023 | - | 75.2 | - | EN | [Han et al.](https://link.springer.com/chapter/10.1007/978-3-031-30672-3_39) | -| NSM | 2021 | - | 74.30 | - | EN | [He et al.](https://arxiv.org/pdf/2101.03737.pdf) | -| Rigel | 2022 | - | 73.3 | - | EN | [Costas Mavromatis and George Karypis](https://arxiv.org/pdf/2210.13650.pdf) | -| SGM | 2022 | 72.36 | - | - | EN | [Ma L et al.](https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=9747229) | -| CBR-SUBG | 2022 | 72.1 | - | - | EN | [Yu et al.](https://arxiv.org/pdf/2210.00063.pdf) | -| NPI | 2022 | - | 72.6 | - | EN | [Cao et al.](https://aclanthology.org/2022.acl-long.559.pdf) | -| TextRay | 2022 | - | 72.2 | - | EN | [Cao et al.](https://aclanthology.org/2022.acl-long.559.pdf) | -| CBR-SUBG | 2022 | - | 72.10 | - | EN | [Das et al.](https://arxiv.org/pdf/2202.10610.pdf) | -| KGQA Based on Query Path Generation | 2022 | - | 71.7 | - | EN | [Yang et al.](https://link.springer.com/chapter/10.1007/978-3-031-10983-6_12) | -| STAGG_SP | 2022 | 71.7 | - | - | EN | [Wang et al.](https://aclanthology.org/2022.naacl-main.294.pdf) | -| SSKGQA | 2022 | - | 71.4 | - | EN | [Mingchen Li and Jonathan Shihao Ji](https://arxiv.org/pdf/2204.10194.pdf) | -| TransferNet | 2022 | - | 71.4 | - | EN | [Shi et al.](https://arxiv.org/pdf/2104.07302.pdf) | -| SeqM | 2020 | 71.83 | - | - | EN | [Ma L et al.](https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=9747229) | -| ReTraCK | 2021 | 71.0 | 71.6 | - | EN | [Shu et. al.](https://aclanthology.org/2022.emnlp-main.555.pdf) | -| REAREV | 2022 | 70.9 | 76.4 | - | EN | [Costas Mavromatis and George Karypis](https://arxiv.org/pdf/2210.13650.pdf) | -| HGNet | 2021 | 70.3 | 70.6 | - | EN | [Yu et al.](https://arxiv.org/pdf/2210.00063.pdf) | -| GrailQA Ranking | 2021 | 70.0 | - | - | EN | [Shu et. al.](https://aclanthology.org/2022.emnlp-main.555.pdf) | -| SQALER | 2022 | - | 70.6 | - | EN | [Costas Mavromatis and George Karypis](https://arxiv.org/pdf/2210.13650.pdf) | -| STAGG | 2015 | 69.00 | - | - | EN | [Ma L et al.](https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=9747229) | -| UHop | 2019 | 68.5 | - | - | EN | [Ma L et al.](https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=9747229) | -| KBIGER | 2022 | 68.4 | 75.3 | - | EN | [Du et al.](https://arxiv.org/pdf/2209.03005.pdf) | -| NSM | 2022 | - | 69.0 | - | EN | [Cao et al.](https://aclanthology.org/2022.acl-long.559.pdf) | -| GraftNet-EF+LF | 2018 | - | 68.7 | - | EN | [Sun et al.](https://aclanthology.org/D18-1455.pdf) | -| PullNet | 2019 | - | 68.1 | - | EN | [Sun et al.](https://arxiv.org/pdf/1904.09537.pdf) | -| KBQA-GST | 2022 | 67.9 | - | - | EN | [Wang et al.](https://aclanthology.org/2022.naacl-main.294.pdf) | -| Topic Units | 2019 | 67.9 | - | - | EN | [Ma L et al.](https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=9747229) | -| NSM | 2022 | 67.4 | 74.3 | - | EN | [Du et al.](https://arxiv.org/pdf/2209.03005.pdf) | -| Relation Learning | 2021 | 64.5 | 72.9 | - | EN | [Shu et. al.](https://aclanthology.org/2022.emnlp-main.555.pdf) | -| SR+NSM | 2022 | 64.1 | 69.5 | - | EN | [Yu et al.](https://arxiv.org/pdf/2210.00063.pdf) | -| NSM | 2022 | 62.8 | 68.7 | - | EN | [Costas Mavromatis and George Karypis](https://arxiv.org/pdf/2210.13650.pdf) | -| ARN_ConvE | 2023 | - | 68.0 | - | EN | [Cui et al.](https://www.sciencedirect.com/science/article/abs/pii/S0020025522013317) | -| GraftNet | 2022 | 62.8 | 67.8 | - | EN | [Du et al.](https://arxiv.org/pdf/2209.03005.pdf) | -| PullNet | 2019 | 62.8 | 67.8 | - | EN | [Yu et al.](https://arxiv.org/pdf/2210.00063.pdf) | -| DCRN | 2021 | - | 67.8 | - | EN | [Cai et al.](https://aclanthology.org/2021.findings-acl.19.pdf) | -| ARN_TuckER | 2023 | - | 67.5 | - | EN | [Cui et al.](https://www.sciencedirect.com/science/article/abs/pii/S0020025522013317) | -| NRQA | 2022 | - | 67.1 | - | EN | [Guo et al.](https://link.springer.com/content/pdf/10.1007/s10489-022-03927-0.pdf) | -| GraftNet | 2022 | - | 66.4 | - | EN | [Mingchen Li and Jonathan Shihao Ji](https://arxiv.org/pdf/2204.10194.pdf) | -| EmbedKGQA | 2020 | - | 66.6 | - | EN | [Saxena et al.](https://aclanthology.org/2020.acl-main.412.pdf) | -| GraftNet | 2022 | 62.4 | 66.7 | - | EN | [Costas Mavromatis and George Karypis](https://arxiv.org/pdf/2210.13650.pdf) | -| HR-BiLSTM | 2022 | 62.3 | - | - | EN | [Wang et al.](https://aclanthology.org/2022.naacl-main.294.pdf) | -| GraftNet-EF+LF | 2018 | 62.30 | - | - | EN | [Sun et al.](https://aclanthology.org/D18-1455.pdf) | -| TextRay | 2019 | 60.3 | - | - | EN | [Bhutani et al.](https://dl.acm.org/doi/pdf/10.1145/3357384.3358033) | -| SGReader | 2022 | 57.3 | 67.2 | - | EN | [Costas Mavromatis and George Karypis](https://arxiv.org/pdf/2210.13650.pdf) | -| ARN_ComplEx | 2023 | - | 65.3 | - | EN | [Cui et al.](https://www.sciencedirect.com/science/article/abs/pii/S0020025522013317) | -| ARN_DistMult | 2023 | - | 61.7 | - | EN | [Cui et al.](https://www.sciencedirect.com/science/article/abs/pii/S0020025522013317) | -| FLAN-T5 | 2023 | - | - | 59.87 | EN | [Tan et al.](https://arxiv.org/pdf/2303.07992.pdf) | -| KGT5 | 2022 | 56.1 | - | - | EN | [Yu et al.](https://arxiv.org/pdf/2210.00063.pdf) | -| FILM | 2022 | 54.7 | - | - | EN | [Yu et al.](https://arxiv.org/pdf/2210.00063.pdf) | -| ReifKB | 2020 | - | 52.7 | - | EN | [Cohen et al.](https://arxiv.org/pdf/2002.06115.pdf) | -| KV-Mem | 2022 | 38.6 | 46.7 | - | EN | [Du et al.](https://arxiv.org/pdf/2209.03005.pdf) | -| KGQA-RR(GPT2) | 2023 | - | - | 18.11 | EN | [Hu et al.](https://arxiv.org/pdf/2303.10368.pdf) | +| Model / System | Year | F1 | Hits@1 | Accuracy | Language | Reported by | +| :---------------------------------: | :--: | :--------: | :--------: | :------: | :------: | :---------------------------------------------------------------------------------------: | +| chatGPT | 2023 | - | - | 83.70 | EN | [Tan et al.](https://arxiv.org/pdf/2303.07992.pdf) | +| FRED | 2023 | 86 ± 5 | - | - | EN |[Lamott et al.](https://recap.uni-trier.de/static/377a488cc4cee95714b3ad713aa22fa7/88.pdf) | +| TIARA | 2022 | 78.9 | 75.2 | - | EN | [Shu et. al.](https://aclanthology.org/2022.emnlp-main.555.pdf) | +| DECAF (DPR + FiD-3B) | 2022 | 78.8 | 82.1 | - | EN | [Yu et al.](https://arxiv.org/pdf/2210.00063.pdf) | +| GPT-3.5v3 | 2023 | - | - | 79.60 | EN | [Tan et al.](https://arxiv.org/pdf/2303.07992.pdf) | +| DECAF (DPR + FiD-large) | 2022 | 77.1 ± 0.2 | 80.7 ± 0.2 | - | EN | [Yu et al.](https://arxiv.org/pdf/2210.00063.pdf) | +| UniK-QA | 2022 | - | 79.1 | - | EN | [Yu et al.](https://arxiv.org/pdf/2210.00063.pdf) | +| SQALER+GNN | 2022 | - | 76.1 | - | EN | [Costas Mavromatis and George Karypis](https://arxiv.org/pdf/2210.13650.pdf) | +| EmQL | 2020 | - | 75.5 | - | EN | [Yu et al.](https://arxiv.org/pdf/2210.00063.pdf) | +| GMT-KBQA | 2022 | 76.6 | - | 73.1 | EN | [Hu et al.](https://aclanthology.org/2022.coling-1.145.pdf) | +| GPT-3.5v2 | 2023 | - | - | 72.34 | EN | [Tan et al.](https://arxiv.org/pdf/2303.07992.pdf) | +| Program Transfer | 2022 | 76.5 | 74.6 | - | EN | [Yu et al.](https://arxiv.org/pdf/2210.00063.pdf) | +| RnG-KBQA (T5-large) | 2022 | 76.2 ± 0.2 | 80.7 ± 0.2 | - | EN | [Yu et al.](https://arxiv.org/pdf/2210.00063.pdf) | +| RnG-KBQA | 2022 | 75.6 | - | 71.1 | EN | [Hu et al.](https://aclanthology.org/2022.coling-1.145.pdf) | +| ArcaneQA | 2022 | 75.3 | - | - | EN | [Yu et al.](https://arxiv.org/pdf/2210.00063.pdf) | +| QNRKGQA+h | 2022 | - | 75.7 | - | EN | [Ma et al.](https://link.springer.com/chapter/10.1007/978-3-031-10983-6_11) | +| DECAF (BM25 + FiD-large) | 2022 | 74.9 ± 0.3 | 79.0 ± 0.4 | - | EN | [Yu et al.](https://arxiv.org/pdf/2210.00063.pdf) | +| MRP-QA-marginal_prob | 2022 | 74.9 | - | - | EN | [Wang et al.](https://aclanthology.org/2022.naacl-main.294.pdf) | +| QNRKGQA | 2022 | - | 74.9 | - | EN | [Ma et al.](https://link.springer.com/chapter/10.1007/978-3-031-10983-6_11) | +| ReTrack | 2022 | 74.7 | - | - | EN | [Hu et al.](https://aclanthology.org/2022.coling-1.145.pdf) | +| ReTrack | 2021 | 74.6 | 74.7 | - | EN | [Yu et al.](https://arxiv.org/pdf/2210.00063.pdf) | +| BART-large | 2022 | 74.6 | - | - | EN | [Hu et al.](https://aclanthology.org/2022.coling-1.145.pdf) | +| Subgraph Retrieval | 2022 | 74.5 | 83.2 | - | EN | [Shu et. al.](https://aclanthology.org/2022.emnlp-main.555.pdf) | +| QGG | 2022 | 74.0 | - | - | EN | [Yu et al.](https://arxiv.org/pdf/2210.00063.pdf) | +| CBR-KBQA | 2021 | 72.8 | - | 69.9 | EN | [Yu et al.](https://arxiv.org/pdf/2210.00063.pdf) | +| GPT-3 | 2023 | - | - | 67.78 | EN | [Tan et al.](https://arxiv.org/pdf/2303.07992.pdf) | +| KGQA-RR(Roberta) | 2023 | - | - | 64.59 | EN | [Hu et al.](https://arxiv.org/pdf/2303.10368.pdf) | +| KGQA-RR(Luke) | 2023 | - | - | 64.52 | EN | [Hu et al.](https://arxiv.org/pdf/2303.10368.pdf) | +| KGQA-RR(Kepler) | 2023 | - | - | 64.46 | EN | [Hu et al.](https://arxiv.org/pdf/2303.10368.pdf) | +| KGQA-RR(Bert) | 2023 | - | - | 64.11 | EN | [Hu et al.](https://arxiv.org/pdf/2303.10368.pdf) | +| KGQA-RR(Albert) | 2023 | - | - | 63.89 | EN | [Hu et al.](https://arxiv.org/pdf/2303.10368.pdf) | +| KGQA-RR(XLnet) | 2023 | - | - | 63.87 | EN | [Hu et al.](https://arxiv.org/pdf/2303.10368.pdf) | +| KGQA-RR(DistilBert) | 2023 | - | - | 63.59 | EN | [Hu et al.](https://arxiv.org/pdf/2303.10368.pdf) | +| KGQA-RR(DistilRoberta) | 2023 | - | - | 62.57 | EN | [Hu et al.](https://arxiv.org/pdf/2303.10368.pdf) | +| KGQA-CL(Roberta) | 2023 | - | - | 62.32 | EN | [Hu et al.](https://arxiv.org/pdf/2303.10368.pdf) | +| KGQA-CL(Luke) | 2023 | - | - | 62.31 | EN | [Hu et al.](https://arxiv.org/pdf/2303.10368.pdf) | +| KGQA-CL(Kepler) | 2023 | - | - | 62.02 | EN | [Hu et al.](https://arxiv.org/pdf/2303.10368.pdf) | +| KGQA-CL(Bert) | 2023 | - | - | 61.76 | EN | [Hu et al.](https://arxiv.org/pdf/2303.10368.pdf) | +| KGQA-CL(DistilBert) | 2023 | - | - | 61.49 | EN | [Hu et al.](https://arxiv.org/pdf/2303.10368.pdf) | +| KGQA-CL(Albert) | 2023 | - | - | 61.47 | EN | [Hu et al.](https://arxiv.org/pdf/2303.10368.pdf) | +| KGQA-CL(XLnet) | 2023 | - | - | 61.46 | EN | [Hu et al.](https://arxiv.org/pdf/2303.10368.pdf) | +| KGQA-CL(DistilRoberta) | 2023 | - | - | 61.05 | EN | [Hu et al.](https://arxiv.org/pdf/2303.10368.pdf) | +| KGQA-CL(GPT2) | 2023 | - | - | 60.49 | EN | [Hu et al.](https://arxiv.org/pdf/2303.10368.pdf) | +| NSM | 2021 | - | 74.30 | - | EN | [He et al.](https://arxiv.org/pdf/2101.03737.pdf) | +| Rigel | 2022 | - | 73.3 | - | EN | [Costas Mavromatis and George Karypis](https://arxiv.org/pdf/2210.13650.pdf) | +| SGM | 2022 | 72.36 | - | - | EN | [Ma L et al.](https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=9747229) | +| CBR-SUBG | 2022 | 72.1 | - | - | EN | [Yu et al.](https://arxiv.org/pdf/2210.00063.pdf) | +| NPI | 2022 | - | 72.6 | - | EN | [Cao et al.](https://aclanthology.org/2022.acl-long.559.pdf) | +| TextRay | 2022 | - | 72.2 | - | EN | [Cao et al.](https://aclanthology.org/2022.acl-long.559.pdf) | +| CBR-SUBG | 2022 | - | 72.10 | - | EN | [Das et al.](https://arxiv.org/pdf/2202.10610.pdf) | +| KGQA Based on Query Path Generation | 2022 | - | 71.7 | - | EN | [Yang et al.](https://link.springer.com/chapter/10.1007/978-3-031-10983-6_12) | +| STAGG_SP | 2022 | 71.7 | - | - | EN | [Wang et al.](https://aclanthology.org/2022.naacl-main.294.pdf) | +| SSKGQA | 2022 | - | 71.4 | - | EN | [Mingchen Li and Jonathan Shihao Ji](https://arxiv.org/pdf/2204.10194.pdf) | +| TransferNet | 2022 | - | 71.4 | - | EN | [Shi et al.](https://arxiv.org/pdf/2104.07302.pdf) | +| SeqM | 2020 | 71.83 | - | - | EN | [Ma L et al.](https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=9747229) | +| ReTraCK | 2021 | 71.0 | 71.6 | - | EN | [Shu et. al.](https://aclanthology.org/2022.emnlp-main.555.pdf) | +| REAREV | 2022 | 70.9 | 76.4 | - | EN | [Costas Mavromatis and George Karypis](https://arxiv.org/pdf/2210.13650.pdf) | +| HGNet | 2021 | 70.3 | 70.6 | - | EN | [Yu et al.](https://arxiv.org/pdf/2210.00063.pdf) | +| GrailQA Ranking | 2021 | 70.0 | - | - | EN | [Shu et. al.](https://aclanthology.org/2022.emnlp-main.555.pdf) | +| SQALER | 2022 | - | 70.6 | - | EN | [Costas Mavromatis and George Karypis](https://arxiv.org/pdf/2210.13650.pdf) | +| STAGG | 2015 | 69.00 | - | - | EN | [Ma L et al.](https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=9747229) | +| UHop | 2019 | 68.5 | - | - | EN | [Ma L et al.](https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=9747229) | +| KBIGER | 2022 | 68.4 | 75.3 | - | EN | [Du et al.](https://arxiv.org/pdf/2209.03005.pdf) | +| NSM | 2022 | - | 69.0 | - | EN | [Cao et al.](https://aclanthology.org/2022.acl-long.559.pdf) | +| GraftNet-EF+LF | 2018 | - | 68.7 | - | EN | [Sun et al.](https://aclanthology.org/D18-1455.pdf) | +| PullNet | 2019 | - | 68.1 | - | EN | [Sun et al.](https://arxiv.org/pdf/1904.09537.pdf) | +| KBQA-GST | 2022 | 67.9 | - | - | EN | [Wang et al.](https://aclanthology.org/2022.naacl-main.294.pdf) | +| Topic Units | 2019 | 67.9 | - | - | EN | [Ma L et al.](https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=9747229) | +| NSM | 2022 | 67.4 | 74.3 | - | EN | [Du et al.](https://arxiv.org/pdf/2209.03005.pdf) | +| Relation Learning | 2021 | 64.5 | 72.9 | - | EN | [Shu et. al.](https://aclanthology.org/2022.emnlp-main.555.pdf) | +| SR+NSM | 2022 | 64.1 | 69.5 | - | EN | [Yu et al.](https://arxiv.org/pdf/2210.00063.pdf) | +| NSM | 2022 | 62.8 | 68.7 | - | EN | [Costas Mavromatis and George Karypis](https://arxiv.org/pdf/2210.13650.pdf) | +| ARN_ConvE | 2023 | - | 68.0 | - | EN | [Cui et al.](https://www.sciencedirect.com/science/article/abs/pii/S0020025522013317) | +| GraftNet | 2022 | 62.8 | 67.8 | - | EN | [Du et al.](https://arxiv.org/pdf/2209.03005.pdf) | +| PullNet | 2019 | 62.8 | 67.8 | - | EN | [Yu et al.](https://arxiv.org/pdf/2210.00063.pdf) | +| DCRN | 2021 | - | 67.8 | - | EN | [Cai et al.](https://aclanthology.org/2021.findings-acl.19.pdf) | +| ARN_TuckER | 2023 | - | 67.5 | - | EN | [Cui et al.](https://www.sciencedirect.com/science/article/abs/pii/S0020025522013317) | +| NRQA | 2022 | - | 67.1 | - | EN | [Guo et al.](https://link.springer.com/content/pdf/10.1007/s10489-022-03927-0.pdf) | +| GraftNet | 2022 | - | 66.4 | - | EN | [Mingchen Li and Jonathan Shihao Ji](https://arxiv.org/pdf/2204.10194.pdf) | +| EmbedKGQA | 2020 | - | 66.6 | - | EN | [Saxena et al.](https://aclanthology.org/2020.acl-main.412.pdf) | +| GraftNet | 2022 | 62.4 | 66.7 | - | EN | [Costas Mavromatis and George Karypis](https://arxiv.org/pdf/2210.13650.pdf) | +| HR-BiLSTM | 2022 | 62.3 | - | - | EN | [Wang et al.](https://aclanthology.org/2022.naacl-main.294.pdf) | +| GraftNet-EF+LF | 2018 | 62.30 | - | - | EN | [Sun et al.](https://aclanthology.org/D18-1455.pdf) | +| TextRay | 2019 | 60.3 | - | - | EN | [Bhutani et al.](https://dl.acm.org/doi/pdf/10.1145/3357384.3358033) | +| SGReader | 2022 | 57.3 | 67.2 | - | EN | [Costas Mavromatis and George Karypis](https://arxiv.org/pdf/2210.13650.pdf) | +| ARN_ComplEx | 2023 | - | 65.3 | - | EN | [Cui et al.](https://www.sciencedirect.com/science/article/abs/pii/S0020025522013317) | +| ARN_DistMult | 2023 | - | 61.7 | - | EN | [Cui et al.](https://www.sciencedirect.com/science/article/abs/pii/S0020025522013317) | +| FLAN-T5 | 2023 | - | - | 59.87 | EN | [Tan et al.](https://arxiv.org/pdf/2303.07992.pdf) | +| KGT5 | 2022 | 56.1 | - | - | EN | [Yu et al.](https://arxiv.org/pdf/2210.00063.pdf) | +| FILM | 2022 | 54.7 | - | - | EN | [Yu et al.](https://arxiv.org/pdf/2210.00063.pdf) | +| ReifKB | 2020 | - | 52.7 | - | EN | [Cohen et al.](https://arxiv.org/pdf/2002.06115.pdf) | +| KV-Mem | 2022 | 38.6 | 46.7 | - | EN | [Du et al.](https://arxiv.org/pdf/2209.03005.pdf) | +| KGQA-RR(GPT2) | 2023 | - | - | 18.11 | EN | [Hu et al.](https://arxiv.org/pdf/2303.10368.pdf) | diff --git a/systems.md b/systems.md index 5e7446af..943a9811 100644 --- a/systems.md +++ b/systems.md @@ -139,4 +139,5 @@ | TIARA | Shu et al. | [Link](https://aclanthology.org/2022.emnlp-main.555.pdf) | yes | [Link](https://github.com/microsoft/KC/tree/main/papers/TIARA) | [Link](https://aclanthology.org/2022.emnlp-main.555.pdf) | In this paper, we present a new KBQA model, TIARA, which addresses those issues by applying multi-grained retrieval to help the PLM focus on the most relevant KB contexts, viz., entities, exemplary logical forms, and schema items. Moreover, constrained decoding is used to control the output space and reduce generation errors. | Shu et al. | | MACRE | Xu et al. | [Link](https://link.springer.com/chapter/10.1007/978-3-031-30672-3_40) | no | - | [Link](https://link.springer.com/chapter/10.1007/978-3-031-30672-3_40) | MACRE is a novel approach for multi-hop question answering over KGs via contrastive relation embedding (MACRE) powered by contrastive relation embedding and context-aware relation ranking. | Xu et al. | | KGQAcl/rr | Hu et al. | [Link](https://arxiv.org/pdf/2303.10368.pdf) | yes | [Link](https://github.com/HuuuNan/PLMs-in-Practical-KBQA) | [Link](https://arxiv.org/pdf/2303.10368.pdf) | KGQA-CL and KGQA-RR are tow frameworks proposed to evaluate the PLM's performance in comparison to their efficiency. Both architectures are composed of mention detection, entity disambiguiation, relation detection and answer query building. The difference lies on the relation detection module. KGQA-CL aims to map question intent to KG relations. While KGQA-RR ranks the related relations to retrieve the answer entity. Both frameworks are tested on common PLM, distilled PLMs and knowledge-enhanced PLMs and achieve high performance on three benchmarks. | Hu et al. | -| W. Han et al. | Han et al. | [Link](https://link.springer.com/chapter/10.1007/978-3-031-30672-3_39) | no | - | [Link](https://link.springer.com/chapter/10.1007/978-3-031-30672-3_39) | This model is based on machine reading comprehension. To transform a subgraph of the KG centered on the topic entity into text, the subgraph is sketched through a carefully designed schema tree, which facilitates the retrieval of multiple semantically-equivalent answer entities. Then, the promising paragraphs containing answers are picked by a contrastive learning module. Finally, the answer entities are delivered based on the answer span that is detected by the machine reading comprehension module. | Han et al. | \ No newline at end of file +| FRED | Lamott et al. | [Link](https://recap.uni-trier.de/static/377a488cc4cee95714b3ad713aa22fa7/88.pdf) | no | - | [Link](https://recap.uni-trier.de/static/377a488cc4cee95714b3ad713aa22fa7/88.pdf) | FRED combines pattern-based entity retrieval with a transformer-based question encoder. The method uses an evolutionary approach to learn SPARQL patterns, which retrieve candidate entities from a knowledge base. The transformer-based regressor is then trained to estimate each pattern’s expected F1 score for answering the question, resulting in a ranking of candidate entities. Unlike other approaches, FRED can attribute results to learned SPARQL patterns, making them more interpretable. | Lamott et al. | +| W. Han et al. | Han et al. | [Link](https://link.springer.com/chapter/10.1007/978-3-031-30672-3_39) | no | - | [Link](https://link.springer.com/chapter/10.1007/978-3-031-30672-3_39) | This model is based on machine reading comprehension. To transform a subgraph of the KG centered on the topic entity into text, the subgraph is sketched through a carefully designed schema tree, which facilitates the retrieval of multiple semantically-equivalent answer entities. Then, the promising paragraphs containing answers are picked by a contrastive learning module. Finally, the answer entities are delivered based on the answer span that is detected by the machine reading comprehension module. | Han et al. |