QaldGen: Towards Microbenchmarking of Question Answering Systems over Knowledge Graphs

Kuldeep Singh; Muhammad Saleem; Abhishek Nadgeri; Felix Conrads; Jeff Z. Pan; Axel-Cyrille Ngonga Ngomo; Jens Lehmann

doi:10.1007/978-3-030-30796-7_18

QaldGen: Towards Microbenchmarking of Question Answering Systems over Knowledge Graphs

Kuldeep Singh, Muhammad Saleem, Abhishek Nadgeri, Felix Conrads, Jeff Z. Pan, Axel-Cyrille Ngonga Ngomo, Jens Lehmann

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Abstract

Over the last years, a number of Knowledge Graph (KG) based Question Answering (QA) systems have been developed. Consequently, the series of Question Answering Over Linked Data (QALD1–QALD9) challenges and other datasets have been proposed to evaluate these systems. However, the QA datasets contain a fixed number of natural language questions and do not allow users to select micro-benchmarking samples of the questions tailored towards specific use-cases. We propose QaldGen, a framework for micro-benchmarking of QA systems over KGs which is able to select customised question samples from existing QA datasets. The framework is flexible enough to select question samples of varying sizes and according to the user-defined criteria on the most important features to be considered for QA benchmarking. This is achieved using different clustering algorithms. We compare state-of-the-art QA systems over knowledge graphs by using different QA benchmarking samples. The observed results show that specialised micro-benchmarking is important to pinpoint the limitations of the various QA systems and its components.

Original language	English
Title of host publication	The Semantic Web -- ISWC 2019
Subtitle of host publication	18th International Semantic Web Conference, Auckland, New Zealand, October 26–30, 2019, Proceedings, Part II
Editors	Chiara Ghidini, Olaf Hartig, Maria Maleshkova, Vojtech Svátek, Isabel Cruz, Aidan Hogan, Jie Song, Maxime Lefrançois, Fabien Gandon
Place of Publication	Cham
Publisher	Springer International Publishing
Pages	277–292
Number of pages	16
ISBN (Electronic)	978-3-030-30795-0
ISBN (Print)	978-3-030-30796-7
DOIs	https://doi.org/10.1007/978-3-030-30796-7_18
Publication status	Published - 17 Oct 2019
Event	16th International Symposium on Wireless Communication Systems - Oulu, Finland Duration: 27 Aug 2019 → 30 Aug 2019 http://iswcs2019.org/

Publication series

Name	Lecture Notes in Computer Science
Publisher	Springer, Cham
Volume	11779
ISSN (Print)	0302-9743
ISSN (Electronic)	1611-3349

Conference

Conference	16th International Symposium on Wireless Communication Systems
Abbreviated title	ISWCS 2019
Country/Territory	Finland
City	Oulu
Period	27/08/19 → 30/08/19
Internet address	http://iswcs2019.org/

Access to Document

10.1007/978-3-030-30796-7_18Licence: All Rights Reserved

https://link.springer.com/chapter/10.1007/978-3-030-30796-7_18Licence: All Rights Reserved

Cite this

Singh, K., Saleem, M., Nadgeri, A., Conrads, F., Pan, J. Z., Ngonga Ngomo, A-C., & Lehmann, J. (2019). QaldGen: Towards Microbenchmarking of Question Answering Systems over Knowledge Graphs. In C. Ghidini, O. Hartig, M. Maleshkova, V. Svátek, I. Cruz, A. Hogan, J. Song, M. Lefrançois, & F. Gandon (Eds.), The Semantic Web -- ISWC 2019: 18th International Semantic Web Conference, Auckland, New Zealand, October 26–30, 2019, Proceedings, Part II (pp. 277–292). (Lecture Notes in Computer Science; Vol. 11779). Springer International Publishing. https://doi.org/10.1007/978-3-030-30796-7_18

Singh, Kuldeep ; Saleem, Muhammad ; Nadgeri, Abhishek et al. / QaldGen: Towards Microbenchmarking of Question Answering Systems over Knowledge Graphs. The Semantic Web -- ISWC 2019: 18th International Semantic Web Conference, Auckland, New Zealand, October 26–30, 2019, Proceedings, Part II. editor / Chiara Ghidini ; Olaf Hartig ; Maria Maleshkova ; Vojtech Svátek ; Isabel Cruz ; Aidan Hogan ; Jie Song ; Maxime Lefrançois ; Fabien Gandon. Cham : Springer International Publishing, 2019. pp. 277–292 (Lecture Notes in Computer Science).

@inproceedings{347a0fca1257412d9981cbf8e015061b,

title = "QaldGen: Towards Microbenchmarking of Question Answering Systems over Knowledge Graphs",

abstract = "Over the last years, a number of Knowledge Graph (KG) based Question Answering (QA) systems have been developed. Consequently, the series of Question Answering Over Linked Data (QALD1–QALD9) challenges and other datasets have been proposed to evaluate these systems. However, the QA datasets contain a fixed number of natural language questions and do not allow users to select micro-benchmarking samples of the questions tailored towards specific use-cases. We propose QaldGen, a framework for micro-benchmarking of QA systems over KGs which is able to select customised question samples from existing QA datasets. The framework is flexible enough to select question samples of varying sizes and according to the user-defined criteria on the most important features to be considered for QA benchmarking. This is achieved using different clustering algorithms. We compare state-of-the-art QA systems over knowledge graphs by using different QA benchmarking samples. The observed results show that specialised micro-benchmarking is important to pinpoint the limitations of the various QA systems and its components.",

author = "Kuldeep Singh and Muhammad Saleem and Abhishek Nadgeri and Felix Conrads and Pan, {Jeff Z.} and {Ngonga Ngomo}, Axel-Cyrille and Jens Lehmann",

year = "2019",

month = oct,

day = "17",

doi = "10.1007/978-3-030-30796-7_18",

language = "English",

isbn = "978-3-030-30796-7",

series = "Lecture Notes in Computer Science",

publisher = "Springer International Publishing",

pages = "277–292",

editor = "Chiara Ghidini and Olaf Hartig and Maria Maleshkova and Vojtech Sv{\'a}tek and Isabel Cruz and Aidan Hogan and Jie Song and Maxime Lefran{\c c}ois and Fabien Gandon",

booktitle = "The Semantic Web -- ISWC 2019",

note = "16th International Symposium on Wireless Communication Systems , ISWCS 2019 ; Conference date: 27-08-2019 Through 30-08-2019",

url = "http://iswcs2019.org/",

}

Singh, K, Saleem, M, Nadgeri, A, Conrads, F, Pan, JZ, Ngonga Ngomo, A-C & Lehmann, J 2019, QaldGen: Towards Microbenchmarking of Question Answering Systems over Knowledge Graphs. in C Ghidini, O Hartig, M Maleshkova, V Svátek, I Cruz, A Hogan, J Song, M Lefrançois & F Gandon (eds), The Semantic Web -- ISWC 2019: 18th International Semantic Web Conference, Auckland, New Zealand, October 26–30, 2019, Proceedings, Part II. Lecture Notes in Computer Science, vol. 11779, Springer International Publishing, Cham, pp. 277–292, 16th International Symposium on Wireless Communication Systems , Oulu, Finland, 27/08/19. https://doi.org/10.1007/978-3-030-30796-7_18

QaldGen: Towards Microbenchmarking of Question Answering Systems over Knowledge Graphs. / Singh, Kuldeep; Saleem, Muhammad; Nadgeri, Abhishek et al.

The Semantic Web -- ISWC 2019: 18th International Semantic Web Conference, Auckland, New Zealand, October 26–30, 2019, Proceedings, Part II. ed. / Chiara Ghidini; Olaf Hartig; Maria Maleshkova; Vojtech Svátek; Isabel Cruz; Aidan Hogan; Jie Song; Maxime Lefrançois; Fabien Gandon. Cham : Springer International Publishing, 2019. p. 277–292 (Lecture Notes in Computer Science; Vol. 11779).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

TY - GEN

T1 - QaldGen: Towards Microbenchmarking of Question Answering Systems over Knowledge Graphs

AU - Singh, Kuldeep

AU - Saleem, Muhammad

AU - Nadgeri, Abhishek

AU - Conrads, Felix

AU - Pan, Jeff Z.

AU - Ngonga Ngomo, Axel-Cyrille

AU - Lehmann, Jens

PY - 2019/10/17

Y1 - 2019/10/17

N2 - Over the last years, a number of Knowledge Graph (KG) based Question Answering (QA) systems have been developed. Consequently, the series of Question Answering Over Linked Data (QALD1–QALD9) challenges and other datasets have been proposed to evaluate these systems. However, the QA datasets contain a fixed number of natural language questions and do not allow users to select micro-benchmarking samples of the questions tailored towards specific use-cases. We propose QaldGen, a framework for micro-benchmarking of QA systems over KGs which is able to select customised question samples from existing QA datasets. The framework is flexible enough to select question samples of varying sizes and according to the user-defined criteria on the most important features to be considered for QA benchmarking. This is achieved using different clustering algorithms. We compare state-of-the-art QA systems over knowledge graphs by using different QA benchmarking samples. The observed results show that specialised micro-benchmarking is important to pinpoint the limitations of the various QA systems and its components.

AB - Over the last years, a number of Knowledge Graph (KG) based Question Answering (QA) systems have been developed. Consequently, the series of Question Answering Over Linked Data (QALD1–QALD9) challenges and other datasets have been proposed to evaluate these systems. However, the QA datasets contain a fixed number of natural language questions and do not allow users to select micro-benchmarking samples of the questions tailored towards specific use-cases. We propose QaldGen, a framework for micro-benchmarking of QA systems over KGs which is able to select customised question samples from existing QA datasets. The framework is flexible enough to select question samples of varying sizes and according to the user-defined criteria on the most important features to be considered for QA benchmarking. This is achieved using different clustering algorithms. We compare state-of-the-art QA systems over knowledge graphs by using different QA benchmarking samples. The observed results show that specialised micro-benchmarking is important to pinpoint the limitations of the various QA systems and its components.

U2 - 10.1007/978-3-030-30796-7_18

DO - 10.1007/978-3-030-30796-7_18

M3 - Conference contribution

SN - 978-3-030-30796-7

T3 - Lecture Notes in Computer Science

SP - 277

EP - 292

BT - The Semantic Web -- ISWC 2019

A2 - Ghidini, Chiara

A2 - Hartig, Olaf

A2 - Maleshkova, Maria

A2 - Svátek, Vojtech

A2 - Cruz, Isabel

A2 - Hogan, Aidan

A2 - Song, Jie

A2 - Lefrançois, Maxime

A2 - Gandon, Fabien

PB - Springer International Publishing

CY - Cham

T2 - 16th International Symposium on Wireless Communication Systems

Y2 - 27 August 2019 through 30 August 2019

ER -

Singh K, Saleem M, Nadgeri A, Conrads F, Pan JZ, Ngonga Ngomo A-C et al. QaldGen: Towards Microbenchmarking of Question Answering Systems over Knowledge Graphs. In Ghidini C, Hartig O, Maleshkova M, Svátek V, Cruz I, Hogan A, Song J, Lefrançois M, Gandon F, editors, The Semantic Web -- ISWC 2019: 18th International Semantic Web Conference, Auckland, New Zealand, October 26–30, 2019, Proceedings, Part II. Cham: Springer International Publishing. 2019. p. 277–292. (Lecture Notes in Computer Science). doi: 10.1007/978-3-030-30796-7_18

QaldGen: Towards Microbenchmarking of Question Answering Systems over Knowledge Graphs

Abstract

Publication series

Conference

Access to Document

Fingerprint

Cite this