QaldGen: Towards Microbenchmarking of Question Answering Systems over Knowledge Graphs

Kuldeep Singh, Muhammad Saleem, Abhishek Nadgeri, Felix Conrads, Jeff Z. Pan, Axel-Cyrille Ngonga Ngomo, Jens Lehmann

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

Over the last years, a number of Knowledge Graph (KG) based Question Answering (QA) systems have been developed. Consequently, the series of Question Answering Over Linked Data (QALD1–QALD9) challenges and other datasets have been proposed to evaluate these systems. However, the QA datasets contain a fixed number of natural language questions and do not allow users to select micro-benchmarking samples of the questions tailored towards specific use-cases. We propose QaldGen, a framework for micro-benchmarking of QA systems over KGs which is able to select customised question samples from existing QA datasets. The framework is flexible enough to select question samples of varying sizes and according to the user-defined criteria on the most important features to be considered for QA benchmarking. This is achieved using different clustering algorithms. We compare state-of-the-art QA systems over knowledge graphs by using different QA benchmarking samples. The observed results show that specialised micro-benchmarking is important to pinpoint the limitations of the various QA systems and its components.
Original languageEnglish
Title of host publicationThe Semantic Web -- ISWC 2019
Subtitle of host publication18th International Semantic Web Conference, Auckland, New Zealand, October 26–30, 2019, Proceedings, Part II
EditorsChiara Ghidini, Olaf Hartig, Maria Maleshkova, Vojtech Svátek, Isabel Cruz, Aidan Hogan, Jie Song, Maxime Lefrançois, Fabien Gandon
Place of PublicationCham
PublisherSpringer International Publishing
Pages277–292
Number of pages16
ISBN (Electronic)978-3-030-30795-0
ISBN (Print)978-3-030-30796-7
DOIs
Publication statusPublished - 17 Oct 2019
Event16th International Symposium on Wireless Communication Systems - Oulu, Finland
Duration: 27 Aug 201930 Aug 2019
http://iswcs2019.org/

Publication series

NameLecture Notes in Computer Science
PublisherSpringer, Cham
Volume11779
ISSN (Print)0302-9743
ISSN (Electronic)1611-3349

Conference

Conference16th International Symposium on Wireless Communication Systems
Abbreviated titleISWCS 2019
CountryFinland
CityOulu
Period27/08/1930/08/19
Internet address

Fingerprint

Dive into the research topics of 'QaldGen: Towards Microbenchmarking of Question Answering Systems over Knowledge Graphs'. Together they form a unique fingerprint.

Cite this