Making Test Corpora for Question Answering More Representative

Andrew Walker; Andrew Starkey; Jeff Z. Pan; Advaith Siddharthan

doi:10.1007/978-3-319-11382-1_1

Making Test Corpora for Question Answering More Representative

Andrew Walker, Andrew Starkey, Jeff Z. Pan, Advaith Siddharthan

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Abstract

Despite two high profile series of challenges devoted to question answering technologies there remains no formal study into the representativeness that question corpora bear to real end-user inputs. We examine the corpora used presently and historically in the TREC and QALD challenges in juxtaposition with two more from natural sources and identify a degree of disjointedness between the two. We analyse these differences in depth before discussing a candidate approach to question corpora generation and provide a juxtaposition on its own representativeness. We conclude that these artificial corpora have good overall coverage of grammatical structures but the distribution is skewed, meaning performance measures may be inaccurate.

Original language	English
Title of host publication	Information Access Evaluation. Multilinguality, Multimodality, and Interaction
Subtitle of host publication	5th International Conference of the CLEF Initiative, CLEF 2014, Sheffield, UK, September 15-18, 2014. Proceedings
Editors	Evangelos Kanoulas, Mihai Lupu, Paul Clough, Mark Sanderson, Mark Hall, Allan Hanbury, Elaine Toms
Place of Publication	Cham
Publisher	Springer
Pages	1-6
Number of pages	6
ISBN (Electronic)	978-3-319-11382-1
ISBN (Print)	978-3-319-11381-4
DOIs	https://doi.org/10.1007/978-3-319-11382-1_1
Publication status	Published - 18 Sept 2014
Event	2014 Cross Language Evaluation Forum Conference, CLEF 2014 - Sheffield, United Kingdom Duration: 15 Sept 2014 → 18 Sept 2014

Publication series

Name	Lecture Notes in Computer Science
Publisher	Springer, Cham
ISSN (Print)	0302-9743
ISSN (Electronic)	1611-3349

Conference

Conference	2014 Cross Language Evaluation Forum Conference, CLEF 2014
Country/Territory	United Kingdom
City	Sheffield
Period	15/09/14 → 18/09/14

Access to Document

10.1007/978-3-319-11382-1_1Licence: All Rights Reserved

https://link.springer.com/chapter/10.1007/978-3-319-11382-1_1Licence: All Rights Reserved

Cite this

Walker, A., Starkey, A., Pan, J. Z., & Siddharthan, A. (2014). Making Test Corpora for Question Answering More Representative. In E. Kanoulas, M. Lupu, P. Clough, M. Sanderson, M. Hall, A. Hanbury, & E. Toms (Eds.), Information Access Evaluation. Multilinguality, Multimodality, and Interaction: 5th International Conference of the CLEF Initiative, CLEF 2014, Sheffield, UK, September 15-18, 2014. Proceedings (pp. 1-6). (Lecture Notes in Computer Science). Springer. https://doi.org/10.1007/978-3-319-11382-1_1

Walker, Andrew ; Starkey, Andrew ; Pan, Jeff Z. et al. / Making Test Corpora for Question Answering More Representative. Information Access Evaluation. Multilinguality, Multimodality, and Interaction: 5th International Conference of the CLEF Initiative, CLEF 2014, Sheffield, UK, September 15-18, 2014. Proceedings. editor / Evangelos Kanoulas ; Mihai Lupu ; Paul Clough ; Mark Sanderson ; Mark Hall ; Allan Hanbury ; Elaine Toms. Cham : Springer, 2014. pp. 1-6 (Lecture Notes in Computer Science).

@inproceedings{f41ae560c81845a88c23a37311845e7f,

title = "Making Test Corpora for Question Answering More Representative",

abstract = "Despite two high profile series of challenges devoted to question answering technologies there remains no formal study into the representativeness that question corpora bear to real end-user inputs. We examine the corpora used presently and historically in the TREC and QALD challenges in juxtaposition with two more from natural sources and identify a degree of disjointedness between the two. We analyse these differences in depth before discussing a candidate approach to question corpora generation and provide a juxtaposition on its own representativeness. We conclude that these artificial corpora have good overall coverage of grammatical structures but the distribution is skewed, meaning performance measures may be inaccurate.",

author = "Andrew Walker and Andrew Starkey and Pan, {Jeff Z.} and Advaith Siddharthan",

year = "2014",

month = sep,

day = "18",

doi = "10.1007/978-3-319-11382-1_1",

language = "English",

isbn = "978-3-319-11381-4",

series = "Lecture Notes in Computer Science",

publisher = "Springer",

pages = "1--6",

editor = "Evangelos Kanoulas and Mihai Lupu and Paul Clough and Mark Sanderson and Mark Hall and Allan Hanbury and Elaine Toms",

booktitle = "Information Access Evaluation. Multilinguality, Multimodality, and Interaction",

address = "United Kingdom",

note = "2014 Cross Language Evaluation Forum Conference, CLEF 2014 ; Conference date: 15-09-2014 Through 18-09-2014",

}

Walker, A, Starkey, A, Pan, JZ & Siddharthan, A 2014, Making Test Corpora for Question Answering More Representative. in E Kanoulas, M Lupu, P Clough, M Sanderson, M Hall, A Hanbury & E Toms (eds), Information Access Evaluation. Multilinguality, Multimodality, and Interaction: 5th International Conference of the CLEF Initiative, CLEF 2014, Sheffield, UK, September 15-18, 2014. Proceedings. Lecture Notes in Computer Science, Springer, Cham, pp. 1-6, 2014 Cross Language Evaluation Forum Conference, CLEF 2014, Sheffield, United Kingdom, 15/09/14. https://doi.org/10.1007/978-3-319-11382-1_1

Making Test Corpora for Question Answering More Representative. / Walker, Andrew; Starkey, Andrew; Pan, Jeff Z. et al.
Information Access Evaluation. Multilinguality, Multimodality, and Interaction: 5th International Conference of the CLEF Initiative, CLEF 2014, Sheffield, UK, September 15-18, 2014. Proceedings. ed. / Evangelos Kanoulas; Mihai Lupu; Paul Clough; Mark Sanderson; Mark Hall; Allan Hanbury; Elaine Toms. Cham: Springer, 2014. p. 1-6 (Lecture Notes in Computer Science).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

TY - GEN

T1 - Making Test Corpora for Question Answering More Representative

AU - Walker, Andrew

AU - Starkey, Andrew

AU - Pan, Jeff Z.

AU - Siddharthan, Advaith

PY - 2014/9/18

Y1 - 2014/9/18

N2 - Despite two high profile series of challenges devoted to question answering technologies there remains no formal study into the representativeness that question corpora bear to real end-user inputs. We examine the corpora used presently and historically in the TREC and QALD challenges in juxtaposition with two more from natural sources and identify a degree of disjointedness between the two. We analyse these differences in depth before discussing a candidate approach to question corpora generation and provide a juxtaposition on its own representativeness. We conclude that these artificial corpora have good overall coverage of grammatical structures but the distribution is skewed, meaning performance measures may be inaccurate.

AB - Despite two high profile series of challenges devoted to question answering technologies there remains no formal study into the representativeness that question corpora bear to real end-user inputs. We examine the corpora used presently and historically in the TREC and QALD challenges in juxtaposition with two more from natural sources and identify a degree of disjointedness between the two. We analyse these differences in depth before discussing a candidate approach to question corpora generation and provide a juxtaposition on its own representativeness. We conclude that these artificial corpora have good overall coverage of grammatical structures but the distribution is skewed, meaning performance measures may be inaccurate.

U2 - 10.1007/978-3-319-11382-1_1

DO - 10.1007/978-3-319-11382-1_1

M3 - Conference contribution

SN - 978-3-319-11381-4

T3 - Lecture Notes in Computer Science

SP - 1

EP - 6

BT - Information Access Evaluation. Multilinguality, Multimodality, and Interaction

A2 - Kanoulas, Evangelos

A2 - Lupu, Mihai

A2 - Clough, Paul

A2 - Sanderson, Mark

A2 - Hall, Mark

A2 - Hanbury, Allan

A2 - Toms, Elaine

PB - Springer

CY - Cham

T2 - 2014 Cross Language Evaluation Forum Conference, CLEF 2014

Y2 - 15 September 2014 through 18 September 2014

ER -

Walker A, Starkey A, Pan JZ, Siddharthan A. Making Test Corpora for Question Answering More Representative. In Kanoulas E, Lupu M, Clough P, Sanderson M, Hall M, Hanbury A, Toms E, editors, Information Access Evaluation. Multilinguality, Multimodality, and Interaction: 5th International Conference of the CLEF Initiative, CLEF 2014, Sheffield, UK, September 15-18, 2014. Proceedings. Cham: Springer. 2014. p. 1-6. (Lecture Notes in Computer Science). doi: 10.1007/978-3-319-11382-1_1

Making Test Corpora for Question Answering More Representative

Abstract

Publication series

Conference

Access to Document

Fingerprint

Cite this