A Framework for Evaluating Snippet Generation for Dataset Search

Xiaxia Wang; Jinchi Chen; Shuxin Li; Gong Cheng; Jeff Z. Pan; Evgeny Kharlamov; Yuzhong Qu

doi:10.1007/978-3-030-30793-6_39

A Framework for Evaluating Snippet Generation for Dataset Search

Xiaxia Wang, Jinchi Chen, Shuxin Li, Gong Cheng, Jeff Z. Pan, Evgeny Kharlamov, Yuzhong Qu

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Abstract

Reusing existing datasets is of considerable significance to researchers and developers. Dataset search engines help a user find relevant datasets for reuse. They can present a snippet for each retrieved dataset to explain its relevance to the user’s data needs. This emerging problem of snippet generation for dataset search has not received much research attention. To provide a basis for future research, we introduce a framework for quantitatively evaluating the quality of a dataset snippet. The proposed metrics assess the extent to which a snippet matches the query intent and covers the main content of the dataset. To establish a baseline, we adapt four state-of-the-art methods from related fields to our problem, and perform an empirical evaluation based on real-world datasets and queries. We also conduct a user study to verify our findings. The results demonstrate the effectiveness of our evaluation framework, and suggest directions for future research.

Original language	English
Title of host publication	The Semantic Web -- ISWC 2019
Subtitle of host publication	18th International Semantic Web Conference, Auckland, New Zealand, October 26–30, 2019, Proceedings, Part I
Editors	Chiara Ghidini, Olaf Hartig, Maria Maleshkova, Vojtech Svátek, Isabel Cruz, Aidan Hogan, Jie Song, Maxime Lefrançois, Fabien Gandon
Place of Publication	Cham
Publisher	Springer International Publishing
Pages	680-697
Number of pages	18
ISBN (Electronic)	978-3-030-30793-6
ISBN (Print)	978-3-030-30792-9
DOIs	https://doi.org/10.1007/978-3-030-30793-6_39
Publication status	Published - 17 Oct 2019
Event	16th International Symposium on Wireless Communication Systems - Oulu, Finland Duration: 27 Aug 2019 → 30 Aug 2019 http://iswcs2019.org/

Publication series

Name	Lecture Notes in Computer Science
Publisher	Springer, Cham
Volume	11778
ISSN (Print)	0302-9743
ISSN (Electronic)	1611-3349

Conference

Conference	16th International Symposium on Wireless Communication Systems
Abbreviated title	ISWCS 2019
Country	Finland
City	Oulu
Period	27/08/19 → 30/08/19
Internet address	http://iswcs2019.org/

Keywords

Snippet generation
Dataset search
Evaluation metric

Access to Document

10.1007/978-3-030-30793-6_39Licence: All Rights Reserved

https://link.springer.com/chapter/10.1007/978-3-030-30793-6_39Licence: All Rights Reserved

Fingerprint Dive into the research topics of 'A Framework for Evaluating Snippet Generation for Dataset Search'. Together they form a unique fingerprint.

Cite this

Wang, X., Chen, J., Li, S., Cheng, G., Pan, J. Z., Kharlamov, E., & Qu, Y. (2019). A Framework for Evaluating Snippet Generation for Dataset Search. In C. Ghidini, O. Hartig, M. Maleshkova, V. Svátek, I. Cruz, A. Hogan, J. Song, M. Lefrançois, & F. Gandon (Eds.), The Semantic Web -- ISWC 2019: 18th International Semantic Web Conference, Auckland, New Zealand, October 26–30, 2019, Proceedings, Part I (pp. 680-697). (Lecture Notes in Computer Science; Vol. 11778). Springer International Publishing. https://doi.org/10.1007/978-3-030-30793-6_39

Wang, Xiaxia ; Chen, Jinchi ; Li, Shuxin ; Cheng, Gong ; Pan, Jeff Z. ; Kharlamov, Evgeny ; Qu, Yuzhong. / A Framework for Evaluating Snippet Generation for Dataset Search. The Semantic Web -- ISWC 2019: 18th International Semantic Web Conference, Auckland, New Zealand, October 26–30, 2019, Proceedings, Part I. editor / Chiara Ghidini ; Olaf Hartig ; Maria Maleshkova ; Vojtech Svátek ; Isabel Cruz ; Aidan Hogan ; Jie Song ; Maxime Lefrançois ; Fabien Gandon. Cham : Springer International Publishing, 2019. pp. 680-697 (Lecture Notes in Computer Science).

@inproceedings{94d5816981fd46609e8f34caa82f925d,

title = "A Framework for Evaluating Snippet Generation for Dataset Search",

abstract = "Reusing existing datasets is of considerable significance to researchers and developers. Dataset search engines help a user find relevant datasets for reuse. They can present a snippet for each retrieved dataset to explain its relevance to the user{\textquoteright}s data needs. This emerging problem of snippet generation for dataset search has not received much research attention. To provide a basis for future research, we introduce a framework for quantitatively evaluating the quality of a dataset snippet. The proposed metrics assess the extent to which a snippet matches the query intent and covers the main content of the dataset. To establish a baseline, we adapt four state-of-the-art methods from related fields to our problem, and perform an empirical evaluation based on real-world datasets and queries. We also conduct a user study to verify our findings. The results demonstrate the effectiveness of our evaluation framework, and suggest directions for future research.",

keywords = "Snippet generation, Dataset search, Evaluation metric",

author = "Xiaxia Wang and Jinchi Chen and Shuxin Li and Gong Cheng and Pan, {Jeff Z.} and Evgeny Kharlamov and Yuzhong Qu",

year = "2019",

month = oct,

day = "17",

doi = "10.1007/978-3-030-30793-6_39",

language = "English",

isbn = "978-3-030-30792-9",

series = "Lecture Notes in Computer Science",

publisher = "Springer International Publishing",

pages = "680--697",

editor = "Chiara Ghidini and Olaf Hartig and Maria Maleshkova and Vojtech Sv{\'a}tek and Isabel Cruz and Aidan Hogan and Jie Song and Maxime Lefran{\c c}ois and Fabien Gandon",

booktitle = "The Semantic Web -- ISWC 2019",

note = "16th International Symposium on Wireless Communication Systems , ISWCS 2019 ; Conference date: 27-08-2019 Through 30-08-2019",

url = "http://iswcs2019.org/",

}

Wang, X, Chen, J, Li, S, Cheng, G, Pan, JZ, Kharlamov, E & Qu, Y 2019, A Framework for Evaluating Snippet Generation for Dataset Search. in C Ghidini, O Hartig, M Maleshkova, V Svátek, I Cruz, A Hogan, J Song, M Lefrançois & F Gandon (eds), The Semantic Web -- ISWC 2019: 18th International Semantic Web Conference, Auckland, New Zealand, October 26–30, 2019, Proceedings, Part I. Lecture Notes in Computer Science, vol. 11778, Springer International Publishing, Cham, pp. 680-697, 16th International Symposium on Wireless Communication Systems , Oulu, Finland, 27/08/19. https://doi.org/10.1007/978-3-030-30793-6_39

A Framework for Evaluating Snippet Generation for Dataset Search. / Wang, Xiaxia; Chen, Jinchi; Li, Shuxin; Cheng, Gong; Pan, Jeff Z.; Kharlamov, Evgeny; Qu, Yuzhong.

The Semantic Web -- ISWC 2019: 18th International Semantic Web Conference, Auckland, New Zealand, October 26–30, 2019, Proceedings, Part I. ed. / Chiara Ghidini; Olaf Hartig; Maria Maleshkova; Vojtech Svátek; Isabel Cruz; Aidan Hogan; Jie Song; Maxime Lefrançois; Fabien Gandon. Cham : Springer International Publishing, 2019. p. 680-697 (Lecture Notes in Computer Science; Vol. 11778).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

TY - GEN

T1 - A Framework for Evaluating Snippet Generation for Dataset Search

AU - Wang, Xiaxia

AU - Chen, Jinchi

AU - Li, Shuxin

AU - Cheng, Gong

AU - Pan, Jeff Z.

AU - Kharlamov, Evgeny

AU - Qu, Yuzhong

PY - 2019/10/17

Y1 - 2019/10/17

N2 - Reusing existing datasets is of considerable significance to researchers and developers. Dataset search engines help a user find relevant datasets for reuse. They can present a snippet for each retrieved dataset to explain its relevance to the user’s data needs. This emerging problem of snippet generation for dataset search has not received much research attention. To provide a basis for future research, we introduce a framework for quantitatively evaluating the quality of a dataset snippet. The proposed metrics assess the extent to which a snippet matches the query intent and covers the main content of the dataset. To establish a baseline, we adapt four state-of-the-art methods from related fields to our problem, and perform an empirical evaluation based on real-world datasets and queries. We also conduct a user study to verify our findings. The results demonstrate the effectiveness of our evaluation framework, and suggest directions for future research.

AB - Reusing existing datasets is of considerable significance to researchers and developers. Dataset search engines help a user find relevant datasets for reuse. They can present a snippet for each retrieved dataset to explain its relevance to the user’s data needs. This emerging problem of snippet generation for dataset search has not received much research attention. To provide a basis for future research, we introduce a framework for quantitatively evaluating the quality of a dataset snippet. The proposed metrics assess the extent to which a snippet matches the query intent and covers the main content of the dataset. To establish a baseline, we adapt four state-of-the-art methods from related fields to our problem, and perform an empirical evaluation based on real-world datasets and queries. We also conduct a user study to verify our findings. The results demonstrate the effectiveness of our evaluation framework, and suggest directions for future research.

KW - Snippet generation

KW - Dataset search

KW - Evaluation metric

U2 - 10.1007/978-3-030-30793-6_39

DO - 10.1007/978-3-030-30793-6_39

M3 - Conference contribution

SN - 978-3-030-30792-9

T3 - Lecture Notes in Computer Science

SP - 680

EP - 697

BT - The Semantic Web -- ISWC 2019

A2 - Ghidini, Chiara

A2 - Hartig, Olaf

A2 - Maleshkova, Maria

A2 - Svátek, Vojtech

A2 - Cruz, Isabel

A2 - Hogan, Aidan

A2 - Song, Jie

A2 - Lefrançois, Maxime

A2 - Gandon, Fabien

PB - Springer International Publishing

CY - Cham

T2 - 16th International Symposium on Wireless Communication Systems

Y2 - 27 August 2019 through 30 August 2019

ER -

Wang X, Chen J, Li S, Cheng G, Pan JZ, Kharlamov E et al. A Framework for Evaluating Snippet Generation for Dataset Search. In Ghidini C, Hartig O, Maleshkova M, Svátek V, Cruz I, Hogan A, Song J, Lefrançois M, Gandon F, editors, The Semantic Web -- ISWC 2019: 18th International Semantic Web Conference, Auckland, New Zealand, October 26–30, 2019, Proceedings, Part I. Cham: Springer International Publishing. 2019. p. 680-697. (Lecture Notes in Computer Science). https://doi.org/10.1007/978-3-030-30793-6_39