New Metrics for Meaningful Evaluation of Informally Structured Speech Retrieval

Maria Eskevich, Walid Magdy, Gareth J. F. Jones

Research output: Chapter in Book/Report/Conference proceedingChapter

Abstract

Search effectiveness for tasks where the retrieval units are clearly defined documents is generally evaluated using standard measures such as mean average precision (MAP). However, many practical speech search tasks focus on content within large spoken files lacking defined structure. These data must be segmented into smaller units for search which may only partially overlap with relevant material. We introduce two new metrics for the evaluation of search effectiveness for informally structured speech data: mean average segment precision (MASP) which measures retrieval performance in terms of both content segmentation and ranking with respect to relevance; and mean average segment distance-weighted precision (MASDWP) which takes into account the distance between the start of the relevant segment and the retrieved segment. We demonstrate the effectiveness of these new metrics on a retrieval test collection based on the AMI meeting corpus.
Original languageEnglish
Title of host publicationAdvances in Information Retrieval
Subtitle of host publication34th European Conference on IR Research, ECIR 2012, Barcelona, Spain, April 1-5, 2012. Proceedings
EditorsRicardo Baeza-Yates, Arjen P. de Vries, Hugo Zaragoza, B. Barla Cambazoglu, Vanessa Murdock, Ronny Lempel, Fabrizio Silvestri
Place of PublicationBerlin, Heidelberg
PublisherSpringer Berlin Heidelberg
Pages170-181
Number of pages12
ISBN (Electronic)978-3-642-28997-2
ISBN (Print)978-3-642-28996-5
DOIs
Publication statusPublished - 2012

Publication series

NameLecture Notes in Computer Science (LNCS)
PublisherSpringer Berlin Heidelberg
Volume7224
ISSN (Print)0302-9743

Cite this