Extrinsic summarization evaluation: A decision audit task

Gabriel Murray, Thomas Kleinbauer, Peter Poller, Tilman Becker, Steve Renals, Jonathan Kilgour

Research output: Contribution to journalArticlepeer-review

Abstract / Description of output

In this work we describe a large-scale extrinsic evaluation of automatic speech summarization technologies for meeting speech. The particular task is a decision audit, wherein a user must satisfy a complex information need, navigating several meetings in order to gain an understanding of how and why a given decision was made. We compare the usefulness of extractive and abstractive technologies in satisfying this information need, and assess the impact of automatic speech recognition (ASR) errors on user performance. We employ several evaluation methods for participant performance, including post-questionnaire data, human subjective and objective judgments, and a detailed analysis of participant browsing behavior. We find that while ASR errors affect user satisfaction on an information retrieval task, users can adapt their browsing behavior to complete the task satisfactorily. Results also indicate that users consider extractive summaries to be intuitive and useful tools for browsing multimodal meeting data. We discuss areas in which automatic summarization techniques can be improved in comparison with gold-standard meeting abstracts.
Original languageEnglish
Article number2
Pages (from-to)1-29
Number of pages29
JournalACM Transactions on Speech and Language Processing
Issue number2
Publication statusPublished - 1 Oct 2009


Dive into the research topics of 'Extrinsic summarization evaluation: A decision audit task'. Together they form a unique fingerprint.

Cite this