Evaluation of Extractive Voicemail Summarization

Konstantinos Koumpis, Steve Renals

Research output: Chapter in Book/Report/Conference proceedingConference contribution


This paper is about the evaluation of a system that generates short text summaries of voicemail messages, suitable for transmission as text messages. Our approach to summarization is based on a speech-recognized transcript of the voicemail message, from which a set of summary words is extracted. The system uses a classifier to identify the summary words, with each word being identified by a vector of lexical and prosodic features. The features are selected using Parcel, an ROC-based algorithm. Our evaluations of the system, using a slot error rate metric, have compared manual and automatic summarization, and manual and automatic recognition (using two different recognizers). We also report on two subjective evaluations using mean opinion score of summaries, and a set of comprehension tests. The main results from these experiments were that the perceived difference in quality of summarization was affected more by errors resulting from automatic transcription, than by the automatic summarization process.
Original languageEnglish
Title of host publication2003 ISCA Workshop on Multilingual Spoken Document Retrieval, Proceedings of the
Subtitle of host publicationMSDR 2003
Publication statusPublished - 2003
Event2003 ISCA Workshop on Multilingual Spoken Document Retrieval (MSDR2003) - Hong Kong, Hong Kong
Duration: 4 Apr 20035 Apr 2003


Workshop2003 ISCA Workshop on Multilingual Spoken Document Retrieval (MSDR2003)
Country/TerritoryHong Kong
CityHong Kong


Dive into the research topics of 'Evaluation of Extractive Voicemail Summarization'. Together they form a unique fingerprint.

Cite this