MILLA-Multimodal Interactive Language Learning Agent

Joao P Cabral, Nick Campbell, Shree Ganesh, Emer Gilmartin, Fasih Haider, Eamonn Kenny, Mina Kheirkhah, Andrew Murphy, Neasa Ni Chiaráin, Thomas Pellegrini, Odei Rey Orozko

Research output: Chapter in Book/Report/Conference proceedingChapter (peer-reviewed)peer-review

Abstract / Description of output

The goal of this project was to create a multimodal dialogue system which provides some of the advantages of a human tutor, not normally encountered in self-study material and systems. A human tutor aids learners by:
•Providing a framework of tasks suitable to the learner’s needs
•Continuously monitoring learner progress and adapting task content and delivery style
•Providing a source of speaking practice and motivation
MILLA is a prototype language tuition system comprising tuition management, learner state monitoring, and an adaptable curriculum, all mediated through speech. The system enrols and monitors learners via a spoken dialogue interface, provides pronunciation practice and automatic error correction in two modalities, grammar exercises, and two custom speech-to-speech chat bots for spoken interaction practice. The focus on speech in the tutor’s output and in the learning modules addresses the current deficit in spoken interaction practice in Computer Aided Language Learning (CALL) applications, with different text-to-speech (TTS) voices used to provide a variety of speech models across the different modules. The system monitors learner engagement using Kinect sensors and checks pronunciation and responds to dialogue using automatic speech recognition (ASR).A learner record is used in conjunction with the curriculum to provide activities relevant to the learner’s current abilities and first language, and to monitor and record progress.
Original languageEnglish
Title of host publicationProceedings of eNTERFACE’14
Subtitle of host publication10th International Summer Workshop on Multimodal Interfaces
EditorsDaniel Erro, Inma Hernáez
Place of PublicationBilbao, Spain
Publication statusPublished - 2014


Dive into the research topics of 'MILLA-Multimodal Interactive Language Learning Agent'. Together they form a unique fingerprint.

Cite this