A demonstration of multimodal debrief generation for AUVs, post-mission and in-mission

Helen Hastie; Xingkun Liu; Pedro Patron

doi:10.1145/2993148.2998525

A demonstration of multimodal debrief generation for AUVs, post-mission and in-mission

Helen Hastie, Xingkun Liu, Pedro Patron

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Abstract / Description of output

A prototype will be demonstrated that takes activity and sensor data from Autonomous Underwater Vehicles (AUVs) and automatically generates multimodal output in the form of mission reports containing natural language and visual elements. Specifically, the system takes time-series sensor data, mission logs, together with mission plans as its input, and generates descriptions of the missions in natural language, which would be verbalised by a Text-to-Speech Synthesis (TTS) engine in a multimodal system. In addition, we will demonstrate an in-mission system that provides a stream of real-time updates in natural language, thus improving situation awareness of the operator and increasing trust in the system during missions.

Original language	English
Title of host publication	Proceedings of the 2016 International Conference on Multimodal Interaction
Publisher	ACM Association for Computing Machinery
Pages	404-405
Number of pages	2
ISBN (Electronic)	9781450345569
DOIs	https://doi.org/10.1145/2993148.2998525
Publication status	Published - 31 Oct 2016
Event	18th ACM International Conference on Multimodal Interaction - Tokyo, Japan Duration: 12 Nov 2016 → 16 Nov 2016 Conference number: 18 https://icmi.acm.org/2016/index.php?id=home

Conference

Conference	18th ACM International Conference on Multimodal Interaction
Abbreviated title	ICMI 2016
Country/Territory	Japan
City	Tokyo
Period	12/11/16 → 16/11/16
Internet address	https://icmi.acm.org/2016/index.php?id=home

Keywords / Materials (for Non-textual outputs)

Multimodal output
natural language generation
autonomous systems

Access to Document

10.1145/2993148.2998525Licence: All Rights Reserved

https://dl.acm.org/doi/10.1145/2993148.2998525Licence: All Rights Reserved

Cite this

@inproceedings{8fdb7aac7c714000b5e2f9ee09db2eaf,

title = "A demonstration of multimodal debrief generation for AUVs, post-mission and in-mission",

abstract = "A prototype will be demonstrated that takes activity and sensor data from Autonomous Underwater Vehicles (AUVs) and automatically generates multimodal output in the form of mission reports containing natural language and visual elements. Specifically, the system takes time-series sensor data, mission logs, together with mission plans as its input, and generates descriptions of the missions in natural language, which would be verbalised by a Text-to-Speech Synthesis (TTS) engine in a multimodal system. In addition, we will demonstrate an in-mission system that provides a stream of real-time updates in natural language, thus improving situation awareness of the operator and increasing trust in the system during missions.",

keywords = "Multimodal output, natural language generation, autonomous systems",

author = "Helen Hastie and Xingkun Liu and Pedro Patron",

year = "2016",

month = oct,

day = "31",

doi = "10.1145/2993148.2998525",

language = "English",

pages = "404--405",

booktitle = "Proceedings of the 2016 International Conference on Multimodal Interaction",

publisher = "ACM Association for Computing Machinery",

note = "18th ACM International Conference on Multimodal Interaction, ICMI 2016 ; Conference date: 12-11-2016 Through 16-11-2016",

url = "https://icmi.acm.org/2016/index.php?id=home",

}

Hastie, H, Liu, X & Patron, P 2016, A demonstration of multimodal debrief generation for AUVs, post-mission and in-mission. in Proceedings of the 2016 International Conference on Multimodal Interaction. ACM Association for Computing Machinery, pp. 404-405, 18th ACM International Conference on Multimodal Interaction, Tokyo, Japan, 12/11/16. https://doi.org/10.1145/2993148.2998525

TY - GEN

T1 - A demonstration of multimodal debrief generation for AUVs, post-mission and in-mission

AU - Hastie, Helen

AU - Liu, Xingkun

AU - Patron, Pedro

N1 - Conference code: 18

PY - 2016/10/31

Y1 - 2016/10/31

N2 - A prototype will be demonstrated that takes activity and sensor data from Autonomous Underwater Vehicles (AUVs) and automatically generates multimodal output in the form of mission reports containing natural language and visual elements. Specifically, the system takes time-series sensor data, mission logs, together with mission plans as its input, and generates descriptions of the missions in natural language, which would be verbalised by a Text-to-Speech Synthesis (TTS) engine in a multimodal system. In addition, we will demonstrate an in-mission system that provides a stream of real-time updates in natural language, thus improving situation awareness of the operator and increasing trust in the system during missions.

AB - A prototype will be demonstrated that takes activity and sensor data from Autonomous Underwater Vehicles (AUVs) and automatically generates multimodal output in the form of mission reports containing natural language and visual elements. Specifically, the system takes time-series sensor data, mission logs, together with mission plans as its input, and generates descriptions of the missions in natural language, which would be verbalised by a Text-to-Speech Synthesis (TTS) engine in a multimodal system. In addition, we will demonstrate an in-mission system that provides a stream of real-time updates in natural language, thus improving situation awareness of the operator and increasing trust in the system during missions.

KW - Multimodal output

KW - natural language generation

KW - autonomous systems

U2 - 10.1145/2993148.2998525

DO - 10.1145/2993148.2998525

M3 - Conference contribution

SP - 404

EP - 405

BT - Proceedings of the 2016 International Conference on Multimodal Interaction

PB - ACM Association for Computing Machinery

T2 - 18th ACM International Conference on Multimodal Interaction

Y2 - 12 November 2016 through 16 November 2016

ER -

A demonstration of multimodal debrief generation for AUVs, post-mission and in-mission

Abstract / Description of output

Conference

Keywords / Materials (for Non-textual outputs)

Access to Document

Fingerprint

Cite this