Restructuring Multimodal Interaction Data for Browsing and Search

This chapter presents work on techniques for indexing and structuring recordings of interactive activities, such as collaborative editing, computer-mediated and computer-assisted meetings, and presentations. It argues that a model that can minimally account for such recursive inter-media relations based on different levels of text and speech segmentation is needed. The chapter proposes a model based on temporal links that induce a graph structure on multimedia records of multiparty communication and collaboration. It examines several examples of meeting recording and browsing activities and the technologies that support these activities in relation. Once restricted to denoting face-to-face interaction, the word “meeting” has, with the spread of information and communication technologies, taken on a much broader meaning. The requirements and issues discussed suggest that, in addition to technologies capable of capturing and analyzing interaction data such as speech, gestures, facial expressions, and text editing, one needs to be able to integrate these capabilities under a unified information structure.