Multimodal meaning making: The annotation of nonverbal elements in multimodal corpus transcription

  • The article discusses how to integrate annotation for nonverbal elements (NVE) from multimodal raw data as part of a standardized corpus transcription. We argue that it is essential to include multimodal elements when investigating conversational data, and that in order to integrate these elements, a structured approach to complex multimodal data is needed. We discuss how to formulate a structured corpus-suitable standard syntax and taxonomy for nonverbal features such as gesture, facial expressions, and physical stance, and how to integrate it in a corpus. Using corpus examples, the article describes the development of a robust annotation system for spoken language in the corpus of Video-mediated English as a Lingua Franca Conversations (ViMELF 2018) and illustrates how the system can be used for the study of spoken discourse. The system takes into account previous research on multimodality, transcribes salient nonverbal features in a concise manner, and uses a standard syntax. While such an approach introduces a degree of subjectivity through the criteria of salience and conciseness, the system also offers considerable advantages: it is versatile and adaptable, flexible enough to work with a wide range of multimodal data, and it allows both quantitative and qualitative research on the pragmatics of interaction.

Download full text files

Export metadata

Author:Marie-Louise BrunnerORCiD, Stefan DiemerORCiD
Parent Title (English):Research in Corpus Linguistics
Publisher:Asociación Española de Lingüística de Corpus, Spain
Document Type:Article (specialist journals)
Date of OPUS upload:2023/03/01
Year of first Publication:2021
Publishing University:Hochschule Trier
Release Date:2023/03/01
Tag:corpus annotation; corpus transcription; gestures; multimodality; nonverbal elements; spoken discourse; video-mediated communication
GND Keyword:Korpus <Linguistik>; Transkription; Annotation; Multimodalität; Mündliche Kommunikation; Nichtverbale Kommunikation; Visuelle Kommunikation; Geste
Page Number:26
First Page:63
Last Page:88
Departments:FB Umweltwirtschaft/-recht (UCB)
Licence (German):License LogoCreative Commons - CC BY - Namensnennung 4.0 International