Multimodal Language:
theoretical perspectives and research methods

Asli Özyürek

Multimodal Language Department, Max Planck Institute for Psycholinguistics; Donders Institute for Brain Cognition and Behavior; Radboud University

A Multimodal Language Framework (MLF)  best situates language in cognition and communication: Insights from Italian and other spoken and signed languages

Since the publication of seminal works of Kendon and McNeill it is now an accepted view that language is inherently multimodal and that many properties of visual expressions in spoken and sign languages constitute fundamental design features of language. These include modality specific features such as visual iconicity, indexicality and simultaneity expressed by the manual, bodily and facial expressions.  The next challenge in linguistics is then how to define a framework that explains how these can be  integrated along with  arbitrary, categorical and sequential properties of linguistic expressions  typical of spoken/textual expressions and in a way that grounds language in human cognition and communication. To do so I will sketch a Multimodal Language Framework (MLF) (e.g., Holler and Levinson 2019; Hagoort and Özyürek, 2024) that attempts to offer a way for how characterize language in such a  multidimensional way.  Based on  empirical data from spoken and sign languages, including Italian and Italian Sign language, I will outline how producing and perceiving linguistic expressions with this added complexity of multimodal expressions allows us to revisit  typologies of language structures, language-(neuro)cognition interface, production and comprehension processes, models of communication (based on efficiency, relevance, conversation) and language acquisition . The take home message from more than a decade of such research is that a multimodal but not a unimodal view of language structure and use best characterizes human language as an adaptive system to human cognition, learning and communication principles. I will end the talk with a call for more multimodal language research in Italian which is rooted typologically in rich multimodal use of language. ​

​

Maria Graziano

Lund University Humanities Lab

​

Aspects of multimodal language development 

Given the substantial evidence supporting the close link between speech and gesture in human communication, research on language development has increasingly embraced a multimodal approach. While earlier studies primarily concentrated on the initial stages of language acquisition, more recently research has shifted to examining later stages, aiming to understand how children learn to integrate linguistic structures with bodily resources in extended complex discourse, such as storytelling.

I will illustrate how children’s ability to integrate the expressive potentials of spoken and gestural modalities in narrative discourse changes over time and increases with the development of linguistic and pragmatic skills. I will argue that, in later stages of language acquisition, the development of speech and gesture proceed in parallel, and I will maintain that this supports the view that two modalities form an integrated system. Finally, I will also highlight the need for further research on children’s multimodal behaviour, particularly the importance of extending the studied age range beyond childhood, as well as the sort of discourse. Additionally, I will suggest that combining a qualitative and a quantitative perspective could offer valuable insights into how the two modalities develop and interact in children’s spoken discourse.