Multimodal Resources

Collecting audio-visual communication data

An important part of my research deals with the development of multimodal corpora in several languages. Such corpora allow us to study and model human language as used in different communicative situations. Below is a list of a few relevant projects. Get in touch if you are interested in these resources.

• MUMIN:
• Annotation guidelines for multimodal corpora
• NOMCO:
• Multimodal resources in the Nordic countries
• MAMCO:
• Multimodal face-to-face dialogues in Maltese
• Online meetings in English:
• The GEHM Zoom meeting corpus

Gesture and Speech

Studying the form and function of gesture in communication

Face-to-face communication is inherently multimodal. In other words, language is embodied, and speakers and listeners make use of gesture in conjunction with speech. This is a reatively young research area, which means that many questions about the interaction of the speech and gestural modalities are still open. Below are a few relevant links.

• GEHM: The international network on Gesture and Head Movements in Language
• Multimodal Prosody: What is the contribution of gesture to the way speakers express prominence in speech? Some recent results on this issue wil be posted later.

Language Processing

Natural Language Processing (NLP) is the branch of AI that works with the way language is dealt by computers.

NLP is used in many different applications, from word prediction in mobile phones to machine translation. Over the years I have worked with many sub-fields, including processing of historical text and the automatic detection of head movements from multimodal signals (visual and acoustic features).