Multimodální interakce

Course guarantor:

Lecturer:

Tutor:

Supervisor:

Department of Computer Graphics and Interaction

Synopsis:

This course introduces students to modern ways in which people can control and interact with technology. The course focuses on new types of interaction that go beyond the usual methods of using a keyboard, mouse, or touchscreen.

Requirements:

The course will be concluded with an exam (a written multiple-choice test followed by an oral exam). Throughout the semester, students will complete several graded individual assignments and will work on a graded semester-long project in small teams.

Syllabus of lectures:

1. Interactive Systems and Multimodal Interaction: Introduction to multimodal interaction; differences from conventional interaction models.

2. Human Senses and Multimodal Interaction: Human senses, information processing, and user-to-system communication in the context of multimodal interaction.

3. Architectural Models of Multimodal Systems: Models of interactive systems enabling multimodal interaction. Relationship between modalities (CASE, CARE), interaction loop in multimodal systems

4. Rapid prototyping for multimodal interaction: Selected rapid prototyping methods applicable to the design and testing of multimodal UIs (3D printing, Arduino, sensors, actuators).

5. Methods for evaluating multimodal UIs: Selected methods for testing multimodal UIs, focusing on both with- and without-users methods.

6. Sound- and Voice-Based Interaction: Methods utilizing sound and voice for both system input and output. Application of these methods in the context of multimodal UIs.

7. Tactile Interaction I: Basic methods of interaction based on the use of touch. Design and testing of these methods.

8. Tactile Interaction II: Advanced methods utilizing touch, force feedback. Touch-based methods in the context of multimodal systems.

9. Emotion Detection and Utilization in Interaction: Methods for detecting the users emotional state, application in multimodal interaction.

10. Advanced User Input Methods: Eye tracking, measurement of physiological functions (HR, HRV, EEG). Application in multimodal interaction.

11. Multimodal Fusion and Segmentation I: Processing of complex multimodal input from multiple system inputs. Generating output utilizing a combination of modalities.

12. Multimodal Fusion and Segmentation II: Advanced methods for processing complex multimodal input from multiple system inputs. Generating output utilizing a combination of modalities. Adaptivity and adaptability of these methods.

13. AI and Multimodal Interaction: Application of artificial intelligence methods in multimodal interaction.

14. Invited Lecture/Reserve

Syllabus of tutorials:

The exercises will consist of three main thematic blocks focused on the practical implementation of multimodal user interfaces.

I. Methods for designing multimodal user interfaces. Introduction to the MIR Kit (pre-prepared interactive modules for developing and testing multimodal interfaces with an emphasis on haptic interaction), practical methods for rapid prototyping of multimodal UIs (basics of 3D printing, electronics prototypingArduino, etc.). Individual homework assignment.

II. Design, implementation, and testing of interaction methods utilizing non-traditional modalities (touch, gestures, eye-tracking, monitoring of physiological functions). Individual homework assignment.

III. Design, implementation, and testing of multimodal UIs, multimodal fusion and decomposition. Assignment of a team-based semester project.

1. Introduction to the course. Overview of interactive systems and multimodal interaction. Differences from conventional user interfaces.

2. Introduction to the MIR Kit (architecture, components, GUI). Examples of interactive methods using less common modalities.

3. Advanced techniques for using the MIR Kit (API). Examples of using combinations of multiple modalities for input and output

4. Design of multimodal user interfaces.

5. Assignment of individual or team semester projects.

6. Tactile interaction.

7. Voice input and output.

8. Emotion detection.

9. Multimodal fusion and decomposition.

10. Testing of multimodal user interfaces

11. Work on the semester project, consultations.

12. Work on the semester project, consultations.

13. Presentation of semester projects.

14. Reserve.

Study Objective:

Students will gradually learn how technologies respond to various forms of human expression, such as voice, touch, body movements, or gestures. They will also become familiar with less common forms of interaction, such as emotion recognition or the measurement of physiological responses (e.g., heart rate or skin conductance). Based on this knowledge, students will try to design and evaluate so-called multimodal systems - systems that combine multiple methods of control and feedback simultaneously.

Study materials:

The course is based on the following core literature; however, students are not expected to use it directly or need to have access to it. Therefore, this is recommended reading.

Grifoni, P. (Ed.). (2009). Multimodal human computer interaction and pervasive services. IGI Global.

Oviatt, S., Schuller, B., Cohen, P., Sonntag, D., & Potamianos, G. (2017). The handbook of multimodal-multisensor interfaces, volume 1: Foundations, user modeling, and common modality combinations. Morgan & Claypool.

Oviatt, S., Schuller, B., Cohen, P. R., Sonntag, D., Potamianos, G., & Krüger, A. (Eds.). (2018). The Handbook of Multimodal-Multisensor Interfaces: Signal Processing, Architectures, and Detection of Emotion and Cognition - Volume 2. Association for Computing Machinery and Morgan & Claypool.

Throughout the course, students will have access to materials in the form of slides for lectures and exercises, as well as additional reference materials (selected academic articles, development platform documentation, etc.).

The course will utilize learning aids in the form of the MIR Kita set of pre-prepared blocks that can be used to facilitate the implementation of multimodal interfaces.

Note:

In the fall semester of the 202627 academic year, the MIR course will be offered as a pilot program for a limited number of students. Students are encouraged to enroll in the course during the third semester of their masters program.

Further information:

No time-table has been prepared for this course

The course is a part of the following study plans:

• Open Informatics - Extended Reality and Game Development (compulsory elective course)