The handbook of multimodal-multisensor interfaces. Volume 2, Signal processing, architectures, and detection of emotion and cognition / Sharon Oviatt, Björn Schuller, Philip R. Cohen, Daniel Sonntag, Gerasimos Potamianos, Antonio Kruger.

Format:

Book

Author/Creator:

Oviatt, Sharon, author.

Schuller, Bjorn, author.

Cohen, Philip R., author.

Sonntag, Daniel, author.

Potamianos, Gerasimos, author.

Krüger, Antonio, author.

Series:

ACM books ; 2374-6777 #21.

ACM books, 2374-6777 ; #21

Language:

English

Subjects (All):

Multimodal user interfaces (Computer systems).

Human-computer interaction.

Signal processing.

Genre:

Electronic books.

Physical Description:

1 online resource (xxiii, 515 pages) : illustrations.

Edition:

First edition.

Other Title:

Signal processing, architectures, and detection of emotion and cognition

Place of Publication:

[New York] : Association for Computing Machinery ; [San Rafael, California] : Morgan & Claypool, 2019.

System Details:

Mode of access: World Wide Web.

System requirements: Adobe Acrobat Reader.

Summary:

The content of this handbook is most appropriate for graduate students and of primary interest to students studying computer science and information technology, human-computer interfaces, mobile and ubiquitous interfaces, affective and behavioral computing, machine learning, and related multidisciplinary majors. When teaching graduate classes with this book, whether in a quarter- or semester-long course, we recommend initially requiring that students spend two weeks reading the introductory textbook, The Paradigm Shift to Multimodality in Contemporary Interfaces (Morgan & Claypool Publishers, Human-Centered Interfaces Synthesis Series, 2015). With this orientation, a graduate class providing an overview of multimodal-multisensor interfaces then could select chapters from the current handbook, distributed across topics in the different sections.

Contents:

Introduction: Trends in intelligent multimodal-multisensorial interfaces: cognition, emotion, social signals, deep learning, and more

Part I. Multimodal signal processing and architectures

1. Challenges and applications in multimodal machine learning / Tadas Baltrusaitis, Chaitanya Ahuja, Louis-Philippe Morency

1.1 Introduction

1.2 Multimodal Applications

1.3 Multimodal Representations

1.4 Co-learning

1.5 Conclusion

Focus questions

References

2. Classifying multimodal data / Ethem Alpaydin

2.1 Introduction

2.2 Classifying multimodal data

2.3 Early, late, and intermediate integration

2.4 Multiple kernel learning

2.5 Multimodal deep learning

2.6 Conclusions and future work

Acknowledgments

3. Learning for multimodal and affect-sensitive interfaces / Yannis Panagakis, Ognjen Rudovic, Maja Pantic

3.1 Introduction

3.2 Correlation analysis methods

3.3 Temporal modeling of facial expressions

3.4 Context dependency

3.5 Model adaptation

3.6 Conclusion

4. Deep learning for multisensorial and multimodal interaction / Gil Keren, Amr El-desoky Mousa, Olivier Pietquin, Stefanos Zafeiriou, Björn Schuller

4.1 Introduction

4.2 Fusion models

4.3 Encoder-decoder models

4.4 Multimodal embedding models

4.5 Perspectives

Part II. Multimodal processing of social and emotional states

5. Multimodal user state and trait recognition: an overview / Björn Schuller

5.1 Introduction

5.2 Modeling

5.3 An overview on attempted multimodal stait and trait recognition

5.4 Architectures

5.5 A modern architecture perspective

5.6 Modalities

5.7 Walk-through of an example state

5.8 Emerging trends and future directions

6. Multimodal-multisensor affect detection / Sidney K. D'Mello, Nigel Bosch, Huili Chen

6.1 Introduction

6.2 Background from affective sciences

6.3 Modality fusion for multimodal-multisensor affect detection

6.4 Walk-throughs of sample multisensor-multimodal affect detection systems

6.5 General trends and state of the art in multisensor-multimodal affect detection

6.6 Discussion

7. Multimodal analysis of social signals / Alessandro Vinciarelli, Anna Esposito

7.1 Introduction

7.2 Multimodal communication in life and human sciences

7.3 Multimodal analysis of social signals

7.4 Next steps

7.5 Conclusions

8. Real-time sensing of affect and social signals in a multimodal framework: a practical approach / Johannes Wagner, Elisabeth Andre

8.1 Introduction

8.2 Database collection

8.3 Multimodal fusion

8.4 Online recognition

8.5 Requirements for a multimodal framework

8.6 The social signal interpretation framework

8.7 Conclusion

9. How do users perceive multimodal expressions of affects? / Jean-Claude Martin, Celine Clavel, Matthieu Courgeon, Mehdi Ammi, Michel-Ange Amorim, Yacine Tsalamlal, Yoren Gaffary

9.1 Introduction

9.2 Emotions and their expressions

9.3 How humans perceive combinations of expressions of affects in several modalities

9.4 Impact of context on the perception of expressions of affects

9.5 Conclusion

Focus Questions

Part III. Multimodal processing of cognitive states

10. Multimodal behavioral and physiological signals as indicators of cognitive load / Jianlong Zhou, Kun Yu, Fang Chen, Yang Wang, Syed Z. Arshad

10.1 Introduction

10.2 State-of-the-art

10.3 Behavioral measures for cognitive load

10.4 Physiological measures for cognitive load

10.5 Multimodal signals and data fusion

10.6 Conclusion

Funding

11. Multimodal learning analytics: assessing learners' mental state during the process of learning / Sharon Oviatt, Joseph Grafsgaard, Lei Chen, Xavier Ochoa

11.1 Introduction

11.2 What is multimodal learning analytics?

11.3 What data resources are available on multimodal learning analytics?

11.4 What are the main themes from research findings on multimodal learning analytics?

11.5 What is the theoretical basis of multimodal learning analytics?

11.6 What are the main challenges and limitations of multimodal learning analytics?

11.7 Conclusions and future directions

12. Multimodal assessment of depression from behavioral signals / Jeffrey F. Cohn, Nicholas Cummins, Julien Epps, Roland Goecke, Jyoti Joshi, Stefan Scherer

12.1 Introduction

12.2 Depression

12.3 Multimodal behavioral signal processing systems

12.4 Facial analysis

12.5 Speech analysis

12.6 Body movement and other behavior analysis

12.7 Analysis using other sensor signals

12.8 Multimodal fusion

12.9 Implementation-related considerations and elicitation approaches

12.10 Conclusion and current challenges

13. Multimodal deception detection / Mihai Burzo, Mohamed Abouelenien, Veronica Perez-Rosas, Rada Mihalcea

13.1 Introduction and motivation

13.2 Deception detection with individual modalities

13.3 Deception detection with multiple modalities

13.4 The way forward

Part IV. Multidisciplinary challenge topic

14. Perspectives on predictive power of multimodal deep learning: surprises and future directions / Samy Bengio, Li Deng, Louis-Philippe Morency, Björn Schuller

14.1 Deep learning as catalyst for scientific discovery

14.2 Deep learning in relation to conventional machine learning

14.3 Expected surprises of deep learning

14.4 The future of deep learning

14.5 Responsibility in deep learning

14.6 Conclusion

Index

Biographies

Volume 2 Glossary.

Notes:

Includes bibliographical references and index.

Title from PDF title page (viewed on November 10, 2018).

Other Format:

Print version:

ISBN:

9781970001693

OCLC:

1062373656

Access Restriction:

Restricted for use by site license.