Systematic Review of Multimodal Emotion Recognition in the Wild: Integrating Facial Expressions, Speech, and Physiological Signals for Enhanced Context-Aware Applications

Tinjauan Sistematis Pengenalan Emosi Multimodal di Lingkungan Alami: Mengintegrasikan Ekspresi Wajah, Ucapan, dan Sinyal Fisiologis untuk Aplikasi yang Lebih Sadar Konteks

Abstract

Facial Emotion Recognition (FER) has become a critical component of affective computing, human–computer interaction, intelligent healthcare, adaptive education, and assistive technologies. This systematic literature review synthesizes recent developments in multimodal emotion recognition in the wild by examining how facial expressions, speech, physiological signals, deep learning architectures, and deployment technologies shape robust context-aware FER systems. Following the PRISMA protocol, literature was identified from Scopus using FER-related deep learning keywords, resulting in 202 initial records and 61 eligible studies for thematic synthesis, trend analysis, methodological classification, and qualitative interpretation. The findings show that FER research has shifted strongly from handcrafted features toward CNN-based deep learning, transfer learning, hybrid architectures, attention mechanisms, and bio-inspired optimization. Human–computer interaction emerged as the dominant research context, while healthcare, autism spectrum disorder screening, education, assistive technology, mining safety, and smart services represent increasingly important application domains. Transfer learning dominated robustness strategies, while multimodal fusion using facial images, speech, EEG, wearable sensors, and audio-visual signals gained stronger academic attention because it improves contextual understanding and reduces the limitations of unimodal FER. The synthesis also reveals persistent challenges, including poor generalization in uncontrolled environments, dataset imbalance, cultural variation, micro-expression recognition, computational complexity, real-time deployment, and limited explainability. The review contributes a multidimensional conceptual perspective that integrates multimodal sensing, deep learning optimization, edge/IoT deployment, and ethical-aware application design. Future research should prioritize lightweight multimodal FER, cross-cultural datasets, explainable AI, privacy-preserving learning, real-world clinical validation, and adaptive systems capable of operating reliably under noisy, dynamic, and socially sensitive conditions.