Privacy-preserving Facial Emotion Classification with Visual Micro-Doppler Signatures for Hearing Aid Applications

Facial expressions are a crucial aspect of nonverbal communication and often reflect underlying emotional states. Researchers often use facial emotion detection as a tool to gain insights into cognitive processes, emotional states and cognitive load. The conventional camera-based methods to sense human emotions are privacy intrusive, lack adaptability, and are sensitive to variability. These technologies have limited generalization and may not adapt well to variations in ambient lighting, facial landmark localization, facial occlusions and emotion intensity. Radio Frequency (RF) sensing offers promising avenues for improvement with contactless, non-invasive, privacy-preserving and reliable radar-based measurements. The proposed framework utilizes deep-learning techniques to classify facial micro-doppler signatures, generated from an ultra-wideband (UWB) radar. The method relies on continuous multi-level feature learning from radar time-frequency Doppler measurements. The spatiotemporal facial features are extracted from the radar data to train deep learning models. The proposed system achieves a high multiclass classification accuracy of 77% on the continuous streamed data covering basic emotions of anger, disgust, fear, happy, neutral and sadness. The system can transform next-generation multi-modal hearing aids with emotion-aware listening effort and cognitive load detection. This can be particularly useful in translating the emotion-assisted cognitive effort for real-time speech enhancement and personalized auditory experience.