Deep Learning Classification of Neuro-Emotional Phase Domain Complexity Levels Induced by Affective Video Film Clips

Creative Commons License

Aydın S.

IEEE JOURNAL OF BIOMEDICAL AND HEALTH INFORMATICS, vol.24, no.6, pp.1695-1702, 2020 (Peer-Reviewed Journal) identifier identifier identifier

  • Publication Type: Article / Article
  • Volume: 24 Issue: 6
  • Publication Date: 2020
  • Doi Number: 10.1109/jbhi.2019.2959843
  • Journal Indexes: Science Citation Index Expanded
  • Page Numbers: pp.1695-1702
  • Keywords: Electroencephalography, Complexity theory, Principal component analysis, Emotion recognition, Informatics, Trajectory, Brain biophysics, emotion recognition, affective neuroscience, deep learning, PARKINSONS-DISEASE, FEATURE-EXTRACTION, OCULAR ARTIFACTS, EEG SIGNALS, FEATURES, RECOGNITION, PERFORMANCE, POTENTIALS, RECURRENT, REDUCTION



In the present article, a novel emotional complexity marker is proposed for classification of discrete emotions induced by affective video film clips. Principal Component Analysis (PCA) is applied to full-band specific phase space trajectory matrix (PSTM) extracted from short emotional EEG segment of 6 s, then the first principal component is used to measure the level of local neuronal complexity. As well, Phase Locking Value (PLV) between right and left hemispheres is estimated for in order to observe the superiority of local neuronal complexity estimation to regional neuro-cortical connectivity measurements in clustering nine discrete emotions (fear, anger, happiness, sadness, amusement, surprise, excitement, calmness, disgust) by using Long-Short-Term-Memory Networks as deep learning applications. In tests, two groups (healthy females and males aged between 22 and 33 years old) are classified with the accuracy levels of 68.52% and 79.36% through the proposed emotional complexity markers and and connectivity levels in terms of PLV in amusement. The groups are found to be statistically different (p << 0.5) in amusement with respect to both metrics, even if gender difference does not lead to different neuro-cortical functions in any of the other discrete emotional states. The high deep learning classification accuracy of 98.00% is commonly obtained for discrimination of positive emotions from negative emotions through the proposed new complexity markers. Besides, considerable useful classification performance is obtained in discriminating mixed emotions from each other through full-band connectivity features. The results reveal that emotion formation is mostly influenced by individual experiences rather than gender. In detail, local neuronal complexity is mostly sensitive to the affective valance rating, while regional neuro-cortical connectivity levels are mostly sensitive to the affective arousal ratings.



In the present study, a new emotional recognition methodology has been presented. The close relationship between affective stimulus parameters and neuro-cortical activities in young females and males in nine discrete emotional states. For this purpose, PCA is applied to PSTM of short EEG segments. The primary concept was to observe the gender effect on emotional neuro-complexity levels, and then, the second concept was to observe the usefulness of the proposed complexity markers for emotion recognition. In all applications, EEG measurements were segmented into short epochs of 6 s and 12 s in order to investigate the influence of analysis interval for emotion recognition. Conventional and deep learning networks were trained in not only instant classification but also subject classification manners for both segmentation steps. Regarding those main concepts, the results were compatible with to each other: Females were differed from males in E4 (amusement) through CL-4 for both shorter and larger segments. Thus, CL-4 provided the relatively lower performance for discrimination of E4 from baseline in comparison to recognition of the other emotional states for both segment statements. For recognition of E4 (amusement) and E8 (calmness), classification performances were increased when the subjects were classified through CL-4 instead of instant classification. When the analysis interval was largened from 6 s to 12 s, emotion recognition performances were decreased through all classifiers. Considering emotional EEG complexity levels mediated by audio-visual affective video films, gender differences became slightly un-avoidable when subjects were classified instead of instant classification. Similarly, each emotional states were differed from baseline with very high CA levels when subjects were classified instead of instant classification. Among conventional and deep learning networks, the most useful classifier was CL-4, i.e., CNN. Although, SVM classifiers provided slightly better performance when the number of features was low, deep learning algorithms, CNNs and LSTMNs were found to be better when the large number of features were examined. In conclusion, mixed emotions highly modulate the functional connectivity of the amygdala with the other regions of the brain. In particular, regional PCPSTM estimations characterize the dynamic signature of emotion formation depending on individual experiences driven by ongoing perception and cognition processes. From EEG signal processing point of view, primary extraction of PSTM reduces the background EEG, i.e., increases the signal-to-noise ratio. Then, application of PCA on PSTM highlights the main harmonics in association with audio-visual evoked potentials embedded in short epochs. Due to ongoing combination of excitatory and inhibitory post-synaptic potentials as well as Action Potentials (APs), EEG series are time-varying psychophysiological signals. In particular, timevarying audio-visual affective stimuli continuously cause both generation and propagation of nerve APs at auditory, visual and cognitive cortices simultaneously. Therefore, EEG segmentation and analysis of short non-overlapped epochs are crucial pre-processes for emotion recognition. As well, the most useful length of short epoch is found to be 6 s due to AP phases lasting about 2 s. In recent neuroscience studies, it is highlighted that emotional responsiveness of individuals can be clinical support in not only rare disease so called pre-symptomatic Huntington’s disease [101] but also several important and widespread psychiatric conditions such as unipolar depression [102], Parkinson’s disease with lack of dementia [103] and autism spectrum [104] as well as amygdalar lesions [105]. Full-band PSTMCs can be proposed as single-channel emotion recognition system.