IEEE JOURNAL OF BIOMEDICAL AND HEALTH INFORMATICS, cilt.24, sa.6, ss.1695-1702, 2020 (SCI-Expanded)
https://ieeexplore.ieee.org/document/8933102
In
the present article, a novel emotional complexity marker is proposed for
classification of discrete emotions induced by affective video film clips.
Principal Component Analysis (PCA) is applied to full-band specific phase space
trajectory matrix (PSTM) extracted from short emotional EEG segment of 6 s,
then the first principal component is used to measure the level of local
neuronal complexity. As well, Phase Locking Value (PLV) between right and left
hemispheres is estimated for in order to observe the superiority of local
neuronal complexity estimation to regional neuro-cortical connectivity
measurements in clustering nine discrete emotions (fear, anger, happiness,
sadness, amusement, surprise, excitement, calmness, disgust) by using Long-Short-Term-Memory
Networks as deep learning applications. In tests, two groups (healthy females
and males aged between 22 and 33 years old) are classified with the accuracy
levels of 68.52% and 79.36% through the proposed emotional complexity markers
and and connectivity levels in terms of PLV in amusement. The groups are found
to be statistically different (p << 0.5) in amusement with respect to
both metrics, even if gender difference does not lead to different
neuro-cortical functions in any of the other discrete emotional states. The
high deep learning classification accuracy of 98.00% is commonly obtained for
discrimination of positive emotions from negative emotions through the proposed
new complexity markers. Besides, considerable useful classification performance
is obtained in discriminating mixed emotions from each other through full-band
connectivity features. The results reveal that emotion formation is mostly
influenced by individual experiences rather than gender. In detail, local
neuronal complexity is mostly sensitive to the affective valance rating, while
regional neuro-cortical connectivity levels are mostly sensitive to the
affective arousal ratings.
DISCUSSION AND CONCLUSION:
In the present study, a
new emotional recognition methodology has been presented. The close
relationship between affective stimulus parameters and neuro-cortical
activities in young females and males in nine discrete emotional states. For
this purpose, PCA is applied to PSTM of short EEG segments. The primary concept
was to observe the gender effect on emotional neuro-complexity levels, and
then, the second concept was to observe the usefulness of the proposed
complexity markers for emotion recognition. In all applications, EEG
measurements were segmented into short epochs of 6 s and 12 s in order to
investigate the influence of analysis interval for emotion recognition.
Conventional and deep learning networks were trained in not only instant
classification but also subject classification manners for both segmentation
steps. Regarding those main concepts, the results were compatible with to each
other: Females were differed from males in E4 (amusement) through CL-4 for both
shorter and larger segments. Thus, CL-4 provided the relatively lower
performance for discrimination of E4 from baseline in comparison to recognition
of the other emotional states for both segment statements. For recognition of
E4 (amusement) and E8 (calmness), classification performances were increased
when the subjects were classified through CL-4 instead of instant classification.
When the analysis interval was largened from 6 s to 12 s, emotion recognition
performances were decreased through all classifiers. Considering emotional EEG
complexity levels mediated by audio-visual affective video films, gender
differences became slightly un-avoidable when subjects were classified instead
of instant classification. Similarly, each emotional states were differed from
baseline with very high CA levels when subjects were classified instead of
instant classification. Among conventional and deep learning networks, the most
useful classifier was CL-4, i.e., CNN. Although, SVM classifiers provided
slightly better performance when the number of features was low, deep learning
algorithms, CNNs and LSTMNs were found to be better when the large number of
features were examined. In conclusion, mixed emotions highly modulate the
functional connectivity of the amygdala with the other regions of the brain. In
particular, regional PCPSTM estimations characterize the dynamic signature of
emotion formation depending on individual experiences driven by ongoing
perception and cognition processes. From EEG signal processing point of view,
primary extraction of PSTM reduces the background EEG, i.e., increases the
signal-to-noise ratio. Then, application of PCA on PSTM highlights the main
harmonics in association with audio-visual evoked potentials embedded in short
epochs. Due to ongoing combination of excitatory and inhibitory post-synaptic
potentials as well as Action Potentials (APs), EEG series are time-varying
psychophysiological signals. In particular, timevarying audio-visual affective
stimuli continuously cause both generation and propagation of nerve APs at
auditory, visual and cognitive cortices simultaneously. Therefore, EEG
segmentation and analysis of short non-overlapped epochs are crucial
pre-processes for emotion recognition. As well, the most useful length of short
epoch is found to be 6 s due to AP phases lasting about 2 s. In recent
neuroscience studies, it is highlighted that emotional responsiveness of
individuals can be clinical support in not only rare disease so called
pre-symptomatic Huntington’s disease [101] but also several important and
widespread psychiatric conditions such as unipolar depression [102],
Parkinson’s disease with lack of dementia [103] and autism spectrum [104] as
well as amygdalar lesions [105]. Full-band PSTMCs can be proposed as
single-channel emotion recognition system.