基于面部表情的学习者情绪自动识别研究——适切性、现状、现存问题和提升路径
|
摘要
面部表情是表达情绪的主要通道,也是用于情绪识别的一种重要信号。以计算机视觉、人工智能、情感计算等新兴技术为支撑,计算机可以通过识别学习者外显的面部表情,来判断学习者内隐的情绪状态,从而获取识别、理解学习者情绪的能力。实现基于面部表情的学习者情绪识别,首先需要在对不同情绪表征方法进行对比分析的基础上,确定适用的情绪表征方法,再对基于面部表情的学习者情绪识别的适切性进行论证。作为面部表情识别流程中的核心环节,面部表情特征提取方法分为传统的计算机视觉方法和深度学习方法两大类。梳理不同特征提取算法的特点及局限性,可以为探索适合学习者面部表情识别的特征提取算法提供借鉴,并推动学习者面部表情识别研究的发展和有效应用。当前,学习者情绪面部表情识别相关研究仍存有局限性,需要从大规模的自然的学习者情绪面部表情数据库的共建共享,并融合多种特征识别学习者情绪面部表情;从结合人工设计和自动学习两种方法,提取面部表情特征等多种路径,来提升研究深度。
The facial expression is the main channel for expressing emotion and an important signal for emotion recognition.Supported by computer vision, artificial intelligence, affective computing, and other emerging technologies, the computer can estimate the emotion of learners by the means of recognizing the facial expressions of learners. It makes computers have the ability of recogniz-ing the emotion of students. To realize the recognition of learners ' emotion based on facial expressions, firstly, it is necessary to determine the representation methods based on the comparative analysis of different emotional representation methods, then demonstrate the suitability of learner emotion recognition based on facial expression. As the core of facial expression recognition process, the ex-traction methods of facial expression features can be divided into traditional computer vision methods and deep learning methods.The pros and cons of different feature extraction algorithms can provide useful reference for finding the appropriate feature extraction algo-rithm for learners facial expression recognition, and promote the effective development and application of learners facial expression recognition research. Finally, the existing problems in the related researches were pointed out, and the proposals to promote researches on learners facial expression recognition were presented, including co-construction and sharing of the large-scale natural learner fa-cial expression database, recognizing the learner facial expression based on multimodal features, and extracting features of learner fa-cial expressions with feature descriptors and feature learning.
The facial expression is the main channel for expressing emotion and an important signal for emotion recognition.Supported by computer vision, artificial intelligence, affective computing, and other emerging technologies, the computer can estimate the emotion of learners by the means of recognizing the facial expressions of learners. It makes computers have the ability of recogniz-ing the emotion of students. To realize the recognition of learners ' emotion based on facial expressions, firstly, it is necessary to determine the representation methods based on the comparative analysis of different emotional representation methods, then demonstrate the suitability of learner emotion recognition based on facial expression. As the core of facial expression recognition process, the ex-traction methods of facial expression features can be divided into traditional computer vision methods and deep learning methods.The pros and cons of different feature extraction algorithms can provide useful reference for finding the appropriate feature extraction algo-rithm for learners facial expression recognition, and promote the effective development and application of learners facial expression recognition research. Finally, the existing problems in the related researches were pointed out, and the proposals to promote researches on learners facial expression recognition were presented, including co-construction and sharing of the large-scale natural learner fa-cial expression database, recognizing the learner facial expression based on multimodal features, and extracting features of learner fa-cial expressions with feature descriptors and feature learning.
引文
[1]傅小兰.情绪心理学[M].上海:华东师范大学出版社.2016.
[2]刘智,方常丽,刘三女牙等.物理学习空间中学习者情绪感知研究综述[J].远程教育杂志,2019(2):33-44.
[3]Ekman P,Friesen W V,Ellsworth P.Emotion in the Human Face[M].New York:Cambridge University Press,1982.
[4]Alvarado N.Arousal and Valence in the Direct Scaling of Emotional Response to Film Clips[J].Motivation&Emotion,1997,21(4):323-348.
[5]Lewis P A,Critchley H D,Rotshtein P,et al.Neural Correlates of Processing Valence and Arousal in Affective Words[J].Cerebral Cortex,2007,17(3):742-748.
[6]孟昭兰.情绪心理学[M].北京:北京大学出版社,2005.
[7]Yamauchi H,Mashiko S,Kimura M,et al.Verification of Acousical Correlates of Emotional Speech using Formant-Synthesis[J].Proc Isca Workshop on Speech&Emotion,2007,52(1):119-130.
[8]Wallbott H G.Bodily Expression of Emotion[J].European Journal of Social Psychology,1998,28(6):879-896.
[9]Ekman P,Friesen W V.Detecting Deception from the Body or Face[J].Journal of Personality&Social Psychology,1974,29(3):288-298.
[10]Keltner D,Ekman P.Facial Expression of Emotion[M].Handbook of Emotions 3rd.New York:The Guilford Press,2010:173-183.
[11]Lancet T.Communicationwithout Words[J].University of East London,1968,24(23):1084-5.
[12]Scherer K R.Vocal Communication of Emotion:A Review of Research Paradigms[J].Speech Communication,2003,40(1/2):227-256.
[13]Monkaresi H,Bosch N,Calvo R,et al.Automated Detection of Engagement using Video-Based Estimation of Facial Expressions and Heart Rate[J].IEEE Transactions on Affective Computing,2017,8(1):15-28.
[14]Grafsgaard J F,Wiggins J B,Boyer K E,et al.Automatically Recognizing Facial Expression:Predicting Engagement and Frustration[C]//The 6th International Conference on Educational Data Mining EDM 2013.ELSEVIER:43-50.
[15][19]Whitehill J,Serpell Z,Lin Y C,et al.The Faces of Engagement:Automatic Recognition of Student Engagementfrom Facial Expressions[J].IEEE Transactions on Affective Computing,2014,5(1):86-98.
[16]Wang J,Ma X,Sun J,et al.Puzzlement Detection from Facial Expression Using Active Appearance Models and Support Vector Machines[J].International Journal of Signal Processing Image Processing&Pattern Recognition,2014,7(5):349-360.
[17]江波,李万健,李芷璇,等.基于面部表情的学习困惑自动识别法[J].开放教育研究,2018(4):101-108.
[18]韩丽,李洋,周子佳,等.课堂环境中基于面部表情的教学效果分析[J].现代远程教育研究,2017(4):97-103+112.
[20]Happy S L,Dasgupta A,Patnaik P,et al.Automated Alertness and Emotion Detection for Empathic Feedback during e-Learning[C]//2013 IEEE Fifth International Conference on Technology for Education.IEEE:47-50.
[21]童莹.基于空间多尺度HOG特征的人脸表情识别方法[J].计算机工程与设计,2014(11):3918-3922+3979.
[22]Anderson K,Mcowan P W.A Real-time Automated System for the Recognition of Human Facial Expressions[J].IEEE Transactions on Systems Man&Cybernetics Part B Cybernetics A Publication of the IEEE Systems Man&Cybernetics Society,2006,36(1):96-105.
[23]Liu Y J,Zhang J K,Yan W J,et al.A Main Directional Mean Optical Flow Feature for Spontaneous Micro-Expression Recognition[J].IEEE Transactions on Affective Computing,2016,7(4):299-310.
[24]Tie Y,Guan L.A Deformable 3-D Facial Expression Model for Dynamic Human Emotional State Recognition[J].IEEE Transactions on Circuits and Systems for Video Technology,2013,23(1):142-157.
[25]Cheon Y,Kim D.Natural Facial Expression Recognition using Differential-AAM and Manifold Learning[J].Pattern Recognition,2009,42(7):1340-1350.
[26]Pantic M,Patras I.Dynamics of Facial Expression:Recognition of Facial Actions and Their Temporal Segments from Face Profile Image Sequences[J].IEEE Transactions on Systems,Man,and Cybernetics,Part B(Cybernetics),2006,36(2):433-449.
[27]Niese R,Al-Hamadi A,Farag A,et al.Facial Expression Recognition based on Geometric and Optical Flow Features in Colour Image Sequences[J].Iet Computer Vision,2012,6(2):79-89.
[28]Zhao G,Pietik¨Ainen M.Dynamic Texture Recognition using Local Binary Patterns with an Application to Facial Expressions[J].IEEETrans Pattern Anal Mach Intell,2007,29(6):915-928.
[29]Wang Y,See J,Phan C W,et al.LBP with Six Intersection Points:Reducing Redundant Information in LBP-TOP for Microexpression Recognition[C]//Computer Vision-ACCV 2014.Springer,Cham:525-537.
[30]Polikovsky S,Kameda Y,Ohta Y.Facial Micro-Expression Detection in Hi-Speed Video Based on Facial Action Coding System(FACS)[J].Ieice Transactions on Information&Systems,2013,96(1):81-92.
[31]Alexander Klser,Marszalek M,Schmid C.A Spatio-Temporal Descriptor Based on 3D-Gradients[C]//Proceedings of the British Machine Vision Conference 2008,Leeds.2008.
[32]Sun B,Li L,Zuo T,et al.Combining Multimodal Features with Hierarchical Classifier Fusion for Emotion Recognition in the Wild[C]//ACM on International Conference on Multimodal Interaction.ACM:481-486.
[33]He L,Jiang D,Yang L,et al.Multimodal Affective Dimension Prediction Using Deep Bidirectional Long Short-Term Memory Recurrent Neural Networks[C]//International Workshop on Audio/visual Emotion Challenge 2015.ACM:73-80.
[34]Scovanner P,Ali S,Shah M.A 3-dimensional Sift Descriptor and Its Application to Action Recognition[C]//Proceedings of the 15th ACM international conference on Multimedia.ACM:357-360.
[35][43]Zhang K,Huang Y,Du Y,et al.Facial Expression Recognition Based on Deep Evolutional Spatial-Temporal Networks[J].IEEETransactions on Image Processing,2017,26(9):4193-4203.
[36][45]Kim D H,Baddar W,Jang J,et al.Multi-Objective based Spatio-Temporal Feature Representation Learning Robust to Expression Intensity Variations for Facial Expression Recognition[J].IEEETransactions on Affective Computing(Early Access),2017:223-236.
[37]Lan Goodfellow Y B,Aaron Courville.Deep learning[M].Massachusetts:MIT Press,2016.
[38]Hamester D,Barros P,Wermter S.Face Expression Recognition with a 2-channel Convolutional Neural Network[C]//2015 International Joint Conference on Neural Networks(IJCNN).IEEE:1787-1794.
[39]Liu K,Zhang M,Pan Z.Facial Expression Recognition with CNNEnsemble[C]//2016 International Conference on Cyberworlds(CW).IEEE,2016:163-166.
[40]Liu Y,Chen Y.Recognition of Facial Expression based on CNN-CBP Features[C]//2017 29th Chinese Control And Decision Conference(CCDC).IEEE,2017:2139-2145.
[41]Ijjina E P,Mohan C K.Facial Expression Recognition Using Kinect Depth Sensor and Convolutional Neural Networks[C]//Proceedings of the 2014 13th International Conference on Machine Learning and Applications.IEEE,2014:392-396.
[42]Ebrahimi Kahou S,Michalski V,Konda K,et al.Recurrent Neural Networks for Emotion Recognition in Video[C]//Proceedings of the2015 ACM on International Conference on Multimodal Interaction.ACM,2015:467-474.
[44]卿粼波,熊文诗,周文俊,等.基于多流CNN-LSTM网络的群体情绪识别[J].计算机应用研究,2018(12):3828-3831.
[46][52]Lin K C,Huang T C,Hung J C,et al.Facial Emotion Recognition towards Affective Computing‐based Learning[J].Library Hi Tech,2013,31(2):294-307.
[47]魏刃佳,丁亦喆,张莉,等.在线学习系统中情感识别模块的设计与实现[J].现代教育技术,2014(3):115-122.
[48]Ray A,Chakrabarti A.Design and Implementation of Technology Enabled Affective Learning Using Fusion of Bio-physical and Facial Expression[J].Journal of Educational Technology&Society,2016,19(4):112-125.
[49]Mcdaniel B,D’mello S,King B,et al.Facial Features for Affective State Detection in Learning Environments[C]//Proceedings of the Cognitive Science Society 2007.Open Access Publication from University of California,2007:467-472.
[50]D’mello S K,Craig S D,Sullins J,et al.Predicting Affective States expressed through an Emote-Aloud Procedure from AutoTutor's Mixed-Initiative Dialogue[J].International Journal of Artificial Intelligence in Education,2006,16(1):3-28.
[51]Kaliouby R.,Robinson P.Real-Time Inference of Complex Mental States from Facial Expressions and Head Gestures[C]//Real-Time Vision for Human-Computer Interaction.Springer,2005:181-200.
[53]Wang S,Wu C,He M,et al.Posed and Spontaneous Expression Recognition through Modeling their Spatial Patterns[J].Machine Vision&Applications,2015,26(2-3):219-231.
[2]刘智,方常丽,刘三女牙等.物理学习空间中学习者情绪感知研究综述[J].远程教育杂志,2019(2):33-44.
[3]Ekman P,Friesen W V,Ellsworth P.Emotion in the Human Face[M].New York:Cambridge University Press,1982.
[4]Alvarado N.Arousal and Valence in the Direct Scaling of Emotional Response to Film Clips[J].Motivation&Emotion,1997,21(4):323-348.
[5]Lewis P A,Critchley H D,Rotshtein P,et al.Neural Correlates of Processing Valence and Arousal in Affective Words[J].Cerebral Cortex,2007,17(3):742-748.
[6]孟昭兰.情绪心理学[M].北京:北京大学出版社,2005.
[7]Yamauchi H,Mashiko S,Kimura M,et al.Verification of Acousical Correlates of Emotional Speech using Formant-Synthesis[J].Proc Isca Workshop on Speech&Emotion,2007,52(1):119-130.
[8]Wallbott H G.Bodily Expression of Emotion[J].European Journal of Social Psychology,1998,28(6):879-896.
[9]Ekman P,Friesen W V.Detecting Deception from the Body or Face[J].Journal of Personality&Social Psychology,1974,29(3):288-298.
[10]Keltner D,Ekman P.Facial Expression of Emotion[M].Handbook of Emotions 3rd.New York:The Guilford Press,2010:173-183.
[11]Lancet T.Communicationwithout Words[J].University of East London,1968,24(23):1084-5.
[12]Scherer K R.Vocal Communication of Emotion:A Review of Research Paradigms[J].Speech Communication,2003,40(1/2):227-256.
[13]Monkaresi H,Bosch N,Calvo R,et al.Automated Detection of Engagement using Video-Based Estimation of Facial Expressions and Heart Rate[J].IEEE Transactions on Affective Computing,2017,8(1):15-28.
[14]Grafsgaard J F,Wiggins J B,Boyer K E,et al.Automatically Recognizing Facial Expression:Predicting Engagement and Frustration[C]//The 6th International Conference on Educational Data Mining EDM 2013.ELSEVIER:43-50.
[15][19]Whitehill J,Serpell Z,Lin Y C,et al.The Faces of Engagement:Automatic Recognition of Student Engagementfrom Facial Expressions[J].IEEE Transactions on Affective Computing,2014,5(1):86-98.
[16]Wang J,Ma X,Sun J,et al.Puzzlement Detection from Facial Expression Using Active Appearance Models and Support Vector Machines[J].International Journal of Signal Processing Image Processing&Pattern Recognition,2014,7(5):349-360.
[17]江波,李万健,李芷璇,等.基于面部表情的学习困惑自动识别法[J].开放教育研究,2018(4):101-108.
[18]韩丽,李洋,周子佳,等.课堂环境中基于面部表情的教学效果分析[J].现代远程教育研究,2017(4):97-103+112.
[20]Happy S L,Dasgupta A,Patnaik P,et al.Automated Alertness and Emotion Detection for Empathic Feedback during e-Learning[C]//2013 IEEE Fifth International Conference on Technology for Education.IEEE:47-50.
[21]童莹.基于空间多尺度HOG特征的人脸表情识别方法[J].计算机工程与设计,2014(11):3918-3922+3979.
[22]Anderson K,Mcowan P W.A Real-time Automated System for the Recognition of Human Facial Expressions[J].IEEE Transactions on Systems Man&Cybernetics Part B Cybernetics A Publication of the IEEE Systems Man&Cybernetics Society,2006,36(1):96-105.
[23]Liu Y J,Zhang J K,Yan W J,et al.A Main Directional Mean Optical Flow Feature for Spontaneous Micro-Expression Recognition[J].IEEE Transactions on Affective Computing,2016,7(4):299-310.
[24]Tie Y,Guan L.A Deformable 3-D Facial Expression Model for Dynamic Human Emotional State Recognition[J].IEEE Transactions on Circuits and Systems for Video Technology,2013,23(1):142-157.
[25]Cheon Y,Kim D.Natural Facial Expression Recognition using Differential-AAM and Manifold Learning[J].Pattern Recognition,2009,42(7):1340-1350.
[26]Pantic M,Patras I.Dynamics of Facial Expression:Recognition of Facial Actions and Their Temporal Segments from Face Profile Image Sequences[J].IEEE Transactions on Systems,Man,and Cybernetics,Part B(Cybernetics),2006,36(2):433-449.
[27]Niese R,Al-Hamadi A,Farag A,et al.Facial Expression Recognition based on Geometric and Optical Flow Features in Colour Image Sequences[J].Iet Computer Vision,2012,6(2):79-89.
[28]Zhao G,Pietik¨Ainen M.Dynamic Texture Recognition using Local Binary Patterns with an Application to Facial Expressions[J].IEEETrans Pattern Anal Mach Intell,2007,29(6):915-928.
[29]Wang Y,See J,Phan C W,et al.LBP with Six Intersection Points:Reducing Redundant Information in LBP-TOP for Microexpression Recognition[C]//Computer Vision-ACCV 2014.Springer,Cham:525-537.
[30]Polikovsky S,Kameda Y,Ohta Y.Facial Micro-Expression Detection in Hi-Speed Video Based on Facial Action Coding System(FACS)[J].Ieice Transactions on Information&Systems,2013,96(1):81-92.
[31]Alexander Klser,Marszalek M,Schmid C.A Spatio-Temporal Descriptor Based on 3D-Gradients[C]//Proceedings of the British Machine Vision Conference 2008,Leeds.2008.
[32]Sun B,Li L,Zuo T,et al.Combining Multimodal Features with Hierarchical Classifier Fusion for Emotion Recognition in the Wild[C]//ACM on International Conference on Multimodal Interaction.ACM:481-486.
[33]He L,Jiang D,Yang L,et al.Multimodal Affective Dimension Prediction Using Deep Bidirectional Long Short-Term Memory Recurrent Neural Networks[C]//International Workshop on Audio/visual Emotion Challenge 2015.ACM:73-80.
[34]Scovanner P,Ali S,Shah M.A 3-dimensional Sift Descriptor and Its Application to Action Recognition[C]//Proceedings of the 15th ACM international conference on Multimedia.ACM:357-360.
[35][43]Zhang K,Huang Y,Du Y,et al.Facial Expression Recognition Based on Deep Evolutional Spatial-Temporal Networks[J].IEEETransactions on Image Processing,2017,26(9):4193-4203.
[36][45]Kim D H,Baddar W,Jang J,et al.Multi-Objective based Spatio-Temporal Feature Representation Learning Robust to Expression Intensity Variations for Facial Expression Recognition[J].IEEETransactions on Affective Computing(Early Access),2017:223-236.
[37]Lan Goodfellow Y B,Aaron Courville.Deep learning[M].Massachusetts:MIT Press,2016.
[38]Hamester D,Barros P,Wermter S.Face Expression Recognition with a 2-channel Convolutional Neural Network[C]//2015 International Joint Conference on Neural Networks(IJCNN).IEEE:1787-1794.
[39]Liu K,Zhang M,Pan Z.Facial Expression Recognition with CNNEnsemble[C]//2016 International Conference on Cyberworlds(CW).IEEE,2016:163-166.
[40]Liu Y,Chen Y.Recognition of Facial Expression based on CNN-CBP Features[C]//2017 29th Chinese Control And Decision Conference(CCDC).IEEE,2017:2139-2145.
[41]Ijjina E P,Mohan C K.Facial Expression Recognition Using Kinect Depth Sensor and Convolutional Neural Networks[C]//Proceedings of the 2014 13th International Conference on Machine Learning and Applications.IEEE,2014:392-396.
[42]Ebrahimi Kahou S,Michalski V,Konda K,et al.Recurrent Neural Networks for Emotion Recognition in Video[C]//Proceedings of the2015 ACM on International Conference on Multimodal Interaction.ACM,2015:467-474.
[44]卿粼波,熊文诗,周文俊,等.基于多流CNN-LSTM网络的群体情绪识别[J].计算机应用研究,2018(12):3828-3831.
[46][52]Lin K C,Huang T C,Hung J C,et al.Facial Emotion Recognition towards Affective Computing‐based Learning[J].Library Hi Tech,2013,31(2):294-307.
[47]魏刃佳,丁亦喆,张莉,等.在线学习系统中情感识别模块的设计与实现[J].现代教育技术,2014(3):115-122.
[48]Ray A,Chakrabarti A.Design and Implementation of Technology Enabled Affective Learning Using Fusion of Bio-physical and Facial Expression[J].Journal of Educational Technology&Society,2016,19(4):112-125.
[49]Mcdaniel B,D’mello S,King B,et al.Facial Features for Affective State Detection in Learning Environments[C]//Proceedings of the Cognitive Science Society 2007.Open Access Publication from University of California,2007:467-472.
[50]D’mello S K,Craig S D,Sullins J,et al.Predicting Affective States expressed through an Emote-Aloud Procedure from AutoTutor's Mixed-Initiative Dialogue[J].International Journal of Artificial Intelligence in Education,2006,16(1):3-28.
[51]Kaliouby R.,Robinson P.Real-Time Inference of Complex Mental States from Facial Expressions and Head Gestures[C]//Real-Time Vision for Human-Computer Interaction.Springer,2005:181-200.
[53]Wang S,Wu C,He M,et al.Posed and Spontaneous Expression Recognition through Modeling their Spatial Patterns[J].Machine Vision&Applications,2015,26(2-3):219-231.