Review of Facial Expression Detection using Deep Learning

Review of Facial Expression Detection using Deep Learning

Nova Dileep

CSE

Jyothi Engineering College, Cheruthuruthy Thrissur, India

Sreelakshmi C CSE

Jyothi Engineering College, Cheruthuruthy Thrissur, India

Shinoz Mohammed P P

CSE

Jyothi Engineering College, Cheruthuruthy Thrissur , India

Reshma K V

CSE

Jyothi Engineering College, Cheruthuruthy Thrissur , India

AbstractIn the past few years, deep artificial neural networks have received the greatest attention in computer science. Pattern recognition, machine vision and machine learning are some of them. One of its outstanding applications is through facial ex- pression recognition expression area. Facial expression analysis is useful for many tasks, and the application of deep learning in this field is also very important for fast development. Facial expression detection which is part of the facial recognition is becoming more and more important and needed because it has greatly increased. Although there is a way to use machine learning and manual recognition of expressions intelligence technology, this paper tries to point out deep learning and image classification methods to recognize expressions and categorize expressions based on images.

Index TermsFacial Expression Recognition, Deep Learning, CNN Architecture, Emotion Classification.

1. INTRODUCTION

   Facial expressions are essential to human social communi- cation, as this communication is both verbal and non-verbal. Facial expressions are one aspect of non- verbal communica- tion, as the face expresses prominent signals of communica- tion, which includes eye contact. Other aspects of non-verbal communication are gestures and body language. It is easy for humans to notice and understand faces and facial expressions. However, it still proves difficult to develop an automated system that accomplishes the same understanding. There are several problems related to this issue, such as the detection of an image segment as an actual face, due to occlusions or illumination, as well as variations in head poses, extraction of facial expression information, facial landmark detection, or classification of expression.

   Facial Expression Recognition (FER) is an active research area in the field of Artificial Intelligence and applied in vast domains, such as security, monitoring and law enforce-

   ment, marketing and entertainment, e-learning and medicine, emotionally intelligent robotic interfaces, or social humanoid robots. Various fields, like data analytics, psychological re- search, social gaming, and others that include human-computer interactions, can benefit from the ability to recognize facial expressions automatically.It is said that the emotional expres- sion through face can convey 55% of information. It has been determined that certain facial expressions have universal mean- ing. By 1978, Ekman and Friesen had finalized and developed the Facial Action Coding System (FACS), and the authors found that there are six facial expressions, including happiness, sadness, surprise, fear, anger, and disgust, that appear to be universal across all culture. However, it is challenging for computers to recognize these common human expressions in natural conditions because of differences in lighting various head poses.

   Deep learning is an artificial intelligence (AI) function that imitates the workings of the human brain in processing data and creating patterns for use in decision making. Deep learning is a subset of machine learning in artificial intelligence that has networks capable of learning unsupervised from data that is unstructured or unlabeled. Also known as deep neural learning or deep neural network. We have read few articles and papers on facial expression recognition using deep learning and understood different deep learning techniques. Of these different techniques, CNN algorithm will be best suited for the proposed system.

2. DISCUSSION

   Different types of deep learning algorithms are available for facial expression detection such as Deep Belief Network (DBN) and Convolutional Neural Network (CNN). Of these, the steps that have taken by seven papers that used Convolu-

   tional Neural Network (CNN) for facial expression detection have been discussed here.

   The main objective of [1] is to identify 6 basic facial expres- sions such as happy, sad, fear, anger, surprise and disgust. It has used JAFFE dataset for training and testing. Convolutional Neural Network is used for extracting features of eyebrow, eye, mouth, cheeks and nose and also for classification. The layers used are two convolutional layers, two pooling layers and two fully connected layers. ReLU activation function is used in all layers.

   In [2], it classifies facial expressions into 7 expressions like angry, disgust, fear, happy, sad, surprise and neutral. FER-2013 dataset is used for training and testing. Three models are used which are a baseline model, a Resnet50 model and a five layer model. Of these, highest accuracy is obtained for Resnet50 model and five layer model. Five layer model consists of four stages of convolutional layer together with maxpooling layer, 1 fully connected layer and a softmax layer. Resnet50 consists of 50 layers. From that output layer is replaced with two fully connected layers and a softmax layer. To get the final emotion detection network called ED NET, five layer model and Resnet50 model are ensembled. ReLU activation function is used in all layers.

   In [3], it mainly consists of two steps. In first step, the CNN architecture is fine tuned by VGG model to get the first model. In second step, inorder to improve classification the CNN architecture is fine tuned by first model and final model is obtained. It uses a combination of CK+, RaFD, MUG and KDEF datasets. The main objectives of this model is to to improve the accuracy in classifying emotions and to classify the facial expressions to angry, happy, fear, neutral, sad and surprise using CNN architecture. It uses four convolutional layers, three maxpooling layers, two fully connected layers and a softmax layer for classification of facial images and the regularization technique used is dropout.

   The main objectives of [4] are to avoid the explicit fea- ture extraction process and to classify facial expressions to worried, angry, disgust, surprise, anxious, happy, sad, neutral emotions. The dataset used is Chinese Linguistic Data Consor- tium(CLDC), which consists of multimodal emotional audio and video data. In order to avoid the complexity of explicit feature extraction process, the Faster R-CNN is proposed for facial expression recognition here. The facial expression image is normalized and the features are extracted by using convolution kernel. And maxpooling is used for reducing dimensions of the features extracted. The Region Proposal Networks (RPNs) are used for predicting accurate region proposals. The proposed method uses only one convolutional neural network (CNN) for all purposes. The softmax layer is used for classificaton of facial expression images and regressor layer is used for obtaining bounding box coordinates.

   The main objective of [5] is to classify facial expression into seven facial expressions like angry, disgust, fear, happy, sad, surprise and neutral. FER-2013 dataset is used

   for training and testing. It uses six convolutional layers, three maxpooling layers, one fully connected layer, one dense layer and a softmax layer for classification. ReLU activation function is used in all layers and the regularization technique used is dropout.

   In [6], it classifies facial expression into happiness, sur- prise, sadness, anger, fear, disgust and neutral. Preprocessing is performed using normalization and augmentation. Features are extracted using trainable convolutional layers. It uses three convolutional layers, two maxpooling layers, one fully connected layer and a softmax layer. ReLU activation function is used in all layers and dropout regularization is used to prevent overfitting.

   In [7], it classifies facial expressions into anger, disgust, fear, happy, sad and surprise. It uses four CNN networks. First three CNN networks are used to segment facial components like eyebrow, eye and mouth respectively. The fourth CNN network is used for recognition of facial expressions. After post processing, for each face, a final iconized image is obtained combining the segmented image. The fourth CNN classifies facial expressions combining raw facial image and iconized facial image.

3. CONCLUSION

In this paper, we have discussed the deep learning ap- proaches for facial expression recognition of six different papers. All papers are using Convolutional Neural Network (CNN) for feature extraction. Of these, highest accuracy obtained is 99.33% and lowest accuracy obtained is 61%. Since people come from different races, skin tones, skin textures, they may have different facial expressions, it is too difficult to eliminate all these differences. We sincerely hope that further research in this field may solve these problems and provide us with robust, easy-to-use and widely applicable facial expressions recognition system.

Fig. 1. comparison table

REFERENCES

1. S. Harshitha, N. Sangeetha, A. P. Shirly and C. D. Abraham, Human facial expression recognition using deep learning technique, 2019 2nd International Conference on Signal Processing and Communica- tion (ICSPC), Coimbatore, India, 2019, pp. 339-342, doi: 10.1109/IC- SPC46172.2019.8976876.

2. Raghu N and Bharathi S. Facial expression recognition using deep learning, 06 2020.

3. A. Fathallah, L. Abdi and A. Douik, Facial Expression Recognition via Deep Learning, 2017 IEEE/ACS 14th International Conference on Computer Systems and Applications (AICCSA), Hammamet, Tunisia, 2017, pp. 745-750, doi: 10.1109/AICCSA.2017.124.

4. Jiaxing Li, Dexiang Zhang, Jingjing Zhang, Jun Zhang, Teng Li, Yi Xia, Qing Yan, Lina Xun, Facial Expression Recognition with Faster R-CNN, Procedia Computer Science, Volume 107, 2017, Pages 135-140, ISSN 1877-0509, https://doi.org/10.1016/j.procs.2017.03.069. (https://www.sciencedirect.com/science/article/pii/S187705091730 3447)

5. S. Singh and F. Nasoz, Facial Expression Recognition with Convolu- tional Neural Networks, 2020 10th Annual Computing and Communi- cation Workshop and Conference (CCWC), Las Vegas, NV, USA, 2020, pp. 0324-0328, doi: 10.1109/CCWC47524.2020.9031283.

6. L. Xu, M. Fei, W. Zhou and A. Yang, Face Expression Recognition Based on Convolutional Neural Network*, 2018 Australian New Zealand Control Conference (ANZCC), Melbourne, VIC, Australia, 2018, pp. 115-118, doi: 10.1109/ANZCC.2018.8606597.

7. Gozde Yolcu Ismail Oztel Serap Kazan Cemil Oz Kannappan Palaniappan Teresa E. Lever Filiz Bunyak, Facial expression recog- nition for monitoring neurological disorders based on convolutional neural network Received: 6 September 2018 / Revised: 13 June 2019 / Accepted: 10 July 2019 Springer Science+Business Media, LLC, part of Springer Nature 2019