Human Emotion Recognition using Neural Network Technique.

Human Emotion Recognition using Neural Network Technique.

Miss Pooja A Choudhari#1, Dr. M .R. Dixit#2

Department of Electronics and Telecommunication, KITs College of Engineering, Kolhapur, Gokul-Shirgaon,

Shivaji University, Maharashtra, India

Abstract- Emotions play an important role in human to human interaction to have smoother communication. By using emotion recognition systems human computer interaction can also be improved.Due to increasing demand and requirement of improvement in human computer interface in recent years, emotion recognition systems are developed.In this proposed work image database is collected and seven basic emotions are classified using neural network as classifier.

Keywords: Emotion recognition, Face detection, Preprocessing, Feature Extraction, Classification, Neural Network.

1. INTRODUCTION

   In Human society, nowadays there are lot of applications in human computer relationship. So as to improve relationship many solutions have been developed between human and computers such as robots that have ability to understand, interpret and react to emotions. Using various experimental results such as facial expressions, body and hand gestures, acoustic data, and bio physiological data, the human mental state could be understood [5]. In different disciplines, the importance of knowing this mental states appears. For example, Human computer interaction is required to be improved as good as human to human interaction [11]. Hence, recognizing the emotions of human is considered as one of the important step forward.

   Animation and graphic designs at movies and computer games are other applications for interface .As human computer interaction (HCI) started to increase involvement in our life and as HCI become more sophisticated, it becomes even more important that we are able to interact with computers in more natural way [7]. Intelligent factors like detecting and deciding influence the capabilities of emotion recognition. As in the past decades, due to advances of artificial intelligence techniques it is possible to have communication in more natural way, similar to every day interaction between humans using an automatic facial emotions recognition system. Emotion recognition system plays an important role in adapting one to one tutoring to student performance through nonverbal behavior recognition. Human emotion recognition system can be used in forensics to recognize the face of criminal or dead bodies.

   Feature extraction and classification are the two important steps used in recognition system

   1. Features Extraction

      FE is a special form of the dimensional reduction. Feature extraction is used to simplify the number of resources which are required to describe large set of data correctly. Feature extraction methods can be supervised form or unsupervised form, depending on whether or not class labels are used. Principal Component Analysis, Independent Component Analysis, Multi-dimensional scaling are the most used and popular unsupervised methods.

      Supervised FE methods (and also Feature Selection methods) either use information about classification performance called as wrappers or indirect measures, called filters.

   2. Classification and Prediction

   This final step and technique involves classification of segmented image under various labels based on the features generated. The various data mining techniques are used for classification. The classification includes assignment of class label to the set of unclassified cases. They two different classes of classification are supervised classification and unsupervised classification .In supervised methods, the possible set of classes are known in advance where as in unsupervised classification they are not known in advance and after classification we can assign name to the class. Unsupervised classification is also known as clustering.

2. LITERATURE REVIEW

   The origin of facial expression was proposed in 19th century in human and animals by Darwin. Ekman and Frisen later developed the techniques in 1975; they have performed extensive study in human facial expression and human facial recognition.

   Different techniques are developed for the system including different methods and algorithms

   Caifeng Shan et al. evaluated facial representation based on local binary patterns, statistical facial features for Individual independent facial emotion recognition. They have formulated Boosted -LBP to extract the LBP features and obtained the recognition by using the different machine learning techniques. Among all of this techniques they obtained the best result with the combination of Boosted LBP feature and SVM technique.They conducted the experiment on Cohn-Kanade database, from which they selected 320 image sequence from the database which includes six basic emotions of the different university

   student. They partitioned data set into the ten groups consisting of equal number of subjects, from which nine groups were used for training the classifier, while remaining group was used for testing [10].

   Spirous Ioannou et al. worked on the recognition system in which they have used support vector machine for face detection, then the facial points are defined for feature extraction by estimating position of eyes, nose, eyebrows and mouth masks. Different techniques are used for facial feature mask extractions. Neuro fuzzy technique was used for the facial emotion recognition which showed accuracy up to 58% [8].

   Guoying Zhao et al. proposed an approach for Dynamic texture recognition. A local binary pattern method was developed to combine the motion and appearance together.They performed experiments on MIT and Dyntex database by using LBP TOP (Three orthogonal planes). They also experimented results on CohnKanade database by using linear binary pattern three orthogonal planes [17].

   Maja Pantic et al. developed an automatic system to recognize facial action units in static, frontal and / or profile view color face image.For face region extraction they used watershed segmentation with markers. They extracted 19 fiducial points of contours of face components and 10 profile contour fiducial points. They have used eyes, eyebrows, nose, lips and chin to obtain the feature points. Based on this they have recognized 32 action units occurring alone or together. The data base used by them was self-generated which includes true colorimage with 300 to 450 pixels across the width of subjects face [9].

   Another new approach was proposed by [13] in which automatic recognition of basic emotions from the image database. The system includes image processing, pattern recognition and low level 3 dimensional computer vision techniques. Feature extraction includes color and gradient information.Geometric features are computed and extracted for three dimensional processing. Further this features are applied to artificial neural network for classification. They used two hidden layers in neural network with 6 neurons. Result obtained shows that there is slight misclassification between disgust class and anger, while happy class also has misclassification with disgust. Neutral, Happy and surprise shows least mixing.

3. METHODS IMPLEMENTED

   1. Face detection: Proposed Algorithm

      First step in this algorithm is to extract the useful section of face of a human being so that it is easier to work for feature extraction and classification process. Two techniques are used in this system to detect face: Adaboost classifier, skin segmentation process.

      Segmentation process includes detection of skin coor and edge detection. Image Compensation is done then the skin color is extracted from an image. After extracting skin color, noise is removed and edges are detected and dilated. After dilation, holes are filled and region properties like bounding box (BB), area are obtained and face is detected.

      To improve the accuracy rate and process timing other adaboost classifier is used.

      Fig 1. Entire image

      Fig 2. Face detected image

   2. Feature extraction

      After the detection of face from an entire image, next step is a feature extraction which is one of the important step in emotion detection system.Two methods can be used for feature extraction: Analytic approach, Holistic approach [16]. Raw facial image is used as input in holistic approach. While in the analytic approach, some of the important facial features are been detected and extracted from face .we used analytic approach in this system , where we send the extracted selected features from the image as input to a classifier. In our proposed system the methods used for feature extraction are Eigen value, discrete wavelet transform and face structure feature methods.

      Eigen value: Any vector change in magnitude but not in direction is called EV (Eigen Vector).While in the Eigen value the magnitude that vector is changed is called Eigen value. We obtain an Eigen value of a face detected images from database. And the mean value of all the images from database is measured to obtain a highest feature value. The highest Eigen value does contains important feature of data.

      DWT features: Discrete wavelet transform is the second method used for extraction of features. We use two filters i.e. low pass and high pass for decomposition and reconstruction. The second level DWT function is applied so as to obtain better results. Mean of a DWT features of an

      entire images is then calculated and applied as one of the input to the neural network input layers.

   3. Classifier: Neural networks

   An artificial neural network is interconnected group of nodes inspired by neurons in the brain.ANN are a family of statistical learning models and they are generally presented as systems of interconnected neurons which exchange messages between each other[14]. One of the important advantage of neural network is that it is adaptive to inputs and capable of learning easily. Neural network basically consists of three layers: Input layer, hidden layer and output layer. Hidden layers can be of any numbers, layers are increased so as to improve the accuracy and recognition rate.Input layers can be of N nodes where the output from feature extraction is applied[16].Output nodes can have number of nodes as per the requirement of number of output class.

   Neural network is applied as a classifier in this system.One hidden layer is used while input layers consists of the feature extracted from above process.ie eigen values, DWT features and face features. Then the output is the classified emotion of the input one.

   There are two phases in neural network training phase and testing phase. We have used 70% of images from our database for training phase and remaining is used for testing.

   Training phase:

   Input to the training phase is collection of images showing human faces. Feature extraction step is then applied to these. In this step key attributed of the image is computed and stored as feature vector.These feature vector represent the most important properties observed in the face.There are two advantages of this step. First the data size is reduced to few selected important features from an entire image. Second, more structured information is obtained from selection of features than just basic pixel values of images. Hence vectors (features) can be considered as the minimal set which is sufficient to represent the required face image.

   Testing phase:

   Measurement of classification rate is usually done in testing phase. The inputs to this phase are the models that were built during training phase and the images (test) for which the results are expected i.e. emotion to be recognized. In this phase again only the face extracted region is been used as rest of image do not contribute in classifying the different. Feature extraction phase is the first step again in this phase where the key features from the face image are extracted. Extraction method used for this phase must be same as that used in training phase

   .Feature vector of face image is the output of this step which would then be applied to testing step. In this step of testing, the vectors of feature are tested against models built in the training phase. Output of this step is a score which indicates the emotion which is detected by the model. During testing a simple input image, it can be

   correctly classified as accurate emotion expressed. In another case the system can also incorrectly classify an input sample image as expressed correct emotion. This case constitutes as false positives. Sometimes it could be the case that it classifies wrongly a given sample images as emotion expressed incorrect. These are false negative case.

4. RESULT

Here we computed four emotions from 6 basic emotions of 22 subjects by using neural networks as a classifier. Table I shows the confusion matrix of four emotions.

Table I

Emotions	Ha	An	Di	Su
Happy(Ha)	21.6	2.3	6.8	0.0
Anger(An)	2.3	19.3	1.1	0.0
Disgust(Di)	0.0	0.0	15.9	0.0
Surprise(Su)	1.1	3.4	1.1	25

Total accuracy: 81.8%

Here obtained recognition rate of happy emotion is 86.4%, anger emotion is 77.3%, disgust emotion is 66.6% and surprise emotion is highest i.e. 100%.

CONCLUSION

Recognition of facial emotion is investigated in this paper to classify four emotions (happy, anger, disgust surprise) from six basic emotions (happy, sad, fear, anger, disgust, surprise).Neural network and other image processing methods as mentioned above are used in this system to classify the correct emotion. With the self-made database an image is initially, applied to the face detector (Adaboost classifier or skin color extraction method) to obtain required area from an entire image. Eigen values, DWT features and face structure features are extracted and are applied to input layer of neural network which processes to obtain the classification of required emotion. For our database we obtained 81.8% accuracy result with recognition rate of surprise emotion to be of highest and disgust emotion to be lowest.

ACKNOWLEDGEMENT

I wish to thank my guide Dr. M.R. Dixit madam for her valuable support and guidance. The work presented here could not have been completed without her guidance and encouragement during the course of my research work. I express my sincere thanks to Prof. A.R.Nigavekar, Head of Department of Electronics Engineering. I offer my humble and sincere thanks to Dr. Y.M.Patil and other faculty members and non-teaching staff of our department for their valuable support.

I express my sincere thanks to Dr. V.V. Karjinni, Principal, KITs college of Engineering, Kolhapur for his cooperation.

I also thank my family, friends and relatives or their feedback, thoughtful suggestion and the help which they provided me to obtain and collect the emotion image database for this system.

REFERENCES

[1]. Wenming Zheng, Multi-View facial Expression Recognition Based On Group Sparse Reduced Rank Regression,, IEEE Transactions on Affective Computing , Vol 5,No. 1,January March 2014.

[2]. Ognjen Rudvoic, Maja Pantic and Ioannis (Yiannis) Patras, Coupled Guassian Processes fo Pose Invariant Facial Expression Recognition, IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol 35, No.6, June 2013.

[3]. Songfan Yang and Bir Bhanu, Understanding Discrete Facial Expressions in Video Using An Emotion Avatar Image, IEEE Transactions On Systems, Man, and Cybernetics Part B: Cybernetics, Vol 42, No.4, and August 2012.

[4]. Zhihong Zeng, Maja Pantic Glenn I. Roisman, and Thomas S Huang, A Survey of Affect Recognition Metods: Audio Visual and Spontaneous Expressions, IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol 31, No.1, January 2009.

[5]. Yan Tong, Jixu Chen and QiangJi, A Unified Probabilistic Framework for Spontaneous Facial Action Modeling and Understanding, IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol.32, No.2, February 2010.

[6]. Amir Jamshidnejad and A Janshideined, Facial Emotion Recognition For Human Computer Interaction Using A Fuzzy Model in the E-Business,2009 conference on Innovative Technologies in Intelligent systems and Industrial Applications, July 2009.

[7]. Yoshihiro Miyakoshi and Shohei Kato, Facial Emotion Detection Considering Partial Occlusion of face using Bayesian Network.

[8]. Spiros V. Ioannou, Amaryllis T. Raouzaiou, Vasilis A Tzouvvaras

,Theofilos P. Mailis , Kostas C. Karpouzis and Stefanos D. Kollias, Emotion Recognition through Facial expression analysis based on neurofuzzy network, Elsevier, Neural Networks, March 2005.

[9]. Maja Pantic and Leon J. M. Rothkrantz, Facial Action Recognition for Facial Expression Analysis from Static Face Images, IEEE Transactions on Systems, Man and Cybernetics- Part B: Cybernetics, Vol 34, No. 3, June 2004.

[10]. Caifeng Shan, Shaogang Gong and peter W. Mcowan, Facial Expression Recognition Based on Local Binary Patterns.

[11]. Fatima Guney, Emotion Recognition Using Facial Images.

[12]. Dilbag Singh, Human Emotion Recognition System, I.J, Image, Graphics and Signal Processing, 2012.

[13]. Robert Niese, Ayoub Al-Hmadi, Axel Panning and Bernd Michaelis, Emotion Recognition based on 2D-3D facial feature extraction from color image sequence.

[14]. Yafei Sun,Neural Networks for Emotion Classifcation, August 2013.

[15]. Robert Niese, Ayoub Al- Hamadi, Axel Panning and Bernd Michaelis, Emotion Recognition based on 2D-3D Facial Feature Extraction from Color Image Sequences, Journal Of Multimedia, Vol. 5, No. 5, October 2010.

1. Devi Arumugam and Purushothaman, Emotion Classification Using Facial Expression, International Journal of Advanced Computer Science and Applications, Vol. 2, No.7, 2011.

2. Guoying Zhao and Matti Pietikainen, Dynamic Texture Recognition Using Local Binary Patterns with an Application to Facial Expressions, IEEE Transactions on pattern Analysis and Machine Intelligence, Vol. 29, No.6,June 2007.