Effective Emotion Recognition of Expressions from Facial Features

Effective Emotion Recognition of Expressions from Facial Features

Jason Elroy Martis

Department of ISE NMAM Institute of Technology

Nitte 574110

Dr. Balasubramani Professor, Department of ISE NMAM institute of Technology

Nitte 574110

AbstractFacial Expressions convey non-verbal cues, which play an important role in interpersonal relationships. Automatic recognition of facial expressions can be an important component of natural human-machine interfaces; it may also be used in behavioral science and in clinical practical practice. Although humans recognize facial expressions virtually without effort or delay, reliable expression recognition by machines is still a challenge, So, by using a Reliable depth sensor a system was designed that could detect a persons facial expression and display it. This system is designed to read a persons face using Depth Camera which computes various parameters of the persons face. Upon detecting and registering these parameters, the system deduces the persons emotional state. The final outcome will be an automatic feature based approach to facial expression recognition.

1. INTRODUCTION

   Humans can easily identify a large range of facial expressions with incompatible ability via a computer machine. Over the last two decades, the advances in computer vision and pattern recognition power have opened the door to new opportunities of automatic facial recognition system.

   The shape and motion of facial components such as eyebrows, eyes, nose and mouth manifest expressions. Emotion recognition via facial expressions has attracted a great deal of interest with recent advances in artificial intelligence and pattern recognition, mainly because of its various applications in video games, medicine, security, intelligent human- computer interaction and effective computing. Facial expressions constitute the main channel which is used to communicate emotions.

2. LITERATURE SURVEY

   In our research, we came across a paper which extracted 1347 3D facial points by Kinect V2.0. Key facial points were selected and feature extraction was conducted. Feature dimensionality reduction was implemented by Principal Component Analysis (PCA). The emotion recognition models were constructed using several classical classifiers. [1]

   In another research, Evaluation of the degree of characterization J of the distances results from the database; emotion classification using neural networks marks the start of identification of the facial images and their main distinctive features. The task was achieved by using 18 feature points. [2]

   In another work, low dimension weighted Gabor filter bank are used to detect emotions on segmented image. The space domain is reduced by segmentation and only those facial features are focused that reflects expressions accurately. A RBF network classifies selected feature values. [3]

   In another paper, feedback learning theory with Neural Network theory was combined with 3-D Local Binary Pattern feature extraction process to explain the universality problem of feature extraction algorithm and system identification performance. [4]

   We also came across a research paper which revolves around development of an intelligent system that employs ActiveShape Models (ASM) structured with a Support Vector Machine (SVM) classifier to detect human emotions by observing the facial characteristics. FACS Action Units were used to take four ratios from features in the human face to classify emotions. [5]

   In another paper, emotion detection is done based on two components of a computational system. Storage of Links between bodily expressions and emotion values by PEMs (Personalized Emotion Maps). Each persons emotion profile is individually calibrated by these PEMs. Scherers theoretical complex system model of emotion is used to implement them [6], [7]. Individual PEMs were used to present a regression algorithm that determines a persons emotional feeling from sensor measurements of their bodily expressions. [8]. In another paper, physiological measurements of skin conductance response and heart rate reported the level of rejoice and regret based on trading outcomes. Three labelling methods – binary, tri-and tetra-state blended models were compared by means of C4.5, CART, and random forest algorithms, across different window lengths for heart rate to detect emotions.[9] Similarly, another work was about the observation of the facial characteristics by detecting the human emotions through the development of an intelligent system. Active Shape Models (ASM) structured with a Support Vector Machine (SVM) classifier is employed to address this problem. Emotions are classified by defining 4 ratios from features in the human face, by making use of FACS Action Units .[10]

3. FEATURE POINT DETECTION

   Table. 1: 20 Types of Emotions and their Features.

   From the above literature we can detect human emotions by mapping and also uses neural networks, but these techniques shown above do not provide accuracy as well as efficiency due to technological limitations. So we present a technique by which we can increase the accuracy and efficiency of human emotions from facial expressions.

   Every emotion associated with a certain facial expression as shown in Table 1 makes us recognize the mood of the person. These special characteristics that give unique identity to the given expressions provide an insight of how the mechanism of detection of facial emotion recognition works.

   We know that emotion is a state of mind that is expressed through facial expressions. When we take all emotions into account, the expression shows changes in the prominent facial region being,

   • Mouth and lip region

   • Forehead region

   • Eye and Eye brow region.

     In this paper, a new feature based approach of facial expression recognition is being introduced. A Depth based camera is used to detect facial features namely eyes, eyebrows, and mouth using vertical and horizontal projections. An array of vertices define the facial points. A 3D point (x, y and z coordinates) that describes the corner of a geometric triangle is called a vertex. These vertices are used to construct a 3D mesh of a face as shown in fig 1. Using these points the facial features are calculated in the respective regions mentioned above. These features are used to recognize basic expressions like Happy, Sad, Fear, Anger, Disgust and Surprise.

     Fig. 1: Highlighted feature points

     Table 1 shows how the emotions can be diversified from each other. It also shows the fact that mixed emotions which are the combination of two basic emotions have certain characteristics in common. These features can be studied closely by carefully analyzing a face to detect an emotion.

4. IMPLEMENTATION

   The kinect xbox sensor is a new product from Microsoft. It is a device with depth sensing technology, a built in color camera, an infrared emitter and a microphone array enabling it to sense the location and movements of people. It also provides significant improvements over the old sensor to sense objects more clearly.

   Microsoft provides with this a Development Kit with new facilities, drivers, tools, APIs, a Device interface, and many sample codes in C#, C++ to help developers.

   The enhancements in the new kinect sensor are rated to deliver better response time and an improves voice and gesture experience for users, the sensor additionally includes full HD-video

   For the implementation of the project, we use the Kinect SDK 2.0 for face tracking. It gives us the3D facial points which can be used to predict features by training the computer to analyze facial expressions.

   1. To get points on HD Face Mask

      1. If the perimeter crosses a predefined threshold, the eye is classified as big or open

      2. If the perimeter is below the threshold, the eye is classified as small or closed.

      3. The space between these two thresholds is the neutral region.

5. RESULTS AND CONCLUSIONS

   We compared our system to calculate the classification with the animation unit based detection and found out the results as shown in the graph below.

   HD face API is one of the powerful tracking library. Along with detecting the face it also gives us the facial points in 3D space. Although it is powerful, it is badly documented and makes us hard to understand whats going on inside the API because it gives us low level functionality. It also provides us with raw facial data.

   First we tracked which Facial point belonged to which region on the face. This log helped us to identify the main features like Eyebrows, Mouth, and Eyes. These points now were used to find out the features in the table 2.

   The mouth features were detected as follows

   • First the points are assumed to be in a straight line.

     100

     80

     Accuracy

     60

     40

     20

     0

     0 50 100

     No. of Trials

     Eye Brow Mouth Curve Mouth

     Eyes

     Fore Head

     1. If the upper lip and the lower lip points are close. This forms to be the Closed Position

     2. It the Upper lip and lower lip points are far exceeding a preset threshold. This forms to be the open position.

   • The lip curve is collected as follows

     1. The average of the lower lip and upper lip points are4 calculated. This forms to be the straight line which is neutral

     2. If these average points fall below the preset neutral threshold it is low curve down feature.

     3. If these average points fall above the preset neutral threshold it is low curve up feature.

     The Eye brow features are calculated as follows

   • First the brow points are assumed to be neutral line which is curved

     1. If it falls below the neutral curve it is called low

     2. If it rises above the neutral curve it is called high

   • Since it becomes difficult to perform eye tracking using a depth sensor camera we use an extra dimension for accuracy an perform pattern analysis.

   • The eye brow features are calculated as: big, small and neutral.

   Classification of the eye in these categories is done by using the following procedure:

   • Get the feature points of the eye i.e. the surrounding parts of the eye.

   • Join all the points to form a polygon.

   • Calculate the perimeter of the polygon formed.

   Fig 5.1: Graph depicting the accuracy of different feature points

   From the above graph, we found out that classification of mouth features require only 30 trials. This proves that this information can be collected quickly.

   For mouth curve calculation, a minimum of 50 trials are required for learning and classification. Eye brows form an prominent feature and a minimum amount of 30 trials are required to give accurate results. We also observed that the accurate value for classification based on forehead was very low and formed a waveform of crests and troughs between 20% and 30%. This is due to the fact that forehead curves are difficult to identify because of the limitations in technology and precision. We have seen a slow learning curve with eyes having an accuracy of 30%. It was due to the lag and a rise after 40 trials. We predicted that it is due to perimeter analysis. Hence we have concluded that pattern based analysis approach is effective and has better accuracy compared to animation based on unit detection of emotion.

   VII. FUTURE WORK

   We can further apply our pattern analysis technique to identify micro expressions and further classify our emotion accurately based on extra features as shown in table 1.

   1. ACKNOWLEDGEMENTS

      We would like to thank Nidhisha Suvarna, Prajna Shetty, Krithika Marla, Neha Marla, Shwetha Rao from the Department of Information Science and Engineering, Ajey Pai K from the Department of Electronics and Communication Engineering for helping us to implement the

      idea and turn the vision into reality. We would like to thank Joy Martis, Assistant Professor, Department of Humanity, NMAMIT for language assistance.

   2. REFERENCES

1. Liqing Cui, Xiaoqian Liu, Tingshao Zhu Zhan Zhang, "Emotion detection using Kinect 3D facial points," in IEEE/WIC/ACM International Conference on Web Intelligence, 2016, p. 4.

2. Mounir Sayadi, Philippe Gorce Ahmed Fnaiech, "Feature Points Tracking and Emotion Classification," in 2nd International Conference on Advanced Technologies, Monastir, Tunisia, 2016, p. 5.

3. A. Verma, K. Juneja and S. Goel P. Sisodia, "An Improved Method for Emotion Detection," in International Conference on Communication and Signal Processing, April 3-5, India, 2014, p. 6.

4. Yang Li, "Novel Face Recognition Algorithm based on Adaptive 3D Local Binary Pattern Features and Improved Singular Value Decomposition Method,".

5. Arkapravo Bhaumik Sugandha Sangal, "A Robust Method For Automatic Emotion Detection,".

6. K. R. Scherer, "Emotions as episodes of subsystem synchronization driven by nonlinear appraisal processes," 2000.

7. "Emotions are emergent processes: they require a dynamic computational architecture," Philosophical Transactions of the Royal Society B: Biological Sciences, vol. vol. 364, no. 1535, pp. pp. 34593474, 2009.

8. Peter Sander Franck Berthelon, "Regression algorithm for emotion detection," in 4th IEEE International Conference on Cognitive Infocommunications, Budapest, Hungary, December 25, 2013, p. 6.

9. Anuja Hariharan and Marc Thomas Philipp Adam, "Blended Emotion Detection for Decision Support," IEEE TRANSACTIONS ON HUMAN-MACHINE SYSTEMS, vol. VOL. 45, p. NO. 4, AUGUST 2015.

10. Sugandha Sangal and Sugandha Sangal, "A Robust Method For Automatic Emotion Detection," in IEEE International Advance Computing Conference (IACC), 2014.