A Local Binary Pattern Based Facial Expression Recognition using K- Nearest Neighbor (KNN) Search

A Local Binary Pattern Based Facial Expression Recognition using K- Nearest Neighbor (KNN) Search

Goutami Panchal

PG Scholar,

Dept. of Electronics and Comm.

Dayananda Sagar College of Engineering, Bangalore-78, India.

K N Pushpalatha

Professor,

Dept. of Electronics and Comm.

Dayananda Sagar College of Engineering, Bangalore-78, India.

Abstract – The face is the primary focus of attention in social interaction plays a major role in conveying identity and emotion. Detection and recognition of facial expression from images are not restricted in the real-world situations has many potential applications. The face detection and recognition can be used for the security purpose, voter verification etc. The facial expression is a significant key for interpersonal relations where non-verbal communication is conveyed from person to person. The different methodologies and techniques are introduced for detection, extraction, classification and recognition of the face. The proposed method is K-Nearest neighbor along with Local Binary Pattern and Asymmetric Region LBP are used for extracting features and feature classification of the image. This proposed method is best among the existing methods in terms of recognition rate. The method is implemented in MATLAB.

Keywords: LBP, feature vector, emotion detection, expression classification, KNN, face recognition.

1. INTRODUCTION

   The increase need in researches in the area of digital processing and pattern recognition give rise to the development of algorithms for different applications such as automated access control, human-computer interaction, and surveillance [1]. For automated access control, the commonly used method is face recognition but the faces might exhibit different Facial Expressions (FE). Face recognition and FE analysis has attracted the interest of researchers in the field of computer vision. Although human being can recognize different FE exhibited by face very easily [2], it is not so for computers. In a survey on automatic Facial Expression Recognition (FER) [3], Scientists conducted experiments in order to recognize the FE from un-occluded facial images taken under controlled

   Facial expression detection and recognition plays an important role in nonverbal communication in Human Computer Interaction. Mehrabian, a psychologist has claimed that verbal languages convey 7 % of the message in face to face communication and facial expressions provide 55 % of the message [7].

   The emotion grouping is a two-class arrangement errand. There are two passionate condition of the individual [8]. In the first place, positive feeling including Happy and Surprise expressions. Second, negative feelings involving Disgust, Unhappy, Fear and Angry expressions. Free facial expression location and acknowledgment is a multi-class grouping undertaking. A few research endeavors have been made for multi class facial expression acknowledgment.

2. SYSTEM ARCHITECTURE

   Baseline Facial Expression Classification system consists of three important blocks. First, face detection or Pre-processing from input image, second, facial feature extraction from cropped face pattern and third, facial expression classification.

   Feature Extraction

   Training

   Load the image from database

   Pre- processing

   Feature

   laboratory conditions.

   But at times, the human subject may be talking, thus changing his facial features or his face may be partially occluded. Rosenblum et al. [4] applied a technique of networks where the complexity of recognizing FE was grouped into three layers of decomposition. In [5], FER in

   Testing

   Select an image

   Classification

   Pre- processing

   Feature Extraction

   Recognized Expression

   the presence of occlusion was investigated. The approach is based on a localized representation of FE features and on the fusion of classifier outputs. Abboud and Davoine [6] suggested a bilinear factorization expression classifier for facial recognition.

   Fig.1. Facial Expression Classification System Block Diagram

   1. Pre-processing:

      Face patterns are extracted by utilizing face finders. Face finders are the Viola-Jones face detector and Kanade- Lucas-Tomasi tracker. The Viola-Jones confront locator utilizes adaboost system. AdaBoost strategies give a basic and viable approach for stage wise learning of a nonlinear order function [9]. AdaBoost takes in few feeble classifiers and lifts them iteratively into a solid classifier of more noteworthy precision.

      Input face image is loaded from the database. Face detection or pre-processing module detects the face in the image data and cropped face pattern. Detected face is normalized first or resized into 40X40 pixels. Face pattern is extracted by using face detector i.e., Viola- Jones face detector.

   2. Facial Feature Extraction

      A facial Feature Extraction strategy gets discriminative elements from the face. The principle target of highlight extraction is to change over expansive volume of pixel information into discriminative parameters. Include extraction enormously lessens the dimensionality of information space. The extricated components are utilized further for arrangement. There are two sorts of highlight descriptors: Global Feature Descriptors and Local Feature Descriptors. Worldwide descriptors depend on Geometry of example [11] though Local descriptors depend on appearance of face example [10].

   3. Facial Expression Classification and Emotion Detection

   Machine learning is modifying PCs to streamline procedure of utilizing known arrangement of information. There are two noteworthy sorts of learning: directed learning and unsupervised learning regarding the info information. In administered taking in, the point is to take in a mapping from the contribution to a yield whose right values are given by an administrator. In unsupervised learning, there is no such chief and we just have input information. There are many machine vision systems for learning and characterization, cases incorporate, K-closest Neighbour, Support Vector Machine, Artificial Neural Networks. The K-Nearest Neighbor algorithm is a non-parametric method used for classification and regression. In both cases, the input consists of k-closest training examples in the feature space. The accuracy of KNN algorithm is degraded by the presence of noisy or irrelevant features [18]. KNN comprises of two sections Training and Testing. For expression discovery six general expressions, for example, Happy, Surprise, Disgust, Unhappy, Fear and Angry are considered.

3. FEATURE EXTRACTION TECHNIQUES

   A facial feature extraction technique computes significant and distinctive features from the face with a view to reduce the amount of data to be processed. The choice of the feature extraction relates to the recognition quality and computational effort, several approaches to extract these facial points from digital images or video sequences of faces were proposed, which falls into two categories: geometry

   and appearance based techniques. The geometric based approach calculates the geometric distance between the extracted facial action units. The significant facial features are extracted by using relative positions and sizes of the components of face. Examples of such algorithms are active shape model [12] and active appearance model [13]. On the other hand, the appearance based technique uses and processes the whole face by using linear transformation and sometimes statistical approaches to find the basic feature vectors to represent the face. Examples of such algorithms are principle component analysis [14], local binary pattern

   [15] etc.

   1. Local Binary Pattern

      LBP is an efficient operator for texture representation of Image pattern [1]. The LBP operator considers neighbours in all eight directions [3]. The LBP code is obtained at position into the input image by comparing the pixel value with the neighbouring pixels. The basic LBP code is 8 bit and so there are total 256 possible LBP values. There are various approaches to generate LBP code for every selected pixel of input image.

      1. Central LBP

      2. Mean LBP

      3. Median-LBP

      4. Gradient-LBP

      5. Laplacian-LBP

         The Local Binary Pattern (LBP) operator is a signified robust method of texture description. The LBP operator deals with eight neighbours of a pixel. The operator labels the pixels of an image by thresholding the 3×3 surrounding neighbourhood of every pixel with the center pixel and representing the outcome as a binary number. In the case, a neighbour pixel has a greater or equal grey value to the center pixel, then a logic high bit "1" is allocated to that pixel, otherwise, it gets the logic bit "0" [5]. The LBP code for the center pixel is formed by the cumulative sum of weighted resultant 8-bit of 1's or 0's to a binary code as shown in Figure 2.

         Example: For a given pixel at (xc, yc), LBP code is obtained using the following equation (1)

         (1)

         Where gc is the gray value of the central pixel and gp is the gray value of its neighbors. As there are 8 neighborhoods for every pixel, total 256 different labels can be assigned depending on the gray values of the pixels.

         Fig. 2. The LBP operator

         LBP [15] involves dividing up the face image to regions. The neighbouring pixels are examined based on the central pixel gray scale value which threshold the neighbours to 1 or 0. Hence a binary string representing each pixel will be formed. Histograms of 256 bins for every region will be build which are concatenated to form the feature vector for the face image. The most efficient LBP features are selected by applying the AdaBoost learning algorithm [16].

         Initially a 3×3 neighbourhoods was defined, giving 8 bit codes based on the 8 pixels around the central one, the operator labels the pixels of an image by thresholding the 3×3 neighbourhood of each pixel with the center value and considering the results as a binary number.

         One of which is the uniform patterns. A uniform LBP contains at most one 1-0 and or one 0-1 transition when viewed as a circular bit string. For example: 00000000, 11111111, 000110000 and 11111001are uniform patterns. The uniform patterns were noticed to account formerly 90% of all patterns in the (8, 1) neighbourhood and for about70% in the (16, 2) neighbourhood in texture images, where the notation (P, R) denotes a neighbourhood of P equally spaced sampling points on a circle of radius of R that form a circularly symmetric neighbour set [4].

   2. Asymmetric Region Local Binary Pattern (ARLBP)

      A single operator alone does not address the issues of discriminative ability, dimensionality of the feature histogram and scalability. To address these issues using a single operator called ARLBP. The ARLBP operator considers average intensities of sub-regions of different sizes around a pixel to be labeled. ARLBP is scalable which can capture dominant feature at larger scales. The length of feature histogram obtained is constant and it is equal to the number of grids the image is divided multiplied by 2P where, P is the number of neighborhood regions of the operator. ARLBP consists of different sized regions which reduce the loss of texture information since average of pixel intensities are calculated for different sizes. The average gray-pixel value of ARLBP regions is calculated using summed area tables, making it computationally efficient.

      Fig. 3. Example of 5×5 block Asymmetric Region LBP

      The ARLBP is coding micro patterns at smaller scales as it is similar to LBP operator at the size of 3 X 3, it also codes macro patterns at larger scales as it is scalable without increasing the feature histogram. Robust to noise as it considers averaging of pixel intensities to label and threshold the pixel values.

   3. Histograms

   A histogram is a graphical representation of the distribution of numerical data. It is an estimate of the probability distribution of a continuous variable (quantitative variable). To construct a histogram, the first step is to "bin" the range of values that is, divide the entire range of values into a series of intervals and then count how many values fall into each interval.

4. PROPOSED FACIAL EXPRESSION CLASSIFICATIONALGORITHM

   Step1. Read the input face image.

   Step2. Detect face from input face image.

   Step3. If face detected, go to step 4. Otherwise go to step 1. Step4. Divide a face image into different regions.

   Step5. Extract LBP features.

   Step6. Feature classification (KNN method) Step7. Recognize expression from classification.

   Fig. 4. Flow Chart for Proposed Methodology

   i. K-Nearest Neighbor Algorithm

   The simplest algorithm for identifying a sample from the test set is called the Nearest Neighbor method. The object of interest is compared to every sample in the training set, using a distance measure, a similarity measure, or a combination of measures. The unknown object is then identified as belonging to the same class as the closest sample in the training set. This is indicated by the smallest number if using a distance measure, or the largest number if using a similarity measure. This process is computationally intensive and not very robust.

   We can make the Nearest Neighbor method more robust by selecting not just the closest sample in the training set, but by consideration of a group of close feature vectors. This is called the K-Nearest Neighbor method, where, for example, K = 5. Then we assign the unknown feature vector to the class that occurs most often in the set of K-Neighbors. This is still very computationally intensive, since we have to compare each unknown sample to every sample in the training set, and we want the training set as large as possible to maximize success.

   We can reduce this computational burden by using a method called Nearest Centroid. Here, we find the centroids for each class from the samples in the training set, and then we compare the unknown samples to the representative centroids only. The centroids are calculated by finding the average value for each vector component in the training set.

   The iteration will stop after K-subclasses were obtained and the initial K-vector will become the means of each sub-classes. Figure 4 will show that each class Ci will be divided into number of sub-classes Ci, j represented by the mean _i, j of the data.

   According to the Euclidean distance formula, the distance between two points in the plane with coordinates (x, y) and (a, b) is given by equation (2)

   dist((x, y), (a, b)) = (x – a)² + (y – b)² (2)

   Fig. 5. Classes, sub-class, representative data

5. IMPLEMENTATION AND RESULTS

   1. Pre-processing:

      • Crop the detected face image

      • Resize the cropped face image

      • Dividing different regions of resized face image

        • Top face

        • Middle face

        • Bottom face as shown in Figure 5.

          Fig. 6. The cropped face image divided into three regions

   2. Feature Extraction:

      Feature extraction is done by using LBP and ARLBP methods, features of the face image are obtained for different regions of face image and these features are combined and concatenated histogram gives features of the face image as shown in Figure 7 and Figure 8.

      Fig. 7. Concatenated LBP Histogram

      Fig. 8. Concatenated ARLBP Histogram

   3. Feature Classification:

      /li>

   The samples are taken from Japanese Female Expression Database (JAFFE).

   • Total number of images in the database: 213

   • For training: 150 images

   • For testing: 70 images

   Table1. Japanese Female Expression Database

   Overall recognition efficiency: 95.051%

   Figure 9 shows that as k value increases the accuracy going to decreases, non-matching cases of images occurs in larger value of k.

   Fig. 9. k value versus accuracy

   The proposed method is compared with other methods in terms of recognition rate is shown in table.

   Table 2. Comparison of Proposed Method with Other Methods.

   References	Method	Accuracy
   Muzammil Abdulrahman, Tajuddeen R. Gwadabe [17]	Gabor wavelet with Similarity measure	88.87%
   Sukanya Sagarika Meher, Pallavi Maben [19]	PCA and Euclidean distance	85%
   M. Pantic, and L. Rothkrantz, Edwards [3]	PCA based on Mahalonobis distance and LDA	74%
   The proposed method	LBP, ARLBP and KNN	95.051%

6. CONCLUSION

The facial expression recognition system shows that the face images are detected and features are extracted by using local binary pattern and Asymmetric region local binary pattern method. The preprocessing stage followed by reducing the dimensionality using LBP and ARLBP and then uses the KNN classifier to predict the emotion. The experimental results show that better performance than the other methods. The proposed method is experimented on JAFFE database.

REFERENCES

1. Z. Niu and X. Qiu, Facial Expression Recognition Based on Weighted Principal Component Analysis and Support Vector Machines, IEEE 3rd International Conference on Advanced Computer Theory and Engineering, pp. 174-178, 2010.

2. N. Vretos, and I. Pitas, A Model-Based Facial Expression Recognition Algorithm using Principal Components Analysis, IEEE 16th International Conference on Image Processing, pp. 3301-3304, 2009.

3. M. Pantic, and L. Rothkrantz, Automatic Analysis of Facial Expressions: The State of the Art, IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 22, No. 12, pp. 14241440, 2010.

4. M. Rosenblum, Y. Yacoob, and L. S. Davis, Human Expression Recognition from Motion using a Radial Basis Function Network Architecture, IEEE Transactions on Neural Networks, Vol.7, No.5, pp.1121-1138, 1996.

5. F. Bourel, C. Chibelushi, and A. Low, Recognition of FacialExpressions in the Presence of Occlusion, Proceedings of the12th British Machine Vision Conference, Vol. 1, pp. 213 222,2001.

6. B. Abboud and F. Davoine, Appearance Factorization for Facial Expression Recognition and Synthesis, Proceedings of International Conference on Pattern Recognition, pp.163166, 2004.

7. Manja Pantic and Leon J.M. Rothkrantz,Automatic Facial Expressions : The state of the Art, IEEE Transaction on Pattern Analysis and Machine Intelligence, Vol 22, No 12 December 2000.

8. Porawat Vistutsak, Emotion Classification through Lower Facial Expressions using Adaptive Support Machine, Journal of Man, Machine and Technology, Vol 2 No 1, June 2013.

9. Bin Jiang and Kebina Jia, Semi Supervised Facial Expression Recognition Algorithm on The Condition of Multi-Pose, Journal of Information Hiding and Multimedia Signal Processing, VOL 4, No3, July 2013.

10. Xinbo Gao, Ya Su Xuelong Li, and Dacheng Tao, A Review ofActive Appearance Models, IEEE Transaction on System, Man andCybernetics, Vol 40, No 2 March 2010.

11. Irene Kotsia and Ioannis Pitas, Facial Expression Recognition inImage Sequences Using Geometric Deformation Features andSupport Vector Machines, IEEE Transaction on Image Processing,Vol 16, No 1, January 2007.

12. T. F. Cootes, C. J. Taylor, D. H. Cooper, and J. Graham, ActiveShape Models, Their Training and Application, ComputerVision and ImageUnderstanding, Vol. 61, No. 1, pp. 38-59, 1995.

13. G. J. Edwards, C. J. Taylor, and T. F. Cootes, Active Appearance Models, IEEE Transaction on Pattern Analysis andMachine Intelligence, Vol. 23 No. 5, pp. 681-685, 2001.

14. C. Liu and H. Wechsler, Independent Component Analysis ofGabor Features for Face Recognition, IEEE Transactions on Neural Networks, Vol. 14, No. 4, pp. 919-928, 2003.

15. T. Ojala, M. Pietikainen, and T. Maenpaa, Multi-resolutionGray Scale and Rotation Invariant Texture Analysis with LocalBinary Patterns, IEEE Transactions on Pattern Analysis andMachine Intelligence, Vol. 24, No. 7, pp. 971-987, 2010.

16. Y. Freund and R. E. Schapire, "A Decision-Theoretic Generalization of On-line Learning and an Application to Boosting," Journal of Computer and System Sciences, Vol.55, No. 1, pp. 119-139, 1997.

17. Muzammil Abdulrahman, Tejuddeen R. Gwadabe, Fahad J. Abdu, Alaa Eleyan, Gabor Wavelet Transform Based Facial Expression Recognition using PCA and LBP, IEEE 22nd Signal Processing and Communication Applications conference(SIU), 2014.

18. Treesa George, Sumi P. Potty and Sneha Jose, Smile Detection from Still Images Using KNN Algorithm, International Conference on Control, Instrumentation, Communication and Computational Technologies, pp. 461-465, 2014.

19. Sukanya Sagarika Meher and Pallavi Maben, Face Recognition and Facial Expression Identification using PCA, IEEE International Advance Computing Conference (IACC). 2014