Lie Detection based on Facial Micro Expression, Body Language and Speech Analysis

Lie Detection based on Facial Micro Expression, Body Language and Speech Analysis

Soumya Barathi C.

Computer Science and Engineering SRM University

Chennai, India.

Abstract Facial micro expressions, body language and analysis of speech are powerful tools of lie detection. Facial micro expression is an involuntary reaction that is impossible to fake. Liars experience stress which manifests itself as body language cues and verbal cues. Facial micro expressions are detected and recognized based on Paul Ekmans research using Principal Component Analysis. The speech input of the subject is subjected to speech analysis and checked if it is consistent with the results of the facial micro expression detector. The speech input is further subjected to stress analysis. This paper also discusses the various body cues exhibited by a liar and proposes a method to identify a subjects body language using the SURF (Speeded Up Robust Features) approach to see if the subject exhibits the body language of a liar.

KeywordsLie Detection; Facial Micro expression;Principal Component Analysis, Speech Analysis;Body Language ;

1. INTRODUCTION

   Lying is a despicable act and can have serious consequences. Lie Detection is of utmost importance especially in areas such as security, terrorism, interpersonal relationships and could result in highly catastrophic outcomes if it goes undetected. Present day lie detectors such as polygraph testing, thermal cameras that detect minute changes in temperature of the persons face are very expensive[9]. This paper discusses a model that is efficient, cost effective and non invasive.

2. LITERATURE REVIEW

   Albert Mehrabian, the author of silent messages, says that only 7% of communication is verbal and the 93% is non verbal(55% being body language,38% tone of voice). Paul Ekman[2], a pre-eminent psychologist, put forth the theory of universality of facial expressions which states that facial expressions are unlearned behaviors that develop independently from cultured expressions. Studies show that congenitally blind people make the same expressions even though they have never seen other peoples faces.

   According to the groundbreaking research done by Haggard, Isaacs and Paul Ekman[2], a micro expression is a fleeting involuntary facial expression that flashes on human faces, that occur as fast as 1/15 to 1/25 th of a second. They often display a veiled emotion and are the result of suppression. Contrary to normal facial expressions it is

   difficult to conceal a micro expression and hence they are a reliable tool for lie detection.

   According to the Black Book Of Lie Detection by Martin Soorjoo [5] body language gives away the emotions we experience and liars experience stress which manifest itself through body cues. According to Carl Williams and Kenneth Stevens psychological stress affects speech. This can be measured using a Voice Stress Analyzer[1].

   According to a 2002 study conducted by the University of Massachusetts, 60% of adults cant have a ten minute conversation without lying at least once. Many[9] different types of day lie detectors have been developed using a variety of techniques. A polygraph detects lies based on several physiological indices such as blood pressure, pulse, respiration and skin conductivity during an interrogation. However according the book The Lie behind the lie detector by George W Machke and Gino J. Scalabrini several counter measures are available to cheat a polygraph test[3].

3. PROPOSED MODEL

   The proposed model consists of the following components:

   1. Facial Micro Expression detector

   2. Body Language Analyzer

   3. Speech analysis during interrogation

   Facial Micro expression Detection:

   According to Dr Paul Ekmans research on facial micro expression (Facial Action Coding System) [2]FAAC fear,anger, happiness, surprise, disgust, contempt and sadness have been recognized as facial micro expressions. Fig 1 lists the typical characteristics of the micro expressions.

   Since facial micro expressions occur as fast as 1/15 to 1/25th of a second it is essential to use a high speed camera that captures at least up to 30 frames per second. The video is broken down to a set of frames. Facial Micro Expression is detected using the traditional Principal Component Analysis technique. The speech input is converted to text by a speech to text convertor and is sent for speech analysis. The result of the speech analysis is compared with the results of the facial micro expression detector and is checked for consistency.

   Testing Input :-

   A high speed camera that captures at least up to 30

   Fig. 1. Types Of Facial Micro frames per second is used to capture facial mico

   xpression expressions that occur as fast as 1/15 to 1/25 th of a second.

   Testing Input :-

   A high speed camera that captures at least up to 30

   Fig. 1. Types Of Facial Micro frames per second is used to capture facial mico

   xpression expressions that occur as fast as 1/15 to 1/25 th of a second.

   e

   Fig. 2. Facial Micro Expression detector coupled with Speech Analyzer.

   Fig.2 represents the data flow diagram of Facial Micro Expression Detector coupled with Speech Analyzer.

   Pre Processing :-

   The visual appearance of image is subjected to gray scale conversion, noise removal and enhancement.

   Face Detection and Feature Extraction :-

   Facial region is detected and feature extraction is done to reduce the dimensionality of the input space. Principal Component Analysis is used to recognize statistical pattern for data reduction. These extracted features have great role in distinguishing input patterns [15].

   Covariance matrix is calculated after subtracting the mean after which eigen vectors and eigen values are calculated. The components are chosen and feature vector is formatted. The image is compared with the training image by calculating the Euclidean distance between these feature vectors and the facial expression is classified[8].

   Speech Analysis:-

   The interrogation of the subject is recorded and the recorded speech is converted to text. The converted text is analyzed for verbal cues that are exhibited by a liar and the emotional state of the subject is recognized using the dictionary approach.

   The results of the speech analyzer is checked for consistency with the results of the facial expression detector. If the results are found to be inconsistent the subject is identified as a liar.

   Body Language Analysis:

   According to The Black Book of Lie Detection by Martin Soorjo the stress experienced by a liar usually manifests itself through body language cues.

   Some of the most prominent examples of typical body language of a liar are listed below[5][7]:

   • Hand to face/Hand to mouth gestures increase when a person lies.

   • Touching their nose is a very important cue. The opening of capillaries caused by a rush of adrenaline makes a persons nose itch.During the testimony of Bill Clinton over the Monica Lewinsky affair it was observed that when he was believed to be lying he touched his nose every once in 4 minutes reaching a total of 26 times.

   • Placing of the hand near or over their mouth.

   • Getures such as biting their lips, rubbing their hands together, fidgeting.

   • Clenched fist and crossed arms usually indicate dishonesty.

   The system is trained with different typical body language cues exhibited by a liar and those exhibited by an honest person. The images exhiiting the body language of a liar and an honest person are subjected to Limb Action Model Convertor[13] .

   Fig. 3.Block Diagram Of Speech Analyzer

   Fig. 3.Block Diagram Of Speech Analyzer

   Local Keypoints and Descriptors are found in the training images.The video input of the suspect taken with the help of Kinect is subjected to Limb Action Model Convertor.SURF[4][6](Speeded Up Robust Features) approach is used to classify the testing images.

   Analysis Of Speech :

   Speech of the subject recorded during the interrogation is converted to text by using a speech to text convertor. The converted text is then subjected to speech analysis. According[7] to Carol Kinsey Goman typical verbal cues of liars include rambling, selective wording (in which one avoids answering the question exactly as asked), stammering, and the use of qualifiers (To the best of my knowledge. I could be wrong . . . ). Its also been noted that liars use fewer contractions: I did not rather than I didnt . . . and that they excessively use fillers like um,uh,e.t.c.The converted text is checked for such verbal cues.Fig 3 represents the block diagram for the speech analyzer.

   The Filler Detector detects fillers like um,uh,e.t.c.The qualifier detector detects qualifiers and a detector that detects gulping,clearing of throat ,stammering. Emotional states exhibited in the conversation is also analysed by the dictionary approach[14] and is checked if they are consistent with the micro expression detector. As micro expressions cannot be faked if the results of the speech analysis are not consistent with the micro expression it can be concluded that the subject is lying.

   Voice Stress Analyser :- According to Carl Williams and Kenneth Stevens[1] attempts at deception is an emotional situation that affects the respiratory pattern. The change in the respiratory rate in turn affects various components of speech which can be detected and measured by a voice stress analyzer[11].Some of the modern day voice stress analyzers use LVA the Layered Voice Analysis technique.

   Fig. 4. Block Diagram Of The Proposed Model

   Fig. 4. Block Diagram Of The Proposed Model

   Implementation:

   The facial expression detector part of the proposed model is implemented by MATLAB. The training images are subjected to preprocessing and feature extraction. The input video is broken down in to a set of frames and is subjected to pre processing and feature extraction. The classifier classifies images based on the emotion.

   The speech analyzer is implemented by using Java in BlueJ environment.The speech recorded during interrogation is converted to text and the converted text is checked for fillers,qualifiers, gulping,clearing of throat

   ,stammering etc.If such suspicious verbal cues are found to be frequently used then the user is notified.

   The speech analyzer also recognizes the emotional state that the user conveys during the interrogation by using the dictionary approach [14].

   The emotional state conveyed by the user is checked for consistency with the type of micro expression. If the results do not match then the user is notified that the subject might be lying.

4. RESULTS AND TESTING

   Fig. 5. Micro Expression Detector Program Snippet

   Fig. 6.Speech Analyzer Program Snippet

   Fig. 7. Screen shot Of the Output Of Facial Micro expression Detector when Vanessa Van Edwards video on fear micro expression is fed as input[10].

   Input: A video posted by Vanessa Van Edwards [10] is fed as input with an audio input that says I did not steal it.I do not know anything about ..uh..the theft.<Clears Throat>I was not in..uh.. town when the theft happened .I ..um..came home this morning and was surprised to find it missing.

   Output :

   Facial Microexpression Detector Result : Fear Speech Anayser Result:

   Number of Sentences : 4 Number Of Fillers : 3 Use of Suspicious verbal cues found.

   Excessive Use Of Fillers.

   Sentiment Analysis Result : Surprise Inconsistent with the facial microexpression.

   Fig. 8. Screen shot Of the Output Of Speech Analyzer

   The Contingency Table showing the results of the facial microexpression detector and the Speech analyzer are shown in Table 1 and 2 respectively.

   Efficiency Of The System:

   TABLE I. CONTIGENCY TABLE/CONFUSION MATRIX OF THE RESULTS OF THE FACIAL MICRO EXPRESSION DETECTOR

   The facial microexpression detector is found to be 82% efficient .Out of the 100 video clippings given as input 82 were identified correctly.

   TABLE II. CONTIGENCY TABLE/CONFUSION MATRIX OF THE RESULTS OF THE SPEECH ANALYZER

   This can be detected by comparing the displacement of head between different frames captured by a high speed camera and if detected the direction of displacement is noted to find if the subject is nodding or shaking his or her head from side to side. The verbal output from the speech to text convertor is to be analyzed and if found to be inconsistent with the head gestures, it can be concluded that the subject is lying. Blink rates of a person increases up to eight times the normal rate after uttering a lie. This can also be detected by comparing the displacement of the eyelids across different frames of the video of the subject.

   The system can be made more efficient by training it to accommodate new discoveries in behavioral studies. The future work also includes working towards building a portable non invasive lie detector that is cost effective and efficient.

   The speech analyzer is found to be 92% efficient. Out of 100 inputs 92 were identified correctly.

5. CONCLUSION AND FUTURE WORK

In this paper a system that detects a liar based on facial micro expression, body language and speech analysis of the interrogation has been discussed. The facial micro expression detector of efficiency 82% and the speech analyzer of efficiency 92% have been successfully implemented. The result of the speech analyzer is checked for consistency with that of the facial micro expression detector. The user is notified in case the results do not match. The system proves to be efficient, robust and cost effective .It can be applied in various areas such as security, investigations,anti-terrorism and interpersonal relationships.

A body language analyzer that compares the gestures made by the subject during interrogation with the typical body language of a liar and notifies the user if they are similar is to be constructed. Images representing typical body language cues of a liar and an honest person are to be used as training images. The video clipping of the subject captured by a Kinect is to be converted to a set of frames and the images are to be subjected to Limb Model Convertor[13].Key Descriptors and features are to be identified in the training images and testing images. SURF[4][6](Speeded Up Robust Features) approach is to be used to classify the testing images.

The images are to be matched using K-Means algorithm. If the gestures made by the subject match body language cues of a liar then the user is to be notified.

According to the article 12 Ways to spot a liar at work by Carol Kinsey Goman[7] incongruence between what a person is saying and his/her gestures occurs due to psychological stress under gone while lying. Side to side head shake while saying yes and nodding while saying no are typical examples.

REFERENCES

1. Mahmoud E. Gpdallah, Matar A. Matar, Ayman F. Algezawi Speech Based Automatic Lie Detection 16th National Radio Science Conference , Cairo, Egypt NRSC'99 , Feb. 23-25, 1999

2. Ekman Paul Emotions Revealed New York : Henry Holt and Co (2003)

3. George W Maschke and Gino J.Scalabrini,The Lie Behind the lie detector, 4th Edition 2005

4. Herbert Bay a , Andreas Ess a , Tinne Tuytelaars b , and Luc Van Gool Speeded-Up Robust Features (SURF)

   Computer Vision and Image Understanding Volume 110 Issue 3, June, 2008

5. Martin Soorjoo, Th Black Book of Lie Detection,2009

6. Jacob Toft Pedersen, SURF: Feature Detection and Description,

   Q4 2011

7. Carol Kinsey Goman ,12 Ways to spot a liar at work ,April 11,2012

8. Akshat Garg , Vishakha Choudhary , Facial Expression Recognition Using Principal Component Anaysis (IJSRET),2012

9. Michel Owayjan, Ahmad Kashour, Nancy Al Haddad, Mohamad Fadel, and Ghinwa Al Souki, The design and development of a lie detection system using facial micro-expressions, 2nd International Conference on ACTEA :, 2012

10. Vanessa Van Edwards. Science Of People-A Guide to reading microexpressions,2013

11. Vijay P. Patil*, Krishna Kant Nayak** and Manish Saxena Voice Stress Detection IJEECE 2013

12. Gaurav B. Vasani, Prof. R. S. Senjaliya, Prajesh V. Kathiriya, Alpesh J. Thesiya, 5Hardik H. Joshi, Human Emotional State Recognition Using Facial Expression Detection, 2013

13. Jing Du, Dongfang Chen 3D Action Recognition Based on Limb angle model by,2014.

14. Reshma Bhonde, Binita Bhagwat, Sayali Ingulkar, Apeksha Pande ,

    Sentiment Analysis Based on Dictionary Approach ,2015

15. Murali Krishna kanala1 , CH.D. Umasankar,Murali Krishna kanala, Efficient Facial Expression and Face Recognition using Ranking Method ,Int. Journal of Engineering Research and Applications,June 2015