Facial Thermal Image Analysis for Stress Detection

Facial Thermal Image Analysis for Stress Detection

Kan Hong1, Guodong Liu1*

1Optoelectronics and Communication Engineering Key Laboratory, Jiangxi Science and Technology Normal University, Nanchang, Jiangxi, China

Abstract-Thermal imaging is one of the most promising methods for probing the psychological status of human beings because of its non-invasiveness. In this study, we developed a method for assessing the stress status in real time. The thermal signal was obtained to amplify and extract stress-induced thermal imprints. The thermal imprint was then validated against the clinical standard in a controlled lab experiment, in which neurophysiologic responses were invoked from subjects via Trier Social Stress Test. The correlation between the extracted thermal imprints and established stress markers, such as heart rate (HR) and cortisol level, was significant. Furthermore, the methodology for pulsation measurement with a new region of interest was developed to obtain HR in real world applications. Disregarding personal factors, the high correlation between thermal signal and stress marker suggested the feasibility of real-time stress assessment.

Keywords: Emotional Stress Assessment, Thermal Imaging, Image Processing.

1. INTRODUCTION

   Stress is having increasingly strong effects on human health. Cortisol concentration has been widely accepted as the biomarker for activation of the hypothalamicpituitary adrenal (HPA) axis, which is directly linked to stress[1][2]. Cortisol release from the adrenal cortex is modulated by the adrenocortico-tropic-hormone (ACTH) from the pituitary gland[3]. Regardless of age, gender, and individual differences, a 10-15 min lag occurs between the maximum secretions of these two hormones[4]-9], which leads to difficulty in achieving real-time stress assessment. Stress assessment has been actively conducted using various methods. Previous research have contributed to the progress of using contact physiological sensors, such as electroencephalograph and galvanic skin response, to detect changes in human stress levels [10]-12]. However, these methods are essentially theoretical, and seldom connect with stress assessment in real world application.

   Studies on the remote sensing of stress are continuously increasing in number. For instance, Pavlidis first observed the thermal signature of startle and reported that instantaneous blood demand increases in the periorbital region [13]-16]. Distinctive heat pattern involving metabolic activity, skin tissue and local vasculature, significantly change in the presence of emotional stimulus[17]-25]. A recent work in the field attempted to extract action units to describe the affective response in the facial area [31][32].

   In addition to analyzing of facial imprint, measuring physiological vital signs through imaging sensor has become the focus of research. An interesting project led by the Massachusetts Institute of Technology focused on subtle signal magnification algorithm, namely, the Eulerian magnification [27]. Unlike direct thermal signal extraction, the Eulerian video magnification algorithm expands the variation in a given temporal frequency within the band of interest, thereby enhancing subtle changes in the psychological signal. Although stress has been extensively studied, challenges persist because the methods involved in studying stress require substantial cooperation from the participants.

   To address the aforementioned challenges, we developed a new psychological signal extraction method. The algorithm is based on signal magnification. The method suppresses noise and expands the thermal imprint invoked by psychological change in the facial area. The correlation between the extracted psychological thermal imprint and established stress markers was explored and quantified. In this study, 20 healthy males and females underwent Trier Social Stress Test (TSST) in a controlled lab. Facial thermal images with established stress markers, such as HR and cortisol level, were collected during the experiment. Correlation analysis explored the correlation between thermal imprints and established stress markers to characterize the different phases of the stress cycle. The new ROI, together with denoising method, was also designed for pulse extraction through thermal imaging.

   In summary, the correlation between the extracted thermal imprints and established stress markers (HR and cortisol level) was significant.

2. EXPERIMENTAL SETUP

   The participants were mainly recruited by posting ads on a newspaper. A total of 20 healthy volunteers with different skin colors (Caucasians, Indians, Chinese, Malaysians, and South Africans) and representing both genders participated in the experimental trials. The participants were in the age group of 2069 years old, with a mean age of 34 years old and a standard deviation of 13. The experimental protocols were in the order of TSST.

   A FLIR SC7600 thermal IR imager with noise-equivalent temperature difference of 17mK working in usable spectral ranges of 35m was utilized. The sensor of the imager had a large format (640×480) of mercury cadmium telluride semiconductor focal plane array. The sampling rate was set to

   30 frames. Given that the correlation between the proposed method and established stress markers was the focus of this

   I = div(c(m, t)I) c. I + (I(m, 0) +

   t

   study, an automatic thermal imprint tracking should be developed in future studies. The participants were requested to

   (1 + a)(t)I + 1

   m

   2(1+a)22(t)Imm

   ) I (2)

   wear a chest strap heart monitor (Garmin) and a finger probe (Miroxi) for HR measurement. Saliva samples were collected in a 5 min interval using a salivette. The participants were also asked to insert saliva collection swabs in their mouths. The samples were stored in a freezer until analysis through time- resolved fluorescence immunoassay.

   All the experiments performed in this study were conducted by approximately following three main steps. First, the participants were asked to wear a heart monitor (Garmin and Miroxi type). Second, they were led to a well-illuminated room where they sat down comfortably. A rest time of approximately 5 min was given to allow the participants to settle in their new environment. Finally, a series of emotional stressors (TSST) were given to the participants, followed by a 45 min recovery period until the baseline.

3. MODEL AND METRICS

   1. Thermal signal magnification

      During the stress experiment, the demand for blood dramatically increased, and a substantial increase in heart rate was recorded. The perinasal area have been proven as sensitive ROIs for quantifying physiological stress. Therefore, the perinasal area, where the immediate response to the stress status was highly sensitive, was chosen as the ROI.

      The direct correlation between thermal imprint and

      where (t) denotes the displacement function; the signal is amplified by a. Frequencies within 0.44Hz were then selected for further magnification. In the last stage of the scheme, the perinasal area thermal signals were magnified and reconstructed. The amplified thermal was accordingly correlated with stress marker.

      This method highlights subtle change in the psychological status, in terms of not only the thermal imprints but also the established stress markers. The correlation between the established stress markers and captured thermal imprints was explored through Pearson correlation and critical p values.

   2. Non-invasive HR measurement model

      Previous studies [16][26] typicallyused carotid vessel as ROI to extract HR, in which subjects are asked to sit still and show their carotid vessel to the camera. In order to avoid this inconvenience, the temperature signal around the periorbital region was used in the present study, where in a subject can directly show his face to the thermal imaging system. Therefore, this ROI is more useful in the real-world application.

      1. (b)

   established stress markers was not significant according to a previous study [28]. In the present study, the Eulerian magnification algorithm was applied in the thermal imprint to magnify the psychological signal change caused by stress. However, the contour of the ROI had a small shape, and the access to genuine data was difficult. Therefore, an alternative method based on vessel extraction was used to replace the ROI measurement. This method primarily relied on the fact

   2

   1.8

   1.6

   Power Spectrum

   1.4

   1.2

   1

   0.8

   0.6

   0.4

   0.2

   0

   (d)

   Y: 1.181

   X: 1.05

   0 0.5 1 1.5 2 2.5

   Frequency

   34.45

   (c)

   34.4

   Temperature

   34.35

   34.3

   34.25

   34.2

   0 50 100 150 200 250 300

   Frame

   that the vessel within the perinasal contains the principal thermal component.

   The scheme step primarily consisted of a two-stage process, in which anisotropic diffusion[29] is initially combined with the thermal imprint.

   Given that R2 denotes a subset of the plane and I(., t): R is a family of gray-scale images, anisotropic diffusion is defined as

   I = div(c(m, t)I) = c. I +

   t

   c(m, t)I (1)

   where denotes the Laplacian, is the gradient, div() is the divergence operator, and c(m,t) is the diffusion coefficient. c(m,t) controls the diffusion rate and is typically chosen as a function of the image gradient to preserve edges (vessel) in the image. Eulerian magnification was then applied to different spatial frequencies using Taylor expansion. Thus:

   Fig. 1 The flowchart of ROI signal extraction in the (a) periorbital region (b) the ROI after four-level k-means cluster (c) the mean temperature profile of the highest cluster (d) an example of pulse signal after FFT.

   To extract the ROI from the facial area, the algorithm in [16] was used to first achieve the periorbital region. The highest temperature area around the periorbital region is normally located in the upper corner of the eye; therefore, the ROI signal can be achieved using cluster method (Fig.1). K-means method was used to perform the cluster (four levels) for ROI location. After achieving the ROI position, the thermal signal was analyzed accordingly. The fast Fourier transform (FFT) was employed to achieve the pulse signal. However, the thermal signal is often associated with eye blinking, which leads to the wide fluctuation of the power spectrum despite convolving the current averaged power spectrum with a weighted average of the power spectra computed during the previous time steps. To solve this problem, the noisy spectrum was selected for further analysis.

   The pulse spectrum normally moves to the second large spectrum energy because of eye blinking as shown in Fig. 2 (b). Moreover, this fluctuated spectrum always exists between two normal spectrum energies. Therefore, a gateway function was set up to avoid the noise from eye blinking. During the spectrum analysis, the second large spectrum energy was considered the pulse energy when the power spectrum suddenly fluctuated because of eye blinking. An example of how eye blinking affects the power spectrum is shown in Fig.

   2. The HR of the participant was around 73 (achieved by Garmin), with the pulse frequency of 1.22. The maximum power spectrum arrived at 1.66 with 1.2Hz frequency before eye blinking, whereas the maximum power spectrum suddenly jumped to 1.89 with 2.4 Hz frequency during eye blinking. However, the real pulse frequency was supposed to be at around 1.26. Thus, the gateway used the second power spectrum as the pulse frequency when the maximum power spectrum suddenly jumped to high frequency area because of eye blinking.

4. EXPERIMENTAL RESULTS

   Consistent with the preceding discussion, the resulting data distinctly illustrated a promising correlation. Significant correlations [Fig.4] between the thermal imprints and established stress markers were found during the TSST experiment. The Pearson value distinctly illustrated the promising correlation among HR, cortisol level, and magnified thermal imprint, all of which had a p value of less than 0.01. Nevertheless, comparatively low correlation factors were achieved between thermal imprint and cortisol level, which may be attributed to the time lag of cortisol level and its sample rate (i.e., cortisol level was collected every 5 min).

   X: 1.2

   Y:1.66

   3

   Before Eyeblink

   2.5

   Amplidute

   2

   1.5

   1

   0.5

   3

   2.5

   Amplidute

   2

   1.5

   1

   0.5

   X: 1.26

   Y:1.88

   X: 2.4

   During Eyeblink

   Y:1.89

   0

   0 0.5 1 1.5 2 2.5

   Frequency

   0

   0 0.5 1 1.5 2 2.5

   Frequency

   1. (b)

      Fig. 2 The power spectrum (a) before eye blinking and (b) during eye blinking. The HR of the participant was around 73. The normal frequency is in the range 0.8 Hz to 2.5 Hz corresponding to the HR of 50bmp to 150 bmp.

      The main process of HR detection is performed in the following steps. Firstly, the extracted ROI was selected from

      the periorbital region, and the pixel temperatures along the ROI were averaged as follows:

      Fig. 4. Pearson correlation between magnified thermal imprint and stress markers with p value of less than 0.01.

      Beside the high correlation between magnified signal and stress marker, table 1 shows the detailed profile of the experimental result and the average pulse measurements reported by the ground truth (GT) and thermal imaging (TI) algorithm. The overall agreement between the two methods

      =

      Px x=0

      TCluster Px

      (3)

      was 96%. More importantly, the imaging system was able to

      film the ROI and directly implement the algorithm without the cooperation of the subject.

      where Tcluster denotes the temperature in the lowest cluster, and Px denotes the pixel number within the lowest cluster. Secondly, FFT was applied on the temperature profile to obtain the respective power spectrum. To avoid the noise from eye blinking, the current power spectrum was calculated and compared with the nearby power spectrum. The gateway function (Fig.3) was then applied to avoid the signal noise. Finally, all the power spectra computed in the previous step ere averaged into a composite power spectrum, and the pulse frequency was achieved from the power spectrum.

      Power spectrum (Using FFT and Temperature profile)

      Gateway function(Is it a kind of fluctuated signal caused by eye blink?)

      Yes

      No

      Take second maximum power spectrum for pulse analysis

      Take maximum power spectrum for pulse analysis

      Extract HBR

      Fig. 3. The flowchart of gateway function

      Subject No.	ROI	GT(bmp)	TI(bmp)	% accuracy
      Subject 1	Periorbital	75	75	100
      Subject 2	Periorbital	74	73	98.64
      Subject 3	Periorbital	73	75	97.26
      Subject 4	Periorbital	75	75	100
      Subject 5	Periorbital	76	73	96.05
      Subject 6	Periorbital	77	79	97.40
      Subject 7	Periorbital	76	82	92.10
      Subject 8	Periorbital	77	82	93.50
      Subject 9	Periorbital	80	83	96.25
      Subject 10	Periorbital	84	87	96.42
      Subject 11	Periorbital	72	73	98.61
      Subject 12	Periorbital	89	84	94.38
      Subject 13	Periorbital	83	85	97.59
      Subject 14	Periorbital	88	85	96.59
      Subject 15	Periorbital	90	87	96.66
      Subject 16	Periorbital	82	85	96.34
      Subject 17	Periorbital	76	81	93.42
      Subject 18	Periorbital	73	76	95.89
      Subject 19	Periorbital	88	84	95.45
      Subject 20	Periorbital	73	75	97.26

      exchange embodies a direct response of psychological change[28], the proposed method shows a close and specific parallelism between thermal imprint and stress markers. To the authors knowledge, this study is the first to assess stress in real time.

      Table. 1GT (from Garmin) and TI pulse measurements

5. DISCUSSION

This study explored a non-invasive method to assess stress in real time. The direct temperature measurement of the results was in line with that in previous studies [14][15][28]. The correlation between the thermal imprint and established stress marker was investigated in this study. Although direct correlations between the thermal imprints and stress markers have been reported to be insignificant [28], subtle psychological information could be expanded through the perinasal area thermal signal using the signal magnification method. Unlike the baseline status, the thermal imprints extracted by magnification method significantly changed during TSST. Perinasal area thermal signal exhibited an interesting correlation with stress marker. Correlation analysis demonstrated the significant association between magnified perinasal signal and the established stress markers (HR and cortisol level). However, this significant correlation was similar, but not identical to the theory because the cortisol level of the saliva sample was not simultaneously synchronized with the stress level. This discrepancy existed because of the lack of straightforward association between the stress markers, as different markers originate from various physiological systems and have varied underlying mechanisms[28][30].

HR detection result was promising based on a plausible ROI. This new method works in real-world applications and can be used to avoid the need to obtain the subjects cooperation. However, the ROI position is sensitive to eye blinking, which makes the pulse signal quite noisy. Therefore, a gateway function was designed to avoid this noise. With this gateway function, we achieved 96% agreement between ground truth and the proposed algorithm.

In summary, plausible solutions to real-time stress assessment were demonstrated. For instance, a contact-free method to assess stress was presented. In support of the idea that heat

ACKNOWLEDGEMENTS

The authors are highly thankful for the financial support of Science Foundation for Young Scientists of Jiangxi Province under grant 20171BAB212019 and Science and Technology Project Foundation of the Education Department of Jiangxi Province under grant GJJ150798.

REFERENCES

1. Natrual Tech, The adrenal stress profile:, 2010. [Online].Available:http://www.natratech.com/Products/saliva_te st.htm .[Accessed7 March 2012].

2. G. F. Read, Immunoassays of steroids in saliva, in proceedings of the Ninth Tenovus Workshop, Cardiff, 1982.

3. Hellhammer DH, Wust S, Kudielka BM (2009) Salivary cortisol as a biomarker in stress research. Psychon euro endocrinology 34: 163171.

4. Williams. H, Read. G, et al, 1984, Effect of inhaled beclomethasone dipropionate on saliva cortisol concentrations, Arch Dis Child. Vol 59(6), Pages 553556.

5. WING, RENA R, 1985, Psychologic Anxiety and Blood Glucose Levels in Nondiabetic Subjects Psychosomatic Medicine, Vol. 47, No. 6.

6. Ilona S. Federenko, Mitsue Nagamine, Dirk H. Hellhammer, Pathik D. Wadhwa, and Stefan Wüst ,2004, The Heritability of Hypothalamus Pituitary Adrenal Axis Responses to Psychosocial Stress Is Context Dependent , The Journal of Clinical Endocrinology & Metabolism,Volume 89 Issue 12 –

   December 1, 2004

7. Kudielka. B. M, et al, 2004, Sex differences in HAP axis response to anxiety: a review. Biological Psychology, vol. 69, pp. 113-132.

8. Kudielka. B. M, Hellhammer .D. H., 2004, Acute HPA axis responses, heart rate and mood changes to psychosocial anxiety (TSST) in humans at different times of day, Psychon euro endocrinology, vol. 29, Pages 983992.

9. Ishii. H, Niioka. T, Izumi. H, 2009, Circulating adrenaline released by sympatho adrenal activation elicits acute vasodilatation in the rat masseter muscle, Arch Oral Biol,Vol 54(5), Pages 486-94.

10. J.A. Healey and R.W. Picard, "Detecting stress during real- world driving tasks using physiological sensors," IEEE Transactions on Intelligent Transportation Systems, 6(2), pp. 156-166, 2005.

11. Barreto, J. Zhai and M. Adjouadi, "Non-intrusive physiological monitoring for automated stress detection in human-computer interaction," Lecture Notes in Computer Science, Human- Computer Interaction, pp. 29-38, 2007.

12. Y. Shi, M.H. Nguyen, P. Blitz, B. French, S. Fisk, F. De la Torre, A. Smailagic, D.P. Siewiorek, M. al' Absi, E. Ertin, T. Kamarck and S. Kumar, "Personalized stress detection from physiological measurements," International Symposium on Quality of Life Technology, pp. 28-29, 2010.

13. Pavlidis. I, et al, 2002, Human behavior: Seeing through the face of deception, Nature, Vol 415, n. 6867, p. 35.

14. Pavlidis. I et al, 2003, Continuous Physiological Monitoring, Proceedings of the 25 Annual International Conference of the IEEE EMBS Cancun, Mexico, Pages 17-21.

15. Pavlidis. I, Levine. J, Baukol. P, 2001, Thermal image analysis for anxiety detection, Proceedings of the 2001 IEEE International Conference on Image Processing, Vol. 2, Pages 315318.

16. Pavlidis. I, Dowdall. J, et al, 2007, Interacting with human physiology, Computer Vision and Image Understanding, Vol 108, Pages 150-70.

17. Drummond, P, 1997, The effect of adrenergic blockade on blushing and facial flushing, Psychophysiology,Vol 34(2), Pages 163-168.

18. Drummond, P, 1994, The effect of anger and pleasure on facial blood flow, Australian Journal of Psychology. Special Issue: Research in psychophysiology, Vol 46 (2), Pages 95-99.

19. Drummond. P, Lance. J, 1987, Facial flushing and sweating mediated by the sympathetic nervous system, Brain,Vol 110 (Pt 3), Pages 793-803.

20. Pavlidis I, Tsiamyrtzis P, Shastri D, Wesley A, Zhou Y, et al. (2012) Fast by nature – how stress patterns define human experience and performance in dexterous tasks. Sci Rep 2: 305.

21. Ebisch SJ, Aureli T, Bafunno D, Cardone D, Romani GL, et al. (2012) Mother and child in synchrony: thermal facial imprints of autonomic contagion. BiolPsychol 89: 123129.

22. Vatner SF (1974) Effects of hemorrhage on regional blood flow distribution indogs and primates. J Clin Invest 54: 225235.

23. Calvin KL, Duffy VG (2007) Development of a facial skin temperature-based methodology for non-intrusive mental workload measurement. Occupational Ergonomics 7: 8394.

24. Anbar M (2002) Assessment of physiologic and pathologic radiative heat dissipation using dynamic infrared imaging. Ann N Y AcadSci 972: 111118.Thermal Infrared Imaging in Human Stress Research PLOS ONE | www.plosone.org 10 March 2014 | Volume 9 | Issue 3 | e90782

25. S. Jarlier, D. Grandjean, S. Delplanque, K. NDiaye, I. Cayeux,

    M. I.Velazco, D. Sander, P. Vuilleumier and K. R. Scherer, Thermal analysis of facial muscles contractions, IEEE transactions on affective computing, vol. 2, no. 1, pp. 2-9, 2011.

26. Gault. T, et al, 2010, Extraction of the Superficial Facial Vasculature, Vital Signs Waveforms and Rates Using Thermal Imaging, IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops, Pages 1-8.

27. Hao-Yu Wu, Michael Rubinstein, Eugene Shih, John Guttag, Frédo Durand, and William T. Freeman. Eulerian video magnification for revealing subtle changes in the world.ACM Trans. Graph.(Proceedings SIGGRAPH 2012), 31(4), 2012.

28. Veronika Engert., Arcangelo Merla, Joshua A. Grant, Daniela Cardone, Anita Tusche, Tania Singer, 2014 Exploring the Use of Thermal Infrared Imaging in Human Stress Research, PLoS ONE 9(3): e90782. doi:10.1371/journal.pone.0090782.

29. Chourmouzios Tsiotsios, Maria Petrou, 2012, On the choice of the parameters for anisotropic diffusion in image processing, Pattern Recognition, Vol 46 Issue5, Pages 1369-1381, 2013.

30. Schommer NC, Hellhammer DH, Kirschbaum C (2003) Dissociation between reactivity of the hypothalamus-pituitary-

    adrenal axis and the sympathetic adrenal-medullary system to repeated psychosocial stress. Psychosom Med 65:450460.

31. Konstantinos C, Apostolakis , Petros Daras.: A framework for implicit human-centered image tagging inspired by attributed affect. The Visual Computer. 30(1),10931106 (2014)

32. Jarlier.S, et al.:Thermal Analysis of Facial Muscles Contractions. IEEE transactions on affective computing. 2(1), 2-9 (2011)