Fusion Techniques For Thermal And Visual Face Images

Fusion Techniques For Thermal And Visual Face Images

Navya S Babu

Amrita School of Engineering, Department of CSE, Coimbatore, Tamil Nadu, India.

P Bagavathi Sivakumar Assistant Professor Department of CSE,

Amrita School of Engineering, Coimbatore, Tamil Nadu, India.

Abstract

Image Fusion is the process of combining relevant information from two or more images into a single image. This study presents a concept of image pixel fusion of visual and thermal faces, which helps to improve the overall performance of a face recognition system. Several factors are there which affect the face recognition performance such as pose variations, facial expression changes, occlusions etc. So, image pixel fusion of thermal and visual images is a solution to overcome the drawbacks present in the individual thermal and visual face images. Human face recognition is a challenging task and its domain of applications is very vast, covering different areas like security systems, defense applications, and intelligent machines.

Keywords: image fusion, thermal image fusion, face recognition

1. Introduction

   In this work, fusion of thermal and visual face images is done. Like fusion of test images that are of the same scene here visual quality is not important. The fused image is expected to have more information than the input images. Due to lack of illumination, acquisition of visible light imagery is not practical during night- time. Thermal imaging, acquires mid-wave infrared or long-wave infrared radiation naturally emitted by the human body. It can be utilized in low-light conditions to perform surveillance tasks. Identification of individuals which is captured by thermal imaging would significantly enhance night-time intelligence gathering capabilities[10]. In this paper study of

   various fusion techniques is done on the thermal and visual face images and compare the performances.

   The required quality of the fused image varies with each application and hence, image quality metrics allows to choose from the various fusion techniques, the best one for application. This paper is a brief study of different fusion techniques available, which includes the basic fusion methods (Average method, Select maximum, PCA), pyramid methods-(Morphological Pyramid, Gradient Pyramid) and Wavelet methods (DWT).

   Image fusion methods are broadly classified into two spatial domain fusion and transform domain fusion [2],[3]. The fusion methods like Averaging method, Select Maximum method, Select Minimum method and Principal Component Analysis method are the commonly used spatial domain fusion methods. However, the spatial domain approaches suffer from spatial distortion in the fused image. This spatial distortion becomes a negative factor, when the fused image is used for further processing, such as classification of the fused image. This was overcome by the advent of spectral domain fusion methods. Spectral domain methods include Pyramid based methods and Wavelet based methods.

   Oliver Rockinger et.al in [3] proposed image sequence fusion using Shift- Invariant wavelet transform. In this the input images must be decomposed into a shift invariant wavelet representation. Then compute the wavelet transform for all possible shifts of the input images. The merits include improved temporal stability and consistency of fused image. The data sets used here are image sequences, still images etc [3].

2. Importance of Thermal Images

   Thermal imaging [4] is a method of improving visibility of objects in a dark environment by detecting the objects' infrared radiation and creating an image based on that information. Recently, face recognition on thermal infrared spectrum has gained much popularity because thermal image is formed due to emission not reflection. They do not depend on the external light source and intensity of light and are also less dependent on angle of incident of light. Therefore a face recognition system designed based on thermal infrared face images cannot be evaded [4] or disguise, as can occur using the visible spectrum for facial recognition. On comparing visual face recognition systems with this, it will be less vulnerable to different varying conditions, such as head angle variations, the expression changes, or lighting conditions [4].

   The interpretation of either visual images alone or thermal images alone, is usually an unconstraint problem [8]. The estimates of surface temperature which is obtained by the thermal image, is not specific in distinguishing between the object classes [11]. The features extracted from visual intensity images also lack the specificity required for uniquely determining the identity of the imaged object [11]. The interpretation of each type of image thus leads to ambiguous inferences about the nature of the objects in the scene [11]. The use of thermal data gathered by an infrared camera and the visual image acquired using cameras, is seen as a way of resolving some of these ambiguities [11]. To identify these mechanisms thermal imaging is used. Thus thermal images can provide information about the imaged object which is not available from a visual image.

   A great deal of effort has been expended on automated scene analysis using visual images. Some work has been done in recognizing objects in a scene using infrared images [10]. Based on a study of the mechanism that gives rise to the differences in the thermal behaviour of object surfaces in the scene, little effort is there on interpreting thermal images of outdoor scenes. Also effort been made to integrate information extracted from the two modalities of imaging.

   Example for a Thermal image

   For night-time surveillance, acquisition of visible light imagery is impractical due to the lack of illumination [10]. Thermal imaging, acquires mid-wave infrared or long-wave infrared radiation naturally emitted by the human body. It can be utilized in low-light conditions to perform surveillance tasks. Identification of individuals which is captured by thermal imaging would significantly enhance night-time intelligence gathering capabilities [10]. The visible-light face imagery of individuals of interest will be present in government watch lists and databases. Inorder to match thermal face imagery to the existing databases, it requires the development of across modality face recognition algorithms and methods. The thermal-to- visible face recognition is a challenging problem since the modality gap which is caused by the high wavelength difference between visible and thermal radiation is more.

   Human body emits thermal IR which is an ideal modality for nighttime tasks. But there is large disparateness between the thermal IR and visible spectrum. This results in a wide modality gap which makes the thermal-to-visible face recognition a significantly more challenging problem. The development of an algorithm or transform space that well-correlates the thermal and visible face signatures is the key to solve this problem. This work addressed here is the problem of matching thermal probe images to the gallery images which are visible in nature. Visible images which simulate government watch lists and the thermal IR probe imagery simulates suspect imagery which is acquired during night-time surveillance. This face identification problem of matching thermal probe images to visible gallery

   images is cast as a multimodal face recognition problem [10].

3. Proposed Method

   The various image fusion methods are proposed in [7]. In this work the following methods are used. The datasets used in [7] are natural images. ere the datasets used are thermal and its corresponding visual face images.

   Select Maximum:

   The pixel with maximum intensity is selected and taken as resultant pixel of fused image. It considers the higher pixel intensity as the better information. Therefore either the intensity is taken or it is totally avoided.

   Average Method:

   Pixel intensities of the corresponding pixels in the input images are averaged. It projects both good and bad information from input images. Both good and bad information is minimized, arriving at an average image.

   PCA(Principal Component Analysis):

   Arrange the source images in two – column vector of dimension 2*n. Then compute empirical mean along each column. Then subtract mean from each column. Then find the covariance matrix. Then compute Eigen vectors and Eigen values and sort them by decreasing Eigen value. Then consider first column of Eigen vector which corresponds to larger Eigen value to compute normalized P1 and P2.

   Gradient Pyramid:

   Every pyramid transform consists of three major steps. First one is Decomposition. Then formation of the initial image for the recomposition process. Third step is Recomposition. In decomposition process it uses its predefined low pass filter.

   For Morphological pyramid grayscale opening and closing is used.

   DWT(Discrete Wavelet Transform):

   In this method [1] wavelets are discretely sampled , then decomposition and selection and generation of filters are performed.

   The quality of image fusion techniques can be evaluated using various quality metrics [9],[12],[13] such as PSNR(Peak Signal to Noise Ratio), Normalized cross correlation, Structured content, Mean Squared Error(MSE), Normalized absolute error(NAE).

4. Results

   1. Fusion toolbox

      Here fusion of thermal and visual face images is done using an image fusion toolbox. Fig. a shows the fusion toolbox. Here the datasets used are thermal and visual face images. These datasets are loaded into an image fusion toolbox. The different fusion techniques such as PCA, Select Maximum, Average, DWT, Morphological Pyramid, Gradient Pyramid can be selected from the options available. There is option to save the fused results which helps to evaluate the performance of these methods. Hence the comparison of image fusion techniques can be done easily. Figures b and c shows the input images used for fusion. ie visual and thermal images. Two different face images are used here.

   2. Input Visual Images

   3. Input Thermal Images

   4. Fusion results after Average Method

   5. Fusion results after DWT

   6. Fusion results after select maximum

   7. Fusion results after PCA method

   8. Fusion results after Morphological Pyramid

      1. Fusion results after Gradient Pyramid

Figures from fig. d to fig. i shows the results obtained by the fusion of corresponding thermal and visual face images using each of the specified methods. In the case of natural images [5] we can compare the performance of fusion techniques using quality metrics since there will be a reference image. But here in the case of thermal and visual face image fusion, there is no reference image [6]. So, visual scores are used to evaluate the results. The results are evaluated by a group of 20 people. Each person rank the results obtained by each techniques based on the visual quality. Out of 20 people, 75% of them ranked Morphological Pyramid as the best method and PCA as the second best method. From the rankings available Select Maximum method gives the worst results. The table below shows the ranking done based on the visual scores given by 20 people.

1. Conclusion

   TableNo: 1

   1. Wang, Z. J.; Ziou, D. Armenakis, C.; Li, D. R.; Li,

      Q. Q, A comparative analysis of image fusion Methods, IEEE Trans. Geosci. Remote Sens. 2005, 6, 1391-1402.

   2. Tania Stathaki, Image Fusion: Algorithms and Applications, Academic Press, 2008 edition.

   3. http://en.wikipedia.org/wiki (Last visited: November 2012)

   4. C. Xydeas and V. Petrovic, Objective pixel-level image fusion performance measure, Proc. SPIE,Orlando, FL, 2000, pp. 88-99.

   5. Luciano Alparone, Bruno Aiazzi, Stefano Baronti, Andrea Gaezelli, Filippo Nencini, Information- Theoretic Image Fusion Assessment without Reference, Institute of Applied Physics, Italian NationalResearch Council, Italy.

   6. M. K. Bhowmik, D. Bhattacharjee, M. Nasipuri, D.

   K. Basu & M. Kundu, Fusion of Wavelet

   This work is oriented towards the comparative study of various image fusion techniques which can be applied on the input datasets used for face recognition. The dataset is a pair of visual and thermal face images. The fusion result gives the improved amount of information which helps in person identification to tackle the drawbacks of individual thermal and visual images. Here six fusion techniques such as Average Method, Select Maximum, PCA, Gradient Pyramid, Morphological Pyramid and Discrete Wavelet Transform are applied on the datasets . Since there is no reference image, the fusion results of thermal and visual images are interpreted visually. From the various fusion results, it is found that Morphological pyramid fusion gives the best fused results for face recognition. PCA also gives much better results. The Select maximum method gives the worst result for all the datasets. Fusion is done with the help of image fusion tool box for convenience. In future an effective quality metrics will be formulated.

   .

2. References

coefficients from Visual and Thermal face images for Human Face Recognition , International Journal of Image Processing (IJIP), Volume (4): Issue (1).

1. Debotosh Bhattacharjee,Mrinal Kanti Bhowmik,Mita Nasipuri,Classification of fused face images using Multilayer Perceptron Neural Network,Technical Report.

2. Manjusha Deshmukh and Udhav Bhosale,Image Fusion and Image Quality Assessment of Fused Images, International Journal of Image Processing (IJIP), Volume (4): Issue (5).

3. Jonghyun Choi and Larry S Davis,Thermal-to- Visual Face Recognition,Defense Tech Briefs October 2012.

4. Mohammad Hanif and Usman Ali,Optimized Visual and Thermal Image Fusion for Efficient Face Recognition ,Technical Report.

5. Shivsubramani Krishnamoorthy, K.P Soman. 2010,Implementation and comparative study of image fusion algorithms. Amrita University

6. Asha P Kurian, Bijitha S R, Lekshmi Mohan, Megha M Kartha, K P Soman, 2012, Performance Evaluation of Modified SVD based Image fusion, IJCA, Vol 58, Amrita University.