Multiexposure Composition of Images using Gradient Extraction

Multiexposure Composition of Images using Gradient Extraction

Niraja J Shenoy

Department of Electronics and Communication Marian Engineering College

Trivandrum, India

Monisha Menon

Department of Electronics and Communication Marian Engineering College

Trivandrum, India

AbstractDynamic range exhibited by real world scenes are very high and hence a single image captured turn out to be under or over exposed. To overcome this a set of bracketed sequences are required. Also in such cases camera must be absolutely still between exposures else it may cause misalignment of images. In this paper, we propose a novel algorithm for fusing images using laplacian and gradient information. This algorithm is very simple and experiments illustrate that it generate high quality image. For this initially image registration is performed as a preprocessing step and to these set of images fusion is performed. Here gradient magnitude is used as a quality measure because it highlights the pixels exposure quality and also gradient is associated with the features of image. Thus this algorithm provides a single image of superior quality with all input information. This technique can also be used for concealed weapon detection, fusing of CT and MRI image etc. One of the advantage of this algorithm is it does not require much parameter tweaking.

Keywordsgradient, laplacian, registration, high dynamic range , under or overexposed image

1. INTRODUCTION

   Real world scenes have very high dynamic range so an image turns out to be under or over exposed when captured using a limited dynamic range camera. Dynamic range is defined as the ratio between brightest to darkest part of an image. Real world dynamic range is very high of the order of 100,000:1. But the dynamic range of JPEG format will not exceed 255:1 which is referred as a low dynamic range image. A bracketed exposure sequence can be transformed into a single image using conventional technique. But the resultant image is a high dynamic range that cannot be displayed using normal display device. So again we have to convert this HDR image to LDR image using tone mapping operators [1], [2]. Image fusion is a simple and effective method that skips the step of computing a high dynamic range image and provides a Low dynamic range image using a set of multi exposure sequences. It fuse multi exposures into a high quality, low dynamic range image that is ready for display. Image fusion is the process of combining relevant information from a set of image into a single image. Each of these exposures provide information about different parts of the image. These exposures must be properly aligned [9], misalignment leads to poor fusion result. This paper describes a method for fusing a set of images into a single image with maximum information. Initially image registration is performed and to these image fusion is done by obtaining a weighing function. Fusion is mainly performed to obtain pleasant and complete output when compared to input sequences.

   Main requirements of image fusion process are

   • Should preserve all relevant information from input images

   • Should not introduce artifacts which can lead to a wrong diagnosis

     Broadly image fusion is classified into

   • Pixel level or low level

   • Feature level or medium level

   • Decision level or High level

     Another major classification is

   • Spatial domain

   • Transform domain

   Out of the above pixel level is the most simplest and can be implemented either by using spatial [3] or transform domain

   [4] methods. Pixel level fusion is achieved by directly taking pixel values. In most cases low level information such as intensity is extracted to achieve pixel level fusion. In feature level [6] it segments image into contiguous regions and fuse using their regional properties. Features can be extracted separately from each sequence or they may be obtained by simultaneous processing. Decision level fusion [8] uses output of initial object detection and thus classifies input to perform data integration. Both decision level fusion and feature level fusion may result in incomplete and inaccurate representation of information.

   Several simple to complicated fusion algorithms are available like pixel based to sophisticated wavelets [5] and PCA based. Principle Component Analysis (PCA) is a vector space transform mainly used to reduce multidimensional data to lower dimensions so that analysis becomes simpler. It takes pixel values at each location of source or input image and adds a weight factor to each of these pixel value and finally takes average of this added pixel to obtain a fused image. Discrete Wavelet Transform (DWT) is a frequency domain method that calculates approximate coefficients and detailed coefficients. Now each band is substituted instead of a low resolution component, this process is repeated until all bands are transformed. In transform domain changing a single coefficient using transform may lead to change in all pixel values of fused image in spatial domain.

   Commonly used standard fusion techniques are Brovey Transform (BT), IHS (Intensity Hue Saturation), PCA. PCA provides basis for commonly used fusion techniques .Out of

   these IHS is the oldest method that perform fusion in RGB domain. Initially RGB domain is transformed into IHS domain, match the histogram and perform inverse IHS transform to obtain RGB domain [3]. Brovey Transform is purely based on chromacity. For this RGB image is normalized and multiplied by other image. Resultant image is then added to intensity component of RGB image [3]. PCA is similar to IHS and have no limitation in number of fused band.

   As different from previous work [8] focuses to obtain a fused image without using tone mapping operators [1], [2] instead it finds a weighing function for each of the input. Three quality measures are used for this calculation. They are contrast, well exposedness, and saturation. For this each pixel combines information from all these quality measures into a scalar map using multiplication.

   Image fusion has several advantage over single image and also the resultant output will have high PSNR, increased robustness and reliability. An ideal image fusion technique should have three essential features they are computationally efficient, reduce colour distortion, and preserve high spatial resolution. Main application includes concealed weapon detection, fusion of medical image like CT and MRI, flash no flash photography etc.

   In this paper gradient is used as a quality measure to extract the weighing function. Main goal is to preserve all features present in the input sequence and make them visible in a single image. Gradient is used because it conveys information about the latent scene and correctly detects edges. Section II gives overall idea of how multi exposure composition is performed. Section III gives the method to extract laplacian and gradient to obtain the weighing function.

2. PROPOSED METHOD

   Images captured from different modalities are fused to obtain a complete image of superior quality. And the resultant image is given by

   (1)

   Where N represent number of input exposures,

   Ii (x, y) denote intensity of pixel located at (x, y) in the ith exposure

   Wi (x, y) denote intensity of pixel located at (x, y) in the ith exposure

   H denote the composite image to be generated

   The final composite image mainly depends on the weighing function W, hence W must be properly selected to obtain a perfect result. Since gradint gives information about latent scene and gradient magnitude defines the pixels exposure quality, it is used as a parameter for obtaining the weighing function.

   The block level representation of overall algorithm is shown in fig 1. A set of multiexposure images are registered depending on their misalignment. To these registered images laplacian is performed to enhance the fine details and to sharpen the image and gradient of this laplacian images are taken to obtain the

   edge information as well as to remove the noise that was enhanced during laplacian operation. Gradient magnitude is computed by using a Gaussian filter to obtain the weighing function. To obtain the composite image, each weighing function is multiplied with the corresponding image.

   In addition to this bilateral filtering [10] is performed to these weighing function to eliminate the outlier weights and to preserve the edges. The standard deviation of space Gaussian is set to 3 pixels and range Gaussian is set to 2 pixels. Finally adaptive histogram equalization is performed to the composite image to enhance the contrast as a post processing.

3. MAIN ALGORITHM

   Initially a set of images are registered using there amount of misalignment using proper transformation. To registered image laplacian is computed.

   Generate Gaussian pyramid of the image at different scales. It is obtained by setting G1 = i, and iteratively applying Gi+1 = shrink (Gi).

   Generate the Laplacian pyramid of the image by backward- processing of Gaussian pyramid, setting Lk = Gk and iteratively applying Ii = Gi expand (Gi+1).

   From the obtained laplacian, gradient is extracted so that prominent edge details are correctly obtained. This is obtained by taking first derivative of 2D Gaussian filter g(x, y, d) in x and y directions to extract the gradient information

   (2)

   (3)

   Where and are the partial derivatives of laplacian image along x and y directions .Standard deviation d is set to four pixels. Once gradient is extracted smoothen the gradient using a Gaussian filter g(x, y, ) with

   large value of . Significance of gradient magnitude is that it

   reflects the maximum change in pixel values. The magnitude value is given by

   (4)

   Due to under or over exposure features that are visible in one exposure will not be available in another image so our aim is to combine all these feature and gradient magnitude is a simple and effective parameter whose value becomes high when image is not better exposed and low when an image is under or over exposed. From this gradient magnitude weighing function is estimated using the following equation

   Image registration

   Laplacian and gradient extraction

   Weighing map estimation

   xposures

   put

   Seamless compositi on

   Adaptive histogram equalization

   Composite image

   In e

   Fig1. Block level representation of the algorithm

   (5)

   Where is a positive value given by 10^ (-25) to avoid singularity.

   Once each weighing is obtained using equation (5) filter these functions using a Gaussian filter g1(x, y, ) before performing bilateral filtering to avoid any noise present and also to smoothen the function. Main advantage of Gaussian is it is a separable and rotationally symmetric. Also degree of smoothening is governed by variance . A larger value of means wider Gaussian filter and more smoothening.

   Now to each weight bilateral filtering is performed so that distortion free image is obtained. Bilateral filtering smoothens images while preserving edges, by means of a nonlinear combination of nearby image values. If the three bands of colour images are filtered distinctly from one another, colours are corrupted close to image edges. This is because different bands have different levels of contrast, and they are smoothed differently. Separate smoothing perturbs the balance of colours, and unexpected colour combinations appear. Bilateral filters, on the other hand, operates on the three bands at once. After bilateral filtering of each weighing function adaptive histogram equalization is performed to increase or enhance the contrast of the image. For a high contrast image histogram covers a wide range and distribution of pixels will be uniform. Thus net effect will be an image of high dynamic range.

4. EXPERIMENTAL RESULTS

   The proposed algorithm is tested to various type of input images and experiments shows that it works well for all type of images without any complexity. In case of black and white images the algorithm works without the help of bilateral filtering. The value of Gaussian filter in g(x, y, and ) is an important parameter to obtain a perfect output, setting it to high value produces an output with no edge distortions. Otherwise it produces colour distortions as in [11]. Fig 2a) represents a set of input exposures, Fig 2.b) is the fused output obtained using proposed algorithm. Fig 2.c) is the fused output

   obtained using [11]. From these images it is clear that contrast of the image is poor also have colour distortion at edges. These problems are eliminated by using proper image registration and taking laplacian of the image.

   Fig (i) Fig (ii)

   Fig (iii) Fig (iv) Fig 3.a) set of 4 input exposures

   Fig 3.b) fused result

   Fig (i) Fig (ii)

   Fig (iii) Fig (iv)

   Fig 2. a) Set of 4 input exposures

   Fig 2. b) Fused output using proposed algorithm

   Fig 2. c) fused result using [11]

   This algorithm can be extended to concealed weapon detection. It is a technique of fusing the visual and IR image after registration. We find that the body is brighter than the background in the IR image. Also background is almost black and gives little details because of the high thermal emissivity of body. Also weapon is darker than the surrounding body due to a temperature difference between it and the body (it is colder than human body). The resolution in the visual image is much higher than that of the IR image, but there is no information on the weapon in the visual image.

   1. Visual image b) IR image

      1. Fused image with visible weapon

      2. Detected image with weapon highlighted Fig 4. Concealed weapon detection

   Another main application is fusion of MRI with CT image there by leading to a more perfect image that is informative. Medical image fusion is a branch of data fusion. It is a process of fusing synthetic medical image so as to obtain more information or more suitable for visual sense and computer process through processing the organized information from two or more original medical image in the same scene. Fig.5

   1. shows medical image fusion

      1. CT image b) MRI image

   c) Fused output

   Fig 5. Medical image fusion

   Fig 6. Plot of entropy of for various input exposures

5. PERFORMANCE MEASUREMENT

It is mainly used to find the image quality which is specific for an image and gives the perceived image degradation. There are several techniques and metrics that can be measured objectively and automatically. Quality assessment methods can be broadly classified into two categories: Full Reference Methods (FR) and No Reference Method (NR). In FR image quality assessment methods, the quality of a test image is evaluated by comparing it with a reference image that is assumed to be perfect. Some of the FR methods are PSNR, SSIM etc. In (NR) there is no reference image. Some of the methods are entropy, standard deviation etc.

1. Peak Signal to Noise Ratio (PSNR)

   PSNR is defined as the ratio between the maximum possible power of a signal and power of corrupting noise The PSNR value of fused image is always high.

   In this case, at pixel level, the highest possible value is 255.

2. Structural Content (SC)

   It is the ratio of the net sum of the square of the expected data and the net sum of square of the obtained data.

   m is the heightof the image implying the number or pixel rows

   n is the width of the image, implying the number of pixel columns.

   Aij being the pixel density values of the perfect image.

   Bij being the pixel density values of the fused image.

3. ENTROPY

   It is used to evaluate information quantity contained in an image, high value of entropy indicates that fused image is perfect.

   Fig 7. Plot of PSNR for different fusion techniques

   E=P*log2P

4. Structural similarity index

   The Structural Similarity Index is used to measure similarity between two images. It is an enhanced version of traditional methods such as PSNR, MSE. It is considered as one of the most effective and consistent parameter.

5. Normalized Cross Correlation (NCC)

Normalized Cross Correlation method is used for finding the similarities between fused image and reference image.

TABLE 1: Performance measure for set of images

Plot of entropy of various input exposure is shown in fig 6. It shows that the fused output has high entropy. Entropy is the measure of information content. Large value of entropy implies fused output has large amount of information or in other words fused output ha all information from input images. Also a plot of PSNR for a set of images is shown in fig 7 from which it is clear that proposed algorithm has high value than the others.

V1. CONCLUSION

Human visual system is highly sensitive to gradient and this gradient encodes edges and local contrast quite well. Edges refers to sharp transitions in an image. In this paper, edge information is extracted using gradient because it refers to pixels exposure quality. With the help of this, Gradient magnitude is used as a parameter for fusion. Proposed method does not have much parameter tweaking thus making the system simple and effective. Using this method a single image provides lot of information. Hence this technology is extended to detect weapons in defense field. Similar technique can also be used in medical field to fuse CT and MRI image which will provide a more useful image. This work can also be extended for satellite image fusion.

REFERENCES

1. E. Debevec and J. Malik. Recovering high dynamic range radiance maps from photographs. In SIGGRAPH 97: Proceedings of the 24th annual conference on Computer graphics and interactive techniques, pages 369378, New York,NY, USA, 1997. ACM Press/Addison-

   Wesley Publishing Co

2. E. Reinhard, G. Ward, S. Pattanaik, and P. Debevec. High Dynamic Range Imaging: Acquisition, Display and Image Based Lighting. Morgan Kaufmann Publishers, Dec. 2005.

3. Image fusion techniques in Remote Sensing, Reham Gharbia, Ahmad Tahir Azar [4] VPS Naidu, Discrete Cosine Transform based Image Fusion Techniques, Journal of Communication, Navigation and Signal Processing (January 2012) Vol. 1, No. 1, pp. 35-45[5] Mirajkar Pradnya P., Sachin D. Ruika r Image Fusion Method Based On WPCA 2012, IJARCSSE Volume 2, Issue 5, May 2012

1. G. Piella, A region-based multiresolution image fusion algorithm, in: Proceedings of the 5th International Conference on Information Fusion, Annapolis, MS, July 811, 2002, pp. 15571564.

2. Z. Zhang, R. Blum, Region-based image fusion scheme for concealed weapon detection, in: ISIF Fusion Conference, Annapolis, MD, July 2002.

1. G. Ward, Fast, robust image registration for compositing high dynamic range photographs from handheld exposures, J. Graph. Tools, vol. 8, pp. 1730, 2003.

2. C. Tomasi _ R. Manduchi Bilateral Filtering for Gray and Color Images, Proceedings of the 1998 IEEE International Conference on Computer Vision, Bombay,India

3. Wei Zhang, Wai-Kuen Cham Gradient-Directed Multiexposure Composition, IEEE transactions on image processing, vol. 21, no. 4, april 2012

.