A Novel Approach of Weight Map Generation for Fusion of Multi-Exposed Images

A Novel Approach of Weight Map Generation for Fusion of Multi-Exposed Images

Deep Doshi

Department of Electronics and Telecommunication University of Mumbai

Mumbai, India

Nilesh Singh Bhati

Department of Electronics and Telecommunication

University of Mumbai Mumbai, India

Abstract Multi-Exposure image fusion can be considered as a process of blending a set of variably exposed low dynamic range images in a high detail output well exposed everywhere. There are various low as well as high complexity algorithms developed in order to carry out the process of multi-exposure fusion and researchers are still working on developing low complexity algorithm that produces results with good details and vivid color. A novel low complexity mechanism to develop weight maps is proposed in this paper which uses exposedness of the image as a basis. A Laplacian-Gaussian pyramid based fusion scheme is employed to obtain the final output. The experimental results exhibit that our results have high degree of detail preserving capability in highly over and under exposed regions when compared with some of the popular algorithms. A quantitative analysis also performed which verifies the superior quality of our algorithm.

Keywords Exposure transfer function, Laplacian-Gaussian pyramid, Multi-exposure image fusion, Multi-resolution fusion, Weight maps.

1. INTRODUCTION

   There are various concepts and methodologies developed in order to improve the quality of images and multi-exposure fusion is one amongst them. Image fusion is two or more perfectly spatially aligned input images with different exposures are used to obtain a single output image which contains all the useful parts from the input images. There are two main trends for combining multi-exposure images. The first trend involves computation of photometric response function of cameras from set of low dynamic range (LDR) images and then creating a high dynamic range (HDR) image from it. HDR image is then converted to LDR image for display purpose using tone mapping operations.

   The second trend involves combining LDR images directly using a fusion process to create output LDR image. is popularly called as multi-exposure fusion. The initiative work of multi-exposure fusion was done by Mann and Picard [1]. They used estimation of camera response function and radiance map to select properly exposed regions from the pictures and then generating a high dynamic range image. A decade after this initial work, there was a tremendous growth in the field of multi-exposure fusion with various low and high complexity algorithms proposed by various researchers. The complexity algorithms combined the low dynamic range images directly in the spatial domain without using any

   Sourabh Kulkarni

   Department of Electronics and Telecommunication University of Mumbai

   Mumbai, India

   Chirag Desai

   Department of Electronics and Telecommunication University of Mumbai

   Mumbai, India

   complex transformations or algorithms. Because of their low complexity there is a trade-off between the overall quality and detail recovery.

   Kartalov et al. [2] have proposed a low complexity algorithm with less number of operations per pixel. Their main aim was to restrict to low memory consumption per operation. They used one underexposed and one overexposed image which were directly translated using luminance transfer function, to get weights for averaging. They used saturation maximization process for color images. The main disadvantage of this algorithm is that it works only for two images, and if the images are far apart in exposure beyond

   ±2ev it leads to obnoxious results.

   Vanmali et al. [3] have proposed another low complexity algorithm with balanced exposures, they used pixel by pixel fusion governed by a weight map generated using exposures of image. They used Gaussian weight map curves to generate the corresponding weight maps for the images to be fused. This method has an advantage of good to excellent detail recovery but slightly lagging in case of color reproduction.

   There has been a series of high complexity algorithms proposed by various researchers. Goshtasby [4] has proposed a block based method to select high entropy blocks to form a fused image. A gradient ascent algorithm was then employed to remove the blocky artifacts generated due to block based mechanism. This refinement process is recursive in nature making the algorithm slow.

   Another popular algorithm has been proposed by Mertens et al. [5], where they employed a multi-resolution fusion framework. They used contrast, saturation and well exposedness as measure to generate the weight maps. They used a Laplacian-Gausian pyramid based fusion mechanism. The images were decomposed using Laplacian pyramid and the weight maps were decomposed using Gaussian pyramid. A fused Laplacian pyramid was formed by sum of Laplacian pyramid weighted by Gaussian pyramid. The fused image is then reconstructed from this fused Laplacian pyramid. The result showed a very good detail recovery and an excellent color reproduction, almost for all types of datasets, either balanced or unbalanced. It is also useful to incorporate flash images and sets a bench mark in field of multi-exposure fusion.

   The work of Mertens et al. [5], has been extended in wavelet domain by Malik et al. [6].The variations in wavelet based fusion were then proposed by [7], [8] and [9]. Zhao et al. [7] used a sub-band architecture using quadrature mirror filters (QMFs). Wang et al. [8], has proposed a method based on shift invariant discrete wavelet transform. Abd-el-kader et al. [9], gave a frame work of multi exposure fusion using curvelet transform.

   Kotwal et al. [10], have proposed an optimization based method for multi-exposure fusion. They have optimized the cost function in order to get high entropy and contrast in the result. This iterative algorithm has excellent detail recovery when compared to other algorithms, but lacks in color reproduction. Since the algorithm is iterative in nature the runtime goes in minutes to produce the final output.

   In this paper we propose a new algorithm to generate the weight maps for the multi-exposed images. We blend the method of Kartalov et al. [2] and Vanmali et al. [3] to get more reliable weight map functions. We use a more sophisticated luminance transfer function to calculate weights of each pixel in the image. We use the Laplacian-Gaussian pyramid based blending algorithm proposed by Mertens et al. [5]. The experimental results show that our algorithm has excellent detail recovery. To overcome the precincts of our algorithm we add a post processing step to get more vivid colors. The subjective as well as the quantitative analysis indicate that our results are up to the mark or even better than

   0.75

   Weights

   Weights

   w(i)

   w(i)

   0.25

   1.0

   Weights w(i)

   Weights w(i)

   0.6

   0.2

   0.125 0.375 0.625 0.875

   Location coefficient k i

   1. For N=4

      the most of the popular algorithms. The key feature of our algorithm is that it can produce details in highly under / over exposed regions where most popular algorithms fail to recover the details.

2. ALGORITHM

   The proposed algorithm is divided into five steps. In the first step we calculate the initial weights for the images in the

   0.1

   0.3 0.5 0.7 0.9

   Location coefficient k i

   1. For N=5

   Fig. 1 Calculation of weights

   cues from Vanmali et al. [3] algorithm. In second step we classify the image either as uderexposed or as overexposed.

   The initial weight for the image is then calculated as

   In the next step we calculate the weight maps pixel by pixel

   using luminance transfer function and initial weights

   w(i)

   2(ki )

   0 ki 0.5

   generated in first step. The images are then fused using

   2(1 ki )

   0.5 ki 1

   Laplacian-Gaussian multi-resolution fusion as proposed by Mertens et al. [5]. Finally a post processing step for color enhancement is performed. The details of these steps are as follows:-

   I: Initial weights for the images

   Vanmali et al. [3] has demonstrated that for underexposed images dark region should be given minimal weight and for overexposed images saturated regions should have minimum weight. Using the similar idea we have calculated the initial weights for the images .Consider Ii, i=1,2,3,…N be the set of N

   These weighting functions will then ensure that the mid exposed image will have higher weights compared to that of under / over exposed images. An example for N=4 and N=5 is as shown in Fig.1.

   II: Classification of images

   To classify the image as underexposed or overexposed image we make use of the average value of the grayscale image. Consider Imono,i represents grayscale version of Ii then we have,

   multi-exposed images, arranged in the increasing order of exposure, i.e. I1 indicates least exposed image and IN indicates

   i mean(Imono,i )

   (3)

   maximally exposed image. All these images are spatially aligned and are in dynamic range of 0 to 1. The location of the weight coefficient ki is calculated as,

   If the average value µi is greater than 0.5 we classify the

   image as overexposed image otherwise as underexposed image.

   k 2i 1

   i 2N

   (1)

   Iclass

   overexposed underexposed

   i 0.5

   i 0.5

   (4)

   0.95

   Mo1 Mo2

   Values

   Values

   Mo3

   Mu1 Mu2

   Oversaturated

   Ideal

   value of luminance below 5 percent of maximum luminance value i.e. class of pixels to discarded, 2.pixels having values from 5 percent of maximum luminance value to mean luminance value of underexposed image is the class of pixels to be weighted averaged and the 3.pixels that carries maximum information details in underexposed image is the pixels that have the value of luminance above the mean value of the image to the maximum luminance value called as class of pixels to translated.

   For underexposed image

   Mu3

   Undersaturated

   0 0 Imono,i(x,y) 0.05

   w(x,y,i) w(i) 0.05 Imono,i(x,y) i

   1

   1

   0.05

   Pixels

   Fig. 2 Luminance Transfer Function

   i I

   mono,i(x,y)

   (6)

   III: PIXEL-BY-PIXEL WEIGHT MAPS

   Here we make use of the idea proposed by Kartalov et al. [2]. They used luminance transfer function. For overexposed image the pixels having luminance value less than its mean were assigned a weight of 1, and were translated as it is to the

   For the weighted fusion the weights must lie in the range of 0 to 1 and must add to 1 at every pixel. Therefore we normalize these weights to obtain final weight maps as

   N ,i

   N ,i

   N

   N

   W ( x, y) Wi ( x, y)

   ideal curve. The pixels which are oversaturated were assigned

   Wk (x, y)

   (7)

   a weight of 0. Inverse weighting is done for the underexposed image. The mid-exposed pixels were fused by a weighted sum. Thus they used saturation maximization and hue selection mechanism for color images.

   We extend this idea of two images proposed by Kartalov et al. [2] to N images and we give away the process of saturation maximization to obtain more reliable color reproduction. The mean value of the grayscale image gives a fair amount of information for the image. For example, the mean value of 0.9 indicates that the image is quite bright and the mean value is 0.2 indicates that the image is dark. Thus in

   k 1

   IV: MULTI-RESOLUTION FUSION

   To obtain seamless fusion we use Laplacian-Gaussian pyramid based multi-resolution fusion proposed by Burt et al. [11]. Accordingly, each input image is decomposed using Laplacian pyramid and their corresponding weight maps are decomposed using Gaussian pyramid. Consider L{I}l be the lth level in the Laplacian pyramid and G{WN}l be the lth level of the Gaussian pyramid, then the fused lth level of Laplacian pyramid is calculated as

   this algorithm we use the mean value itself as one of the threshold.

   L{IF

   (x, y)}l G{W

   N

   N

   i1

   N ,i

   (x, y)}l L{I (x, y)}l

   (8)

   i

   i

   It is also quite obvious that in any image that is overexposed, the pixels that have the values above 95 percent of the luminance doesnt carry any significant information and hence can be simply neglected. Similarly for an underexposed image the pixels that have the luminance below 5 percent are going to be visually very dark and thus are neglected.

   Depending on the above criteria the pixels in an overexposed image can be classified into three classes 1.pixels which would carry maximum information consisting of pixels ranging from minimum luminance value to mean called as class of pixels to translated, 2. pixels having values from mean

   The final fused image is then obtained by collapsing the fused pyramid. For color images we process each color channel i.e. R, G & B using equation (8) to obtain the final color output.

   V: POST PROCESSING

   The main aim behind applying post processing is to increase the visual appearance of the obtained output image. In post processing the output image after applying above algorithm is converted into HSV. The saturation values of each pixel are boosted by a factor of

   to 95 percent of maximum luminance value is the class of pixels to be weighted averaged and 3.pixels that carries

   Io,sat

   (x, y) I

   F ,sat

   (x, y)

   (9)

   minimum information in an overexposed image is the pixels that have the value of luminance above 95 percent of maximum luminance i.e. class of pixels to discarded.

   For overexposed image

   The value of is typically kept in the range of 0.02 to 0.2 depending on output saturation level required. We used value of 0.06 for most of the results of our algorithm. The above boosting may result in over-saturating the colors. Hence the saturation values of pixels above 1 are ebbed to 1.

   1 0 Imono,i(x,y) i

   I (x, y) I

   (x, y) 1

   w(x,y,i) w(i)

   i I

   mono,i(x,y) 0.95

   Io, sat

   (x, y)

   o, sat

   1

   o, sat

   I

   (x, y) 1

   (10)

   0

   0

   0.95 I

   mono,i(x,y)

   (5)

   o, sat

   Similarly, the pixels in an underexposed image can be classified into three classes 1.pixels that contain minimum information in an underexposed image is the pixels having the

   Finally the image is again converted back to RGB color space.

   1. Eiffel Tower (Image Courtesy: Jacques Joffre) (b) Grandcanal (Image Courtesy: Jacques Joffre)

   (c) House (Images courtesy: Tom Mertens) (d) Office (Images courtesy: Matlab Toolbox)

   (e) Lizard (Images courtesy: Eric Reinhard) Fig. 3 Input image sequences

3. EXPERIMENTAL RESULTS

   We have implemented the proposed algorithm in MATLAB and tested it for various datasets. We present the results for five different datasets using Eiffel Tower, Grandcanal, House, Office and Lizard of various size and different numbers of input low dynamic range images. These datasets are shown in Fig.3. For brevity of space, we present weight maps generated using algorithm for Eiffel Tower and Grandcanal Fig. 4.

   We compared our results with results generated by algorithm of Mertens et al. [5], Kotwal et al. [10] and Vanmali etal. [3]. Mertens et al. [5] algorithm is a popular one which exhibits effective blend of details and color. Kotwal et al. [10] algorithm has one of the best detail preserving capability found in the literature and the other hand Vanmali et al. [3],algorithm gives a low complexity result with high degree of details with a

   acceptable color quality. The results of these algorithms and our results for Eiffel Tower and Grandcanal sequence are presented in Fig. 5 and 6 respectively. For House, Office and Lizard sequence we present the comparison with the Mertens et al. [5], results in Fig. 7, 8 and 9 respectively. We perform both subjective as well as quantitative analysis of the result and are presented below.

   1. Subjective analysis

      The visual observation of results of Mertens et al. [5], exhibit excellent color and contrast. Their results have very good detail recovery but one can see less detail in highly under / over exposed regions when compared with other algorithms. Overall analysis of Kotwal et al. [10] result show that their algorithm has high degree of detail recovery but their color is very dull when compared with other algorithm.

      1. Weight maps for Eiffel Tower (b)Weight Maps for Grandcanal Fig. 4:Weight maps for our results

         Fig. 5: Results for Eiffel Tower image sequence. L to R; Result of Mertens et al. [5], result of Kotwal et al. [10], result of Vanmali et al. [3], result of our algorithm.

         Fig. 6: Results for Grandcanal image sequence. T to B; Result of Mertens et al. [5], result of Kotwal et al. [10], result of Vanmali et al. [3], result of our algorithm.

         Fig. 7: Results for House image sequence. T to B; Result of Mertens et al. [5], result of our algorithm

         Fig. 8: Results for Office image sequence. T to B; Result of Mertens et al. [5], result of our algorithm

         Fig. 9: Results for Lizard image sequence. T to B; Result of Mertens et al. [5], result of our algorithm

         Vanmali et al. [3] result has detail recovery comparable to that of Kotwal et al. [10] result with better color saturation. But both [10] and [3] lack in contrast when compared with Mertens et al. [5]. On contrary, our result show equivalent degree of detail recovery to that of [10] and [3], with contrast matching the performance of [5]. The overall appearance shows that our results have vivid color and better details captured in under / over exposed regions when compared with [5]. Also one can adjust the color saturation more flexibly in our algorithm using the post processing step which gives an added advantage to the user.

   2. Quantitative analysis

   In our proposed algorithm the key difference is in the mechanism of weight map generation when compared with Mertens et al. [5].

   We used variance as used by Kotwal et.al. [10] which captures the overall details transformed from input images to the fused results. The results for the variances are shown in Table I.

   To measure the contrast richness of our algorithm we used RMS contrast as a measure as used by Vanmali et al. [3] given as

   TABLE I: Comparison of Variance

   Test sequence	Variance
   	Without Post Processing	With Post processing	Mertens et. al.
   Eiffel Tower	0.0687	0.0665	0.0617
   Grandcan al	0.0611	0.0586	0.0581
   House	0.0560	0.0602	0.0669
   Office	0.0634	0.0624	0.0759
   Lizard	0.0553	0.0558	0.0616

   Test sequence	RMS contrast
   	Without Post Processing	With Post processing	Mertens et. al.
   Eiffel Tower	0.2742	0.2745	0.2398
   Grandcan al	0.2922	0.2921	0.2556
   House	0.1982	0.1986	0.1983
   Office	0.1870	0.1874	0.2000
   Lizard	0.1976	0.1978	0.2079

   Test sequence	RMS contrast
   	Without Post Processing	With Post processing	Mertens et. al.
   Eiffel Tower	0.2742	0.2745	0.2398
   Grandcan al	0.2922	0.2921	0.2556
   House	0.1982	0.1986	0.1983
   Office	0.1870	0.1874	0.2000
   Lizard	0.1976	0.1978	0.2079

   TABLE II: Comparison of RMS Contrast

   1 N 1 M 1

   2

   (11)

   ContrastRMS

   (I (x, y) Imean )

   MN

   MN

   x0 y0

   TABLE III: Comparison of Saturation

   Test sequence	Saturation
   	Without Post Processing	With Post processing	Mertens et. al.
   Eiffel Tower	0.2899	0.3443	0.2835
   Grandcan al	0.2218	0.2719	0.2296
   House	0.3725	0.4321	0.3250
   Office	0.2656	0.3139	0.2548
   Lizard	0.3237	0.3853	0.3379

   Test sequence	Saturation
   	Without Post Processing	With Post processing	Mertens et. al.
   Eiffel Tower	0.2899	0.3443	0.2835
   Grandcan al	0.2218	0.2719	0.2296
   House	0.3725	0.4321	0.3250
   Office	0.2656	0.3139	0.2548
   Lizard	0.3237	0.3853	0.3379

   The results for the RMS contrast are shown in Table II. Here also we can see that our results are comparable or in some cases even better to that of Mertens et al. [5] results. The results indicate that our algorithm has better performance for dataset of three images. For higher number of images Mertens et al. [5] gives comparatively better results.

   To measure the chromaticity of the results we use saturation as a measure. For this we convert the resultant images into HSV color plane and use the average value of S components a saturation measure. The results of this measure are as shown in Table III. The results indicate that our algorithm performs better than Mertens et al. [5] algorithm almost in all cases. We see further increase in contrast with the post processing results. After post processing our results always exhibit better color saturation in all the cases. The users can even tune the saturation level by adjusting the value of in the post processing step.

4. CONCLUSION

We have proposed a Laplacian-Gaussian pyramid based fusion for multi-exposed imaes. This algorithm is driven by weight maps where we have developed a new mechanism which takes into account, the exposedness of the images. The experimental results show that our algorithm produces results which are at par or even better than the traditional benchmark algorithms commonly referred in the literature. Our algorithm has a very good detail preservation capability with high degree of details in extremely under / over exposed regions. The post processing step added gives additional freedom to user to adjust the colors and make the results more chromatic. The visual results are verified using three quantitative measure and they are in tune with visual results.

ACKNOWLEDGMENT

We would like to thank Prof. Ashish Vanmali from Department of Electronics and Telecommunication Engineering at Vidyavardhinis College of Engineering and Technology, University of Mumbai, for his valuable guidance and support.

REFERENCES

1. S. Mann and R. W. Picard. Being undigital with digital cameras: Extending dynamic range by combining differently exposed pictures. Technical Report 323, M. I. T. Media Lab Perceptual Computing Section, Bosten, Massachusetts, 1994. Also appears, IS&Ts 48th annual conferance, Cambridge, Massachusetts, May 1995.

2. T. Kartalov, A. Petrov, Z. Ivanovski, and L. Panovski. A real time algorithm for exposure fusion of digital images. In MELECON 2010 –

   2010 15th IEEE Mediterranean Electrotechnical Conference, pages 641

   646, April 2010.

3. A. V. Vanmali, S. S. Deshmukh, V. M. Gadre, "Low complexity detail preserving multi-exposure image fusion for images with balanced exposure," Communications (NCC), 2013 National Conference on ,

   Delhi, India, pp.1-5, 15-17 Feb. 2013

4. A. A. Goshtasby. Fusion of multi-exposure images. Image Vision Comput., 23(6):611618, June 2005

5. T. Mertens, J. Kautz, and F. V. Reeth. Exposure fusion. Computer Graphics and Applications, Pacific Conference on, 0:382390, 2007.

6. M. H. Malik, S. A. M. Gilani, and A. ul Haq. Wavelet based exposure

   fusion. In Proceedings of the World Congress on Engineering 2008 Vol I,WCE 08, July 2 – 4, 2008, London, U.K., pages 688693. International Association of Engineers, 2008.

7. Y. Zhao, J. Shen, and Y. He. "Subband architecture based exposure fusion." In Image and Video Technology (PSIVT), 2010 Fourth Pacific-Rim Symposium on, pp. 501-506. IEEE, 2010.

8. J. Wang, D. Xu, C. Lang and B. Li, 2011. Exposure Fusino Based on Shift-Invariant Discrete Wavelet Transform. J. Inf. Sci. Eng., 27(1), pp.197-211

9. A. Abd-el-Kader, H. E. Moustafa, and S. Rehan. "Performance measures for image fusion based on wavelet transform and curvelet transform." In Radio Science Conference (NRSC), 2011 28th National, pp. 1-7. IEEE, 2011.

10. K. Kotwal and S. Chaudhuri. An optimization-based approach to fusion of multi-exposure, low dynamic range images. In Information Fusion (FUSION), 2011

11. P. Burt and T. Adelson. The Laplacian Pyramid as a Compact Image Code. IEEE Transactions on Communication, COM-31:532540, 1983.