An Empirical Evaluation of Coalescing a Bracketed Sequence of Images for High Quality Image Acquisition

An Empirical Evaluation of
Coalescing a Bracketed Sequence of Images for High Quality Image Acquisition

Jyoti Raghav, Atul Kumar
Department of Computer Science,
Kamrah Institute of Information and Technology
Bhondsi, Gurugram, Haryana, India

Abstract— A technique for coalescing a bracketed sequence of images into a high-quality image is proposed using DCT and luminance, chrominance techniques. DCT involves dividing the images to be coalesced into non-overlapping blocks of size NxN. DCT coefficients are computed for each block and coalesced rules are applied to get coalesced DCT coefficients. IDCT is then applied on the coalesced coefficients to produce the coalesced image/block. The procedure is repeated for each block. The second technique is based on blending the acclivated luminance components of the input images using the maximum inclined magnitude at each pixel location and then obtaining the coalescence luminance using a Haar wavelet-based image reconstruction technique.
The proposed algorithms are simple, easy to implement and could be used for real time applications. Experimental results and comparison show that the visual quality of these two techniques are comparable and produces fast, similar or better results than existing techniques. The implemented techniques is significantly useful for coalescing out of focus images.

Keywords— DCT, Bracketed Sequence, IDCT, Image Coalescence, Acclivation.
I. INTRODUCTION

Off late, different image coalesced algorithms have been developed to merge the multiple images into a single image that contain all useful information. Pixel averaging of the source images (the images to be coalesced) is the simplest image coalesced technique and it often produces undesirable side effects in the coalesced image including reduced contrast. To overcome this side effects many researchers have developed multi resolution, multi scale and statistical signal processing-based image coalesced techniques. In similar line, contrast-based image coalesced algorithm in DCT and Luminance, chrominance has been presented to blend the underexposed, over exposed and out of focus images.
In this paper, we propose two techniques to blend the images taken at various exposures and out of focus images to obtain a high-quality image, ready for display: as shown in Fig. 1 and Fig 2

A plastic container

Description automatically generated
Fig 1. Demonstration of Coalescing out of focus images.

This coalesced is similar to other image coalesced techniques. DCT involves dividing the images to be coalesced into non-overlapping blocks of size NxN. DCT coefficients are computed for each block and coalesced rules are applied to get coalesced DCT coefficients. IDCT is then applied on the coalesced coefficients to produce the coalesced image/block. The procedure is repeated for each block. The second technique is based on blending the acclivated luminance components of the input images using the maximum inclined magnitude at each pixel location and then obtaining the coalescence luminance using a Haar wavelet-based image reconstruction technique.
The proposed algorithms are simple, easy to implement and could be used for real time applications. Experimental results and comparison show that the visual quality of these two techniques are comparable and produces fast, similar or better results than existing techniques. The implemented techniques is significantly useful for coalescing out of focus images.

A screenshot of a cell phone

Description automatically generated
Fig.2.Demonstration of Coalescing bracketed sequence of images.

II. DISCRETE COSINE TRANSFORM

Discrete cosine transform (DCT) is an important transform in image processing. Large DCT coefficients are concentrated in the low frequency region; hence, it is known to have excellent energy compaction properties. It helps separate the image into parts (or spectral sub-bands) of differing importance (with respect to the image’s visual quality). The DCT is similar to the discrete Fourier transform: it transforms a signal or image from the spatial domain to the frequency domain.

Fig 3. Spatial to Frequency domain Transformation.

The 1D discrete cosine transform X(k) of a sequence x(n) of length N is defined as

One can observe that for k=0, the Eqn. (1) becomes

The first transform coefficient is the average of all samples in the sequence and is known as DC coefficient, and other transform coefficients are known as AC coefficients.
The inverse discrete cosine transform is defined as:

Eqn (1) is generally called as analysis formula or forward
transform and Eqn (3) is called as synthesis formula or inverse transform. The basis sequence

is real and discrete time sinusoids.

The 2D DCT is a direct extension of 1D DCT. The
2D discrete cosine transform X(k1,k2) of an image or 2D
signal x(n1,n2) of size N1 ´ N2 is defined as:

Similarly, the 2D inverse discrete cosine transform is
defined as:

Both DCT and IDCT are separable transformation and the advantage of this property is that 2D DCT or 2D IDCT can be computed in two steps by successive 1D DCT or 1D IDCT operations on columns and then on rows of an image x(n1,n2) as shown in Fig. 4.

Fig 4. Computation of 2-D DCT using separability property.

Benefits of DCT over other Transforms: –

• Data de-correlation: The ideal transform completely de-correlates the data in a sequence/block; i.e., it packs the most amount of energy in the fewest number of coefficients.

• Data-independent basis functions: Owing to the large statistical variations among data, the optimum transform usually depends on the data, and finding the basic functions of such transform is a computationally intensive task. This is particularly a problem if the data blocks are highly nonstationary, which necessitates the use of more than one set of basic functions to achieve high de-correlation. Therefore, it is desirable to trade optimum performance for a transform whose basis functions are data-independent.

• Fast implementation: The number of operations required for an n-point transform is generally of the order O(n2). Some transforms have

• implementations, which reduce the number of operations to O(n log n). For a separable n × n 2-D transform, performing the row and column 1-D transforms successively reduces the number of operations from O(n4) to O(2n2log n).

Fig. 5 showed that frequency distribution of the image which is converted by Discrete Cosine Transformation (DCT). According to the Fig. 5 images converted can be distributed by 3 parts, the coefficient on the left-top named DC value, others are named AC values. The DC value represents the average illumination and the AC values are coefficients of high frequency.

Fig.5. Frequency distribution of DCT

III. LUMINANCE & CHROMINANCE BASED TECHNIQUE 3.1 Luminance coalesce

 

 

In this section, a new image coalescence algorithm is proposed. The proposed algorithm can be applied to blend together a sequence of either color or greyscale images (minimum two images). The proposed algorithm operates in the YCbCr color space.The luminance (Y) channel represents the image brightness information and it is in this channel where variations and details are most visible, since the human visual system is more sensitive to luminance (Y ) than to chrominance (Cb, Cr). This important observation has two main consequences for the proposed coalesce algorithm. Firstly, it indicates that the coalesce of the luminance and chrominance channels should be done in a different manner, and that it is in the luminance channel where the most advanced part of the coalesce is to be performed. Secondly, it reveals that the same procedure used for the luminance channels coalesce can be used to blend single channel images (i.e., images in greyscale representation). In what follows, the proposed luminance coalesce technique is described, followed by chrominance coalesce

Luminance coalesce is performed taking the gradients with the maximal magnitude at each pixel position will lead to an image which has much more detail than any other image in the stack.

Let the luminance channels of a stack of N input images be I, where N >= 2. According to a commonly employed discretization model, the gradient of the luminance channel of the nth image in the stack may be defined as:

The magnitude of the gradient may be defined as

Let the image number having the maximum gradient magnitude at the pixel location

the blend luminance gradient may be represented as

A close up of a device

Description automatically generated

3.2. Chrominance coalesce
Chrominance coalesce is to be carried out for the coalesce of the input chrominance channels of color images in YCbCr representation (i.e., greyscale images). If the input images are in RGB representation, conversion to YCbCr should be performed first, to obtain the luminance (Y) and chrominance (Cb, Cr) channels representation. If the input images are in single channel (e.g., greyscale representation), this step does not apply.
Inherently different than luminance coalesce, chrominance coalesce operates directly on chrominance values, as opposed to their gradients. Specifically, the chrominance coalesce is done by taking a weighted sum of the input chrominance channels. The values in the chrominance channels have a range from 16/240 and carry information about color. These channels are such that when both Cb and Cr are equal to 128, the image is visually similar to a greyscale image, and thus carries the least amount of color information. This motivates selecting the weights for the chrominance channels such that at each pixel position they depend on how far from 128 the chrominance value is. Let us denote the chrominance channels of the input images by

The blend chrominance channels may be represented as follows:

where

 

Where

3.3 Haar Wavelet
Haar wavelet is a sequence of rescaled “square-shaped” functions which together form a wavelet family or basis. Wavelet analysis is similar to Fourier analysis in that it allows a target function over an interval to be represented in terms of an orthonormal basis. The Haar sequence is now recognized as the first known wavelet basis and extensively used as a teaching example.
The Haar sequence was proposed in 1909 by Alfred Haar. Haar used these functions to give an example of an orthonormal system for the space of square-integrable functions on the unit interval [0, 1]. The study of wavelets, and even the term “wavelet”, did not come until much later. As a special case of the Daubechies wavelet, the Haar wavelet is also known as Db1.
The Haar wavelet is also the simplest possible wavelet. The technical disadvantage of the Haar wavelet is that it is not continuous, and therefore not differentiable. This property can, however, be an advantage for the analysis of signals with sudden transitions, such as monitoring of tool failure in machines.
The Haar wavelet’s mother wavelet function can be described as follows

A close up of a clock

Description automatically generated

IV. EXISTING COALESCED TECHNIQUES

A. Laplacian Pyramid
Several approaches to Laplacian fusion techniques have been documented since Burt and Andelson introduced this transform back in 1983. The basic idea is to perform a pyramid decomposition on each source image, then integrate all these decompositions to form a composite representation, and finally reconstruct the fused image by performing an inverse pyramid transform. A schematic diagram of the Laplacian Pyramid fusion method is shown in Fig 6.

Fig.6 Laplacian Pyramid Technique

The first step is to construct a pyramid for each source image. The coalesced is then implemented for each level of the pyramid using a feature selection decision mechanism. It can be used several modes of combination, such as selection or averaging. In the first one, the combination process selects the most salient component pattern from the source and copies it to the composite pyramid, while discarding the less salient pattern. In the second one, the process averages the sources patterns. This averaging reduces noise and provides stability where source images contain the same patter information. The former is used in locations where the source images are distinctly different, and the latter is used in locations where the source images are similar. One other possible approach, chosen in this research, is to select the most salient component, following next equation

where Al, Bl and Fl are the two input and coalesced signals for levels 0 <=l <=N . Then a consistency filter is applied. The aim of this consistency filter is to eliminate the isolated points. Finally, for level N it is performed an average of both source components.

B. Wavelet Transform
An alternative to coalesced using pyramid based multi resolution representations is coalesced in the wavelet transform domain. The wavelet transform decomposes the image into low-high, high-low, high-high spatial frequency bands at different scales and the low-low band at the coarsest scale. The L-L band contains the average image information whereas the other bands contain directional information due to spatial orientation. Higher absolute values of wavelet coefficients in the high bands correspond to salient features such as edges or lines. With these premises, Li et al. propose a selection-based rule to perform image coalesced in the wavelet transform domain. Since larger absolute transform coefficients correspond to sharper brightness changes, a good integration rule is to select, at every point in the transform domain, the coefficients whose absolute values are higher. This idea is represented in Figure 7.

C:\Users\Sameer\Desktop\Untitled-3.jpg
Fig.7 Wavelet transform Technique

C. CEMIF
This fusion system proposed by Petrovic and Xydeas is based on an adaptive, multi resolution approach with a reduced number of levels. The goal of this technique is to reduce the computational complexity of multi resolution systems, such as
the Laplacian pyramid and Wavelet transform, while preserving the robustness and high image quality of multi resolution fusion. Actually, the spectral decomposition employed in this system represents a simplified version of the conventional Gaussian- Laplacian pyramid approach.
Multiscale structure is simplified into two levels of scale only, the background and the foreground levels. The former contains the DC component and the surrounding baseband and represents large scale features, on the other hand, the latter contains the high frequency information, which means small scale features. Signal fusion is performed at both levels independently. Background signals, obtained as the direct product of the average filtering, are combined using an arithmetic fusion approach. Foreground signals produced as the difference between the original and background signals are fused using a simple pixel-level feature selection technique. Finally, the resulting, fused, foreground and background signals are summed to produce the fused image. CEMIF fusion system is shown in fig 8.

Fig.8. CEMIF Fusion Technique.

D. Multi-resolution Singular Valued Decomposition
Multi-resolution singular value decomposition is very similar to wavelets transform, where signal is filtered separately by low pass and high pass finite impulse response (FIR) filters and the output of each filter is decimated by a factor of two to achieve first level of decomposition. The decimated low pass filtered output is filtered separately by low pass and high pass filter followed by decimation by a factor of two provides second level of decomposition. The successive levels of decomposition can be achieved by repeating this procedure. The idea behind the MSVD is to replace the FIR filters with singular value decomposition (SVD).
Let X = [x(1), x(2),…, x(N)] represent a 1D signal of length N and it is assumed that N is divisible by 2K for K &ge;1 21-26. Rearrange the samples in such a way that the top row contains the odd number indexed samples and the bottom row contains the even number indexed samples. Let the resultant matrix called data matrix is:

Denote the scatter matrix

Let U1 be the eigenvector matrix that brings T1 into diagonal matrix as:

The diagonal matrix

contains the squares of the singular values, with s1(1) > s2( 2) .The schematic diagram for the MSVD image fusion scheme is shown in Fig. 9.

Fig.9. MSVD Fusion process.

VI. IMPLEMENTATION

A. DCT

The full quality image of a real-world scene is generally much larger than that of the sensing devices used to capture it, as well as the imaging devices used to reproduce it. When a large quality range must be processed using a limited-range device, one is forced to split the large quality range into several smaller &rdquo;strips&rdquo;, and handle each of them separately. This process produces a sequence of images of the same scene, covering different portions of the large quality range. When capturing a high quality range scene, the sequence is obtained by varying the exposure settings of the sensor. When reproducing an high quality scene, the sequence is obtained by splitting the full range of the image into several sub-ranges, and displaying each separately.

C:\Users\Sameer\Desktop\1.jpg
Fig.10. Implementation Process

C:\Users\Sameer\Documents\major\Flow.jpeg
Fig.11. Flow Chart

The implementation process involves the following steps:

1. The images taken at various exposures (over-exposed, under- exposed, slightly exposed) or out of focus images are uploaded.

2. Next the images are divided into blocks of sizes NxN.

3. Then the DC and AC coefficients of each block are calculated.

4. The calculated DC coefficients are then averaged.
Xf ( 0,0 )= 0.5 (X1( 0,0 ) +X2 ( 0,0)) )

5. AC coefficients corresponding to the frequency band having largest energy are chosen.

6. IDCT is applied on the resultant matrix obtained from the final DC and AC coefficients.

7. Finally the coalesced image is obtained.

B. LUMINANCE & CHROMINANCE

A screenshot of a cell phone

Description automatically generated

VII. COALESCED EVALUATION METRICS

The following coalesced evaluation metrics are used to evaluate the performance coalesced algorithms when the reference image or the auto-focus image is available:
1. Root Mean Square Error (RMSE): It is computed as the root mean square error of the corresponding pixels in the reference image and the coalesced image . It will be zero when the reference and coalesced images are alike.

2. Peak signal to Noise Ration (PSNR): This value will be high when the coalesced and reference images are similar.

3. Coalesced quality index: The range of this metric is 0 to 1. One indicates the coalesced image contains all the information from the source images.

C:\Users\Sameer\Desktop\Untitled.jpg
4. Structural Similarity (SSIM) Index: It is a method for measuring the similarity between the coalesced and reference images. Its value may vary from -1 to 1. The value 1 implies that both images are identical. The coalesced image with high SSIM would be considered. The SSIM is computed as:

 

 

VIII. EXPERIMENTAL RESULTS

The coalescing process was implemented on a number of image sets and comparable results were obtained. The technique was successfully tested for both colour and grey-scale images. The results of the experiments are effectively shown as follows.

Fig 12. Front End

Fig.13. Screen for selecting the image for coalescing

Fig.14. GUI with uploaded color images (1st Sample) and image coalescing using DCT

A picture containing sitting, small, child, little

Description automatically generated
Fig.15. Error image obtain as result of the difference between the pixel values of the reference coloured image and the coalesced result

Fig.16.GUI with the uploaded colour images (2nd Sample) and coalesced result obtained using Exposure Fusion technique.

A close up of a bottle

Description automatically generated
Fig.17. Error image obtain as result of the difference between the pixel values of the reference coloured image and the coalesced result.

Fig.18.GUI with the uploaded grey-scale images (3rd Sample) and fused result obtained using DCT.

A picture containing photo, standing, sitting, man

Description automatically generated
Fig.19. Error image obtain as result of the difference between the pixel values of the reference grey-scale image and the coalesced result.

A screenshot of a cell phone

Description automatically generated

A screenshot of a cell phone

Description automatically generated

A screenshot of a cell phone

Description automatically generated
ACKNOWLEDGEMENT

With profound respect and gratitude, we would like to take this opportunity to convey our deep gratitude to Prof. Shravan Kumar (HoD, Computer Science and Engineering) for providing the infrastructure and labs in KIIT. We would like to thank our project guide Mr. Atul kumar (Associate Professor, Computer Science and Engineering) for his continuous help and support throughout the development of this project. Without his help, the project could not have been accomplished. We also acknowledge our sincere thanks to our Lab Assistant and all the faculty of KIIT for providing us with the guidance to steer our efforts in right direction. Last but not the least; we are thankful to our parents without whose efforts and blessing we could not come this far in our pursuit for success.

CONCLUSION

A technique for coalescing a bracketed sequence of images into a high-quality image based on discrete cosine transform (DCT) has been successfully carried out. The technique involves dividing the images to be coalesced into non-overlapping blocks of size NxN. DCT coefficients are computed for each block and blend rules are applied to get coalesced DCT coefficients. IDCT is then applied on the coalesced coefficients to produce the coalesced image/block. The procedure is repeated for each block. Experimental results show that the visual quality of the coalesced image obtained using this technique is comparable to other techniques for grey scale images but slightly inferior as compared to Luminance and Chrominance blend technique in case of colour images. The implemented technique is significantly useful for coalescing out of focus images. This technique is better than pixel averaging as it retains the original image contrast. As DCT coefficients obtained are real, which improve the time and
computational complexity. Blend performance is not good while using block sizes less than 8×8 and also if the block size is equivalent to the image size itself.

REFERENCES

[1] VPS Naidu, &ldquo;Discrete Cosine Transform-based Image Fusion Techniques&rdquo;, Journal of Communication, Navigation and Signal Processing , Vol. 1, No.1, pp.35-45, Jan. 2012.

[2] VPS Naidu, &ldquo;Discrete Cosine Transform-based Image Fusion&rdquo;, Special Issue on Mobile Intelligent Autonomous System, Defence Science Journal, Vol. 60, No.1, pp.48-54, Jan. 2010.

[3] VPS Naidu, Bindu Elias, &ldquo;A Novel Image Fusion Technique using DCT based Laplacian Pyramid&rdquo;, International Journal of Inventive Engineering and Sciences, Vol. 1, Issue 2, 2013.

[4] G. Strang, &ldquo;The Discrete Cosine Transform&rdquo;, SIAM Review, Vol.41, pp.135-147, 1999.

[5] Jinshan Tang, &ldquo;A Contrast based Image Fusion Technique in the DCT Domain&rdquo;, Digital Signal Processing, Vol. 14, pp.218-226, 2004.

[6] Tom Mertens and Jan Kautz, &ldquo;Exposure Fusion&rdquo;, University College London, Journal of Exposure Coalesced, UK, vol. 58, no. 2, pp. 468-493, 2011.

[7] Takao Jinno, and Masahiro Okuda, &ldquo;Multiple Image Fusion to get a High Dynamic Range Image&rdquo;, Journal of Multiple Image Coalesced vol. 17 , no. 21, pp. 238-260, 2011.