A Hardware Modelling of Motion based Super Resolution Image Reconstruction

A Hardware Modelling of Motion based Super Resolution Image Reconstruction

Sahana. S

Department of Electronics and Communication Siddaganga Institute of Technology

Tumkur-572103, India

1. S. Shilpashree Asst.Professor, Department of ECE Siddaganga Institute of Technology

   Tumkur- 572103, India

   Abstract-A computational process known as super resolution (SR) is used to reconstruct the High Resolution (HR) image from the blurred, noisy and aliased low resolution (LR) images. SR reconstruction is an effective approach for increasing the spatial resolution of image, there is no need of changing the hardware of the existing imaging system. In a sequence of image captured from a camcorder there is a substantial overlap between the successive frames. In this paper, the HR image is obtained by the LR observation by the SR method known as iterative back projection (IBP). In IBP the simulated images are compared with the observed images and the difference between these two will be considered as the residual error. The obtained residual error will be the update estimate for the successive iteration. The iterative process will be continued until the error get minimum. The simulations tools such as MATLAB and XILINX are used to test the IBP process. The XILINX has the comparative higher speed than MATLAB. From all the advantages of the SR, it can mentioned as the most useful and practical algorithm, which can be used to improve the image resolution.

   Keywords Super Resolution, High Resolution, Image restoration, Iterative back projection.

   1. INTRODUCTION

      Pixel density in an image is referred as spatial resolution. If the pixel density within an image is high then it is known as HR, which will offer fine details. Charge couple device (CCD) and complementary metal-oxide semiconductor (CMOS) are the commonly used image sensors to capture digital images. When capturing the image from a LR camera of a scene there will be natural loss of spatial resolution caused due to the optical blur, motion blur, noise and the environmental artifacts.

      The most direct method to increase the spatial resolution is to reduce the pixel size by the sensor manufacturing technique. A reduction in sensor size leads to the collection of photons over a smaller area and hence, an increased sensitivity to shot noise. Thus, resolution enhancement by sensor manufacturing techniques is limited by the minimum sensor area beyond which shot noise degrades the image. The imaging industry has already reached this pixel size limit of

      number of pixels in the image may not contribute towards enhancing fine details in the image content.

      Another approach for enhancing the spatial resolution is to increase the chip size, which leads to an increase in capacitance [2]. Large capacitance makes it difficult to speed up a charge transfer rate; this approach is not considered effective. The precision optics and image sensors cost is too high. Therefore, by looking at the difficulties in the hardware based solutions, a new approach towards increasing spatial resolution based on software is required to overcome these limitations of the sensors and optics manufacturing technology.

      A software-based method is more attractive because it can be applied after the image sequence has already been captured and can make the trial and error method in choosing the parameter used in the process. There is no need of changing the components or the experimental set up. The 1980s saw the beginning of a growing area of signal processing called super resolution (SR) which is a promising alternative to obtain HR images. The advantages of SR are that it may cost less and the existing low-resolution (LR) imaging systems can still be utilized.

      SR [2] is a process obtaining the HR image by fusing information from a sequence of images, which are captured in one or more of several possible ways. To construct a HR version at least two non identical images are required. The LR frames can be obtained by displacement with respect to the reference frame, for e.g., in the satellite imaging, where there will be considerable distance between the scene and camera, blur induced due to optical aberration, relative motion between the camera and object, and finally degraded by additive noise

      This paper is organized as follows: in section II literature survey, III a motion based super resolution model is described. This section gives basic premise for SR. The observation model is described in section IV. SR reconstruction using IBP method is described in section V. In section VI, the SR reconstruction is evaluated through experimental results. Finally, conclusion will be discussed in section VII.

      40 m2 for a 0.35 m [1] CMOS process. Increasing the

   2. LITERATURE SURVEY

      The survey of SR begins with the pioneering algorithm which was presented by Huang & Tsai, who made explicit use of the aliasing effect, assuming the image is band limited, and the images are noise-free. Kim et.al. generalized the work to noisy and blurred images, using least square minimization. Spatial domain algorithm was presented by Ur & Gross. Assuming a known 2D translation, a fine sample grid image was created from the input images, using interpolation, and the camera blur was cancelled using deblurring technique. The above methods assumed blur function which is uniform over all the images, and identical on different images. They were also restricted to global 2D translation.

      A Different Approach was suggested by Irani & Peleg, based on previous work by Peleg et al [3]. The basic idea, Iterative Backward Projecting IBP, was adopted from computer-aided Tomography (CAT) [4]. An initial guess is made and the imaging process is iteratively simulated by reprojecting the error back to the SR image. The IBP algorithm can handle general motion and non-uniform blur function, assuming they can be approximated accurately.

      SR falls under the group of image restoration techniques which have a wide variety of applications. The more popular applications of SR include surveillance applications like facial and license plate recognition; medical imaging applications [5] fusing images from different modalities; resolution enhancement of astronomical and satellite imagery; and converting low resolution video to high resolution video amongst others.

   3. MOTION BASED SUPER RESOLUTION

      The main idea of the motion based SR techniques is the availability of multiple LR images capture from the same scene. Usually the LR images represent different information of the same scene. That is, LR images are sub sampled (aliased) as well as shifted with sub pixel precision. If there is an integer shift in LR, then there is no new information, which will not help to reconstruct the HR. So in motion based SR, the LR images have different sub pixel shifts from each other. In this case, the new information contained in each LR image can be fused to obtain an HR image.

      To obtain the different information of the same scene, one camera with several captures or from multiple cameras located in different positions. There exists the scene motion in successive frames. These scene motions can occur due to the controlled motions in imaging systems, e.g., images acquired from orbiting satellites. The same is true of uncontrolled motions, e.g., movement of local objects or vibrating imaging systems. The scene motion are estimated and the different information of these LR images are fused together to get the HR. The basic idea for the SR reconstruction is illustrated in Fig. 1 [6].

      Fig. 1. Basic idea of SR [6].

   4. OBSERVATION MODEL

      The observation model consists of LR images. The LR image is obtained by wrapping, blur, decimating and adding the noise to the HR image. The general approach to obtain the LR images is a forward model, in which the known HR image will be considered. The goal of the SR reconstruction is to reconstruct back the HR image; this is known as the inverse model. Choosing the imaging model is very important in SR reconstruction; if the imaging model goes wrong then the image will be degraded further. After that the SR procedure can then be applied to a region of interest. The key to successful SR consists of accurate alignment i.e. registration (observed images are mapped to a common reference) and formulation of an appropriate forward image model.

      A. Building blocks for obtaining the LR images

      The vital success of the SR algorithm is an accurate description of the observation model. The imaging sensors are characterized and the type of scene is being imaged in the description. Such assumptions impact the choice of displacement model used and the prior knowledge incorporated into the SR algorithm. A skeleton diagram of a general observation model is shown in Fig. 2 [7].

      The degradation model has the fundamental components which comprises the warp operator, the blur operator and the down-sampling operator. The displacement that takes places between two images in a sequence is described by the warp operator. The camera motion, object motion in the scene or a combination of both will induce the displacement in the images. The blurring operator describes the cumulative blurring effects from sensor averaging, motion blur and an out-of-focus blur etc. These cumulative effects can be combined into a single blurring operator represented by a single point spread function.

      Fig. 2. Degradation model for the SR reconstruction problem [7].

      2

      The image formation process [8] can be formulated in the following way:

      imaging system. A residual error is formulated by comparing the observed data and simulated image. A small residual error suggests that the scene estimate is an accurate one, whereas a significant residual error indicates that the estimate is poor. The error information can be used to improve the scene estimate. By a process called back-projection," [11] the error residual is used to form an updated estimate of the scene which is a better approximation of the original. The process is iterated until the simulation residual error becomes minimum. The iterative process thus comprises two steps: simulation of the observed images, and back-projection of the error to correct the estimate of the original scene.

      In the iterative methods, for practical reasons it is assumed that the noise is uncorrelated and has uniform variance. In this case, the maximum likelihood solution [12] is found by minimizing the functional:

      x n

      DH W x e

      (1)

      E x H

      = 1

      2

      x L -Ax H

      (4)

      L n n H n

      H L

      A single matrix vector is obtained by stacking the vector equations from the different images. The extension of the equation (1) up to n images is as shown in equation (2).

      Taking the derivative of E with respect to equation (4) and setting the gradient to zero.

      E=0

      x H for the

      x L

      1

      DH1W1

      e1

      AT Ax -x

      =0

      2

      DH W

      e

      K T T T n

      (5)

      x L

      2 2 x

      2

      (2)

      Wn Hn D

      DHn Wn xH -xL

      =0

      H

      n=1

      x

      n

      L

      DHnWn

      en

      The iterative methods can be used without explicit construction of these large matrices. Instead, the

      xL =AxH +en (3)

      multiplication with A and AT . A is implemented using only image operations such as warp, blur and sampling. The

      Where:

      matrices Wn , Hn and D model the image formation process,

      xH is the HR image XH of size N1N2×1.

      and their implementation is simply the image warping, blurring and respectively [13].

      1 2

      L

      K

      x n is the nth image of size M M ×1.

      en is the normally distributed additive noise in the nth image.

      x {m+1}=x {m}+WTHTDT (x(n) -DH W x {m})

      -H -H n n L n n -H

      n=1

      Wn is the geometric warp matrix, of size N1N2×N1N2 .

      Hn is the blurring matrix, of size N1N2×N1N2 .

      D is the decimation matrix, of size M1M2×N1N2 .

      As per the equation (1) [9] the four LR images are obtained. Each of which differs in the warp parameter, keeping the blur and noise parameter constant. To reconstruct the HR image from this LR images IBP algorithm is used.

   5. ITERATIVE BACK PROJECTION

      The IBP is a simulate-and-correct approach to restoration. It has a widespread use in computed topographic reconstruction [10]. The objective of IBP is to determine the HR scene. The assumed initial estimate of the scene is available. If the prior knowledge of the observed scene is available then it is possible to simulate the output of the

      (6)

      This is a version of the Iterated Back Projection [14], using a specific blur kernel and forward warping. In equation (6),

      {m} represents the previous iteration, {m+1} represents the present iteration. xH mis the HR image of the previous iteration. In proceeding iterations the obtained image is used as the original image and the error is back projected.

   6. EXPERIMENTAL RESULTS AND DISCUSSION In this section, we obtained the SR image by applying the

      IBP. The algorithm is tested on the synthetic images in image domain only. The performance measure of the IBP is demonstrated by the Mean square error (MSE) and Improve mental signal-to-noise ratio (ISNR) which are calculated as

      MSE

      1

      N1 N 2

      || X X ||

      7

      ISNR

      10 log 10

      || X X (0) ||

      || X X ||

      8

      In the above equations,

      N ,N are the size of the original

      1 2

      image, X is the original image, X(0) is the initial estimate of

      the super resolved image which is obtained by averaging the

      bilinear up-sampled aligned images, and X

      resolved image.

      is the super

      We demonstrate the performance of the IBP by considering the three different cases which are mentioned below:

      Case 1: Ideal case

      In the first case, a single high quality image of size

      1. (d)

         Fig. 3. (a) Original Image. (b) LR image. (c) Bilinearlly interpolated image.

      2. SR image using IBP.

         256 256 pixels as shown in the Fig. 3(a) is considered for

         which the four geometrical warped, blurred and down sampled image of size 128 128 pixels are generated. Bilinear interpolation is used to perform warping to sub pixel shift with motion parameters (0, 0), (0, 0.5), (0.5, 0.5) and (0.5, 0) in horizontal and vertical direction. Each of the LR text images was synthetically blurred by Gaussian blur of standard deviation of 0.05, down sampled by a factor of 2 and added Gaussian noise of zero mean and 0.01 variance. Fig. 3(b) shows one such degraded LR image. The image quality is not improved by using bilinear interpolation as shown in the Fig. 3(c) which does not help to readability, some blur is still present. Fig. 3(d) shows the SR image which is obtained using the IBP algorithm for 100 iterations which gives the good quality image as good as original image and the readability is improved. The performance results of this case are as shown in the Fig. 4(a) & (b). The MSE value and the ISNR value are 6.4973 and 21.3851 dB respectively.

         (a) (b)

         (a) (b)

         Fig. 4. (a) MSE plot. (b) ISNR plot.

         Case 2: High blur

         In the second case all the constraints are same except the blur value. Here the standard deviation of 0.5 is considered which is as shown in the Fig. (5). here it is seen that the original image and the SR image is almost similar and the text in the image is readable. The performance of the second case is as shown in the Fig. 5 (d), where the MSE is exponentially decreasing which in turn specifies that the IBP is having the better performance. The MSE value is 10.73 and ISNR value is 18.2059 dB.

         (a) (b)

         1. (d)

            Fig. 5. (a) Original image. (b) LR image. (c) SR image. (d) MSE plot.

            Case 3: High noise

            In this case the experiment is repeated by choosing the less camera defocus parameter ( 0.05) . The Gaussian noise with zero mean and 0.1 variance is added to each LR images. The warping parameters are same as in case 1. The bilinear interpolation output as shown in Fig. 6(b) is still degraded which is having the blur and some noise. After SR reconstruction by IBP the output is as shown in the Fig. 6(c). The MSE value and ISNR value for the high noise is 20.3012 and 10.2472 dB respectively. From both case output we infer that less noise much better than the high noise. Even though the small amount of noise is present in the image, the text is readable and visible clearly. So IBP works better for all cases under some constraints.

            1. (b)

               (c) (d)

               Fig. 6. (a) Original image. (b) LR image for noise of variance = 0.1. (c) SR image. (d) MSE plot.

               The comparison between these three cases is tabulated in Table I. It infers that the IBP performance better when the value of noise added will be less. Whereas the text in the image is concerned, it can be readable in all the cases. This will serves the purpose in typical applications such as license plate recognition in surveillance.

               TABLE I. MSE AND ISNR VALUES FOR DIFFERENT CASES

               Performance parameters	Different cases considered
               	Ideal case	High blur	High noise
               MSE	6.4973	10.73	20.3012
               ISNR (in dB)	21.3851	18.2059	10.2472

   7. HARDWARE MODELLING AND SIMULATION

      RESULTS

      Xilinx ISE (Integrated Software Environment) [15] is a software tool produced by Xilinx for synthesis and analysis of HDL designs, in which the developer can synthesize their designs, perform timing analysis, examine RTL diagrams, simulate a design's reaction to different stimuli, and configure the target device with the programmer.

      The IBP equation is simulated in Xilinx 14.2 carried out for 20 iterations. For this process the four LR image pixel values are imported from the MATLAB directly, then all other process are continued in verilog coding which is written in ISE design suite. The comparison between the MATLAB and XILINX in terms of speed is shown in Fig. 7 and Fig. 8 respectively. In Fig. 7 the elapsed time is 11.3936 seconds and in Fig. 8 the entire simulation time is 1.675ms. From this it can be inferred that the XILINX has more speed comparatively with MATLAB. The XILINX provide only the pixel values of the reconstructed output image as shown in the Fig. 8. The coding is done in such a way that the text file of the reconstructed image has generated. The generated outfile is read in the MATLAB and the corresponding reconstructed image has displayed. The SR image and reconstructed image are as shown in the Fig. 9 (a) and Fig. 9

      1. respectively. Both the images resemble the same. To verify the output of both MATLAB and XILINX the performance parameter called MSE and the timing analysis has been used which is tabulated in Table II. from the values it can be said that the MSE values are almost same in MATLAB and XILINX. The timing analysis tells that the XILINX is faster than the MATLAB.

         Fig. 7. MATLAB simulation elapsed time in seconds

         Fig. 8. Simulation results and timings of XILINX.

         1. (b)

      Fig. 9. (a)SR image obtained by MATLAB. (b) Reconstructed image obtained by XILINX.

      TABLE II. COMPARISON BETWEEN MATLAB AND XILINX OF MSE AND TIMING ANALYSIS.

      Simulation platform	Parameters considered for comparison
      	MSE	Total simulation time for 20 iterations
      MATLAB	6.4973	11.3936s
      XILINX	6.8876	1.675ms

   8. CONCLUSION

This paper presents the SR reconstruction problem. In this problem, an SR image has recovered from warped, blurred, down sampled and noisy LR image. The simulations are carried out on the synthetic images. It is observed that IBP gives the good results compared with the bilinear interpolation. The bilinear interpolation over smoothened the

image where as it does not provide readability but IBP improve the readability. The results of the IBP for different cases are compared which shows that the algorithm works best for the ideal and high blur conditions. Even though for high noise the results comparatively degrade, it provides the readability. Additional to this the plot and the tabulated values of MSE and ISNR demonstrate the effectiveness of the algorithm. The MSE values of both the simulation platform are similar. The timing analysis between the simulations result of XILINX is comparatively speeder than the MATLAB simulation results.

REFERENCES

1. Mustafa H. Fanaswala, a thesis on Regularized Super-Resolution of Multi-View Images, Carleton University Ottawa, August 2009.

2. Michael K and Andy C, Super Resolution Image Restoration from Blurred LR Images in Journal of Mathematical Imaging and Vision 23: pp. 367378, 2005.

3. Assaf Zomet and shmuel Peleg, Super Resolution from multiple images having Arbitrary Mutual Motion. Kluwer Academic Publishers, New York, 2002.

4. Xizhang Wei ; Zhen Liu ; Xiaofeng Ding ; Meimei Fan Super- Resolution Reconstruction of Radar Tomographic Image Based on Image Decomposition, IEEE Vol. 11 , Issue. 3 Digital Object Identifier, pp. 607 611, 2014.

5. Ashikaga, H, Estner, H.L and Mcveigh, E.R, Quantitative Assessment of Single-Image Super-Resolution in Myocardial Scar Imaging, in IEEE Journal of Translational Engineering in Health and Medicine, Vol. 2 , Issue. 1, 30 January 2014.

6. Sung Cheol Park, Min Kyu Park, and Moon Gi Kang, SR Image Reconstruction: A Technical Overview in IEEE Signal Processing Magazine, May 2003.

7. Baikun Wan and Weijie Wang Super Resolution Reconstruction based on Totla Variation regularization, in International Conference on Robotics and Biometics, Tianjin, China, December 2010.

8. K. V. Suresh, G.Mahesh Kumar and A. N. Rajagopalan, Superresolution of Licence plate in Real Traffic Videos in IEEE transaction on Intelligent Transportation System, Vol. 8, pp. 2, June 2007.

9. Georgis, G, Lentaris, G and Reisis, D, Single-image super-resolution using low complexity adaptive iterative back-projection in 18th International Conference on digital signal processing (DSP), pp. 1-6, 1- 3 July 2013.

10. Milan. N. Bareja and Chintan. K. Modi, An Effective Iterative Back Projection based Single Image Super Resolution Approach, in International Conference on Communication System and Network Technologies, 2011.

11. Hailiang Li and Kin-Man Lam, Guided iterative back-projection scheme for single-image super-resolution in Global High Tech Congress on Electronics (GHTCE), 2013 IEEE, pp. 175 180, 17-19 November 2013.

12. Haiying, Song Xiaohai, Weilong Chen and Yanyue Sun, An Improved Iterative Back-Projection Algorithm for Video Super-Resolution Reconstruction in Photonics and Optoelectronic (SOPO), Symposium, pp. 1 4, 19-21, June 2010.

13. Subhasis Chauhuri, Super-Resolution Imaging Kluwer Academic Publishers, New York, 2002.

14. Xiuju Liang and Z ongliang Gan, Improved Non-local Iterative Back- Projection Method for Image Super-Resolution, in Sixth International Conference on Image and Graphics (ICIG), pp. 176 181, 12-15, August 2011.

15. Soares dos and Ferreira, Improvement on control performance using FPGAs over software-based platforms in 17th Conference on Emerging Technologies & Factory Automation (ETFA), pp. 1 4, 17- 21, September 2012.