A Survey On Super Resolution Image Reconstruction Techniques

A Survey On Super Resolution Image Reconstruction Techniques

Abstract

The aim of this survey is to review various super resolution image reconstruction techniques, classify them into different categories and also discuss the recent issues in these methods(super resolution) and provide solution to handle them. Super Resolution is first, low-level and important task in computer vision. In this survey we categorize the super resolution techniques, provide detailed information of each technique in each category, as well as describe the pros and cons of that method. We define 3 basic steps for super resolution image reconstruction.

Keywords: Image Registration, Interpolation, Restoration.

1. Introduction

   The basic assumption for increasing the spatial resolution is the availability of multiple LR images captured from the same scene [5]. The LR images represent different looks at the same scene so LR images are sub sampled as well as shifted with sub pixel precision. If the LR images are shifted by integer units, then each image contains the same information and thus there is no new information that can be used to reconstruct an HR image. If the LR images have different sub pixel shifts from each other and if aliasing is present, then each image cannot be obtained from the others.[3] Here new information contained in each LR image can be exploited to obtain an HR image. If we combine these LR images, SR image reconstruction is possible. There is a natural loss of spatial resolution caused by the optical distortions because of out of focus, diffraction limit, motion blur due to limited shutter speed, noise that occurs within the sensor or during transmission and insufficient sensor density in the process of recording a digital image. A related problem to SR techniques is image restoration, which

   is a well-established area in image processing applications [6]. The goal of image restoration is to recover a degraded image, but it does not change the size of image. Restoration and SR reconstruction are closely related theoretically and SR reconstruction can be considered as a second-generation problem of image restoration. One more problem related to SR reconstruction is image interpolation that has been used to increase the size of a single image. Comparisons between various SR techniques have been primarily concerned with what assumptions are made in modeling the SR problem. Some of these assumptions include assuming the blurring process to be known or that regions of interest among multiple frames are related through global parametric transformations [7]. Signal-to-noise ratio, peak signal to noise ratio (PSNR), root mean squared error, mean absolute error, and mean square error (MSE) of super-resolved images versus interpolated images have all been used as objective measures of SR accuracy; however, the prominent method of presenting results in literature has clearly been subjective visual quality. Figure -1 shows the basic work process of super resolution image reconstruction.

   Figure 1: Common SR flow diagram

   In section 2, we categorize and describe the existing super resolution image reconstruction techniques and compare the techniques based on PSNR values. In section 3, we describe the related

   issues to the super resolution techniques. Section 4 concludes the reviews.

2. Super Resolution Algorithms.

   There are many different methods available for Super Resolution Image Reconstruction. I did comparison of some of the methods available. Algorithms are as follows:

   1. Nonuniform Interpolation

This approach is the most intuitive method for SR image reconstruction. In this approach three stages are performed successively.

1. Relative motion is estimated, i.e., registration if the motion information is not known.

2. Nonuniform interpolation is done to produce an improved resolution image, and

3. Process of deblurring is done depending on the observation model.

Non-uniform interpolation has relatively low computational complexity and it assumes that the blur and noise characteristics are identical across all LR images. The HR image on non uniformly spaced sampling points is obtained with the relative motion information estimated. Then, the direct or iterative reconstruction procedure is followed to produce uniformly spaced sampling points. Once an HR image is obtained by non uniform interpolation, we tackle the restoration problem to remove blurring and noise.

1. Frequency Domain

   Aliasing which exists in all LR images is explicitly used to reconstruct HR image. Tsai and Huang [1] first derived a system equation that describes the relationship between LR images and a desired HR image by using the relative motion between LR images. The frequency domain approach is based on three principles:

   1.The shifting properties of the Fourier transform. 2. The aliasing relationship between the continuous Fourier transform (CFT) of an original HR image and the discrete Fourier transform (DFT) of observed LR images.3.The assumption that an original HR image is band limited.

   These properties make it possible to formulate the system equation relating the aliased DFT coefficients of the observed LR images to a sample of the CFT of an unknown image. Main advantage of the frequency domain approach is theoretical simplicity, it means the relationship between LR images and the HR image is clearly demonstrated in the frequency domain. This method is also convenient for parallel

   implementation capable of reducing hardware complexity.

2. Nearest Neighbor algorithm

   The Nearest Neighbor interpolation is the fastest and simplest option. It simply takes the color of a pixel and assigns it to the new pixels that are created from that pixel. Due to this simplistic approach, it does not create an anti-aliasing effect. This leads to problems with jaggies. Consequently, Nearest Neighbor interpolation is considered to be incapable of producing photographic quality work. It selects the value of the nearest pixel by rounding the coordinates of the desired interpolation point x 0.With an obvious extension to the two-dimensional case. Let

   · is the floor operator: the largest integer less than or equal to the argument. As a result of this simplistic interpolation scheme, nearest neighbor doesn't have subpixel accuracy and generates strong discontinuities, especially when arbitrary rotations and scale changes are involved. The only interesting property of this algorithm is the fact that it preserves the original noise distribution in the transformed image, which can be useful in some image analysis applications. This is most commonly used in-camera when reviewing and enlarging images to view details. It simply makes the pixels bigger, and the color of a new pixel is the same as the nearest original pixel.

3. Bilinear Interpolation algorithm

   Bilinear interpolation uses the information from a pixel (let's call it the original pixel) and four of the pixels that touch it to determine the color of the new pixels that are reated from the original pixel. Bilinear uses rather simple, linear calculations to do this. The Bilinear interpolation does have an anti- aliasing effect. However, it is not considered good enough for photo quality images. This takes the information from an original pixel, and four of the pixels that touch it, to decide on the color of a new pixel. It produces fairly smooth results, but it reduces the quality significantly. Images can become blurry.

4. Bicubic Interpolation algorithm

   Bicubic interpolation uses the information from an original pixel and sixteen of the surrounding pixels to determine the color of the new pixels that are created from the original pixel. Bicubic interpolation is a big improvement over the nearest neighbour interpolation and bilinear interpolation methods for two reasons:

   (1) Bicubic interpolation uses data from a larger number of pixels and (2) Bicubic interpolation uses a Bicubic calculation that is more sophisticated than

   the calculations of the previous interpolation methods. Bicubic interpolation is capable of producing photo quality results and is probably the method most commonly used. This is the most sophisticated of the bunch, as it takes information from the original pixel and 16 surrounding pixels to create the color of a new pixel. Bicubic calculation is far more advanced than the other two methods, and it is capable of producing print quality images. Bicubic interpolation also offers the two variants of "Smoother" and "Sharper" for finely tuned results.

5. Regularized SR Reconstruction

   The SR image reconstruction approach is generally an ill-posed problem because of an insufficient number of LR images and ill-conditioned blur operators. Procedures adopted to stabilize the inversion of ill-posed problem are called regularization. There are two regularization approaches for SR image reconstruction, deterministic and stochastic. The deterministic regularized SR approach solves the inverse problem by using the prior information about the solution which can be used to make the problem well posed. Stochastic SR image reconstruction, typically a Bayesian approach, provides a flexible and convenient way to model a priori knowledge concerning the solution. Constrained least squares (CLS) and maximum a posteriori (MAP) SR image reconstruction methods are introduced.

6. Projection onto Convex Sets

   Low resolution images usually suffer from blurring caused by a sensors point spread function (PSF) and additionally from aliasing caused by under-sampling. The POCS method describes an alternative iterative approach to incorporate prior knowledge about the solution into the reconstruction process. With the estimates of registration parameters, this algorithm simultaneously solves the restoration and interpolation problem to estimate the SR image. The POCS formulation of the SR reconstruction was first suggested by Stark and Oskoui [2]. An estimate of the high-resolution version of the reference image is determined iteratively starting from some arbitrary initialization. Successive iterations are obtained by projecting the previous estimate onto the consistency set with an amplitude constraint set that restricts the gray levels of the estimate to the range [0, 255]. The advantage of POCS is that it is simple, and it utilizes the powerful spatial domain observation model. It also allows a convenient inclusion of a priori information. These methods have the disadvantages

   of non uniqueness of solution, slow convergence, and a high computational cost.

7. Papoulis-Gerchberg Algorithm

   This method assumes two things:

   • Some of the pixel values in the high-res grid are known.

   • The high frequency components in the high- res image are zero.

     It works by projecting HR grid data on the two sets described above. The steps are:

     1. Form a high res grid. Set the known pixels values from the low-res images (after converting their pixel position to the ref frame of first low- res image). The position on the HR grid is calculated by rounding the magnified pixel positions to nearest integer locations.

     2. Set the high-freq components to zero in the freq domain.

     3. Force the known pixel values in spatial domain. Iterate.

8. Iterative Back-Projection (IBP)

First proposed in [11], IBP is based on a similar idea as the computer-aided tomography where a 2-D object is reconstructed from its 1-D projections. The method involves a registration procedure, an iterative refinement for displacement estimation, and a simulation of the imaging process (the blurring effect) using a PSF. This approach begins by guessing an initial HR image. This initial HR image can be generated from one of the LR images by decimating the pixels. This initial HR image is then down-sampled to simulate the observed LR images. The simulated LR images are subtracted from the observed LR images. If the initial HR image was the real observed HR image, then the simulated LR images and the observed LR images would be identical and their difference zeros. Hence, the computed differences can be back-projected to improve the initial guess. The back-projecting process is repeated iteratively to minimize the difference between the simulated and the observed LR images, and subsequently produce a better HR image. IBP is intuitive hence easy to understand. However, its ill-posed nature means that there is no unique solution. The choice of back-projection filter is arbitrary. Compared to other approaches such as regularization approaches, it is more difficult to incorporate prior information.

Table-I: Results of some algorithms based on PSNR.

1. Advanced Issues in SR

   1. SR Considering Registration Error Registration is a very important step to the success of the SR image reconstruction as

      mentioned earlier. Therefore, accurate registration methods, based on robust motion models including multiple object motion, occlusions, transparency, etc., should be needed.

   2. Blind SR Image Reconstruction and Computationally Efficient SR Algorithm[10] In many practical situations the blurring process is generally unknown or is known only to within a set of parameters. So, it is necessary to incorporate the blur identification into the reconstruction procedure. To apply the SR algorithm to practical situations, it is important to develope an efficient algorithm that reduces the computational cost.

   3. A PDE Approach to Super resolution with Contrast Enhancement[8]

      They present a fast partial differential equation (PDE) model for multi-frame image super resolution reconstruction. Then combine their proposed super resolution model with the local histogram equalization (LHE), which perform super resolution and enhance image contrast simultaneously. It overcomes the shortcomings of recent promising super resolution methods dealt with super resolution and contrast enhancement separately.

   4. A High-efficiency Super-resolution Reconstruction Algorithm from Image/Video sequences[9]

      So far, existin super-resolution reconstruction methods are all confronted with the problem of slow convergence and expensive computation. To satisfy the requirement of real-time application, They

      propose a high-efficiency super-resolution reconstruction algorithm that solves two key bottlenecks in the multi-frame MAP framework. The first breakthrough is to select the Armijo rule to identify the step length instead of the exact line search. The second one is to approximately compute the gradient of the MAP objective function using analytic representation instead of numerical calculation.

   5. Super Resolution Reconstruction of Compressed Low Resolution Images using Wavelet Lifting Schemes[4]

      Here, They propose lifting schemes for intentionally introducing down sampling of the high resolution image sequence before compression and then utilize super resolution techniques for generating a high resolution image at the decoder. Lifting wavelet transform has its advantages over the ordinary wavelet transform by way of reduction in memory required for its implementation. This is possible because lifting transform uses in-place computation. The lifting coefficients replace the image samples present in the respective memory locations.

2. CONCLUSION

In this article, we have presented a survey of super resolution techniques. We classify super resolution image reconstruction techniques into various categories. We provide details of each category, with its pros and cons. Here, we also present the result of some of the methods for the comparison purpose. We describe the various issues for super resolution image reconstruction. This article gives valuable insight into this important research topic and encourages the new research in the area of super resolution image reconstruction as well as in the field of computer vision.

REFERENCES

1. R.Y. Tsai and T.S. Huang, Multipleframe image restoration and registration, in Advances in Computer Vision and Image Processing. Greenwich, CT: JAI Press Inc.,1984, pp. 317-339.

2. H. Stark and P. Oskoui, High resolution image recovery from image-plane arrays, using convex projections, J. Opt. Soc. Am. A, vol. 6, pp.1715-1726, 1989.

3. R.R. Schulz and R.L. Stevenson, Extraction of high- resolution frames from video sequences, IEEE Trans. Image Processing, vol. 5, pp. 996-1011,June 1996.

4. Liyakathunisa, C.N .Ravi Kumar, V.K.Ananthashayana, Super Resolution Reconstruction of Compressed Low Resolution Images using Wavelet Lifting Schemes, 2009 Second International Conference on Computer and Electrical Engineering.

5. S. Chaudhuri, Ed., Super-Resolution Imaging. Norwell, MA: Kluwer, 2001.

6. S.P. Kim, N.K. Bose, and H.M. Valenzuela, Recursive reconstruction of high resolution image from noisy undersampled multiframes, IEEE Trans.Acoust., Speech,

   Signal Processing, vol. 38, pp. 1013-1027, June 1990

7. W. Zhao and H. S. Sawhney, Is super-resolution with optical flow feasible?, inECCV 02: Proceedings of the 7th European Conference on Computer Vision-Part I, pp. 599613, Springer-Verlag, 2002.

8. A High-efficiency Super-resolution Reconstruction Algorithm from Image/Video Sequences, Third International IEEE Conference on Signal-Image technologies and Internet-Based System, Chuangbai Xiao1, Jing Yu1, and Yi Xue2.

9. A PDE Approach to Super-resolution with Contrast Enhancement, 2010 3rd International Congress on Image and Signal Processing (CISP2010), Weili Zeng,Xiaobo Lu.

10. Ayers G. R., and Dainty J. C. Iterative blind deconvolution methods and its applications. Optics Letter, vol. 13, no. 7, July 1988.

11. E.D. Castro, C. Morandi, Registration of translated and rotated images using finite Fourier transform , IEEE Transactions on Pattern Analysis and Machine Intelligence 700703,1987.