Super Resolution – A Review

Super Resolution – A Review

M. Sangeetha

Alagappa University

Dr. A. Senthilrajan Director,Computer Centre Alagappa University,Karaikudi Tamil Nadu,India

Abstract—- Super resolution reconstruction is the technique of increasing the resolution of the image. The set of low resolution images (LR) are combined to form one or more high resolution image (HR).The super resolution reconstruction is possible only when the LR images acquired need to have different information of the same scene. In this paper existing super resolution reconstruction techniques of spatial and frequency domain methods are analyzed. Recent contributions were also studied.

Keywords: Super Resolution; Non-uniform interpolation; SR reconstruction.

1 .INTRODUCTION

Super resolution (SR) is a method to create high resolution image (HR) from one or more low resolution images (LR).In digital image processing the extraction of high frequency components of an image is of great importance. An application of SR includes satellite imaging where multiple images are taken from the same location and SR could be used to get more information for a certain object. In surveillance, region of interest is desired. Further, it is used in medical imaging such as in magnetic resonance imaging (MRI) and computer tomography (CT) with the possibility of more images with limited resolution. Another useful application would be conversion of SDTV to HDTV signal to provide the demand for HD quality.SR algorithms can be classified as i) Interpolation based ii) Example based iii) Reconstruction based methods. Interpolation- based SR algorithms are fast but the results may lack some of the fine details. In example-based SR algorithms detailed textures are elucidated by searching through a training set of LR/HR images. They need a careful selection of the training images, otherwise erroneous details may be found. Alternatively, reconstruction-based SR algorithms apply various smoothness priors and impose the constraint that when properly down sampled, the HR image should reproduce the original LR image[1].

The effort to attain the very high resolution coincides with technical limitations. Charged coupled device (CCD) or complementary metal-oxide-semiconductor (CMOS) sensors are widely used to capture image signals. Spatial

resolution of the image is determined mainly by the number of sensor elements per unit area. The straight solution to increase spatial resolution is to increase the density of the pixel per unit area by reducing the size of the pixel. However, as the pixel size decreases, the amount of light impact on each sensor element also decreases and more shot noise is generated [2]. Another way to enhance the spatial resolution is to increase the chip size. However, the increase in the capacitance decreases charge transfer rate. Due to the cost of high precision optics and image sensors limits scientific applications. Super-resolution is the term generally applied to the problem of transcending the limitations of optical imaging systems through the use of image processing algorithms, which presumably are relatively inexpensive to implement [7].

The major advantage of the signal processing approach is that it may cost less and the existing LR imaging systems can still be used. The SR technique is possible only with the availability of multiple LR images of the same scene. If the LR images are shifted by integer units, then each image contains the same information, and thus there is no new information that can be used to reconstruct an HR image [3].Each LR image should provide new information to acquire a HR image. The sub-pixel shift between the images is obtained by capturing the same scene from multiple cameras positioned at different angles or by multiple captures by using single camera. The sub-pixel accuracy is estimated and pixels are aligned in the HR grid using non uniform interpolation as presented in the figure.

1. OBSERVATION MODEL

   When capturing a digital image, the original image of the scene cannot be obtained. Generally the captured image has several kinds of degradations due to the hardware limitation of the optical system. The acquired image suffers from various defects such as optical blur caused by the size of the lens, minimal sensor size leads to sensor blur, motion blur due to limited aperture time. The inadequate sensor density results in aliasing effects

   kth warped image

   kth warped image

   Desired HR image X

   Continuous Scene

   Sampling without Aliasing

   Sampling without Aliasing

   Sampling

   Translation Rotation, etc.

   Translation Rotation, etc.

   Motion

   Blur Down Sampling

   LR

   Optical Blur Motion Blur Sensor Blur, etc

   Optical Blur Motion Blur Sensor Blur, etc

   Down sampling with aliasing

   Down sampling with aliasing

   Observations

   Sensor noise

   Sensor noise

   Fig 1. The Observation model relating HR images to LR images.

   thus create a reduced spatial resolution. A better observation model is required that relates the HR image of the original scene to the acquired LR images. The observation model is presented in Fig1

   Let the size of the desired HR image be I×J that is sampled beyond the Nyquist sampling rate from continuous scene which is supposed to be band-limited signal. The HR image can be represented in lexicographical order as the vector X=[x1, x2… xN]T where N=I×J. Consider the LR image of size I×J. Let the kth LR image in lexicographical notation be yk=[yk,1,yk,2,….,yk,M]T for k=1,2,……,p and M=I×J. Now, it is assumed that X remains constant during the acquisition of the multiple LR images, except for any motion and degradation allowed by the model. Therefore, the observed LR images result from warping, blurring, and sub-sampling operators performed on the HR image X .The observation model can be formulated [3] as

   yk =DBkMkX + nk for 1kp (1)

   where Mk is the motion model,Bk is the blur model and D is the down sampling matrix and nk is the additive noise vector. It can be written [4] as

   yk = WkX + nk. (2)

   Wk is sometimes referred to as the warp matrix and provides a mapping from the high-resolution image to the kth low-resolution image. The involved matrices Mk, Bk, D or Wk are very sparse, and this linear system is typically ill- posed [9]. The matrices are unknown in real systems and it is necessary to estimate them from available LR images.

2. NON UNIFORM INTERPOLATION

   The method for performing super-resolution is to map pixels from the low-resolution images onto a common plane and then interpolate over a more nely sampled grid to obtain a higher-resolution image [4].The super resolution process has three stages as shown in Fig 2. First

   the estimation of relative motion is computed from one or more observed LR images (known as registration) second, the registered samples are aligned in a HR grid to increase the resolution. The non uniform second, the registered samples are aligned in a HR grid to increase the resolution. The non uniform interpolation is applied to the HR grid of non-uniformly spaced samples followed by removal of blur and noise as a final process for restoration.

   The simple method is the bilinear interpolation. The nearest pixel is found first. Then the algorithm detects which LR image the pixel belongs to and then selects three neighbouring pixels from the same LR image. It is presented in Fig 3.

   Gilman and Bailey [5] used near optimal non uniform interpolation that a synthetic image is used to derive the optimal weights to be used for the input mages. The weights are computed to minimize the mean square error. It is assumed that the weights are independent of the image content but depend only on the relative position of the available samples in the grid. Therefore the weights depend only weakly on the image content. The optimal weights derived on known image can be used for other images with the same offset. This method provides better results with the computational overhead of calculating optimal coefficients.

   Panagiotopoulou and Anastassopoulos [6] presented a method of non uniform interpolation of SR reconstruction. Gradient based motion estimation is performed. The kriging interpolation method is carried out to estimate the value of each HR grid point with the information from non- uniformly placed LR frames to construct uniformly positioned samples. This is a direct reconstruction procedure as opposed to iterative. For every pair of two LR pixels the distance is calculated. Then the semivariance matrix for each LR frame is created. A semivariance matrix is obtained from computing the differences between the two matrices. A semi-variogram is constructed from the distance and semi-variance pairs. A Gaussian model is employed to fit data on semi-variogram. Wiener filter is used for restoration.

   1 N1 1 N2 1

   2 r

   2 r

   yk(r1,r2)= Xk

   1 m1 ,

   2 m2

   (3)

   T1,T2 m1 0 m2 0

   T1 N1

   T2 N2

   Fig 2. Super Resolution process (Figure taken from [Park et al. 2003])

   The disadvantage is that the number of known values should be more than 5 to calculate every HR grid point. In addition the computational difficulty is high in performing non-uniform interpolation.

   Directional bi-cubic interpolation is [10] presented to preserve the sharpness of the image. First local variance is computed. If the variance is less than the threshold bilinear interpolation is performed. Otherwise gradient method is performed along the direction of the edges. The two biggest gradients are extracted and interpolated by bi-cubic interpolation. The bilinear interpolation is used based on local image variance to reduce the computational complexity.

   In hybrid interpolation [11] linear interpolation for flat area and max relativity edge interpolation for edges are performed in parallel. The cubic B-spline linear interpolation is executed for low frequency region and edge interpolation algorithms are applied in the high frequency information. The method has low computation and higher PSNR value. The advantage of the non-uniform interpolation approach is that it takes relatively low computational load and makes real-time applications possible [3].

3. FREQUENCY DOMAIN APPROACH

   The frequency domain approach by Tsai and Huang is based on the following three principles: i) the shifting property of the Fourier transform, ii) the aliasing relationship between the continuous Fourier transform (CFT) of an original HR image and the discrete Fourier transform(DFT)of observed LR images ,iii)and the assumption that an original HR image is band limited.[3].

   Let x(t1,t2) be a continuous image and xk(t1,t2) be the kth shifted images where k=1,2….p. Then the continuous Fourier transform (CFT) of the shifted image will be xk(t1,t2) = x(t1+x, t2+y) where x, y are the arbitrary

   The matrix form can be obtained as Y=X where Y is a p×1 column vector with the kth element of the DFT coefficient yk(r1,r2),X is a N1N2 × 1 column vector with the samples of the unknown CFT coefficients of x(t1,t2) and is a p × N1N2 which relates the DFT of the LR images to the samples of the continuous HR image.

   Fig 3. Bilinear non-uniform interpolation and near optimal non-uniform interpolation

   (Figure taken from [15])

4. REGULARIZED SR RECONSTRUCTION APPROACH

   Many of the SR reconstruction methods are inverse problem because of numerical instability. The inverse problem is often ill posed that generates high resolution content from the set of observed LR images whereas the forward model is a method of deriving LR images from the given HR image. The measures generated to stabilize the inverse nature of the system and to provide a unique solution, regularization is necessary .The methods are classified in to Deterministic and stochastic approaches.

   The Deterministic method solves the inverse problem by means of prior information about the solution space to get a well posed problem. Many standard techniques exist for doing this, but perhaps the most common approach is to impose a smoothness prior via Tikhonov regularization on top of a least-squares optimization function [4].The matrix Wk in equation (2) is estimated for each LR image yk. The HR image can be estimated by minimizing the function

   known values. Then the continuous Fourier transform p 2

   (CFT) of the shifted image x

   (u , u ) can be written as x

   yk Wk X CX

   (4)

   k 1 2 k

   (u1, u2) =exp [j2(xu1+yu2)]x(u1, u2) where x(u1, u2) is the CFT of x(t1,t2).The shifted images xk(t1,t2) are sampled with the sampling period T1 and T2 and LR images are generated yk(n1,n2)=xk(n1t1+x,n2t2+y) .From assuming the band limitedness of xk (u1, u2) xk (u1, u2) =0 for u1 (N1) / T1, u2 (N2) / T2.The relationship between the CFT of the HR image and the DFT of the kth observed LR image can be written as [3].Let yk(r1,r2) be the DFT of the LR images

   k 1

   where C is a high pass filter , is the regularization parameter and controls how much weight is given to the regularization constraint. The larger value of will smooth the image.

   Stochastic Approach relates the SR reconstruction steps stochastically toward optimal reconstruction. The HR image and motions among low-resolution inputs can be both regarded as stochastic variables [9].The Bayesian model is the commonly used approach for super-resolution.

   The PDF of the original image must be recognized. There are two techniques Maximum likelihood (ML) and Maximum a posterior (MAP).The MAP is considered over ML because of its prior distribution. Consider the equation

   (2) in the observation model. The LR images yk, noise n, the HR image X are assumed to be stochastic and Wk is known. The estimator X maximizes the probability of the HR image of the observed LR images is expressed as

   X=arg max P(X|y1,y2,……..yp) (5)

   The Bayes theorem is applied to the conditional probability and by taking logarithm we get

   X=arg max(log P(y1,y2,….yp)|X)+logP(X)) (6)

   The P(X) prior density of X and P(y1,y2,….yp)|X) the conditional density has to be defined by the HR image X and the statistical information of noise.

5. PROJECTION ON TO CONVEX SETS

   This is a set-theoretic approach where each piece of a priori knowledge is formulated as a constraining convex set. Once the group of convex sets is formed, an iterative algorithm is employed to recover a point on the intersection of the convex sets [4].

   g i+1 = PMPM-1.P2P1gi (7)

   Where g0 is the initial guess, Pj is a projection of a given point on to the jth convex sets and M is the number of convex sets.

   The advantage of the POCS technique lies in its simplicity to include any kinds of restriction and prior knowledge. The disadvantage of the POCS is well-known for its heavy computation, slow convergence. The solution is not unique.

6. ITERATIVE BACK PROJECTION

   The SR reconstruction algorithm of iterative back projection (IBP) was proposed by Irani and Peleg [12].The simulated LR images of the subsequent observed LR images are produced from the HR image. The estimated SR image is iteratively refined by back projecting the error (i.e., the difference) between synthetically created LR images and observed LR images until the error is minimized. The IBP function is formulated as

   1

7. CHALLENGES AND ISSUES IN SR

   Many algorithms have been proposed for SR reconstruction works theoretically but have implementation limitation. Ther are challenges and issues for applications of SR techniques.

   1. Image Registration

      The registration step is crucial to SR reconstruction. Therefore accurate registration methods, based on robust motion models including multiple object motion, occlusions, transparency, etc., should be needed [13]. The SR techniques treat registration as separate step and the recovered HR image depends on the accuracy of the registration process. On one hand, accurate sub-pixel motion estimation benefits HR image estimation. On the other hand, high quality HR image can facilitate accurate motion estimation. Therefore, tailored to the SR reconstruction problem, the LR image registration can be addressed together with the HR image reconstruction, leading to joint ML or MAP framework for simultaneous estimation. These joint estimation algorithms capture the dependence between LR image registration and HR image estimation, and performance improvements are observed [9]. However estimation of registration parameters jointly results in over fitting. A simultaneous registration and reconstruction approach is also expected to reduce the effect of registration error in the SR estimates, since registration and reconstruction process are closely interdependent [3].

   2. Computational Efficiency

      The SR algorithms should be fast enough to apply in real time applications. The great number of unknown values leads to exhaustive computation thus reduces efficiency. The non-uniform interpolation is suitable for its low computational cost. Many of the algorithms proposed for efficiency requires precise registration. Ce Liu and Dequing Sun proposed a Bayesian approach that simultaneously estimates underlying motion, blur kernel and noise level while reconstruction the original HR images [15].It has huge computation.

   3. Robustness

      The SR technique suffers from noise, motion blur, motion errors etc. Robustness is important because the image degradation model parameters cannot be estimated perfectly, and sensitivity to outliers may results in visual artifacts, which are not suitable for many applications. Several algorithms are proposed to treat outliers but are of limited practical use.

   4. Performance Limitations

   X(n+1)=X(n)+c Mk

   k

   [ hbpf * s ( k – yk ) ] (8)

   The analysis of the performance limits for all SR techniques is difficult. The SR reconstruction is a complex

   where c is constant, bpf is the back-projection kernel,

   s is the up sampling operator and k is the simulated kth LR image from the current HR image estimate. The IBP is easily understood. Due to the ill-conditioned nature, the IBP solution is not unique. It depends on choosing the IBP kernel.

   task which consists of many interdependent components. Second, it is still unknown what is the most informative prior given the SR task, especially for the example-based approaches. Last, a good measure instead of simple MSE is still needed for performance evaluation [9].Also there is a lack of objective metrics for the measurement of the quality of the SR image.

8. CONCLUSION

This paper reviews the super resolution techniques. The observation model is covered. The existing methods of non-uniform interpolation are presented followed by frequency domain method. Furthermore some of the challenges and limitations of SR techniques is also presented.

REFERENCES

1. Sundaresh Ram and Jeffrey J. Rodriguez,Single Image Super-

   Resolution Using Dictionary-Based Local Regression IEEE 2014

2. MILANFAR, Peyman. Super-resolution Imaging, CRC Press, 2011. 472s PARK, S. C., PARK, M. K., AND KANG, M.G.2003. "Super resolution image reconstruction: A technical overview". IEEE Signal Processing Magazine 20, 3 (May), 2136.

3. Neil Alldrin , ´´Super-Resolution, University of California, San Diego

4. A. Gilman, D.G. Bailey, Near optimal non-uniform interpolation for image super resolution from multiple images

5. A. Panagiotopoulou and V. Anastassopoulos, Super-resolution image reconstruction employing Kriging interpolation technique 2007 IWSSIP & EC-SIPMCS, Slovenia

6. Sina Farsiu, Dirk Robinson,Michael Elad,Peyman Milanfar, Advances and Challenges in Super-Resolution 2004 Wiley Periodicals, Inc

7. Marco Bevilacqua. Algorithms for super-resolution of images and videos based on learning methods. Image Processing. Universit´e Rennes 1, 2014. English.

8. Jianchao Yang, Thomas Huang Image super-resolution: Historical overview and future challengesJianchao Yang, Thomas Huang University of Illinois at Urbana-Champaign

9. Liu J ing, Gan Zongliang, Zhu Xiuchang Directional Bicubic Inter polation A New Method of Image Super-Resolution3rd International Conference on Multimedia TechnologyICMT 2013)

10. Xiangguang Zhang, Yun Liu, A Computationally Efficient Super- Resolution Reconstruction Algorithm Based On the Hybird Interpolation JOURNAL OF COMPUTERS, VOL. 5, NO. 6, JUNE 2010

1. S. Borman and R.L. Stevenson, Super-resolution from image sequencesA review, inProc. 1998 Midwest Symp. Circuits and Systems, 1999, pp. 374-378

2. LIU, C.; SUN, D., A Bayesian Approach to Adaptive Video Super Resolution, IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 209 -216, 2011

3. KOC, U.; LIU, K., Interpolation-free subpixel motion estimation techniques in dct domain. IEEE Trans. Circuits Syst. Video Technol., vol. 8 pp. 460-487, Aug. 1998.