A Review on Image Super Resolution Techniques

A Review on Image Super Resolution Techniques

Nirali Harsoda

Department of Electronics and Communication R.K.University

Rajkot,india

Namarata Joshi

Department of Electronics and Communication R.K.University

Rajkot,india

Abstract Enhancing image quality or Images with high resolution (HR) has always been a continuous ongoing process in image Technology.In order to restore an image into a HR image correctly,it is necessary to infer high frequency components of a low resolution image.In some applications it becomes essential to extract the useful information from the images.For example in video surveillance, forensic investigation, in medical diagnosis and even the satellite images and pattern recognition in computer vision.During this process, enlarging the image beyond a certain limitresults in a blurred image with no peculiar information.Hardware limitations of sensors is one of the main cause behind this problem which includes sampling rates of Charged-Coupled Devices(CCD). Also, the main idea behind achieving the HR images is not to tamper with the observable quality of image.Many super resolution techniques have been proposed to overcome the hardware limitations in order to achieve the best results .In this electronic document a comprehensive review of all well-known super resolution techniques has been presented.

Keywords restoration;Super resolution ;iterative back projection;POCS;interpolation.

1. INTRODUCTION

   The goal of super resolution techniques is to reconstruct a high resolution image from a single or multiple low resolution images. Multiple low resolution images of the same scene can be obtained by using either single sensor or many sensors as depicted in fig.1.[1].HR images contains high pixel density and thus they offer improved details of the image that be precious in various applications. As the pixel density increases, image quality degrades due to shot noise[3].Another approach is to increase chip size,this in turn increases the capacitance i.e. higher charge transfer rate and also the size of device. Therefore, limits exists on prevailing sensors and optics processors[3].This can be overcome by increasing spatial resolution and this can be achieved using various signal processing techniques.

   SR techniques consists the main features of image interpolation and image restoration. Image interpolation changes the dimension of an image and image restoration recovers a degraded image. Thus, image SR is a technique that restores the degraded image and also increases the size of the image[1].

   Figure.1.(a)HR image obtained from four different cameras.

   Figure.1.(b)Multiple frames obtained by a single camera

   The SR image reconstruction consists of three stages. By means of image registration, all pixels from available LR images can be mapped back onto the reference image. Then via interpolation a uniformly spaced up sampled image is obtained. In the end, image restoration is applied to the up- sampled image to remove the effect of sensorPSF blur and noise [4].

   To represent the SR technique, a commonly used model describes the relationship between original HR and a set of LR images. Let us consider the desired HR image represented as

   x of size 11 × 22.

   It is assumed that x is to be sampled at or above the Nyquist rate from a continuous scene to be bandlimited. Let the parameters L1 and L2 represent the down sampling factors for the horizontal and vertical directions in the observed model.

   Thus,each observed LR image is of size 1 ×

   2.Let denotes the kth LR image represented as

   =[,1, ,2, . . ,] (1) Where, k=1,2,3p and m= 1 × 2.

   The observation model is mathematically represented as:

   = + (2) Here, is a M× decimating subsampling matrix.

   is blur matrix of size × .is warp matrix of size

   × and is white Gaussian noise being encountered in the observation model.

   The aim of image SR reconstruction is to estimate the HR imagex from the LR images .

2. INTERPOLATION

   This technique is based on the non-uniform sampling theory. Clark et.al.[5] was the first one to introduce this idea. Estimation of HR image using interpolation follows 3 steps: Registration, Interpolation and restoration.

   In the registration step the relative motion information i.e. relative shifts is estimated between LR images in comparision with reference LR images with sub-pixel accuracy[2]. Then the direct or iterative reconstruction procedure is followed to produce uniformly spaced sampling points[6]-[9]. Next step is to apply non-uniform interpolation because the registered HR image does not always match up to a uniformly spaced grid so non-uniform interpolation results a uniformly spaced HR image[2].In the restoration step noise and blurring effect in HR image is removed.

   This technique has advantage of less computational complexity and also real-time applications are possible. However there are some limitations like errors occurred in the interpolation step are ignored by restoration step, so optimally of the whole reconstruction algorithm is not guaranteed [2].

3. ITERATIVE BACK PROJECTION(IBP)

   Irani and Peleg [10] formulated the iterative back-projection SR reconstruction approach. In this approach, the HR image is estimated by back projecting the error difference between simulated LR image and observed LR images as shown in Figure 2. This process is repeated iteratively to minimize the energy of error[3].This method uses multiple simulated LR image of similar scene to find corresponding HR image.

   (0) is the high frequency data of the image X(0) that is obtained from the interpolation of initial LR image[11][12]. This technique is very easy to understand but SR reconstruction is not unique due to ill posed nature of inverse problem.

   It should be noted that the selection of back-projection matrix affects the resolution. Afterwards, extended version of the approach was proposed by Van Cittert and Tikhonov-Miller [13].The results of this comprehensive work are compared in [14]. In [14], the author has also discussed the approach based on Adaptation of conjugate gradient that fastlyconverges the iterations. The original back-projection method is suffering from chessboard effect or ringing effect, especially at edges.

4. PROJECTION ONTO CONVEX SETS (POCS)

   The projection onto a convex set (POCS) approach was originally developed by D.C.youla [16] and H.Webb[17] for image restoration problems, and has been widely employed indeveloping image superresolution construction[18,19,20]. The method of POCS requires definition of closed convex sets within a well-defined vector space that contain the actual SR image [19]. An estimate of the SR image is then defined as a point in the intersection of these constraint sets, and is determined by projecting an arbitrary initial estimate onto the constraint sets[19].

   A convex set is defined which represents certain tight constraints on the solution image. The POCS yields progressively improved super resoluted construction with assured convergence [15].

   The image formation model is expressed by

   1, 2, =

   (1,2) 1, 2 (1 1, 22, ) +v(1, 2)

   (4)

   Where (1, 2, ) is a linear blur mapping of high resolution source image 1, 2, to the kth measured image

   1, 2, at a lower resolution and v(1, 2) is an additive white Gaussian noise[15].

   A closed, convex constraint sets, one for each pixel within the LR image sequence (1,2, ) is defined as follows:

   = { 1, 2, : (1, 2, ) 0(1, 2, )}

   (5)

   Where,

   1, 2, = 1, 2,

   1, 2, (1 1, 22, )

   (1,2)

   (6)

   is the residual associated with an arbitrary member , f, of the constraint set bounded in magnitude by 0. The projection of an estimate 1, 2, onto is defined by [16][22]:

   1, 2, 1, 2, =

   Fig.2.Pictorial representation of IBP

   ( 0)(1, 2,)

   2 ( , ,)

   ,

   > 0

   Mathematically the SR steps according to IBP are written as:

   1 2 1 2

   ( +1) = ( ) +

   + (0) (3)

   1, 2, +

   0 , 0

   Where,

   ( + )( , ,)

   ( +1)is estimated HR image of n+1th iteration;

   ( )is estimated HR image of nth iteration;

   is error correction;

   0 1 2

   1 2 2 (1, 2,)

   , < 0

   (7)

   Where function is meant to be 1, 2, .Note that we allow the blurring functionh to be slightly different at each frame depending on the small motion between neighboring frames which usually is only a small fraction of a pixel. Theamplitude constraint should always be satisfied, i.e.,

   0 1, 2, 255.

   The estimation , , of the high resolution image

   1. S. C. Park, M. K. Park and M.G.Kang, Super-Resolution Image Reconstruction: A Technical Overview, IEEE signal processing magazine, pp 21-36, May 2003.

   2. S.Chaudhuri, super-resolution imaging, Kluwer Academic Publishers, 1/e, 2002.

   3. N.Nguyen and P.Milanfar, An efficient wavelet-based algorithm for image superresolution,in Proc.Int.Conf.Imageprocessing,vol 2,2000,pp-351- 354.

   1 2 [6] J.J.Cark,M.R.Palmer and P.D.Laurence, A transformation method for

   f(n1,n2;k) is obtained from images gd(m1,m2,q) (q = k-1, k, k+1) through the projections in the backward and forward directions[15].

   An application of Wavelet-based POCS Super resolution for Cardiovascular MRI Image Enhancement is discuss in [15]. Xia Su et al. had combined sparse signal representation with the projection onto convex sets in [22]. Human face image super-resolution techniques have a variety of application such as surveillance, recognition, transmission [23].

   As performance criteria, Mean Square Error (MSE), Peak Signal to Noise Ratio (PSNR) and Mean Absolute Error MAE) are calculated. The mathematical equations for M×N image analysis are as given below.

   [ , , ]2

   the reconstruction of functions from non-uniformly spaced samples,IEEE Trans.Acoust.,Speech,SignalProcessing,vol.ASSP- 33,pp.1151-1165,1985.

   1. S.P.Kim and N.K.Bose, Reconstruction of 2-D banlimited discrete signals from non-uniform samples,Proc,Inst,Elec.Eng.,vol. 137,pt.F,pp,197-204,June 1990.

   2. A.Papoulis,Generalized sampling theorem,IEEE Trans,Circuits,Sys.,vol.24.pp.652-654,Feb.1977.

   3. J.L.Brown, Multi-channel sampling of low pass signals,IEEE Trans,Circuits,Sys.,vol.CAS-28,pp.101- 106,Feb.1981.

   4. M.Irani and S.Peleg, Improving resolution by image registration,CVGIP:Graphical Models and Image Proc.,vol.53,pp.231- 239,May,1991.

   5. R.R.Makwana, N.D.Mehta, Single Image Super- Resolution VIA Iterative Back Projection Based Canny Edge Detection and a Gabor Filter Prior, International Journal of Soft Computing and Engineering (IJSCE)ISSN:2231-2307,Vol-3,pp.379- 384 Issue-

      =

      ×

      255 ×255

      (8)

      1, March 2013.

   6. M.N.Bareja and C.K.Modi, An Effective Iterative Back Projection based Single Image Super Resolution Approach,2012 International

      SNR = 1010

      (9)

      Conference on Communication Systems and Network Technologies.IEEE,pp.95-99.

   7. Marcia L. S. Aguena, Nelson D. A. Mascarenhas, Generalization of Iterative RestorationTechniques for Super-Resolution,SIBGRAPI Conference on Graphics, patterns and Images, pp. 258-265,

5. CONCLUSION

Image super resolution concept and overview of techniques has been discussed here.SR has more optimized the image acquisition process and provided improved content visualization and object recognition.IBP and POCS uses sequence of LR images to extract a SR image.The advantage of interpolation approach is that it takes relatively low computational load and makes real-time applications possible.However,interpolation is not guaranteed, since restoration step ignores the errors that occurs in interpolation step.Projection onto convex sets (POCS) is simple and allows a convenient inclusion of priori information. POCS has disadvantage of non-uniqueness of solution, slow convergence and high computational cost.

Iterative Back Projection (IBP) is understood intuitively and easily. However, this method has no unique solution due to the ill-posed nature of the inverse problem and it has some difficulty in choosing the .In contrast to POCS it is difficult to apply a priori constraints.

REFERENCES

1. A.J.Shah,S.B.Gupta, Image super resolution- A Survey , 1st international conference on emerginf technology and trends in electronics,communication and networking,IEEE ,2012.

2. S.S.Qureshi, X.M.Li, T.Ahmad, Investigating Image Super Resolution Techniques:What to Choose?ICACT 2012

Aug.2011.

1. S. P. Kim and W. Su, Recursive high-resolution reconstruction of blurred multiframe images, IEEE Transactions on Image Processing,vol. 2, pp. 534539, 1993.

2. J. T. Hsu, C. C. Yen, C. C. Li, M. Sun, B. Tian and M. Kaygusuz, Application of Wavelet-based POCS Superresolution for CardiovascularMRI Image Enhancement, Proceedings of the Third International Conference on Image and Graphics (ICIG04) IEEE,2004.

3. D.C. Youla, "Generalized image restoration by the method of alternating orthogonal projections," IEEE Trans.Circuits Syst., vol. CAS-25, pp. 694-702, 1978.

4. D.C. Youla and H. Webb, "Image Restoration by the Method of Convex Projections: Part 1-Theory," IEEE Trans.Medical Imaging, vol. MI- 1, no. 2, pp. 81-94, October 1982.

5. M.K. Ozkan, A.M. Tekalp, and M.I. Sezan, "Pocs BasedRestoration of Space-Varying Blurred Images," IEEETransactions on Image Processing, vol. 3, pp. 450-454,1994.

6. A.J. Patti, M.I. Sezan, and A.M. Tekalp, "Superresolutionvideo reconstruction with arbitrary sampling lattices and nonzero aperture time," IEEE Transactions on ImageProcessing, vol. 6, pp. 1064- 1076, 1997.

7. M. Elad and A. Feuer, "Restoration of a single superresolution image from several blurred, noisy, and undersampled measured images," IEEE Transactions onImage Processing, vol. 6, pp. 1646-1658, 1997.

8. W.K. Pratt, Digital image processing, 2nd ed. New YorkChichester: Wiley, 1991.

9. X.q.Su,S.Li, Multi-frame Image Super-resolutionReconstruction based on Sparse Representation and POCS, International Journal of Digital Content Technology and its Applications. Volume 5, Number 8, pp 127- 135, Aug. 2011.

10. Hong Yu, Ma Xiang, Huang Hua, Qi Chun, Face Image Superresolution Through POCS and Residue Compensation,VIE, pp 494-497, 2008.