Design & Analysis of Blind Super Resolution via Sparse Representation

Design & Analysis of Blind Super Resolution via Sparse Representation

Amruta Sonavale#1

# M Tech Student,ECE,MITE Modabidri, Karnataka, India

Shreya Prabhu K*2

*Assistant Professor,ECE,MITE Moodabidri, Karnataka, India

AbstractThis paper presents a new approach to single-image super resolution, based on sparse signal representation. Research on image statistics suggests that image patches can be well- represented as a sparse linear combination of elements from an appropriately chosen over-complete dictionary. Inspired by this observation, we seek a sparse representation for each patch of the low-resolution input, and then use the coefficients of this representation to generate the high-resolution output. Theoretical results from compressed sensing suggest that under mild condi- tins, the sparse representation can be correctly recovered from the down sampled signals. By jointly training two dictionaries for the low- and high-resolution image patches, we can enforce the similarity of sparse representations between the low resolution and high resolution image patch pair with respect to their own dictionaries. Therefore, the sparse representation of a low resolution image patch can be applied with the high resolution image patch dictionary to generate a high resolution image patch. The learned dictionary pair is a more compact representation of the patch pairs, compared to previous approaches, which simply sample a large amount of image patch pairs [1], reducing the computational cost substantially. The effectiveness of such a sparsity prior is demonstrated for both general image super- resolution and the special case of face hallucination. In both cases, our algorithm generates high-resolution images that are competitive or even superior in quality to images produced by other similar SR methods. In addition, the local sparse modeling of our approach is naturally robust to noise, and therefore the proposed algorithm can handle super-resolution with noisy inputs in a more unified framework.

Keywords: Super resolution, sparsity, image processing, sparse coding, image restoration.

1. INTRODUCTION

   The quality of images can be significantly affected by the situation when capturing. The goal of super-resolution (SR) image reconstruction is to promote the space resolution of captured original images through software. After the first effective approach proposed by Tsai to solve this ill-posed problem, there are diverse methods to handle SR problem which can be classified into two groups the multiple frames based approaches and single frame based approaches. For multiple frames based approaches, spatial domain and frequency domain are two major directions and the former one is more popular these years as it can be applied to more general images with motion blur and noise to produce higher quality result. Some popular approaches belonging to spatial ones are projection-onto convex- sets (POCS), iterative back-

   projection (IBP), maximum a posteriori estimation (MAP). However, the deficiencies of these methods are in that they all need priori information of the point spread function (PSF) which is always difficult to get in practical situation. Therefore, to solve this problem, the blind SR method is proposed which combines the image registration, image restoration and PSF estimation into one framework. The method uses partial differential equation (PDE) as the regularization term of the high resolution image to preserve the edges while suppressing noise. However, the parameter of the regularization term which is significant in determining the smoothness of the image should be adjusted manually, which increases the in definability in generating high quality images with detailed texture. The method we propose here is a self- adaptive blind super resolution image reconstruction approach based on multiple frames. In the project, PDE framework and eigenvector-based alternating minimization (EVAM) constraint are used as the regularization term. In addition, we also design a novel image quality assessment method without reference image for adaptively choosing parameter of this blind super-resolution algorithm.

   Super-resolution (SR) image reconstruction is currently a very active area of research, as it offers the promise of overcoming some of the inherent resolution limitations of low-cost imaging sensors (e.g. cell phone or surveillance cameras) allowing better utilization of the growing capability of high-resolution displays (e.g. high-definition LCDs). Such resolution-enhancing technology may also prove to be essential in medical imaging and satellite imaging where diagnosis or analysis from low-quality images can be extremely difficult. Conventional approaches to generating a super-resolution image normally require as input multiple low-resolution images of the same scene, which are aligned with sub-pixel accuracy. The SR task is cast as the inverse problem of recovering the original high-resolution image by fusing the low-resolution images, based on reasonable assumptions or prior knowledge about the observation model that maps the high-resolution image to the low-resolution ones. The fundamental reconstruction constraint for SR is that the recovered image, after applying the same generation model, should reproduce the observed low resolution images. However, SR image reconstruction is generally a severely ill- posed problem because of the insufficient number of low resolution images, ill-conditioned registration and unknown blurring operators and the solution from the reconstruction constraint is not unique. Various regularization methods have

   been proposed to further stabilize the inversion of this ill- posed problem. However, the performance of these reconstruction-based super-resolution algorithms degrades rapidly when the desired magnification factor is large or the number of available input images is small. In these cases, the result may be overly smooth, lacking important high- frequency details. Another class of SR approach is based on interpolation. While simple interpolation methods such as Bilinear or Bi-cubic interpolation tend to generate overly smooth images with ringing and jagged artifacts, interpolation by exploiting the natural image priors will generally produce more favorable results. It represented the local image patches using the background/foreground descriptors and reconstructed the sharp discontinuity between the two. Explored the gradient profile prior for local image structures and applied it to super-resolution. Such approaches are effective in preserving the edges in the zoomed image. However, they are limited in modeling the visual complexity of the real images. For natural images with fine textures or smooth shading, these approaches tend to produce watercolor- like artifacts. A third category of SR approach is based on machine learning techniques, which attempt to capture the occurrence prior between low-resolution and high-resolution image patches. Proposed an example-based learning strategy that applies to generic images where the low-resolution to high-resolution prediction is learned via a Markov Random Field (MRF) solved by belief propagation. Here it extends this approach by using the Primal Sketch priors to enhance blurred edges, ridges and corners. Nevertheless, the above methods typically require enormous databases of millions of high- resolution and low-resolution patch pairs, and are therefore computationally intensive. Here it adopts the philosophy of Locally Linear Embedding (LLE) from manifold learning, assuming similarity between the two manifolds in the high- resolution and the low-resolution patch spaces. Thir algorithm maps the local geometry of the low-resolution patch space to the high-resolution one, generating high-resolution patch as a linear combination of neighbors. Using this strategy, more patch patterns can be represented using a smaller training database. However, using a fixed number K neighbors for reconstruction often results in blurring effects, due to over- or under-fitting. In our previous work we proposed a method for adaptively choosing the most relevant reconstruction neighbors based on sparse coding, avoiding over or under fitting of and producing superior results.

   However, sparse coding over a large sampled image patch database directly is too time-consuming. While the mentioned approaches above were proposed for generic image super- resolution, specific image priors can be incorporated when tailored to SR applications for specific domains such as human faces. This face hallucination problem was addressed in the pioneering work of Baker and Canada. However, the gradient pyramid-based prediction introduced in does not directly model the face prior, and the pixels are predicted individually, causing discontinuities and artifacts. It proposed a two-step statistical approach integrating the global PCA model and a local patch model. Although the algorithm yields good results, the holistic PCA model tends to yield results like the mean face and the probabilistic local patch model is complicated and computationally demanding. Wei Liu

   proposed a new approach based on Tensor Patches and residue compensation. While this algorithm adds more details to the face, it also introduces more artifacts. This project focuses on the problem of recovering the super resolution version of a given low-resolution image.

2. METHODOLOGY

   1. Self-Adaptive Blind Super-Resolution ImageReconstruction

      1. Methemathical Model:

         SR can be seen as the opposite procedure of observation model in imaging system. The low resolution images are captured through the process of blurring, distortion, down sampling as well as system noise. The blurring- warping model is an appropriate mathematical model in SR can be represented as

         Lk = D[Mk(Bk * H)] + Nk (1)

         Where Lk is the captured frame Kth low resolution image with the size of m × n, H is the corresponding high resolution image,Bk is the PSF of the Kth frame which convolves with the high resolution image and determines the type of blur. Mk is the motion and distortion operator,D is the down-sampling operator, and Nk denotes the system noise.

      2. Approach Pipeline:This approach consists of two main stages. First, PDE is used as the regularization term of high resolution image, which contains Lorentzian function as spread coefficient. In another aspect, it uses EVAM constraint as the regularization term of the extended PSF. Alternating minimization algorithm is used to minimize the cost function. Second, we propose a no reference image quality assessment method which considers the effects of blurring and ringing to guide the choice of parameters in the regularization term. In this way it is possible to preserve the details of the image since the parameter significantly determines the smoothness of the resulting image.

   2. Image Resolution Vai Sparse Representation

      To solve these ill-posed and ill-conditioned proplem two constraints are modeled in this session those are

      1. Reconstruction Constraint: The observed low- resolution image Y is a blurred and down sampled version of the high resolution image X:

         = (2)

         Here, H represents a blurring filter, and S the down sampling operator. Super-resolution remains extremely ill-posed, since for a given low-resolution input Y , infinitely many high- resolution images X satisfy the above reconstruction constraint. We further regularize the problem via the following prior on small patches x of X.

      2. Sparsity Prior: The patches x of the high-resolution image X can be represented as a sparse linear combination in a dictionary Dh trained from high-resolution patches sampled from training images:

         For some for some (3)

         The sparse representation will be recovered by representing patches y of the input image Y, with respect to a low resolution dictionary Dl co-trained with Dh. It applies this approach to both generic images and face images. For generic image super-resolution, we divide the problem into two steps. First, as suggested by the sparsity prior, we find the sparse representation for each local patch, respecting spatial compatibility between neighbors. Next, using the result from this local sparse representation, we further regularize and refine the entire image using the reconstruction constraint. In this strategy, a local model from the sparsity prior is used to recover lost high-frequency for local details. The global model

3. ALGORITHMS

   Algorithm 1: (Super-Resolution via Sparse Representation).

   1: Input: training dictionaries Dh and Dl, a low-resolution image Y.

   2: For each 3 × 3 patch y of Y, taken starting from the upper- left corner with 1 pixel overlap in each direction Compute the mean pixel value m of patch y.

   • Solve the optimization problem with and y that is defined in:

     2 + 1 (4)

     2

     from the reconstruction constraint is then applied to remove

     possible artifacts from the first step and make the image more consistent and natural. The face images differ from the generic images in that the face images have more regular structure and thus reconstruction constraints in the face subspace can be more effective. For face image super-resolution, we reverse the above two steps to make better use of the global face structure as a regularize. We first find a suitable subspace for human faces, and apply the reconstruction constraints to recover a medium resolution image. We then recover the local details using the sparsity prior for image patches.

     1. Generic Image Super-Resolution from Sparsity

     1. Local Model from Sparse Representastion: Similar to the patch-based methods mentioned previously, our algorithm tries to infer the high resolution image patch for each low- resolution image patch from the input. For this local model, we have two dictionaries Dh and Dl, which are trained to have the same sparse representations for each high-resolution and low-resolution image patch pair. We subtract the mean pixel value for each patch, so that the dictionary represents image textures rather than absolute intensities. In the recovery process, the mean value for each high-resolution image patch is then predicted by its low-resolution version. For each input low-resolution patch y, we find a sparse representation with respect to Dl. The corresponding high-resolution patch bases Dh will be combined according to these coefficients to generate the output high-resolution patch x.

     2. Learning the dictionary pairs: This section will focus on learning a more compact dictionary pair for speeding up the computation.The super-resolution problem using sparse prior which states that each pair of high and low-resolution image patches have the same sparse representations with respect to the two dictionaries Dh and Dl.A straightforward

   • Generate the high-resolution patch x = Dh. Put the patch x

     + m

     into a high-resolution image X0. 3: End.

     4: Using gradient descent, find the closest image to X0 which satisfies the reconstruction constraint:

     2 2

     2 2

     = 2 + 02 (5)

     5: Output: super-resolution image

     The entire super-resolution process is summarized as Algorithm 1.

     Face image resolution enhancement is usually desirable in many surveillance scenarios, where there is always a large distance between the camera and the objects (people) of interest. Unlike the generic image super-resolution discussed earlier, face images are more reglar in structure and thus should be easier to handle. Indeed, for face super- resolution, we can deal with lower resolution input images. The basic idea is first to use the face prior to zoom the input to a reasonable medium resolution, and then to employ the local sparsity prior model to recover details. To be precise, the solution is also approached in two steps: 1) global model: use reconstruction constraint to recover a medium high-resolution face image, but the solution is searched only in the face subspace; and 2) local model: use the local sparse model to recover the image details.

     Algorithm 2: (Face Hallucination via Sparse Representation). 1: Input: sparse basis matrix U, training dictionaries Dh and Dl, a

     low-resolution aligned face image Y.

     2: Find a smooth high-resolution face X from the subspace spanned by U through:

     • Solve the optimization problem in:

   2 + 2 . 0 (6)

   way to obtain two such dictionaries is to sample image patch

   pairs directly, which preserves the correspondence between the high resolution and low resolution patch items

   2

   ^ =

   However,such a strategy will result in large dictionaries and hence expensive computation.

   1. Single Dictionary Training: Sparse coding is the problem of finding sparse representations of the signals with respect to an over complete dictionary D. The dictionary is usually learned from a set of training examples X = {x1, x2, …, xt}. Generally, it is hard to learn a compact dictionary which

   3: For each patch y of X, taken starting from the upper-left corner

   with 1 pixel overlap in each direction, Compute and record the mean pixel value of y as m. Solve the optimization problem with D and y defined.

   n~ ~2 + 1 (7)

   guarantees that sparse representation can be recovered 2

   minimization. Fortunately, many sparse coding algorithms

   proposed.

   • Generate the high-resolution patch x = Dh + m. Put the patch

   x into a high-resolution image

   4: Output: super-resolution face . For estimated the high resolution face, and is a parameter used to balance the reconstruction fidelity and the penalty of the prior term. In this project, we simply use a generic image prior requiring that the solution be smooth. Let denote a matrix performing high-pass filtering. The final formulation is:

   = 2 + 2 . 0 (8)

   1. Self-Adaptive Blind Super-Resolution Image Reconstruction GUI

      2

      The medium high-resolution image X is approximated by .

      The prior term in suppresses the high frequency components, resulting in over-smoothness in the solution image. We rectify this using the local patch model based on sparse representation mentioned. The complete framework of our algorithm is summarized as Algorithm 2.

4. EXPERIMENTAL RESULTS

   The main GUI window is designed to have two buttons along With a button to exit the GUI. The two buttons are to select different methods to implement blind resolution. The button that says Self a daptive blind super resolution performs super resolution on the image. The next button Sparse representation performs the sparse representation for single image. Hence we can compare the results of both the methods on same GUI. We can use the button Exit to exit from the GUI window.

   Fig 1:The main GUI of Analysis of Super Resolution

   To select image which was already stored in image file, by click on the brows button, that will show the registered images which are stored in image file and results with a title of original image that is show in below gui.

   However, SR image reconstruction method generally contains ill-posed problem because of the insufficient number of low resolution images and it also contain ill-conditioned problem due to the registration and unknown blurring operators, and the solution from the reconstruction constraint is not unique. To overcome these problems Sparse Representation method is proposed.

   1. Image Super-Resolution via Sparse Representation GUI

5. CONCLUSION

The super-resolution reconstruction problem is an inverse problem, dealing with the recovery of a single high-resolution image from a set of low quality images. In its general form, the super resolution problem may consist of images with arbitrary geometric warp, space variant blur and colored noise.

Self Adaptive Blind Super resolution uses Lorentzian function as spread coefficient and partial differential function as regularization term of resulting image. A generalized version of the eigenvector-based alternating minimization (EVAM) constraint is used to regularize PSF and estimate resulting image and PSF simultaneously. In addition, in order to achieve self-adaptive regularization terms parameter choosing, we also present a new robust no-reference image quality assessment method which provides blurring and ringing effect assessment value as feedback.

Sparse Representation method presented a novel approach toward single image super-resolution based on sparse representations in terms of coupled dictionaries jointly trained from high- and low-resolution image patch pairs. The compatibilities among adjacent patches are enforced both locally and globally. Experimental results demonstrate the effectiveness of the sparsity as a prior for patch-based super- resolution both for generic and face images.

REFERENCES

1. J. Yang, J. Wright, T. Huang, and Y. Ma, Image super-resolution as sparse representation of raw image patches, in IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2010, pp. 18.

2. M. Irani and S. Peleg, Motion analysis for image enhancement: resolution occlusion and transparency, Journal of Visual Communication and Image Representation, vol. 4, no. 4, pp.324335, 1993.

3. R. C. Hardie, K. J. Barnard, and E. A. Armstrong Joint map registration and high resolution image estimation using a sequence of undersampled images, IEEE Transactions on Image Processing, vol. 6, pp. 1621 1633, 1997.

4. S. Baker and T. Kanade, Limits on super-resolution and how to break them, IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 24, no. 9, pp. 11671183, Sep. 2002.

5. S. Farsiu, M. D. Robinson, M. Elad, and P. Milanfar, Fast and robust multi frame super- resolution, IEEE Transactions on Image Processing, vol. 13, pp. 13271344, 2004.

6. M. Elad and M. Aharon, Image denoising via sparse and redundant representations over learned dictionaries, IEEE Transactions on Image Processing, vol.15, pp, 37363745, 2006.

7. J. Mairal, G. Sapiro, and M. Elad, Learning multiscale sparse representations for image and video restoration Multiscale Modeling andSimulation, vol. 7, pp. 214241, 2008. E-mail:julien.mairal@inria.fr

8. Jianchao Yang and Thomas Huang Image super-resolution: Historical overview and future challenges University of Illinois at Urbana- Champaign.

9. Jianchao Yang,J. Wright , Yi. Ma Image super-resolution as sparse representation slide, SIAM imaging science,july 7,2008.

10. Assaf Zomet and Shmuel Peleg super resolution from multiple images having arbitrary mutual motion

11. Wagner, Waagen and Cassabaum Super-Resolution slide.