A Super-Resolution Based Image Inpainting in the Advertence of Examplar Technique

A Super-Resolution Based Image Inpainting in the Advertence of Examplar Technique

Manoj K Y

Department of computer science and engineering

VTU Belgaum, KVGCE Sullia , DK

Mr. Venkatesh U C

Asst Professor

Department of computer science and engineering KVGCE Sullia, DK

Dr. Antony P J Director of PG Studies VTU Belgaum KVGCE Sullia , DK

Abstract – This paper introduces a new examplar-based inpainting framework. A real world images may consisting of some low resolution pixels, blur parts or any unwanted objects. This coarse version of the input image is first inpainted by a non-parametric patch sampling. There have been several approaches proposed for the same. In this paper, an algorithm is introduced that improves and extends a previously proposed algorithm and provides an effective and faster inpainting technique. In this approach, Inpainting is based on examplar technique and super-resolution algorithm that one can inpaint large regions (e.g.: remove an object) as well as recover small areas (e.g.: restore a photographs by removing cracks) in the image and a single-image super- resolution algorithm is applied to recover the details of missing areas from the low-resolution inpainted image. This technique can also be used in restoring old photographs or damaged film. It can also remove superimposed text like dates, subtitles etc.; Compared to existing approaches, some improvements have been done. This inpainting techniques allows reducing the computational complexity, improves visual quality and to be less sensitive to noise.

Key words Super – resolution, Examplar based inpainting, single-image super-resolution.

1. INTRODUCTION

   Image inpainting refers to methods which consist in filling-in missing regions (holes) in an image [1]. First the user will input the low-resolution image that consists of some noisy pixels, blur part or any unwanted objects to the inpainting process that removes unwanted objects [2] without missing any pixels. Existing methods can be categorized into two main types. The first category concerns diffusion-based approach which propagates linear structures by using partial differential equations that tend to introduce some blur when the hole to be filled in is large [3]. The second category concerns examplar- based methods which samples and copy best matching pixels (texture patches) from the known neighborhood pixels in [4] inspired by texture synthesis techniques. Initially, the examplar-based techniques are used for object removal and then a priori improve a search of rough estimate similar patches to fill-in missing regions using multi-scale approach. This converts the image from coarse to fine levels. Now the two types of methods (diffusion and examplar-based) are combined efficiently [8].

   Considering when the whole to be filled is large and high computational time in general required. Difficulties remain although several progresses have been made in the past years on the inpainting technique. These two problems are here addressed by considering hierarchical approach in which a low-resolution pixel in the input image is first computed and inpainted using K-NN (K Nearest Neighbors) examplar-based method. Dictionary is used to store the correspondences between the K-NN low-resolution and high-resolution patches. These pixels are then used to find the missing pixels by using some principles in the single-image Super-resolution methods.

   Super-resolution (SR) refers to that creates one enhanced resolution image from one or multiple input low resolution images. In some earlier work an image may produce some noise even after applying the inpainting technique. So the SR method is used with inpainting technique that applies best matching pixels that removes noise and improves visual quality of the inpainted image at the final resolution. Here the problem is to find the details of missing pixels in the input image. So the proposed SR method is used in the single-image inpainting method.

   The SR still produces a problem since multiple high- resolution images can produce same low-resolution image. To solve this problem some prior information is introduced. This prior information takes the form of example images or corresponding LR-HR pairs of patches from a set of un-related training images present in the external database [9] or from the input low-resolution image itself [10]. This approach is known as example-based SR methods [11]. This consists of K-NN found in the external database [12]. A multi-resolution pyramid is constructed from the input low-resolution image instead of exampler-based SR method which consists of K-NN found in the external database.

   Since the quality of the LR inpainted image has a critical impact on the quality compared at the final resolution, instead of by using simply the best match by template matching and K coherence candidates, using of both linear combination of K most similar patches (K-NN) to the input patch that first improves the inpainting algorithm. A different patch priority term on the quality of the inpainted images is also used.

   In the second step, Particular filling order is considered while filling patches into the input HR image. the inpainting

   thus proceeds by searching for K nearest neighbours to the input vector concatenating the known HR pixels of the patch and the pixels of the corresponding inpainted LR patch. The KNN patches are searched in a dictionary composed of LR- HR patches extracted from known part of the image.

   The rest of the paper is organized as follows: Section 2 describes the related work in this area. Section 3 presents a new framework of the proposed inpainting method. Section 4 elaborates the proposed examplar-based inpainting method and comparison with the existing techniques. Section 5 presents details of the Super-resolution inpainting method. Section 6 describes implementation of algorithm. Section 7 presents the description of performance. Section 8 concludes this paper.

2. RELATED WORK

   The Inpainting technique initially starts as an ancient painting gets older, on certain regions, the pigments start to fall off the canvas, and the painting becomes incomplete. The human work of filling in the missing parts of the painting is called "inpainting [3]". Digital inpainting has much wider applications in image processing and computer vision. In other words, our inpainting scheme is robust to noise, and thus insensitive to pixel values.

   To overcome the problem of Wavelet Transfer, some frameworks describes different types of image inpainting techniques. The various transforms are wavelet transform, contourlet transform, Nonsubsampled transform. The main drawback of wavelet transform is that there is a problem of filling missing data will occur and it has poor directional specificity of the images. In contourlet transform the image improvement cannot capture the geometric information of

3. PROPOSED SYSTEM

   The challenging here is the image completion of large missing regions. In this paper, a new inpainting method is introduced using a single-image SR algorithm. The main idea of this technique and the reasons why this technique is new and innovative is described below.

   1. Motivations

      There are two main and sequential operations are used in the composition of proposed methods. The first one is non- parametric patch sampling method used to fill-in missing regions rather than filling in missing regions at the original resolution, the inpainting algorithm is applied on a coarse version of the input image. There are several reasons for performing theinpainting on a low-resolution image. First, the coarse version of input image is compared with the gist [15] that represents some important structures of the image. Performing inpainting on coarse version of the image is much easier since the inpainting would be less contingent on local orientation or even noise.

      Real image

      n

      Inpainting

      Single- image SR

      Inpain

      -ted Image

      –

      Selected object

      Real image without Object

      Object Select- ed from

      images and be liable to amplify noises when they are applied

      to noisy images also that they cannot distinguish noises from weak edges. The entire drawback is overcome by the Nonsubsampled Contourlet transform.

      In several approaches, the error in Image inpainting refers

      image

      Trainig

      samples

      Figure 1: The frame work of the proposed method.

      to restoring a damaged image with missing information. In recent years, there have been many developments on computational approaches to image inpainting problem while there are many effective algorithms available; there is still lack of theoretical understanding on under what conditions these algorithms work well. This investigates an error bound for inpainting methods, by considering different image spaces such as smooth images, piecewise constant images and a particular kind of piecewise continuous images [14].

      Characteristics for appreciating the image restoration accuracy, in addition to comparing the subjective results with digital photographs. The result is an image in which the selected object has been replaced by a visually plausible background that mimics the appearance of the source region [15].

      In this paper, an extension to earlier inpainting algorithms and techniques has been made with the focus of reducing the computational complexity of the approaches along with some other improvements such as computational speed and accuracy of the method.

      Second, as the picture to inpaint is smaller than the original one, the computational time to inpaint is significantly reduced when compared to inpaint a full resolution image. The output of the first operation is used as input to the next operation i.e. the second operation. Its goal is to improve the quality of the image using a single-image SR approach a low- resolution input image depends on the first step of the inpainting process this obtains a high-resolution using a set training examples, which are taken from the known part of the input picture.

      In figure 1, the new method it represents is generic since there is no constraints on both the number and type of inpainting methods used in the first pass, one could imagine using different settings or methods to fill-in the low resolution images and to fuse results that would increase the robustness and visualization of inpainting obtained by the combination of three methods.

   2. Principles

      Figure 1 illustrates the main concept of proposed method. The two main components are the inpainting and the super- resolution algorithms. The following steps are performed:

      1. A low-resolution image is first built from the original picture.

      2. An inpainting algorithm is applied to fill-in the missing pixels in the image.

      3. The quality of the inpainted regions is improved by using a single-image SR method.

   In this approach, Inpainting is based on examplar technique and super-resolution algorithm of low-resolution images are described in the below sections.

4. EXAMPLAR-BASED INPAINTING OF LOW-RESOLUTION IMAGES

   The Examplar-based inpainting method is used to fill in the low-resolution images. In the two methods, the first one is based on a non-parametric patch sampling whereas the second one is based on partial derivatives equation [2]. Here examplar-based method follows the two classical steps: the filling order computation and the texture synthesis. These are described in the below sections.

   1. Patch priority and filling order

      The filling order computation defines a measure of priority for each patch in order to distinguish the structures from the textures where a high priority indicates the presence of structure. The priority of a patch centered on p is just given by a data term. Three different data terms have been tested: gradient-based priority [4], tensor-based [9] and sparsity-based [15].

      In a search window, a template matching is performed between the current patch and neighboring patches that belong to the known part of the image. By using a non-local means approach, a similarity weight is computed for each pair of patches. The sparsity-based priority is more robust and visually improves the final result compared to the gradient and tensor-based priority. In the following, this method is used to compute the filling order.

   2. Texture synthesis

      The filling process starts with the patch having highest priority. The candidates used to fill in the known part of the current patch are composed of the K most similar patches located in a local neighborhood centered on the current patch. They are combined by using a non-local means approach.

      A major problem of local neighbor search is its tendency to get stuck at a particular place in the sample image and to produce verbatim copying. This kind of regions is often called garbage region. This problem can be addressed by introducing some constraints in terms of spatial coherence.

5. SUPER-RESOLUTION ALGORITHM

After the inpainting of the low-resolution picture is completed, a single-image super-resolution algorithm is used to convert the low-resolution pixels of the image into high- resolution image. The idea is to use low-resolution inpainted areas in order to guide the texture synthesis at the higher resolution from a database of examples. The following steps are showed in the below figure.

1. Dictionary building: it consists of pixels between the low and high resolution image patches. In this algorithm a high-resolution and valid patches are extracted from the known part of the image. The size of the dictionary is a user-parameter which might influence the overall computational speed/quality of the image. An array is used to store the spatial coordinates of high resolution patches.

2. Filling order of the HR picture: it is computed on the HR picture with the sparsity-based method. The filling process starts with the patch having the highest priority. This improves the quality of the image.

3. For the LR patches corresponding to the HR patch having the highest priority, its K-NN in the inpainted images of lower resolution is sought.

4. Weights wp,pj of the image are calculated by using a non-local means method to perform a linear combination of these neighbors. The similarity distance used to compute the weights is composed of two terms: the first one is classical since this is the LR patch and its LR neighbors. The second term is the distance between the known parts of the HR patch and the HR patches corresponding to the LR neighbors.

5. A HR pixel is finally deduced by using a linear combination of HR patches.

6. Stitching: the patch is then pasted into the missing areas. The stitching algorithm is only used when all pixel values in the overlapping region are known or already synthesized. Otherwise, stitching is disabled.

In the below figure 2, the weights are computed using the similarity distances between LR and HR patches. The top image represents the original image with the missing areas whereas the bottom one (image that we get after stitching neighbor pixels) is the result of the low resolution inpainting.

Dictionary

1

Origin

-al Image

Filling order computation 2

LR

Inpainted picture

Similarity Computation 3

Stitching 5

Weight computation 4

VII. PERFORMANCE DISCUSSION

One use is in restorig photographs. With time, photographs get damaged and scratched. Users can then use the software to remove the cracks from the photographs.

Another use of image inpainting is in creating special effects by removing unwanted objects from the image. Unwanted objects may range from microphones, ropes, some unwanted person and logos, stamped dates and text etc. in the image. During the transmission of images over a network, there may be some parts of an image that are missing. These parts can then be reconstructed using image inpainting with super resolution.

The algorithm automatically proceeds that it looks reasonable to the human eye. No mathematical method can recover the hidden details of object to be removed. Therefore the objective for image inpainting is not to recover the original image, but to create some image that has a close resemblance with the original image.

Compared to the existing methods, using this technique it improves the robustness and visual relevance of inpainting. In the image inpainting, by introducing super-resolution algorithm and fast inpainting it reduces the computational

Figure 2: Super-resolution based inpainting.

VI. IMPLEMENTATION

In real world, image resources are often received in poor condition, mostly with noise or defects making the resources hard to read. This implementation proposes an effective algorithm based on super-resolution image inpainting the mechanism can be used in restoring images with very high noise or defect ratio (e.g., 90%). The algorithm is based on the concept of image subdivision and estimation of color variations. Different sizes of blocks which contain Noise are inpainted with different levels of surrounding information. This implementation results showed that an almost unrecognizable image can be recovered with visually good result.

The above algorithm further implements fast inpainting technique that will be useful in the case of high resolution pixels. The parameters which are used in above inpainting technique is

reproducible research: it is possible to reproduce results by using the executable software.

parameters: two versions of the proposed methos are evaluated. One version uses down factor of 4 in both direction whereas the same is set to 2 for the second. The size of the dictionary is same in both versions and contains at most 6000 patches evenly distributed over the picture.

Line front feathering: instead of using stitching method, the front line which is the border between known and unknown areas can still be visible. It can be hide this transition by feathering the pixel values across this seam.

complexity of inpainting and improves the speed and accuracy of the image inpainting technique.

VIII. CONCLUSION

Beyond this first point which demonstrates the effectiveness of the proposed method, this framework can be improved. For instance, one interesting avenue of future work would be to perform several inpainting of the low-resolution images and to fuse them by using a global objective function. After inpainting, a super resolution algorithm is used to improve the visual quality that compares the intensity of the neighbor pixels. If the intensity is same or matched then it copies the best matched pixel into the selected region with same intensity. So the fast inpainting technique is added to fast the inpainting process.

Finally, the proposed framework will be appropriate for video completion. This application is indeed bit time- consuming if the image consists of high resolution pixels for image restoration. So this includes fast inpainting technique that copies block of pixels into the selected region in the image. For instance, one interesting avenue of future work would be to perform several inpainting of the low-resolution images and to fuse them by using a global objective function. The use of this proposed framework could dramatically reduce the computational time. In the future work, if the super resolution algorithm is improved so that the computational complexity is further improved while retaining the quality of inpainting and if possible, this would also like to improve the inpainting algorithm.

REFERENCES

1. Bertalmio, M., Sapiro, G., Caselles, V., Ballester, C.: Image inpainting. In: SIG-GRPAH 2000.

2. Tschumperl´e, D., Deriche, R.: Vector-valued image regularization with pdes: a common framework for different applications. IEEE Trans. on PAMI 27 (2005)

3. Chan, T., Shen, J.: Variational restoration of non-flat image features: models and algorithms. SIAM J. Appl. Math.

4. Criminisi, A., P´erez, P., Toyama, K.: Region filling and object removal by examplar-based image inpainting. IEEE Trans. On Image Processing 13 (2004) 12001212

5. Drori, I., Cohen-Or, D., Yeshurun, H.: Fragment-based image com pletion. ACM Trans. Graph. 22 (2003)

6. Harrison, P.: A non-hierarchical procedure for re-synthesis of complex texture. In:Proc. Int. Conf. Central Europe Comp. Graphics, Visua. and Comp. Vision. (2001)

7. Barnes, C., Shechtman, E., Finkelstein, A., Goldman, D.B.: PatchMatch: A randomized correspondence algorithm for structural image editing.

8. Efros, A.A., Leung, T.K.: Texture synthesis by non-parametric sampling. In: International Conference on Computer Vision. (1999)

1. Freeman,W.T., Jones, T.R., Pasztor, E.C.: Example-based super- resolution. IEEE Computer Graphics and Applications 22 (2002)

2. Glasner, D., Bagon, S., Irani, M.: Super-resolution from a single image. In: In 2009 IEEE 12th International Conference on Computer Vision (ICCV). (2009)

3. Chang, H., Yeung, D.Y., Xiong, Y.: Super-resolution through neighbor embedding. In: Computer Vision and Pattern Recognition. (2004)

4. Ashikhmin, M.: Synthesizing natural textures. In: I3D01. (2001)

5. Oliva, A., Torralba, A.: Building the gist of a scene: the role of global image features in recognition. Progress in Brain Research: Visual perception 155 (2006)

6. Bugeau, A., Bertalm´o, M., Caselles, V., Sapiro, G.: A comprehensive framework for image inpainting. IEEE Trans. on Image Processing 19 (2010)