Image Inpainting Framework with Super-Resolution

Image Inpainting Framework with Super-Resolution

Dhanashree Umale1 , Kalpana Malpe 2, Abdulla Shaik3

1. Tech Student, Computer Science & Engineering Department, Nuva College of Engineering and Technology, Nagpur.

   Abstract This paper introduces a replacement examplar- based inpainting frame-work. a rough version of the input image is initial inpainted by a non-parametric patch sampling. Compared to existing approaches, some enhancements are done (e.g. filling order computation, combination of K nearest neighbours). The inpainted of a rough version of the input image permits to scale back the procedure quality, to be less sensitive to noise and to figure with the dominant orientations of image structures. From the low-resolution inpainted image, a single- image moreover as multiple-image super-resolution is applied to recover the missing areas. Experimental results on natural pictures and texture synthesis demonstrate the activeness of the proposed technique.

   Keywords examplar-based inpainting, super-resolution

   1. INTRODUCTION

      Image inpainting refers to strategies that consist in filling- in missing regions (holes) in an image [1]. Existing strategies is classified into two main classes. the primary class considerations diffusion-based approaches that propagate linear structures or level lines (so-called isophotes) via diffusion based on partial differential equations [1,2] and variational strategies [3] .Unfortunately, the diffusion-based strategies tend to introduce some blur once the outlet to be filled-in is large. The second family of approaches considerations examplar- based strategies that sample and copy best matching texture patches from the renowned image neighborhood [4, 7]. These strategies are galvanized from texture synthesis techniques [8] and square measure renowned to figure well in cases of normal or repeatable textures. the primary attempt to use examplar- based techniques for object removal has been reported in [6]. Authors in [5] improve the look for similar patches by introducing an a priori rough estimate of the inpainted values employing a multi-scale approach that then leads to an repetitious approximation of the missing regions from coarse to fine levels. the two styles of strategies (diffusion- and examplar-based) is combined expeditiously, e.g. by using structure tensors to compute the priority of the patches to be filled as in [9].

      Although tremendous progress has been created within the past years on inpainting, difficulties stay once the outlet to be filled is large and another important facet is that the high process time normally needed. These two issues are here self- addressed by

      considering a hierarchic approach within which a lower resolution of the input image is initially computed and inpainted using a K-NN (K Nearest Neighbours) examplar- based methodology. Correspondences between the K-NN low- resolution and high-resolution patches are first learnt from the input image and hold on during a dictionary. These correspondences are then used to find the missing pixels at the higher resolution following some principles utilized in single- image further as multiple-image super-resolution strategies.

      Super-Resolution (SR) refers to the method of creating one increased resolution image from one or multiple input low resolution images. the two corresponding issues are then mentioned as single or multiple images SR, severally. In each cases, the matter is of estimating high frequency details that are missing within the input image(s). The proposed SR-aided inpainting methodology falls into the context of single-image as multiple-image SR on that we therefore focus during this section.

      The SR drawback is ill-posed since multiple high- resolution images will turn out constant low-resolution image. finding the matter thus needs introducing some previous info. The previous info is Associate in Nursing energy purposeful de-fined on a category of images that is then used as a regularization term in conjunction with interpolation techniques[10] . This previous info may take the shape of example images or corresponding LR-HR (Low Resolution – High Resolution) pairs of patches learnt from a group of un- related coaching images in an external information [11] or from the input low resolution image itself [12]. This latter family of approaches is thought as example-based SR strategies. Associate in Nursing example-based SR methodology embedding K nearest neighbours found in an external patch information has also been represented in [13]. rather than constructing the LR-HR pairs of patches from a group of un-related coaching images in an external information, the authors in [12] extract these correspondences by looking for matches across completely different scales of a multi-resolution pyramid made from the input low-resolution image.

      The proposed methodology therefore builds upon earlier work on examplar-based inpaint-ing especially on the approach proposed in [4] , further as upon earlier work on single-image further as multiple-image examplar-based super-resolution [12]. However, since the standard of the low-resolution inpainted image encompasses a important impact on the standard at the ultimate resolution, the inpainting algorithmic

      rule in [4] is first improved by considering each a linear combination of K most similar patches (K-NN) to the input patch instead of using merely the simplest match by example matching and K-coherence candidates as proposed in [14]. The impact of various patch priority terms on the standard of the inpainted pictures is additionally studied, resulting in retain a sparsity-based priority term. additionally, a replacement similarity live supported a weighted Battacharya distance is introduced. during a second step, the patches to be crammed inside the input time unit image are processed per a specific filling order. The algorithmic rule therefore payoff by looking for K nearest neighbours to the input vector concatenating the renowned time unit pixels of the patch and therefore the pixels of the cor-responding inpainted LR patch. The K-NN patches are searched during a dictionary composed of LR-HR patches extracted from the renowned a part of the image. The similarity metric is once more the weighted Battacharya metric. Similarity weights are computed between the input and K-NN vectors shaped by the LR and renowned pixels of the time unit patches. Finally, since the inpainted time unit patches are overlapping, a seam is searched throughout the overlapping region, and therefore the ini-tially overlapping patches are therefore pasted on this seam.

      In summary, the proposed methodology any advances the progressive in examplar-based inpainting strategies by proposing:

      • a replacement framework which mixes inpainting and super-resolution during a two-step approach rising the trade-o between quality and complexity;

      • enhancements regarding the use of priority terms, the set of candidates (K-NN and K-coherence candidates) and distance metrics.

      The paper is organized as follows. In Section II, the new framework of the proposed inpainting methodology is given. Section III elaborates the proposed examplar-based inpainting and presents experimental leads to comparison with those created by with progressive strategies. In Section IV, the details of the SR-aided inpainting methodology are introduced. Section V presents the performance of the proposed methodology further as comparisons with progressive strategies. Finally, we tend to conclude this work in the SectionV.

   2. ALGORITHM OVERVIEW

      Image completion of huge mising regions may be a difficult task. As presented within the previous section, there are variety of solutions to tackle the inpainting drawback. during this paper, we have a tendency to propose a new inpainting technique using a single-image additionally as multiple-image SR algorithmic rule. within the following sections, we have a tendency to briefly gift the most ideas of this paper and also the reasons why the proposed technique is new and innovative.

      1. Motivations

         The proposed technique consists of two main and sequential operations. the primary one may be a non- parametric patch sampling technique used to fill-in missing regions. However, instead of filling in missing regions at the initial resolution, the inpainting algorithmic rule is applied on a rough version of the input image. There are many reasons for

         performing the inpainting on a low-resolution image. First, the coarse version of the input image may well be compared to a gist [15] representing dominant and necessary structures. performing the inpainting of this coarse version is way easier since the inpainting would be less depending on native singularities (local orientation for instance) or maybe noise. Second, because the image to inpaint is smaller than the initial one, the machine time to inpaint it's considerably reduced compared to the one necessary to inpaint the complete resolution image.

         The second operation is run on the output of the primary step. Its goal is to enhance the resolution and also the subjective quality of the inpainted areas. we have a tendency to use a single-image additionally as multiple-image SR approach. Given a low-resolution input image, that is that the results of the primary inpainting step, we have a tendency to recover its high-resolution using a set of coaching examples, that are taken from the known a part of the input image.

         Fig. 1. The framework of the proposed method

         This new technique is generic since there's no constraint on each the quantity and also the kind of inpainting ways utilized in the primary pass. the higher the inpainting of low-resolution images, the higher the ultimate result should be. concerning the quantity of ways, one might imagine using totally different settings (patch size, search windows etc) or ways to fill-in the low-resolution images and to fuse results. we have a tendency to believe that it might increase the strength and also the visual relevancy of in-painting, as recently proposed by Bugeau et al [16]. they need so shown that the simplest inpainting results are obtained by a mix of 3 different ways. during this paper, owing to the restricted house, we are going to not consider the possibility to mix the results of various inpainting algorithms, however rather concentrate on the proposed SR-aided inpainting framework.

      2. Principle

         Figure one illustrates the most thought underlying the proposed technique. the two main parts are the inpainting and also the super-resolution algorithms. a lot of specifically, the subsequent steps are performed:

         1. a low-resolution image is initial designed from the original picture;

         2. associate inpainting algorithmic rule is applied to fill-in the holes of the low-resolution picture;

         3. the standard of the inpainted regions is improved by using a single-image additionally as multiple-image SR technique.

   3. EXAMPLAR-BASED INPAINTING OF LOW- RESOLUTION IMAGES

      This section presents the inpainting methodology that is used during this paper to fill within the low-resolution images. this can be an adaptation of the Criminisi et al. [4] method. The influence of various priority terms on the standard of the inpainted images is first studied. A similarity metric supported a weighted Bhattacharya distance is proposed. The ensuing inpainting formula is compared against two progressive strategies. the primary one is additionally supported a non- parametric patch sampling (Patch Match, [7]) whereas the second relies on partial derivatives equations [2]. we've chosen these two strategies due to their connectedness and since the code is offered. The proposed examplar-based methodology follows the two classical steps as represented [4] the filling order computation and therefore the texture synthesis. These are represented within the next sections.

      1. Patch priority and filling order

         The filling order computation defines a measure of priority for every patch so as to distinguish the structures from the textures. Classically, a high priority indicates the presence of structure. The priority of a patch targeted on p is simply given by an information term (the confidence term proposed in [4]

         Fig. 2. Inpainting of LR pictures with completely different gradient-based priority (first row), tensor-based priority [9] (second row) and sparsity-based priority [17] (third row).

      2. Texture synthesis

         The filling method starts with the patch having the very best priority. two sets of candidates are used to fill within the unknown a part of this patch. a primary set consists of the K most similar patches situated in a very local neighbourhood targeted on this patch. they're combined by employing a non- local suggests that approach [18]. The coefficient factors are classically outlined as follows:

         isn't used here since it doesn't induce any improvement). three completely different knowledge terms are tested: gradient-

         ,

         = p , p ,p j

         (2) where d()

         based priority [4], tensor-based [9] and sparsity-based [17].

         The sparsity-based priority has been proposed recently by Xu et al. [17] in a very search window, a templet matching is performed between this patch p and close patches p,pj that belong to the known a part of the image. By using a non-local suggests that approach [18], a similarity weight wp,pj (i.e. proportional to the similarity between the two patches targeted on P and Pj ) is computed for every combine of patches. The poorness term is defined as:

         may be a metric indicating the similarity between patches, and h may be a decay issue. These weights are then normalized as wp,pj / k wp,pk . the quantity of neighbours is adapted locally so the similarity of chosen neighbours lies inside a variety (1

         + ) × dmin, wherever dmin is that the distance between this patch and its closest neighbour, is equal to zero.75.

         As mentioned by [8], a significant problem of local neighbourhood search is its ten-dency to get stuck at a selected place within the sample image and to supply ver-batim repeating. this sort of regions is commonly referred to as

         =

         × ()

         ()

         (1)

         garbage region. This drawback is self-addressed by introducing some constraints in terms of spacial coherence. the concept relies on the very fact that patches that are neighbours within

         where Ns and N represent range|the amount|the quantity} of valid patches (having all its pixels known) and therefore the total number of candidates within the search window. once kWpk2 is high, it suggests that larger poorness whereas atiny low worth indicates that this input patch is with efficiency expected by several candidates. As illustrated in figure 2, the sparsity-based priority is additional sturdy and visually improves the final result compared to the gradient and tensor- based priority. within the following, we tend to adopt this methodology to compute the filling order.

         the input image should be also neighbours within the output image [14,19]. Figure three a) illustrates this method. With a 8- connexity neighbourhood targeted on this patch (noted C on figure three a)), Ki patches are used as candidates and compared to the most effective candidate obtained by the local neighbourhood search. Figure three b) and c) show the influence of the k-coherence methodology on the standard of the low-resolution inpainted image. the use of k-coherence candidates improves locally the standad on several elements of the pictures.

         Concerning the similarity measure, we've considered two metrics: the classical sum of square differences (dSSD ) and a weighted Bhattacharya distance because the one proposed in (d(SSD,BC)). The last metric is defined as follows:

         , p , p,pj

         = p , p,pj

         × 1 + p, p,pj

         (3)

         where, dBC p, p,pj is a modified version of the classical Bhattacharya distancenas described in [16]

         p , p,pj = 1 p1 k p2(k)

         k

         where p1 and p2 represent the histograms of patches p, p,pj , respectively). this can be not exactly the same formulation as in [16]: so Bugeau et al. directly multiply the SSD distance with dBC. This presents a drawback: for 2 patches having identical distribution, in spite of however the rotation the worth dBC is null, resulting in a null distance d(SSD,BC). With the proposed metric, the distance is equal to the SSD distance.

         Fig. 3. (a) K-coherence algorithm: candidates shown in orange square measure used as predictors; (b) and (c) are inpainted images once the k-coherence methodology is

         disabled or enabled severally. inexperienced circles stress the most important variations between pictures (b) and (c).

      3. Alternative state-of-the-art strategies

         To compare the proposed inpainting methodology to existing ones, we've chosen strategies that either the ASCII text file or an executable file are available. Criminisi et al. method: Criminisi et al. [4] proposed to guide the filling method with a priority term supported edge strength. Diffusion-based method: as introduced earlier, diffusion-based strategies prop-agate the structures into missing regions. For comparison functions, we tend to use during this paper the approach proposed in inpainting. Patch Match: Patch Match methodology may be a quick formula for computing dense approximate nearest neighbour correspondences between patches of two image regions. This algorithm is offered in Adobe Photoshop CS5 and works well. we are going to consistently compare our results to Patch Matchones. Note that we've also tested a way referred to as tensor completion [20]. However, once the missing space is simply too giant, the inpainting quality is low. we tend to then overlooked this methodology.

      4. Comparison between proposed and progressive strategies

         Figure four (e) and (f) illustrate the performance of the proposed methodology on various low-resolution pictures. These pictures are down sampled versions of the initial one (the down sampling issue is adequate to four in each directions for (e) and adequate to two for (f)) of course, the diffusion-

         based approach retrieves the most structures of the scene (except for the third and fifth pictures). However, it tends to swish the rough regions. Results obtained by the proposed and therefore the Patch Match strategies are comparable though artifacts aren't identical (for Patch Match: a person is duplicated (see second row) and a few grass seems on the rock (last row); for the proposed methodology (e): the image on the third row presents additional artifacts).

         Fig. 4. (a) low-resolution pictures with missing areas in black; (b) Criminisi et al.s results; (c) Patch Match results; (d)

         Diffusion-based results; (e) proposed methodology (the down sampling issue is ready to N = 4; patch size is 11 × 11); (f) proposed methodology with N =2 and patchs size of15×15.

         Fig. 5. flowchart of the super-resolution algorithm. The missing elements of the red block is filled by a linear

         combination of K HR-candidates (green arrows). The weights are computed using the similarity distance between LR and hr patches (green and red arrows, respectively). the highest image represents the original image with the missing areas whereas all-time low one is that the results of the low-resolution inpainting.

   4. SUPER-RESOLUTION ALGORITHM

      Once the inpainting of the low-resolution image is completed, a single-image super-resolution approach is used to reconstruct the high resolution of the image. the concept is to use the low-resolution inpainted areas so as to guide the texture synthesis at the higher resolution. As in [11] the problem is to search out a patch of higher-resolution from a info of examples. the most steps, illustrated in figure 5 are described below:

      1. dictionary building: it consists of the correspondences between low and high resolution image patches. The distinctive constraint is that the high-resolution patches got to be valid, i.e. entirely composed of legendary pixels. within the pro-posed approach, high-resolution and valid patches are equally extracted from the known a part of the image. the dimensions of the dictionary could be a user-parameter which could influence the speed/quality trade-off. associate array is employed to store the spatial coordinates of time unit patches (DHR). Those of LR patches are merely deduced by using the destruction issue.

      2. Filling order of the hr picture: the computation of the filling order is similar to the one delineate in Section 3. it's computed on the hr image with the sparsity-based technique. The filling method starts with the patch pHR having the very high priority. This improves the standard of the inpainted image compared to a raster-scan filling order.

      3. For the LR patch corresponding to the hr patch having the very best priority, its K-NN within the inpainted pictures of lower resolution are wanted. the amount of neighbours is computed as delineate within the previous section. The similarity metric is also identical as previous;

      4. Weights wp,pj are calculated by using a non-local suggests that technique as if we might wish to perform a linear combination of those neighbours. However, the similarity distance used to compute the weights consists of two terms: the primary one is classical since this can be the distance between the present LR patch and its LR neighbours, noted

         d(pLR, LRp,pj). The second term is that the distance between the known components of the hr patch hrp and therefore the HR patches corresponding to the LR neighbors of LRp . Say otherwise, the similarity distance is that the distance between two vectors composed of each pixels of LR and hr patches. the use of pixel values of hr patches permits to constraint the nearest neighbor search of LR patches.

      5. A hr candidate is finally deduced by using a linear combination of hr patches with the weights previously computed:

         with the usual conditions 0 wp,pj 1, and k wp,pk = 1.

      6. Stitching: the hr patch is then pasted into the missing areas. However, as an overlap with the already synthesized areas is feasible, a seam cutting the overlapped regions is decided to additional enhance the patch mixing. The minimum error boundary cut [21] is used to search out a seam that the two patches match best. The similarity measure is that the geometer distance between all pixel values within the

      overlapping region. a lot of complicated metrics are tested however they are doing not considerably improve the ultimate quality. At the most four overlapping cases (Left, Right, prime and Bottom) will be encountered. There are consecutive treated within the aforesaid order. The stitching algorithmic rule is just used once all pixel values within the overlapping region are known or already synthesized. Otherwise, the stitching is disabled.

      After the filling of the present patch, priority value is recomputed and therefore the afore-mentioned steps are iterated whereas there exist unknown areas.

   5. EXPERIMENTAL RESULTS

      In order to assess the performance of the proposed approach, the parameters of the algorithmic program are kept constant for the tests presented during this paper.

      1. Implementation details and parameters

         Reproducible research: it's possible to reproduce results by using the exe-cutable code, the masks and picures accessible on authors web content. Parameters: two versions of the proposed methodology are evaluated. One uses a down sampling issue of 4 in each directions (the patch size is up to 5

         × 5) whereas this issue is about to two for the second version (the patch size is up to 7 × 7). For each versions, the scale of the dictionary is that the same and may contain at the most 6000 patches equally distributed over the image. The LR patch size is 3 × 3 and therefore the hr patch size is 15 × 15. Line front feathering: in spite of the use of stitching methodology, the line that is that the border between known and unknown areas will still be visible. it's possible to cover this transition by feather the element values across this seam. A mathematician kernel is used to perform the filtering.

      2. Comparison with state-of-the-art methods

      Figure 6 illustrates the comparison between the proposed ways and state-of-the-art ways. The proposed methodology (for each settings (e) and (f)) provides similar results to Patch Match and visually outperforms Criminisis approach. Figure 8 offers further results. For these examples, a large missing space has been filled in. Figure 7 presents further results for texture synthesis. a small chunk of texture (in the instance 256 × 256) was placed into the higher left corner of an empty image. Figure 7 illustrates the performance of the proposed methodology for these types of texture. For deterministic textures, results are excellent. For random ones, some artefacts are visible. However, increasing the patch size would address these artefacts, as illustrated on the bottom-right of figure 7. The state-of-the-art on a three rate central processing unit is a smaller amount than one minute for run time having a resolution of 512 ×512.

   6. CONCLUSION

In this paper we've got introduced a replacement inpainting framework which combines non-parametric patch sampling methodology with a super-resolution methodology. we tend to initial propose an extension of a widely known examplar-based methodology (improvements are sparsity-based priority, K- coherence candidates and a similarity metric adapted from [16]) and compare it to existing ways. Then, a super-resolution methodology is used to recover a high resolution version. This

framework is interesting for various reasons. initial the results obtained are among the progressive for a moderate complexness. on the far side this initial purpose that demonstrates the activeness of the proposed methodology, this framework may be improved. as an example, one interesting avenue of future work would be to perform many inpainting of the low-resolution pictures and to fuse them by employing a world objective perform. First, totally different varieties of inpainting ways (patch-based or PDE-based) can be used to fill-in the missing areas of a low-resolution image. Second, for

Fig. 6. Comparison of the proposed methodology with state-of-the-art approaches: (a) Cri-minisi et al. [4]; (b) Patch Match [7]; (c) proposed methodology (N = 4), (N = 2) (f).

a given inpainting methodology, one will envision to stand- in the missing areas by using totally different settings e.g. for the patch size so as to raised handle a range of textures and to raised approach the texture component sizes. Finally, we tend to believe that the proposed framework are acceptable for video completion. This application is so terribly long. the utilization of the proposed framework may dramatically cut back the machine time.

Fig. 7. Texture synthesis (down sampling factor N = 2).

Fig. 8. For these pictures, a large missing region has been inpainted (masks are not intentionally given but are available as supplementary materials).

REFERENCES –

1. Bertalmio, M., Sapiro, G., Caselles, V., Ballester, C.: Image inpainting. In: SIG-

   GRPAH 2000. (2000)

2. Tschumperl´e, D., Deriche, R.: Vector-valued image regularization with pdes: acommon framework for different applications. IEEE Trans. on PAMI 27 (2005)506517

3. Chan, T., Shen, J.: Variational restoration of non-flat image features: models and algorithms. SIAM J. Appl. Math. 61 (2001) 13381361

4. Criminisi, A., P´erez, P., Toyama, K.: Region filling and object removal by examplar-based image inpainting. IEEE Trans. On Image Processing 13 (2004)12001212

5. Drori, I., Cohen-Or, D., Yeshurun, H.: Fragment- based image completion. ACM Trans. Graph. 22 (2003) 303312

6. Harrison, P.: A non-hierarchical procedure for re- synthesis of complex texture. In: Proc. Int. Conf. Central Europe Comp. Graphics, Visua. and Comp. Vision. (2001)

7. Barnes, C., Shechtman, E., Finkelstein, A., Goldman, D.B.: PatchMatch: A randomized correspondence algorithm for structural image editing. ACM Transactions on Graphics (Proc. SIGGRAPH) 28 (2009)

8. Efros, A.A., Leung, T.K.: Texture synthesis by non- parametric sampling. In:International Conference on Computer Vision. (1999) 10331038

9. Le Meur, O., Gautier, J., Guillemot, C.: Examplar- based inpainting based on local geometry. In: ICIP. (2011)

10. Dai, S., Han, M., Xu, W., Wu, Y., Gong, Y., Katsaggelos, A.: Softcuts: a soft edge smoothness prior for color image super-resolution. IEEE Trans. On Image Processing 18 (2009) 969981

11. Freeman,W.T., Jones, T.R., Pasztor, E.C.: Example- based super-resolution. IEEE Computer Graphics and Applications 22 (2002) 5665

12. Glasner, D., Bagon, S., Irani, M.: Super-resolution from a single image. In: In 2009 IEEE 12th International Conference on Computer Vision (ICCV). Volume 10. (2009) 349356

13. Chang, H., Yeung, D.Y., Xiong, Y.: Super-resolution through neighbor embedding. In: Computer Vision and Pattern Recognition. Volume I. (2004) 275282

14. Ashikhmin, M.: Synthesizing natural textures. In: I3D01. (2001)

15. Oliva, A., Torralba, A.: Building the gist of a scene: the role of global image features in recognition. Progress in Brain Research: Visual perception 155 (2006) 2336

16. Bugeau, A., Bertalm´o, M., Caselles, V., Sapiro, G.: A comprehensive framework for image inpainting. IEEE Trans. on Image Processing 19 (2010) 2634 2644

17. Xu, Z., Sun, J.: Image inpainting by patch propagation using patch sparsity. IEEE

    TIP 19 (2010) 11531165

18. Wexler, Y., Shechtman, E., Irani, M.: Space-time video completion. In: Computer Vision and Pattern Recognition (CVPR). (2004)

19. Tong, X., Zhang, J., Liu, L., Wang, X., Guo, B., Shum, H.Y.: Synthesis of bidriec- tional texture functions on arbitrary surfaces. In: SIGGRAPH02. (2002) 6635672

20. Liu, J., Musialski, P., Wonka, P., Ye, J.: Tensor completion for estimating missing values in visual data. In: International Conference on Computer Vision. (2009) 21142121

21. Efros, A.A., Freeman, W.T.: Image quilting for texture synthesis and transfer. In: SIGGRAPH. (2001)