A Review on Image Inpainting Techniques

A Review on Image Inpainting Techniques

Roshani Parashram Sonawane Department of Computer Engineering MET BKC Institute of Engineering, Maharashtra, India

Dr. Prashant M. Yawalkar Department of Computer Engineering MET BKC Institute of Engineering, Maharashtra, India

AbstractVarious factors affect the image that causes image deterioration. The art of restoration of deteriorated parts of image is known as image inpainting. The image inpainting is mainly categorized in 2 sections: Diffusion based and Exemplar- based image inpainting. This paper contains the overview of various techniques of image inpainting its advantages and limitations. Based on the Existing techniques, a new technique is proposed for image inpainting to overcome the drawbacks of existing system.

Keywords Image inpainting, Surface fitting, Angle awareness, Dynamic patch selection, Markov random field model

1. INTRODUCTION

   Image inpainting is also referred as image completion. This is an image processing technique in which image is visually filled at the missing region. It corrects or reconstructs the deteriorated part of image. The image deterioration can happen due to various reasons like: environmental factors, chemical processing, improper storage and more. Image inpainting has various applications like:

   Image restoration: It recovers the degraded or distorted part of image and improves the appearance.

   1. Image coding and transmission: Using this technique image is encoded and transmitted to the receiver end. At the receiver end image is get reconstructed initially to display only highlighted features and as per the user requirement it get refined.

   2. Photo editing: In photo editing image can be altered, newly created or helps to merge 2 or more images.

   3. Virtual restoration: It detects and removes cracks on digitized image.

   Initially Bertalmio et al. [8] proposed an inpainting method to resolve the damaged and occluded parts of an image. The solution is based on formulating higher order partial differential equation i.e. it creates Laplacian of an image. This equation propagates the information in the direction of isophotes. The proposed algorithm follows the direction of gradient for image inpainting.

   The image inpainting is categorized in two sections:

   1. Diffusion based:

      In diffusion-based technique, missing region is filled. The missing region is called as hole. The hole is filled with smooth propagation from boundary to the center of hole. This follows the linear propagation structures. It generates good results in case of inpainting long thin region. But fails to inpaint texture-based image.

   2. Exemplar Based:

   This method is useful for filling large scale missing region. This technique is mainly categorized in 2 sections:

   1. Pixel Based: This system works with pixel by pixel filling of mission region. This method is time consuming and texture synthesis is not properly done at pixel level. To overcome this drawback patch-based exemplar technique is proposed.

   2. Patch Based: This technique fills the image patch by patch. It searches for well-matching replacement parches from undamaged part of image. Such patches can be also called as candidate patches. The best suited patch is copied to the required location.

      As compared to the diffusion-based method patch-based methods generates better results in case of texture-based image inpainting or for inpainting the larger holes.

      The proposed system performs inpainting in dynamic manner. In the process of inpainting initially the unknown region is initialize using fitting surface method. The dynamic patch selection strategy applied for inpainting missing region. Small target patches are applied in the high-frequency region. The patches are selected from context aware source region using angle aware patch matching technique. To improve the patch selection process Markov random field is used. For angle- aware patch matching, Jaccard similarity coefficient is used. For this study secondary data has been collected.

2. LITERATURE SURVEY

   Na Zhang et al. [1], proposes an angle aware patch inpainting. It initially initializes the target patch that is to be removed using surface fitting and MIS method. This system uses dynamic patch selection process. In the high-frequency region, small target patches are applied whereas in low frequency region, large patches are applied. The system uses angle-aware patch matching. For this Jaccard similarity coefficient is used. The patch is searched from all the regions of images. This is a time-consuming process.

   In the following section some image processing techniques are discussed:

   1. Texton histogram [1]:

      This is used for image texture representation. The procedure for texton calculation is as follows:

      1. Extract 5×5 sized image patches

      2. From centre of each image patch

      3. Subtract the mean rgb value of the pixels in that patch

      4. Clucter the image patches using K-means

      5. Centroids of the K Clusters are the textons

      6. Compute a K dimensional histogram of textons

   2. Markov random field model [4]:

      For matching best patch from an image, the image patches are searched from other part of image. To improve the efficiency of searching patches the image patches are arranged using Markov random field model. The network of similar patches is generated using Markov random field. This is undirected

      graph. For Markov random filed generation following steps are carried out:

      Divide image in adaptive size blocks based on intensity and color

      1. Find texton of image

      2. Generate a Graph(V,E) where V are patches and E represents set of edges with similarity in texton values.

   3. Surface Fitting Technique[1]:

   This method is used to initialize the pixel values for missing region of an image. The process of surface fitting technique is as follows:

   1. Find grey value of each pixel

   2. Project the image pixels from 2D space to 3D space by adding height as grey value

   3. Apply moving least square method

   Zhao et al. [2] proposes a GPU based inpainting method. A coherent direction-aware patch alignment algorithm is proposed. GPU speed up the patch searching process and enhances the similarity among matched patches. The execution time reduces as compared to the other methods.

   Prasath et al. [3] proposes a diffusion-based method. The system works on ill-posed image processing problems. The inpainting process works on limited dataset. The system works on lower structures of texture and geometric constructions. The system takes only small-scale natural images. The inpainting area should be small. For large missing region or complex textures, the resultant image contains inconsistency of structure and texture, unpleasant artifacts. The generated image is over-smoothed.

   T. Ruzic et al. [4], proposes a context-aware patch-based image inpainting method. In this technique textural descriptors are used to find best matching candidate patches. This technique divides the image in unequal size of blocks based on its context. This is a global image inpainting method. It uses Markov random field to encode a priori knowledge of other neighboring candidate patches. Due to Markov random field evaluations the number of candidate patches are get filtered and hence candidate size is got reduced. For contextual descriptors texton Histograms is poposed. This histogram finds the texture within a local region.

   Cheng-Shian Lin et al. [5] proposes a multichannel nonlocal total variation model for inpainting. This system deals reconstruction problem of remotely sensed images. In this technique surrounded part and silent region is inpainted first and then the remaining one.

   C. Barnes et al. [6] proposes a fast patch-based image inpainting method called as patch match. The main disadvantage of patch-based method is its efficiency. In patch-based method system searches for different candidate patches. This candidate generation process is time consuming and exhaustive. This is a structural image editing method using randomized algorithm. It quickly finds the best match patch by approximate nearest neighbor match. This paper proposes an interactive image editing tool.

   T. F. Chan et al. [7] proposes an inpainting method for normal non-textural images. This technique uses curvature-driven diffusions mechanism. This technique is used in nonlinear

   Partial differential equation. This paper treats on the disocclusion of image based on human vision. It treats image as 2D projection of 3D world scene. This paper deals the inpainting process as a lower level process as compared to disocclusion.

   Patch based inpainting methods are categorized in following 3 categories:

   1. Greedy:

      Using greedy technique best match is selected for filling the hole. The selected patch is called as target patch. The patch is get selected based on its known pixels. The greedy method is iterative method that fills the hole gradually.

   2. Multiple candidates:

      In this type of methods multiple matched patches are selected from other undamaged part of image. The missing region is filled by weighted average or sparse combination of multiple combinations of patches.

   3. Global:

   This method treats the inpainting as a global optimization problem. Here also multiple candidates are chosen. Such candidates are selected for different positions in an image. These candidates are known as labels. Labels for each position are selected in such way that it minimizes the global optimization function.

3. PROBLEM FORMULATION

   Based on the literature review, patch based inpainting method generates better image inpainting results as compared to the diffusion-based method. Angle-ware rotation patch matching strategy which considers the different angles of the same source patch in order to find multiple candidate patches for every target patch and improves the matching accuracy. The Context aware patch-based method using MRF helps to reduce the number of possible candidate patches and helps to improve efficiency in inpainting method. From the matched contextual block, the matched patch identified. MRF method improves the system efficiency.

   There is need of such system that helps to find angle aware target patches from context aware source region to inpainted texture-based images efficiently.

4. PROPOSED METHODOLOGY

1. System Architecture

   In Figure 1 shows the architecture of a system. The image to be inpainted is input to the system. The block to be inpainted is selected by the user. System generates inpainted image with its PSNR value as an output.

   Figure 1. System Architecture

   The proposed system focuses on context-aware patch-based image inpainting. The proposed system is Exemplar-based inpainting solution. Initially the image is divided in variable size block according to their context. The candidate patches are searched with the matched image block. The Markov random field model is used to manage the access of nearest matching patches. The image inpainting is based on surface fitting as the prior knowledge. This technique uses angle- aware patch matching. For the matching precision between patches, Jaccard similarity coefficient is used. This maintains the consistency of the structures and textures. The system is tested on multiple dataset images for object removal. For image quality analysis, PSNR ratio is calculated. The system results are compared with existing system in terms of efficiency.

2. System Working

   The image is given as input to the system. Based on the user input, initially image is divided in adaptive size blocks. And contextual descriptor using Texton histogram is assigned to the patches. The Markov random field model is used to manage generated blocks in a chain. For patch matching several suitable candidate blocks are selected for patch search. The method focuses on replacing the missing region using textural and structural similarity. Initially the unknown pixels in missing region are initialized by surface fitting technique. In second phase filling order is defined using priority function. Filling of mage is started from boundary point to the inner section. Based on the gradient value target patch is selected. Then for filling inner section of target region various patches are selected using angle rotation strategy from source region.

   The selected patch is placed in the image and then again priority is checked for next filling order this is an iterative

   process. After completing the image inpainting PSNR value of an image is calculated.

   IV. CONCLUSION

   In this paper various existing image inpainting techniques are discussed. Patch based image inpainting techniques generate better results as compared to the other techniques. Angle- aware rotation patch matching strategy improves the matching accuracy and hence improves the accuracy of inpainting technique for texture-based images. The Efficiency of the system can be improved using Markov technique that preserve the possible matching patch collection. Based on the analysis of existing systems a new system is proposed. The proposed system focuses on context-aware patch-based image inpainting. Initially the image is divided in variable size block according to their context. The Markov random field model is used to manage generated matched image blocks. The candidate patches are searched with the matched image block. The image inpainting is based on surface fitting and an angle-aware patch matching. Jaccard similarity coefficient is used for matching precision between patches. The proposed technique maintains the consistency of the structures and textures of inpainting image.

   REFERENCES

   1. Zhang, Na & Ji, Hua & Liu, Li & Wang, Guanhua, Exemplar- based image inpainting using angle-aware patch matching, EURASIP Journal on Image and Video Processing. 2019. 10.1186/s13640-019-0471-2.

   2. H. L Zhao, H. Y Guo, X. G Jin, J. B Shen, X. Y Mao, J. R Liu, Parallel and efficient approximate nearest patch matching for image editing applications, in Neurocomputing.

      S0925231218304703, 2018

   3. V.B.S. Prasath, D.N.H. Thanh, H.H. Hai, N.X. Cuong, Image restoration with total variation and iterative regularization parameter estimation, in the 8th International Symposium on Information and Communication Technology, SoICT – ACM, 2017

   4. T. Ruzic, A. Pizurica, Context-aware patch-based image inpainting using Markov random field modeling, in IEEE Transactions on Image Processing, Vol. 24(1), pp. 444456,2015

   5. Q. Cheng, H. Shen, L. Zhang, P. Li, Inpainting for remotely sensed images with a multichannel nonlocal total variation model, in IEEE Trans. Geosci. Remote Sens. 52(1), pp: 175187, 2014

   6. S. Darabi, E. Shechtman, C. Barnes, D.B. Goldman, P. Sen, Image melding: combining inconsistent images using patch-based synthesis, in ACM Transactions on Graphics. 31(4) ,2012

   7. J. Shen, T.F. Chan, Mathematical models for local nontexture inpaintings, SIAM Journal on Applied Mathematics 62, pp: 1019 1043, 2002

   8. M. Bertalmio, G. Sapiro, V. Caselles, C. Ballester, Image inpainting in Proceedings of ACM SIGGRAPH, ACM Press, 2000, pp. 417424

   9. Datasets:

https://www2.eecs.berkeley.edu/Research/Projecs/CS/vision/bsds/ Datasets: https://icme19inpainting.github.io/