Video Inpainting for Scratch Removal using Exemplar based Method

Video Inpainting for Scratch Removal using Exemplar based Method

Komal S. Mahajan Assistant Professor Information Technology Dept

Atharva College of Engineering, Malad, India

Jyoti Golakia

Assistant Professor Computer Engineering Dept

Atharva College of Engineering, Malad, India

Snehal Kathale

Assistant Professor Information Technology Dept Atharva College of Engineering,

Malad, India

Trupti Shah

Assistant Professor Computer Engineering Dept

Atharva College of Engineering, Malad, India

Abstract Video Inpainting is considered as a natural extension of image inpainting algorithms. This Video Inpainting has great attention due to its potential applications in video transmission, video stabilization, multimedia editing. This paper proposes a video inpainting algorithm for scratch removal using robust exemplar based inpainting. The line scratches are one of the major defects in motion picture films. Scratches are usually intended as missing information in subsequent frames of image sequences in vertical area of each frame. Such problems may produce very annoying effect on the viewer. So restoration of such scratches is important. So that we are going to propose algorithm which will effectively detect the scratches and the remove the scratches from the video. The proposed technique is based on robust exemplar based inpainting. Exemplar-based inpainting methods iteratively search the source region and fill the missing or damaged region, i.e., target region, with the most similar patch in the source region. This paper presents an automatic scratch detection and removal approach for video. We are going to develop the algorithm which not only obtain the accurate restoration of scratches, but also gives better video quality.

Keywords Image Inpainting, Exemplar-based Inpainting, Frame Extraction, Region Segmentation, Scratch Detection, Scratch Removal, Video inpainting.

1. INTRODUCTION

   Nowadays, the Inpainting technology is a hotspot in computer graphics. And it is becoming very interested topic and attracts many researchers in recent years. It has important value in a heritage preservation, Removing redundant objects, film and television special effects production etc. In the fine art museums, this Inpainting concept is used for degraded paintings. Conventionally Inpainting is carried out by professional artist and usually its very time consuming process because it was the manual process. The main goal of inpainting process is to reconstruct damaged(degraded) parts or missing parts of image or motion films. And this process reconstructs image in such a way that the inpainted region from image and video cannot be detected by a casual

   Observer. Inpainting technique has found widespread use in many applications such as restoration of old films, object removal in digital photos, red eye correction, super resolution, compression, image coding and transmission. Inpainting reconstruct the missing parts or damaged region in an image(frame) utilizing spatial information of neighboring region. Video Inpainting refers to a field of Computer Vision that aims to remove objects or restore missing regions present in a motion films by utilizing spatial and temporal information from neighboring scenes. The main objective of this technique is to generate an inpainted area that is merged seamlessly into the video so that visual coherence is maintained throughout and no distortion in the affected area is observable to the human eye when the video is played as a sequence.

   There are number of algorithm are present for inpainting but we are going to use exemplar based inpainting algorithm. The exemplar based approach is an important class of inpainting algorithms. And they have proved to be very effective. Basically it consists of two basic steps: in the first step priority assignment is done and the second step consists of the selection of the best matching patch. The exemplar based approach samples the best matching patches from the known region, whose similarity is measured by certain metrics, and pastes into the target patches in the missing region. Exemplar- based Inpainting iteratively synthesizes the unknown region i. e. target region, by the most similar patch in the source region. According to the filling order, the method fills structures in the missing regions using spatial information of neighboring regions. This method is an efficient approach for reconstructing large target regions. Video films are susceptible to being easily scratched. Typically, a video film goes through several types of degradations. That are caused by time and the frequent use or by accidents or other unexpected events occurred during production. Scratch detection and removal are important subjects for scratch restoration. Line scratches are main problem in motion films. Such problems may produce very annoying effect on the viewer.

   So restoration of such scratches is area of research now a day. Our contribution to this project mainly concerns scratch restoration. For video films a two step method is used: scratch detection followed by the scratch removal using robust exemplar based inpainting algorithm. This paper is organized as follows. The rest of the paper is structured as follows: Section 2 gives brief review of different techniques available for scratch removal from the video as well as for the exemplar based algorithm. Section

   3 describes system architecture. Section 4 includes the results and discussion in which we have considered different parameters to check validity of proposed system. Finally, we draw conclusion in Section 5.

2. RELATED WORK

   Scratches are one of the defects in video films. Scratches are characterized as dark or bright vertical lines spared over several frames. A large body of research has been devoted for scratch removal for video restoration. One major problem in these methods are that number of methods are not able to reproduce the original grain of the movie, when removal of scratches. And they results in non- natural looking restored frames

   1. Nature of line scratches

      Line scratches in motion films are lines of noticeably darker or lighter intensity than the surrounding pixels. These scratches are exactly or close to vertical. They can be visible on all or much of the height of the image, in which two cases they are named either:

      • Principal scratches: Scratches which are visible along more than 90 % of the height of the image.

      • Secondary scratches: all scratches which do not cover at least 90 % of the height of the image (non-primary scratches).

      Scratches are defined in the literature as persistent (meaning they last more than one frame), or non-impulsive. They are also described (empirically) in papers by Bruni [1, 2, 3] as having a width of 3-10 pixels. This is not a definite criterion, however, and Kokaram [4] considers the width as a variable to be estimated with a probability density function. Scratches are also supposed in several papers [1, 2, 4, 5, 6] to have side-lobes on (lines on the left and right of the central part of the scratch which are less extreme).

   2. Causes of line scratches

      It is important to know what the physical causes of line scratches are. Such knowledge helps to understand why these scratches have certain characteristics and is the basis for any line scratch model. Line scratches are caused by abrasions against the film. This can happen in several cases:

      • A particle or a mechanical part rubbing against the film during projection or transportation.

      • A particlecaught between the film roll itself, if the film is rewound carelessly.

   The second point is not considered in the literature, but was seen on the National Film and Sound Archive [7]. Certain models of line scratches [3, 4] use a very precise knowledge of line scratches effects in order to detect them. For instance, the side-lobes on the line scratches are

   explained and modeled by a light diffraction effect. Three groups of authors are notable for having contributed to this area. The first is A. Kokaram, then V. Bruni and L. Joyeux. These authors and their co-authors have introduced the major techniques and models used for line scratch detection. The other authors introduce different ideas, but, as we shall see, these are often so vaguely explained that in practice it is impossible to verify their performance.

   Detection and Removal of Line Scratches in Degraded Motion Picture Sequences in this paper Kokaram [4] used Combination of deterministic (Hough transform) and statistical (Bayesian refinement) tools. It was the First paper on line scratches detection. Despite interesting results the algorithm by design, not able to deal with single isophotes. Also Statistical (Gibbs sampling) step maybe long.

   Line Scratch Detection on Digital Image: An Energy Based Model in this paper D. Vitulano, V. Bruni, P. Ciarlini [5] introduces Webers law to detection: a law to determine visibility of an object. More scratches are detected in this technique (Better precision). And lesser computation cost, because sampling stage avoided. But this technique is not able to detect scratches which are close together.

   Scratch Detection via Underdamped Harmonic Motion in this paper V. Bruni, D. Vitulano [2] this technique model the solution of a second-order differential equation. The advantage of this paper is more scratches detected (not- alone scratches in particular). But in this technique false alarm rejection problem not addressed.

   Line Scratches Detection and Restoration via Light Diffraction in this paper V. Bruni, D. Vitulano, A. Kokaram [3] this technique is based on light diffraction during digitalisation of film (better grounding of cross- section signal shape than in previous papers). But the problem in this paper is white scratches do not follow the model very closely. Also Detection rate worse than previous paper (in the results shown in the paper).

   A Generalised Model for Scratch Detection in this paper V. Bruni and D. Vitulano [1] in this paper Adaptive threshold for detection (with respect to brightness) introduced. But the problem related with this paper is that detection rate worse than previous TWO papers (Comparison with Kokaram only).

   Detection and Removal of Line Scratches in Motion Picture Films in this paper L. Joyeux, O. Buisson, B. Besserer S. Boukir [22] in this technique interesting temporal element introduced. The temporal aspect is very (intuitively) important for distinguishing line scratches from scene features. But the problems not mentioned in the paper are that Kalman filter depends on sinusoidal movement of scratches in time. This hypothesis is far from obvious.

   The above comments and analyses concerned the three main research groups who have considered the line scratch detection problem. Jin Xu et al. [8] use an over-complete Wavelet expansion to detect the line scratches. They compute horizontal wavelet decomposition on the image, and then sum across the y-axis onto the x-axis. Unfortunately, they do not explain why the wavelet expansion is useful in this case. The authors refer to an

   article on line scratch restoration by Bruni et al. [6] which uses wavelet transforms. In the latter article, the wavelet transform is used to target certain frequencies (scales) which supposedly correspond to the line scratch defect, and manipulate it in order to eliminate this effect. This is not the case in the detection presented by Xu et al. who base their detection on a thresholding of the vertical sum of the first scale wavelet transform of the image. This is coherent in that, as in [6], the first scale sub-band is supposed to represent the high frequency elements, but does not justify the use of the wavelet transform, because this could have (and has) been done in the spatial domain. Furthermore, the rejection stage is simply a question of limiting the width of the scratches, which is far from sufficient.

3. PROPOSED SYSTEM ARCHITECURE

   In this paper we mainly concern with the scratch restoration from the motion films. This scratch restoration is two step process, this includes scratch detection followed by the scratch removal technique. The input to our system is a video file (Damaged video file) in AVI format. And after processing our system will generate A video file without errors. All the scratches are restored from the video file. Fig 1. Shows the block diagram of the proposed method. As we are going to work on the damaged video files, so first we have provided the input as a video file (file should be in AVI format). Then as we know that we cannot directly work on the video files. So first we have to extract the frames from the video. For that we are going to apply frame extraction algorithm. In this step numbers of frames are extracted from the video. Then damaged frames are selected automatically.

   Figure 1. Block diagram of proposed system

   Then on that number of frames scratch detection algorithm is applied. This will give you masked image as the output showing the scratch location. Then in next step for removal of scratch we have to apply the exemplar based inpainting algorithm for selecting best patch from the source region, to collect the information of missing region. This step will remove all the scratches the frames. And after repairing the frame we have to combine that frame with the original video. So again we have to rebuild that video. And finally

   we are going to provide the repaired video as output, ant this video will be similar to the original video without scratch.

   As we have seen our proposed system is divided into 5 basic steps.

   1. Frame Extraction from video.

   2. Scratch detection (Masking).

   3. Apply exemplar based image inpainting.

   4. Scratch removal.

   5. Rebuild video.

      So let us see all these steps one by one. First step of this proposed algorithm is frame extraction from the video. As we know that we cannot apply this algorithm on a video we have to divide the video into number of frames. So in our first step of project we are going to extract the frames from the video. For this we have to select any AVI file as input and our system will extract number of frames from the file. Our aim is not only to extract the frames from the video but also to increase the speed of frame extraction. Following figures shows our experimental results for frame extraction:-

      Figure 2. Result of frame extraction.

      Scratch detection (Masking). In the next step automatically the damaged frames are get selected. Then we have to apply scratch detection algorithm. The segmentation algorithm is used to produce an initial segmentation map. The segmentation map in the exemplar based inpainting method that has one function. That it is an indicator of T (i.e. target region). Means we have to find target region using segmentation technique. Segmentation maps are labeled with gray scale value [0, 255]. We update labels of an initial segmentation map to classify the target region. Here we set the label of user-defined target region to 255 (white color).figure a and c are the input image and final masked image is shown in figure b,d.

      1. (b)

   (c) (d)

   Figure 3. Masking result

   In next step we have to apply the exemplar based inpainting for selecting the best patch. Generally, an exemplar-based Inpainting algorithm includes the following four main steps:

   1. Initializing the Target Region

   2. Computing Filling Priorities.

   3. Searching Example and Compositing.

   4. Updating Image Information.

   Figure 4. Scratch removal results

   Using this for steps we are going to select the best patch. Then after that we have to apply scratch removal technique to remove the scratch. And finally we have to rebuild the video.

4. RESULT AND DISCUSSION

   The results from our implementation are promising. The evaluation of the restoration quality is a complex problem. The existing measures like MSE (Mean Squared Error) and PSNR (Peak Signal-to-Noise Ratio) are easy to apply. The calculation of the PSNR index includes the original and restored image. The PSNR computes the peak signal-to-noise ratio, in decibels, between two images. This ratio is often used as a quality measurement between the original and a compressed image. The higher the PSNR means the better the quality of the compressed or reconstructed image.

   Figure 5. PSNR value of 1st image is 21.81 and 2nd one is 39.26.

   In above two examples have calculated different values like PSNR values for two images before inpainting and after inpainting. In example 1figure 4 shows the input image having scratches image after inpainting. In example 2 figure 5 shows the input image having scratches image after inpainting.

   Figure 5.PSNR value of 1st image is 22.57 and 2nd one is 23.47.

   As we know the higher the PSNR value means better quality of compressed image. So our results show that our system proposes the better results. Means quality of our image is maintained.

5. CONCLUSION

In this paper a robust exemplar based inpainting algorithm has been applied to correct the scratches in the damaged movie. Image inpainting is recently very important research area in the field of image processing. Still many works on images and videos can be done. Our scratch removal algorithm is an efficient technique. And also it going to produce plausible results in scratch removal of video. This proposed algorithm will not only obtain more accurate

detection of scratches, but also gives better video quality. The proposed iterative reconstruction method shows a good performance in the reconstruction of the image parts lost by the scratching. The result does not any blurring which frequently occurs for other techniques. This technique will provide good results when the scene or the scratch are moving and when the scratched objects in the frame are unscratched in at least 1 neighbor frame.

So future study will include one case i.e. when the scene and the scratches are motionless within the entire shot. As we are applying this algorithm only on un- compressed AVI file so in future we can apply this technique for HD videos also.

REFERENCES

1. Bruni, Vitulano, A Generalised Model for Scratch Detection, IEEE Trans Image Process. 2004, Vol. 13, pp. 44-50.

2. Bruni, Vitulano, Scratch Detection via Underdamped Harmonic Motion, ICPR 2002, pp. 887-890.

3. Vitulano, Kokaram, Bruni, Line Scratches Detection and Restoration via Light Diffraction, , IPSA 2003, pp. 5-10.

4. Kokaram, Detection and Removal of Line Scratches in Degraded Motion Picture Sequences., 1996, Signal Processing VIII, Vol. I, pp. 5-8.

5. Bruni, Ciarlini, Vitulano, Line Scratch Detection on Digital Images : An Energy Based Model, 2002, WSCG 2002.

6. Bruni, Vitulano, Fast Removal of Line Scratches in Old Movies., ICPR 2004.

7. National Film and Sound Archive – Damage to Films.., National Film and Sound Archive. [Online] http://www.nfsa.gov.au/preservation/film_handbook/physic

8. Wang, Li Xu, An OWE-based Algorithm for Line Scratches Restoration in Old Movies., ICAS 2007.

9. Kim, Film Line scratch Detection using Neural Network and Morphological Filter., IEEE conference on Cybernetics and Intelligent Systems, 2008.

10. Wang, Mirmehdi, HMM based Archive Film Defect Detection with Spatial and Temporal Constraints.,Proceedings of the 20th British Machine Vision Conference,2009.

11. Shai, Ariel Shamir, Seam Carving for Content Aware Image Resizing., Siggraph 2007.

12. Tschumperle, Brun, Non-Local Image Smoothing by Applying Anisotropic Diffusion PDE in the Space of Patches., Proceedings of the 16th IEEE international conference on Image processing, 2009.

13. Tang, Hwang Shih, Exemplar-Based Video Inpainting Without GhostShadow Artifacts by Maintaining Temporal Continuity., 2009, IEEE Trans on Circuits and Systems for Video Technology 2009, Vol. 19, pp.347-360.

14. Shechtman, Irani Wexler, Space-Time Video Completion., proceedings of the 2004 IEEE.

15. Computer Society Conference on Computer Vision and Pattern Recognition.

16. Sapiro, Bertalmio Patwardhan, Video Inpainting Under Constrained Camera Motion., IEEE Image Process. 2007, pp. 545-553.

17. Patwardhan, Sapiro, Bertalmio, Video Inpainting of Occluding and Occluded Objects., Image Processing, 2005. ICIP 2005., pp. 69-72.

18. Moran, Video Inpainting, 2009.

19. Masnou, Level Lines Based Dissoclusion., ICIP 1998.

20. Mallat, Characterisation of Signals from Multiscale Edges., IEEE Trans on Pattern Analysis and Machine Intelligence 1992, Vol. 14, pp. 710-732.

21. Kokaram, Removal of Line Artefacts for Digital Dissemination of Archived Film and Video.,ICMCS 1999.

22. Kokaram, On Missing Data Treatment for Degraded Video and Film Archives: A Survey and a New Bayesian Approach.,IEEE Trans Image Process. 2004, Vol. 13, pp. 397-415.

23. Joyeux Besserer Boukir, Detection and Removal of Line Scratches in Motion Picture Films.,IEEE Int. Conf. on Computer Vision and Pattern Recognition.

24. Peleg, Motion Analysis for Image Enhancement :Resolution Occlusion and Transparency.,VCIR 1993.

25. Grossauer, Harald., Inpainting of Movies Using Optical Flow., Springer Berlin Heidelberg. 2005, Vol. Mathematical Models for Registration and Applications to Medical Imaging, pp. 151-162.

26. Gangal, Dizdaroglu B., Automatic restoration of old motion picture using spatiotemporal exemplar- based inpainting.,2006, Proc. ACIVS., pp. 55-66.

27. Elad, Feuer., Superresolution Restoration of an Image Sequence: Adaptive Filtering Approach.,IEEE Trans Image Process. 1999, Vol. 8, pp. 387-395.

28. Duda, Hart, Use of the Hough Transformation To Detect Lines and Curves in Pictures,Communications of the ACM, pp. 11-15.

29. Doser, Weitbruch Thebault, Scratch Detection Supported by Coherency Analysis of Motion Vector Fields,2009, Proceedings of the 16th IEEE international conference on Image processing, pp. 89-92.

30. Cohen-Or, Yeshurun Drori, Fragment-based Image Completion,Siggraph 2003, Vol. 22.

31. Bugeau, Bertalmio, Combining Texture Synthesis and Diffusion for Image Inpainting,VISAPP 2009.

32. M. Bertalmio, A. L. Bertozzi, and G. Sapiro,Navier-Stokes, fluid dynamics, and image and video inpainting,in Proc. IEEE Conf. Computer Vision and Pattern Recognition, vol. 1, pp. 355a362, Kauai, HI, Dec. 2001.