Survey on Content-Aware Image Retargeting Methods

Survey on Content-Aware Image Retargeting Methods

Madhura. S. Bhujbal , PG Student

Department of E&TC

STESS, Smt. Kashibai Navale College of Engineering Pune, India

Mrs. Pallavi. S. Deshpande, Professor

Department of E&TC

STESS, Smt. Kashibai Navale College of Engineering Pune, India

Abstract Advances in imaging technology has dominated digital images over the previous analog photos, as their advantage is that these images can be displayed on variety of devices ranging from high-resolution TV or PC monitor to low resolution PDAs or cell phones. For displaying digital images on various devices, need to change the aspect ratio and size of the image, keeping the contents and visual saliency of the image intact. This cropped up the need for image retargeting. Various methods are used for resizing the image to a predefined aspect ratio. This paper throws light on some of the content-aware Image Retargeting methods.

Keywords Image retargeting; importance map; target image; content-aware image retargeting.

1. INTRODUCTION

   Diversity of display devices and availability of images is widely increasing. People widely use PDAs (Personal Device Assistants) and mobile phones for transmitting, sharing and exchanging information. The amount of images and videos available from internet has significantly increased since the past decade. Also, the continuous development of new display devices like mobile phones, notebooks, tablets induces a constantly growing consumption of media content. This has led to requirement in convenient and effective data access using portable devices. Also, new demands on digital media have increased due to diversity and versatility of display devices. HTML and other standards dynamically support the changes in page layout and text. Hence effective utilization of display space is becoming crucially important. The existing digital images, as those taken by digital cameras, have high resolution and fixed aspect ratio. Hence, digital images should be adapted for effective viewing on a variable screen and arbitrary aspect ratio. This adaptation of image to catch up with the necessity of varying resolutions and aspect ratios is called as image retargeting. Image retargeting methods aim at shrinking the spatial size of an image while maintaining important parts, so that screen utilization is maximized.

   Effective image retargeting is achieved through many methods. The basic idea of image retargeting algorithms is to adapt the image content to the screen without distorting the important objects in the scene. Various algorithms differ in how they determine the importance of different pixels in the image and how they use this information. In this paper, we review and categorize algorithms for Content-aware image retargeting, is resizing

   an image while taking its content into consideration to preserve important regions and minimize distortions. This requires preserving the relevant information while maintaining an aesthetically pleasing image for the user, and hence is a challenging problem. Image retargeting techniques typically start by computation of an importance map that represents the relevance of every pixel, and then apply an operator that resizes the image while taking into account the importance map and additional constraints. In this paper, we review and categorize algorithms for Content-aware image retargeting[1]-[3]. Fig. 1 shows the general flow that most retargeting methods follow for content-aware image retargeting.

   Input Image

   Compute importance map

   Retargetin g operation

   Output image

   Target aspect ratio

   Fig. 1: Image retargeting system

   The rest of the paper is organized as follows: In next Section II, different techniques used for image retargeting are discussed. Section III describes the various methods to be considered for tone adjustment in the captured video. Finally, Section IV concludes the paper.

2. IMAGE RETARGETING METHODS

   The major challenge of image retargeting is to take into account the contents of the image by resizing attempt to preserve important regions, and also maintaining a visually pleasing image. Some image retargeting methods are as follows:

   • CONTENT-AWARE CROPPING

   1. System Overview

      Cropping can be defined as manually choosing a window of the target size from the source image to maximize the salient information. Important factor is to find the region of interest (ROI) to determine the optimal sub-image to maintain.

      cropping

      Patch based

      Seam carving

      Image re- targeting methods

      scaling

      Image warping

      • SEAM CARVING

   1. System Overview

      Seam carving technique by Avidan and Shamir[8,9] is a popular Image Retargeting method. The general idea of seam carving is to decrease image width or height one pixel at a time, by removing seam of minimal importance. A seam is 8- connected path of pixels (from top to bottom or from right to left, depending on which dimension is to be reduced) that contains only one pixel per row or per column.[10] Seam carving was invented, not only for effective resizing of images with geometric constraints, but also to consider the image content as well. Seam carving supports content-aware image resizing for both reduction and expansion. It works by finding the lowest-energy connected path of pixels from either left to right (horizontal seam) or top to bottom (vertical seam), removing those pixels, and repeating the process. To

      methods

      Fig. 2: Image retargeting methods

      There are two approaches to find the ROI: a)By determining salient parts for automatic extraction of thumbnails from images. [4] b)Cropping based method with sophisticated feature for determining ROI explained by Chen et al[5].

   2. Advantages and disadvantages

   Cropping is the most reliable and simplest method for image retargeting. However, it loses information outside ROI. Cropping can only retarget specific images with good perceptual quality, and as a result, its performance depends on the source image contents. For images containing with small region and having more salient content, cropping method retargets a good quality image. But, for images where all regions contain meaningful information, it retargets images with bad quality.[6]

   • CONTENT-AWARE SCALING

   1. System Overview

      Scaling is a simple method for retargeting images. Scaling images in horizontal or vertical direction can be performed in real-time using interpolation and will preserve the global visual effects and retarget images with medium perceptual quality.

   2. Advantages and disadvantages

   If the aspect ratio between the input and output is different, scaling will cause large distortions. Image scaling isnt sufficient as it is oblivious to image content and typically can be applied only uniformly. Also, scaling will introduce some shape deformation into the retargeted image.[1,7]

   maintain the rectangular structure of an image, each path of

   pixels include exactly one pixel per column for horizontal seams. In a similar manner, pixels can be added or duplicated along these seams to increase the image size. Selection and order of seams protect the content of the image, as defined by the energy function. A vertical seam is defined as an eight- connected path of pixels with one pixel width from top to bottom of an image. Various methods with dynamic programming approach help decide which seams can be removed. Also, various algorithms allw for resizing in both dimensions by choosing optimum vertical and horizontal seams. Seam carving iteratively removes or inserts a seam passing through unimportant regions.

   1. Advantages and disadvantages

      Seam Carving is a popular approach for content aware image resizing. It supports various visual saliency measures for defining the energy of an image, and can also include user input to guide the process. By storing the order of seams in an image, we create multi-size images, that are able to continuously change in real time to fit a given size. However, seam carving method has some drawbacks as with simple row/column removal, it can excessively carve less important parts of an image and result in unwanted visual distortions. Also, the seam method does not consider any approaches to preserve the object shape. Therefore, alone it can exhibit bad perceptual quality, especially for images containing salient objects.

      • IMAGE WARPING

        1. System Overview

           Image warping is a process of digitally manipulating an image such that, any shape portrayed in the image have been significantly distorted. Warping based methods are also called as continuous methods and they perform nonlinear distortions to obtain a resized image.[1,7] This method offers a better possibility of producing a continuous deformation for content aware retargeting. In this method, the local distortion of important areas is constrained to be as small as possible, while unimportant regions are allowed to distort more. In this way, both important and unimportant areas are preserved in the final image, which can prove to be useful to preserve

           context for relevant objects. But, depending on the amount of distortion, unimportant areas can even disappear, which results in content removal.

        2. Advantages and disadvantages

        This approach has an advantage of distributing distortions to homogeneous regions as it forces quads with significant content uniformly and distorts quads with homogenous contents. It can preserve the aspect ratios of local objects. Limitation of this method is that if there are many quads in the image, image gets totally distorted and fail to preserve the aspect ratio of the whole image. Also, depending on the amount of distortion, unimportant areas can even disappear, which results in content removal.

      • PATCH-BASED METHOD

   1. System Overview

      Methods that come under patch-based methods achieve retargeting through the manipulation of patches.[1] The algorithms use distances between image patches, aiming to minimize a distance measure between the input image and the retargeted image. Then, the patches are rearranged to form the final image. Patch transform by Cho et al [11] is a popular patch-based method where an image is broken down into non-overlapping patches, and modifications and constraints are applied in the patch domain. A modified image is then reconstructed from the patches. In patch based methods, images are broken down into small, non-overlapping patches, and image is manipulated in patch domain. User has the freedom to constrain patch positions, and add or remove patches. This allows explicit control of how much of each texture is in image, and where textures and objects appear. From this modified set of patches, image is reconstructed. Simakov et al defines a bidirectional similarity measure between images, which contain both completeness and coherence measures computed from image patches. Completeness measures if the target image contains all visual features present in the source image. This method can incorporate importance functions to retain features such as faces or remove objects from the image.[12]

      Barne et al proposed a randomized algorithm for quickly finding approximate nearest neighbor matches between image patches[13]. For image retargeting, the authors extend Simakovs approach by including constraints such as line preservation and new locations of objects and lines, which are implemented by constraining the nearest neighbor search.

   2. Advantages and disadvantages

   This approach allows many useful image editing operations. User can select regions of the image and move them to new locations. However, the main limitation of these methods is that the control over the patch location is inherently limited by the size of the patch, which can lead to visible artifacts. If patches are too small, the patch assignment algorithm breaks down due to exponential growth in the state dimensionality. One more limitation of patch-based method is that it needs large amount of computations. Image retargeting is possible using patch methods only if the user specifies both, the

   position of some patches and the size of the target image, fitting content from the original image into the new size.

3. CONCLUSIONS

Image retargeting is emerging as an important necessity with the evolution of new digital devices and displays. There has been a lot study carried out in this area by many researchers, and it still continues with demands of digital world. This paper is a survey of some of the Image Retargeting methods, that are usually used. A brief overview of each method with its advantages and limitations are discussed in this paper.

ACKNOWLEDGMENT

I am indeed thankful to my guide Prof. P. S. Deshpande for her able guidance and assistance to complete this paper; otherwise it would not have been accomplished. I extend my special thanks to Head of Department of Electronics & Telecommunication, Dr. S. K. Shah who extended the preparatory steps of this paper-work. I am also thankful to the head & Principle of STESS, SMT. Kashibai Navale College of Engineering, Dr. A.V. Deshpande for his valued support and faith on me.

REFERENCES

1. Daniel Vaquero, Matthew Turk, Kari Pulli, Marius Tico, Natasha Gelfand, A survey of image retargeting techniques, Applications of Digital Image Processing XXXIII, Proc. SPIE 7798, 779814, 2010.

2. Tongwei Ren, Yan Liu, Gangshan Wu, Image Retargeting based on global energy optimization, IEEE Conf. Multimedia and Expo, (ICME 2009), 2009, pp.406409.

3. Sunghun Cho, Hanul Choi, Yasuyuki Matsushita,Seungyong Lee,

   Image retargeting using importance diffusion, 16th IEEE Intl Conf.

   Image Processing (ICIP 2009), pp. 977-980, 2009

4. Bongwon Suh, Haibin Ling, Benjamin B. Bederson, Dravis W. Jacobs,

   Automatic thumbnail cropping and its effectiveness, in Proc. Of UIST, 2003, pp. 95-104

5. Li-Qun Chen, Xing Xie, Xin Fan, Wei-Ying Ma, Hong-Jiang Zhang, and He-Qin Zhou, A visual attention model for adapting images on small displays, ACM Multimedia Systems Journal, pp. 353-364, 2003.

6. Vidya Setlur, Saeko Takagi, Ramesh Raskar, Michael Gleicher, Bruce Gooch, Automatic Image Retargeting, ICMUM, 2005.

7. Michael Rubinstein, Diego Gutierrez, Olga Sorkine, Ariel Shamir, A comparative study of image retargeting, ACM Trans. Graphics, vol. 29, no. 160, pp.1-7 ,2010.

8. S. Avidan, A Shamir, Seam carving for content-aware image resizing, ACM Trans. Graphics (Proc. Of SIGGRAPH), vol. 26(3), no. 1, pp. 2-7, 2007.

9. M. Rubinstein, Avidan, and Shamir, Improved seam carving for video retargeting, TOG, pp. 1-2, 2008.

10. David Conger, Mrutyunjay Kumar, Rodney Miller, Jiebo Luo, Hayder Radha, Improved Seam carving for Image resizing, IEEE Conf. Signal Processing Systems (SIPS), 2010 pp. 345-349.

11. Taeg Sang Cho, Moshe Butman, Shai Avidan, William Freeman, The patch transform and its applications to image editing, IEEE Conf. Computer Vision and Pattern Recognition (CVPR), 2008, pp. 18.

12. Simakov, D., Caspi, Y., Shechtman, E., Irani, M.., Summarizing visual data using bidirectional similarity, in Proc. of IEEE Conf. on Computer Vision and Pattern Recognitio, 2008.

13. Barnes, C., Shechtman, E., Finkelstein, A., Goldman, D., PatchMatch:A randomized correspondence algorithm for structural image editing, ACM Trans. on Graphics (Proc. Of SIGGRAPH), 28(3), 2009.