Stereo Seam Coupling for Minimizing the Depth by Adjusting the Aspect Ratio in 3D Image Retargeting

Stereo Seam Coupling for Minimizing the Depth by Adjusting the Aspect Ratio in 3D Image Retargeting

Mahendra Jagtap*1, Dineshkumar Jawalkar2

1Department of Computer Science & Engineering, Research Scholar2Department of Computer Science & Engineering,

Assistant Professor Lingayas Vidyapeeth, Faridabad, Haryana 121002

Abstract:- Depth distortion in an image produces some geometric errors which leads certain image quality degradation. Therefore, it is important to enhance the depth information in the left as well as right stereo images and accomplish them in a substantial way. The Disparity Map Acquisition (DMA) algorithm gives rise to the depth distortion with improved disparity matrix.In this paper, we emphasis on depth score enrichment in 3D stereo images retargeting to accomplish the acceptable 3D images with improved depth distortion score. The experimentalresults show the stereo seam carving which deconsideres the unwanted image patches in orderto generate an acceptable 3D stereo images. The obtained 3D stereo images are extensively used in the applications of 3D animated movies by abolishing the blurriness in the stereo images and generate the images where the users can delight with the better visual effects. This may lead to the non-usability of 3D sterilize googles and eventually helps and reduces the burden on Indian economy.

Keywords: Depth distortion, Stereo image, Seam carving

1. INTRODUCTION

   In todays digital technological time, display on handheld digital devices gaining admirationdue to visualize the contents over it without garbling the original contents still preserving the semantic information. This is image retargeting. Image retargeting mainly emphases on the foreground object in order to minimize the low significant part of an image by using seam carving technique. Presently, most of the techniques emphasis on single 2D image which is subsequently obsolete nowadays.

   With repaid development of technology, the techniques need to be extended and widely usedfor 3D stereo image. The 3D stereo images have the different lookouts for the left and right images. The stereoscopic 3D images are generated to accomplish the viewers perception.

   The input to this method is the stereo image pairs with disparity map. The Disparity Map Acquisition (DMA) mapping algorithm is used to diminish the depth distortion. This is achievedby taken into consideration, the seam coupling to subordinate the right image to the left using the disparity map. In order to avoid depth distortion, it deliberates both appearance and depth energy. As it eliminates a pair of pixels which is less significant and may worsens the image performance and not seems to be geometrical reliable. So this technique is used to make the retargeted pair suitable for stereoscopic display.

2. RELATED WORK

   2D seam carving method is extended to 3D stereo image retargeting by periodically confiscating a pair of seams from a stereo image pair. Number of methods were projected in thisdirection. These are mainly based on protracted the warping-based 2D image retargeting methodsstereo image pairs by imposing additional depth-preserving constraints. This is achieved by preserving the whole image by maintaining the depths of a set of sparse correspondences. Secondly, the depth editing methods were proposed to remap the depth which accomplish the target up to some extent. Recently in current scenario, one can take an enormous effort in order to discover and explore some image substances for retargeting.

   While applying the stereo seam on the images, the retargeting of couple images must be considered. The appropriate and easiest approach is to compute a seam of left image and map it to the right image by using the disparity map. As a result, it does not exploit the informationpresent in the right image.

   The proposed method in this paper overcomes the problem of retargeting stereo image pairsby smearing stereo seam carving and disparity map. Carving the pair of seams in both the imageswith minimizing image distortion and ultimately the depth distortion. The produced image givesthe pleasant appearance and more visibility in an image. The Figure 1 shows image retargeting on a single image and Figure 2 shows stereo retargeting approach by applying stereo retargetingalgorithm on left and right stereo images.

   Figure 1. Image Retargeting on a Single Input Image

   The real encounter for seam carving lies in simultaneously carve a pair of seams in both images, while minimizing distortion in appearance and depth. Proper selection of seams and energy pixels are subject to geometric construction. The geometric construction focuses on thevisibility relations between the adjoining pixels [1].

3. The Method

   The input to the method is a pair of m x n stereo images {IL, IR} and a disparity map M, we consider the disparity with respect to either left or right image as a reference image. The pixel values of left image are computed and then its corresponding pixels are mapped with right image. The outcome of this method will be obtained as {IL, IR}. The final goal is to achieve and acquired retargeted images that are visually contented and geometrically consistent 3D scene [1]

   1. Seam Coupling

      Seam coupling is used to match each seam in the left image to that of right. The geometricalcoupling of two seams are which is obtained by using Disparity Map M as :

      are the seam pixels in the left and right images at row i. This method contemplates the pieceof the images for matching.

   2. Energy Function

      In this paper, the energy function being used may considers both appearance energy as well as depth energy. The obtained retargeted left and right images rest on the seam pixels in previous row with respect to the left image. The total energy is obtained by considering the appearance energy along with pixels intensity level.

      The appearance energy is the total of left and right images. Here, Eintensity is the appearance energy and E3D is the depth energy. It controls each and every term in the given equation.

      Figure 2. Stereo Retargeting approach on left and right Images

      seam coupling is then managed with the alignment that the corresponding pixels are either bothremoved or remains corresponding in the output images.

   3. Disparity Map

   The Disparity Map M is updated after carving each seam for the next development until theimage pairs match as per the expectations. The initial disparity map will generate by using Constant Space Belief Propagation (CSBP) technique. This method is popular due to its less time and space complexity.

   The energy function calculation is significant here as high energy pixels are taken into consideration because it focused on image foreground object. The disparity map acquisition algorithm is superfluous to calculate the disparity map on the pair of stereo images. Finally, the depth distortion score will be released. The steps for figuring the disparity map are as follows:

   Initially, the input as 3D image pair will be provided there. Then applying the Constant space belief propagation to find out the energy pixels. Assign the disparity values to calculate the score.

4. Proposed Method

   The methods we are proposed here is highly stimulating as number of scenes in this method are considered with the great depth. The dataset considered here is 3D Middlebury and flicker dataset which consists of variety of datasets to be tested as:

   1. Flickr: A set of stereo images with large depth are used from flicker.

   2. Portrait: A pair of images with a great challenge as the salient objects need not be distorted and must be covered.

   In the experimental set up, we use the method to retarget the pair of stereo image pair. Then we use the disparity map on the retargeted image pair. Without loss of originality of the image, the gained image should have the similar disparity map as that of original image. Here we take the left image as a reference image and use the scaling factor to retarget the disparity map M. Finally, the efficient and minimum depth distortion score is produced [2]

   Figure 3. Dataset of Original Images

   Figure 4. The 3D Stereo Left and Right Car Images Dataset

   Figure 5. The 3D Stereo Left and Right Person Images Dataset

   Figure 6. The 3D Stereo Left and Right Snowman Images Dataset

   Figure 7. The 3D Stereo Left and Right People Images Dataset

   Before the 3D image retargeting methods came into existence, the old-style cropping and scaling methods were extensively used. We used these methods manually to set up and adjust theimage without losing its originality.

   We used here the stereoscopic image repository on flicker that comprises the variety of imagessuch as car, person, animals, people, snowman etc. (seeing in Figure 4, Figure 5, Figure 6 and Figure 7) while processing these images, the real challenge lies in conserving the originality of the image as well as preserving the disparity of the image.

   4.1 Depth Similarity

   The objects under considerations are having a relative depth because of the different depth values that the images have. It is pragmatic that the images with a background is always have a greater depth. Depth eventually helps the viewer to understand the image well. Most of the imageretargeting methods are concentrating on depth values rather than any other image parameters.

   As the image is having different depth, the comprehensive image is divided into different layers. The 2D and 3D image can be decomposed into layers according to the depth information.The basic challenge for stereoscopic image retargeting technique is to preserve the relative depthor layers of image fillings. Hence, the depth resemblance concept is introduced to measure the depth

   preservation before and after the retargeting. It helps to find the corresponding position inthe images when mapping is to be performed. The positions may be treated as a key point by comparing the depth values.

   Sometime the depth similarity is not enough for finding the relative depth, the absolute depthvalues need to be considered as well. The absolute disparity can be attained by using some retargeting techniques such as uniform stretching, image warping etc. so that operative 3D imagequality can obtain.

5. EXPERIMENTS AND RESULTS

   Since we have obtained the retargeted stereoscopic images by considering a categorized image from the image dataset.

   Figure 8. Depth Distortion Score in Snowman Image from Flicker

   Figure 9. Depth Distortion Score in People Image from Flicker

   Table 1. Depth Distortion score in the stereo images

   Dataset	ExistingMethod	ProposedMethod
   Car	0.65	0.58
   Man	0.92	0.52
   People	0.55	0.42
   Snowman	1.12	0.80

   The observation table 1. shows the 3D image dataset from flicker. The flicker images of Car, Man, People and Snowman etc. The existing method and the proposed method show the depth dissimilarities. Existing method shows the depth distortion with greater depth which can distortthe image whereas our proposed method shows the depth accuracy which enhances the image with better accuracy. So, if the image depth is less, more accurate and enjoyable image is gained.

   Figure 10. Performance comparison between depth distortion of stereoimages.

   Here we have obtained the retargeted stereoscopic images in order to gratify our proposed method.

   In experimental results of Figure 8 and Figure 9, The disparity map of left and right stereoimage yields the depth distortion score by applying seam carving technique. Then apply on theleft seam and right seam carving to generate the modified depth distortion score.

   From the experimental results, we perceive that our method performs significantly to obtainthe depth score. The traditional method shows a scrawny attachment between the stereo 3D images. On the other hand, our proposed method outstrips with better predicted result by applying the disparity map accurately. The Figure 10 shows the comparison chart of performance between stereo images depth distortion in terms of its depth distortion percentage.

   Our method well conserves the semantic meaning of the original image by considering its foreground object, action context and the background images by smearing the seam carving carefully and yields the visually contented resize images. The estimates become true by its variabilities features comparisons of the existing pair of stereo images.

6. CONCLUSION

In this paper, we have presented a 3D image retargeting for a pair of stereo images to yields the better performing images with minimal depth distortion. As less as the depth distortion scorein 3D images, more and more accuracy is obtained. That means depth distortion in the image isdirectly proportional to its accuracy. The disparity map would provide the better seam couplingand helps to provide the betterment of the obtained image. The layered architecture provided by the depth similarity is highly achievable despite of the challenge of preserving the layered depth.

Experimental results proved that the greater accuracy in the depth score will always be enjoyable and remarkable for obtaining the better image suitable for all kind of handheld devicesas well as big screen images by maintain the proper aspect ratio of the image. Although the method gives the better performance and better result, there are always a scopefor the improvement. Considering the better depth 3D stereo image databases and testing on more and more image will definitely progresses the depth distortion score up to some extent.

ACKNOWLEDGMENTS

I acknowledge earnest thanks to my guide Dr. Dineshkumar Jawalkar for keen guidance andsuggestions in the context. My sincere thanks to the research coordinator, Doctoral programmeDr. Manoj Kumar, Head of CSE Prof. Kiran Kumar for kind support. Special thanks to my former guide Dr. R. C.Tripathi for support and motivate. Finally gratitude to all the faculties and collogues of CSE department.

REFERENCES

1. T. D. Basha, Y. Moses and S. Avidan, Stereo Seam Carving a Geometrically Consistent Approach, IEEETPAMI, 35(10), (2013), pp. 25132525.

2. J. J. Lei, M. Wu, C. Q. Zhang, F. Wu, N. Ling and C.P. Hou, Depth-Preserving Stereo Image Retargeting Basedon Pixel Fusion, IEEE Transaction on Multimedia, 19(7), (2017), pp.1442-1453.

3. Y. Liu, L. Sun and S. Yang, A Retargeting Method for Stereoscopic 3D Video, Computational VisualMedia,1(2), (2015), pp.119127.

4. Che-Han Chang, Chia-Kai Liang and Yung-Yu Chuang, Content-aware Display Adaptation and Interactive Editing for Stereoscopic Images, IEEE Transactions on Multimedia Vol, 13, no. 4, (2011), pp. 589601.

5. Bo Yan, Ke Li, Xiaochu Yang and Tingxiao Hu, Seam Searching Based Pixel Fusion for Image Retargeting, IEEE Transactions on Circuits and Systems for Video Technology, (2014).

6. Y. Fang, K. Zeng, Z. Wang, W. Lin, Z. Fang and C. W. Lin, Objective Quality Assessment forImage RetargetingBased on Structural Similarity, IEEE JETCAS, 4(1), (2014), pp.95105.

7. Y. Jung, H. Sohn, S. Lee and Y. Ro, Visual Comfort Improvement in Stereoscopic 3D Displays Using Perceptually Plausible Assessment Metric of Visual Comfort, Consumer Electronics, IEEE Transactions on, 60(1), (2014), pp.19.

8. J. Wang and S. Wang, Perceptual Depth Quality in Distorted Stereoscopic Images, IEEE Transactions on ImageProcessing, 26(3), (2017), pp.1202-1215.

9. L. Ma, L. Xu, Y. Zhang, Y. Yan and K. N. Ngan, No-reference Retargeted Image Quality Assessment Based onPairwise Rank Learning, IEEE Transactions on Multimedia, 18(11), (2016), pp.22282237.

10. Feng Shao, Libo Shen and Jiang Qiuping, User Controllable Content Retargeting and Depth Adaptation for Stereoscopic Display, IEEE Access PP(99):1- 1, (2019) January.

11. Dandan Li, Heilongjiang, China Jun Cao and Yi Liu, Stereoscopic 3D Image Retargeting Quality Assessment,Acta Microscopica Vol. 27, No.4, (2018),

pp.279-291.