Survey On 3D Interactive Walkthrough

Survey On 3D Interactive Walkthrough

Nikul P.Patel.

ME student, CSE Department, Government Engineering College, Sector28,

Gandhinagar, Gujarat, India

Abstract

Interactive walkthrough of real environment have been a very important area of research for computer vision and graphics field. It gives user a sense of being there virtually. Most 3D walkthrough system use the computer generated images (CGI) and 3d objects. But to create the walkthrough that simulate the real environment is very complex task. It involves 3D modelling and image based rendering (IBR) techniques to simulate real environment and navigate through it. Throughout last decade various approaches have been developed for interactive walkthrough. Some approaches use the Omni directional images, some use panoramic images where as others have developed video based approach to generate the

interactive walkthrough. This paper has evaluated

virtual 3D walkthrough we have to explicitly define geometric architecture of scene rather than extract it automatically from scene. However extract detailed geometry of photographed scene is still an active research area in computer vision. This paper presents some approaches and techniques on image/video based 3D interactive walkthrough. Figure-1 classifies interactive walkthrough system based on various approaches and techniques that have been developed.

3D Interactive Walkthrough

various approach of interactive walkthrough

Keywords: Interactive walkthrough, image based rendering, 3D reconstruction and visualization, virtual tour.

Image Based

Approach

Video Based Approach

Interactive walkthrough is about roaming in a real environment virtually. Consider a real environment in which you physically roaming. At that time continuous set of images are rendering on your retina and your brain construct 3D model out of that 2D images. That gives you sense of being in the 3D environment. The Interactive walkthrough is about mimic this process. It allows user to navigate real environments virtually in 3D. Many computer vision techniques have been developed to construct the 3D model of scene out of 2D images. Interactive walkthrough has many applications ranging from 3D games to virtual touring. Apart from real image based walkthrough, it is possible to develop walkthrough system that use computer generated 3D model and rendering techniques. It is comparatively less complex to develop virtual walkthrough than image based walkthrough. In

Image Based Rendering

Figure 1. Classification of interactive walkthrough system

Both approaches shown in figure-1 for interactive walkthrough use image based rendering (IBR) methods to simulate the real environment. Image based rendering is about synthesize a novel view from a set of reference images. After extracting the geometric information of scene from set of images that is using image based modelling techniques [1, 2, 3] to create 3D model of scene, suitable image based rendering method is used. This will give user better experience of roaming in real environment virtually. The rest of the paper explain both image based and video based approaches in detail and also explain some image based rendering(IBR) techniques to build real environment virtually.

.2. Image Based Approach

Image Based Approach

Walkthrough using image based modeling(IBM)

Panaromic image based walkthrough.

Walkthrough using omni-directional

images.

Figure 2. Classification of image based approach.

Image based approach is further divided into three techniques to generate interactive walkthrough. This classification is based on types of input images used to simulate real environment virtually. In image based modelling 3d model of scene is build using sparse set of input images. Photo tourism [4] is based on image based modelling technique. Build simple 3d model of indoor environment [2] and architectural scene [3] are also used frequently for generating walkthrough. Creating a walkthrough environment using panoramic images [1, 5] is also well known method. Some approach exploit the Omni directional images [7, 8] taken from Omni directional camera. Each of this method is explained in this paper.

.2.1. Walkthrough using Image Based Modelling (IBM). [4, 2, 3]

Image based modelling method use correspondences between set of images to recover 3d points of object. It will extract geometry of scene from set of images. The photo tourism project[4] use image based modelling which is based on the structure from motion(SFM) ,which aims to recover camera parameters, pose estimate and sparse 3D scene geometry from set of images[1].Image based modelling is used to create large scale architectural models. Modeling of indoor environment require considerable effort[2] to generate simple floor plan like 3d model. Walkthrough applications use image based modelling method to extract geometry of scene and use it as geometric proxy for image based rendering[2] to create photo realistic visualization of environment. Interactive 3d modelling for architectural scene[3] is useful for generating architectural walkthrough. Walkthrough of indoor environment can use simple 3d model like floor plan which is generated using structure from

motion, multi view stereo, Manhattan world stereo for axis aligned geometry and novel depth map integration algorithm[2]. This model is then used as a geometric proxy for scene visualization. However image based modelling method cannot recover detailed geometric information of scene. It is current area of research to extract complete geometry of scene. Image based modelling for walkthrough application is time consuming than other approaches. It takes much time for large scale environment to capture the photos of scene and extracting geometric details of it.

1. Panoramic Image Based Walkthrough. [1, 5]

   Recent advancement in image based rendering techniques allow us interactively viewing panoramic images. Tools like QuickTime VR are available to navigate panoramic image based virtual environment[6]. 3D modelling techniques that are mention above have a correspondence matching problem and segmentation problem. So they are applicable to single continuous object. They are not effective to outdoor locations with multiple individual object[1]. Panoramic images usually contain outdoor environments with multiple individual object. Like 3D modelling it is possible to recover the geometry of scene from set of panoramic images[1]. It is designed for complex outdoor environments which contain multiple individual object and non rigid region. Angular range concept is useful to recover geometric proxy [1, 5] gives epipolar geometry to find correspondence between set of panoramic nodes. The system use triangulation method to generate triangulation mesh for the panorama based on given patches. Thus we can interactively navigate panoramic based environment which consist of multiple individual objects.

2. Walkthrough using Omni directional Images [7, 8]

Omni directional images are taken by Omni directional camera which has 360 degree field of view on horizontal plane. This type of camera is frequently used in robotics or panoramic photography where large field of view is needed. Walkthrough of large scale environment often require large no of images to cover whole environment. At that time we can use Omni directional images to cover whole environment. Various approaches have been developed that use Omni directional images to generate the interactive walkthrough.Sea of Images [7] is one such exampe. It use motorized cart to capture Omni directional images at eye height plane of whole environment. This representation provides 4D

Object tracking

Feature tracking

approximation to plenoptic function which is parameterized by camera position(x, y) and incoming ray direction (Ø , ) [7]. It compresses the acquired images in multi resolution hierarchy so that it can be accessed effectively during interactive walkthrough. This approach replaces complex computer vision techniques of 3D reconstruction with simple image compression and motorize cart navigation [7].

Compressi

on

Figure 3. Sea of Images[7].

Data capture Demosaicing Undistortion De-vignetting

Object pose estimation

It is possible to parameterize 4D plenoptic function which is suitable for interactive walkthrough application [8]. Thus for large scale indoor environments where conventional 3D reconstruction techniques [2] are time consuming

,this Omni directional image based approach which parameterize 4D plenoptic function is convenient.

Camera pose estimation

Tone mappin g

Image stitching

HDR

stitching

Data for interacti ve object replace ment

Data for stabilization

Video

1. The image based approaches explained above use still photography and 3D scene modelling method to generate interactive walkthrough.[9] present video based approach in which we directly filming the whole environment that we wish to explore using high resolution camera. Later on using image based rendering techniques we can interactively navigate environment. Figure-4 show simple block diagram of video processing and rendering pipeline which is used for video based interactive walkthrough [9].For detail explanation of process you can refer to [9].This video based interactive walkthrough approach doesnt require to create the 3D model of environment but using image based rendering techniques it render the environment interactively.

2. Figure 1. Block diagram of video processing and rendering pipeline [9].

   Image based rendering is about generating the novel view from set of reference images. It consist of warping the pixel from nearby images to novel view [10] present some image based rendering techniques.IBR generate the photorealistic output from set of input images. Before image based rendering(IBR) traditional 3D geometry based rendering are used in computer graphics. In image based rendering when the depth of every image is available, we can render the image from any nearby viewpoint by projecting the pixel to its proper 3d location and reproject to the synthesized image view[10]. Some image based rendering systems do not require geometry model. For example image interpolation is used to generate novel view of image which establishes feature correspondence between set of images [11]. All the approaches mentioned above, image based approach and video based approach use the image based rendering

   methods to generate interactive photorealistic walkthrough. [10] gives various image based rendering methods on the basis of geometry. It gives the methods that do not require geometry like image interpolation [11]. Thus to simulate the real environment in interactive walkthrough image based rendering is essential method.

3. Interactive walkthrough of real environment is an area of research for both computer vision and computer graphics. This paper presents several approaches and techniques for generating walkthrough environment. These techniques have their own advantages and drawbacks. We classify both advantages and disadvantages for indoor and outdoor environments. Because walkthrough is generally about simulating indoor and outdoor environments.

   1. Indoor Environment

      The indoor environment generally have uniformly painted walls and texture poor surface[2].For this type of environment 3D reconstruction and visualization technique like[2] is useful. But still 3D modelling techniques lack to recover complete geometry of scene. It is also time consuming and labouring due to capturing large number of photos for image based rendering. We can use technique like [7] to capture Omni directional image using motor cart which takes images at eye height. Then render environment using image based rendering. Video based approach as mentioned above also scale well for indoor environments.

   2. Outdoor Environment:

      Outdoor environment generally consist of multiple individual object. 3D modelling has correspondence matching and segmentation problem. So it is effective to single continuous object. It is not effective for outdoor location which has multiple individual object. Panoramic image based techniques [1 ,5] are useful for outdoor location [1] present efficient panoramic based approach for interactive walkthrough. Video based approach has also done good job for outdoor environments.

4. This paper presents several approaches for building interactive 3D walkthrough environment. We have surveyed recent development in image based modelling and rendering techniques which are applicable to walkthrough environment. We then classify the approaches on basis of which type of images/videos they are used to generate the walkthrough environment. It is clear from our survey that walkthrough environment is combination of both computer vision and graphics field. To generate the walkthrough which simulate real indoor or outdoor environment, we have to use fusion of both techniques of indoor and outdoor environment. As a result we believe that to generate the efficient and scalable walkthrough we have to apply combination of methods presents in this paper in such a way that overcome their drawbacks.

1. Angus M.K. Siu, Ada S.K. Wan, Rynson W.H. Lau, Modeling and Rendering of Walkthrough Environments with Panoramic Images , VRST04, November 1012, 2004, Hong Kong.

2. Yasutaka Furukawa, Brian Curless, Steven M. Seitz, Richard Szeliski, Reconstructing Building Interiors from Images, ICCV 2009.

3. Sudipta N. Sinha, Drew Steedly, Richard Szeliski, Maneesh Agarwala, Marc Pollefeys Interactive 3D Architectural Modeling from Unordered Photo Collections, ACM Transactions on Graphics, December 2008.

4. Noah Snavely, Steven M. Seitz, Richard Szeliski, Photo Tourism: Exploring Photo Collections in 3D, SIGGRAPH '06.

5. Yan-Fai Chan, Man-Hong, Fok Chi-Wing Fu, Pheng-Ann Heng, Tien-Tsin Wong, A Panoramic-based Walkthrough System using Real Photos, computer graphics and application,1999.

6. Shenchang Eric Chen, QuickTime VR An Image- Based Approach to Virtual Environment Navigation, SIGGRAPH '95.

7. Daniel G. Aliaga, Thomas Funkhouser, Dimah Yanovsky, Ingrid Carlbom, Sea of Images, IEEE Visualization 2002 Oct. 27 – Nov. 1, 2002, Boston, MA, USA.

8. Daniel G. Aliaga, Ingrid Carlbom, Plenoptic Stitching: A Scalable Method for Reconstructing 3D Interactive Walkthroughs, SIGGRAPH '01.

9. Matthew Uyttendaele, Antonio Criminisi, Sing Bing Kang,Simon Winder, Richard Hartley, and Richard Szeliski, High-quality Image-based Interactive Exploration of Real-World Environments, IEEE Computer Graphics and Applications,2003.

10. Shum, Heung-Yeung, and Sing Bing Kang. "A review of image-based rendering techniques." IEEE/SPIE Visual Communications and Image Processing (VCIP) 213 (2000).

11. Shiaofen Fang, Yasaswy Bhupalam, Image Interpolation by Adaptive Subdivision for Image-Based Walkthrough of Large Scale Real-World Scenes, International Symposium on Virtual Reality, Archaeology and Cultural Heritage VAST (2005)