Sixth Sense Technology : A Brief Literary Survey

Sixth Sense Technology : A Brief Literary Survey

Manab Kumar Saha1, Sirshendu Hore2 1,2Asst. Prof, Department of CSE HETC

Hooghly, India

Abstract

This paper is a brief survey in the field of Sixth Sense Technology. Sixth Sense is an interface of wearable gestures which upgrade the physical world around us with digital information and let us use the natural gestures to interact with that information. The advent of technology over the last decade has established a new dimension in the field of Human computer Interaction (HCI). The aim of this paper is to review various gestures, components used, methodology adopted in sixth sense technology and its application.

1. Introduction

   We are sensitive on the five basic senses i.e. feeling, seeing, tasting, smelling and hearing. Apart from those basic senses there is also one type of sense known as Sixth Sense. Sixth Sense Technology, it is the newest vernacular that has demonstrates its presence in the technical field. This newly introduced technology has developed, which has its relationship to the influence of these six senses. Our everyday computing machines will soon be able to sense the different feelings expand in the surroundings and it is nothing but a gift of the Sixth Sense Technology Sixth Sense is a wearable gesture based device that broaden the real world with information which are of digital type and lets people use a different types of gestures to interact with that information and there by design a environment sometime known as virtual environment. Information is generally written on paper or computer screen but sixth sense technology frees the data from that limitation and seamlessly integrates information & reality. With sixth sense technology the gap between the digital world & human world is being bridged. The human-computer interaction becomes very flexible due to this technology. This technology helps us to virtualizes real world things & know much more

   information regarding some object very effectively, easily and quickly.

   Gestures are used widely in sixth sense technology because they are the most vital and significant type of human communication. Gestures offer several advantages over traditional human-computer interaction system as it offers touch less system therefore it is particularly helpful for physically challenged person, elder users as well as in health care system while maintaining the sterility.

   Gestures can be of different type such as

   • Facial expression

   • Hand

   • Eye

   • Symbolic Responses

   • Speech

   • Sound

   • Gait

     Figure 1 illustrates different types of Gesture: –

     Figure 1. Different types of gesture

     Gestures in sixth sense can be used to

     1. Generate an event or command for practical interaction.

     2. Control object directly.

     Virtual environment (VE) gives a three- dimensional (3D) environment in which a person is able to interact directly with virtual objects naturally and powerfully with the environment.

     Machine learning & Intelligence: Combines a wide variety of advanced technologies to give machines the ability to learn, adapt, make decisions, and display behaviours not explicitly programmed into their original capabilities. Some machine intelligence capabilities, such as neural networks, expert systems, and self-organizing maps, are plug- in components they learn and manage processes at a very high level. Other capabilities, such as fuzzy logic, Bayes Theorem, and genetic algorithms are building blocks they often provide advanced reasoning and analysis capabilities that are used by other machine reasoning components. Machine Intelligence capabilities add powerful analytical, self-tuning, self-healing, and adaptive behaviour to client applications. They also comprise the core technologies for many of Sciantas advanced data mining and knowledge discovery services.

2. Literature Review

   A lot of research is being done in the fields of Human computer Interaction (HCI) and its application in virtual environment. Researchers have tried detecting the virtual object to control system environment using video devices for HCI. By using the web cameras as the input device, various natural gestures can be detected, tracked and analyzed. To help achieve those gestures we used various image features and gesture Templates. Cootes et al [7] used Active Shape Models (ASM) to track deformable objects. M. Isard et al introduced random sampling filters [8] to address the need of represent multiple hypotheses while tracking. G. Kitagawa [9] applied Condensation algorithm in factored sampling to solve the problem of visual tracking in clutter. Hojoon Park

   [10] used index finger for cursor movement and angle between index finger and thumb for clicking events. Chu-Feng Lien [11] used only the finger- tips to control the mouse cursor and his clicking method was based on image density, and required the user to hold the mouse cursor on the desired spot for a short period of time. A.Erdem et al [12], used finger tip tracking to control the motion of the mouse. A click of the mouse button was implemented by defining a screen such that a click occurred when a users hand passed over the region. Robertson et al [13], used another method to click. They used the motion of the thumb (from a thumbs-up position to a fist) to mark a clicking event thumb. Movement of the hand while making a special hand sign moved the mouse pointer. Shahzad Malik [14] developed a real-time system which will trace the 3D position and 2D orientation

   of the thumb and index finger of each hand without the use of special colour object or gloves. In 3D gaming Nasser H. Dardas et al. [15] developed a finger based gesture recognition system to control 3D game.

3. Evolution

   The WUW stands for (Wear UR World) a wearable gesture used in Sixth Sense which is a user friendly interface that provides links with the physical world around us with digital information and it uses hand gestures to interact with them. The initial approach Radio Frequency Identification (RFID) which is an electronic tagging technology that allows for detection and tracing of the tags, and consequently Sixth Sense focuses on the fixed objects applying on RFID to an enterprise setting, such as a university department or corporate office. In short Sixth Sense represents a form of mobile computing which is applied to non-computing entities.

   Figure 2. Initial Approaches

   The initial approach was with the projector attached to the helmet but there are various drawbacks of this process, e.g. the person had to stay still so that the images could be projected on a suitable surface. And further it was difficult to manage the mobile computing device kept in the users pocket. This approach definitely needed few changes. Thus the approach changed when a smaller projector has been introduced and created the pendant prototype to be shabby around the neck. The prototype Sixth Sense is composed of a camera, a mirror and pocket projector. The hardware components are grouped in a pendant-like mobile wearable device. Both the camera and the projector are connected to the mobile computing device in the users pocket.

   Figure 3. Final approach

   The prototype was built from a powered – battery 3M projector and a ordinary web cam, with an attached mirror all connected to an internet- enabled mobile phone. The cost of the setup which is less than $350, allows the user to project information from the phone onto an surface user hand, the body of another person or even wall.

4. Components

   Camera: Camera behaves as a digital eye of the system. It basically used to captures the scene that the user is looking at. The stream of video captured by the camera is passed to mobile computing device which does the appropriate compute vision computation. The major functions of the camera are:

   • Captures users gestures and movement (used in reorganization of user gestures)

   • Captures the scene in front and objects the user is interacting with (used in object reorganization and tracking)

   • When the user performs a framing gesture, it took a photo of the scene from the front.

     Projector: The key output device of the Sixth Sense system is projector. The projector visually augments physical objects, wall and surfaces the user is interacting with by projecting graphical user interfaces and digital information. Projector provides by the mobile computing device with the content to be projected. The projector unit used in prototype runs on a rechargeable battery. The important functions of the projector have been listed below:

     1. Augments related information from the internet and the physical objects the user interacting with by projecting just-in-time.

     2. Projects graphical user interface of the selected application onto walls or surfaces in front.

     Mirror: Mirror used to reflect the projection emergence from the projector and thus helps in projecting onto the desired locations on walls or surfaces. The user can change the gradient of the mirror manually to change the location of the projection. For example in application where the user wants the projection to go on the surface of the wall instead of the ground in front, he can change the tilt of the mirror to change the projection. Thus, the mirror in the Sixth Sense helps to overcome the limitation of the limited projection space of the projector.

     Microphone: The microphone is an elective component of the Sixth Sense. When using a paper as a computing interface, microphone is required. When the user wants to use a sheet of paper as an interactive surface, he or she clips the microphone to the paper. The microphone attached this way captures the sound signals of users touching the paper. After that the data is passed to computing device for processing. Later, combined with the tracking information about users finger, the system is able to identify precise touch events on the paper. Here, through the microphone sound signal has been capture along with the time information whereas the camera performs tracking.

     Mobile Computing Device: The Sixth Sense system uses a mobile computing device in users pocket as the processing device. This device can be a mobile handset or a small device where connection can be wireless or wired.

5. Applications

   The Numerous applications on sixth sense technology already have taken place. Some of the applications are given bellow: –

   Radio Frequency Identification: The radio frequency identification is the platform which is based on the enterprise intelligence which combines Radio Frequency Identification events with information from other enterprise systems and sensors to automatically make inferences about objects, people, workspaces and their interactions. This is basically an electronic class technology that allows tracking of tags & detection & consequently the objects that they are affixed to. It is used for tagging & remote detection with the low cost having widespread adoption of RFID.

   Washing Machine: A remarkable front loader that incorporates the unparalleled sixth Sense

   technology has been placed in a Whirlpool AWOE 8758 White Washing Machine, gives more optimization of resources and also increased saving in terms of energy, water and time.

   Virtual Call: You can use the Sixth Sense to project a keypad onto your hand, and then use that virtual keypad to make a call.

   Figure 4. Virtual keypad

   Virtual Map: We can see the map virtually on a physical surface & can find our destination. One can use his/her index fingers and thumbs to navigate the map, for example, to zoom in and out and do other controls.

   Figure 5. Virtual map

   Virtual Watch: With Sixth Sense all user can draw a circle on their wrist with their index finger to get a virtual watch that gives us the correct time. The computer track the red marker piece of tape or cap, recognizes the gesture, and instructs the projector to flash the image of a watch onto his/her wrist.

   Figure 6. Virtual wrist watch

   Virtual Newspaper in Multimedia Mode:

   Using this technology we can see more information

   about the topic by video streaming system directly on the newspaper. A Newspaper can provide live video news or dynamic information can be provided on a regular piece of paper. Thus a piece of paper converted into a video display.

   Figure 7. Newspaper in multimedia mode

   Figure 8. Take picture without camera

   3D Gaming in virtual environment: Using different hand gesture as arrow key one can control popular 3D game as show in figure 9.

   Figure 9. Playing Fly over the city game

   Touch based interface-Sparsh: Sparsh is an interaction method that lets users conceptually transfer media from one digital device to their body, and pass it to the other digital device by simple touch gestures.

   Misc App: Other than these applications there are some applications like drawing a picture using hand movement on a physical object, checking the flight details only having the flight information or the boarding pass, we can zoom a picture, get production information while buying a product we can also a get any book information and Amazon rating of that book, a more powerful application

   regarding sixth sense is while interacting with someone we can get his/her personal details. Figure 10 – 11 shows some Misc application.

   Draw a picture Zooming Book detail

   Figure 10. Misc application

   Flight updates People Info Product details

   Figure 11. Misc application

6. Software used & Methodology adapted

   To develop an application based on sixth sense technology one can use

   1. Language: Java, C #, C++, OpenCV, JavaCV.

   2. Image processing Software: MatLab, Mathematica, Cylab

   3. Powerful Electronic Gazette: WUW v0.1 beta, Hand Gloves

   Methods:

   Initialization: It is basically a static approach. Different gestures are captured, enhanced, features are extracted and finally a gesture template or cluster model is created using different algorithm of artificial intelligence such as MLP with Feed Forward Network, Visual Memory, and K-Means Clustering etc. Figure 12 illustrate typical hand gesture training process [15] while figure 13 illustrates the use of k-means cluster algorithm.

   Figure 12. Hand gesture training process

   Figure 13. K-means cluster with two cluster

   Acquisition: It is a real time approach where frames are captured using webcam or from a video input file.

   1. Segmentation: In this technique each of the frames is processed separately.

      Before analysis: The image is smoothed, skin pixels are labeled, noise is removed and small gaps are filled. Image edges are found, and then a region detection algorithm is used to segment the target gesture from other background information. Finally features are extracted using popular methodology [Skin detections, Sift, SURF, FAST algorithms].

   2. Pattern Recognition: Once the users gestures has been segmented and related features are extracted, it is compared with stored gesture templates or clustering model using different matching algorithm such as Hausdorff matching, Euclidean distance, hidden Markova model, Bag of Words, Hamming Distance, correlation based approach etc. Figure 14 illustrate the typical process of testing used for hand gesture recognitin.

   Figure 14. Hand gesture recognition process.

   Execution: Finally, the system carries out the corresponding action according to the recognized gesture.

   Figure 15; illustrate the complete methodology adopted for sixth sense technology.

   Figure 15. Block diagram of methodology adapted for gesture based sixth sense application.

7. Acknowledgement

   We would also like to express our appreciation and gratitude to HETC college authority for helping us to develop this paper.

8. Conclusion

   Sixth sense technology is an emerging technology in the field of computer vision and HCI. As a result, its a common fallacy that Sixth Sense is a completely solved problem in this area. Some of the directions for the future research work in the field to users, evaluators and vendors can be

   1. To make our environment more robust especially for low quality and degraded images which is captured using economical image acquisition device?

   2. How to overcome different lighting condition (day light and artificial light) which has a great impact in this area.

   3. Improve Machine learning & Intelligence for gesture recognition through comparative studies.

9. References

1. Dong A Bowman et. al, The Virtual Venue: User Computer Interaction in Information Rich Virtual Environments, 1998, Vol.7, pp. -478 493.

2. Viola and M.Jones, Robust Real time Object Detection. International Journal of Computer Vision, 2(57) 2004, pp. 137 154.

3. T.Kirishima, K.Sato and K.chihara, Real time gesture recognition by learning selective control of visual interest point. Paper presented at the IEEE Trans. on Pattern Analysis and Machine Intelligence, Vol. 27, No. 3, March [2005], pp. 351

   364.

4. Y.Wu and T.Huang, Hand modelling analysis and recognition for the vision based human computer interaction. Paper presented at the IEEE Signal Processing Magazine, Special Issue on Immersive Interactive Technology, Vol.18, No. 3, 2001,pp.- 51 60.

5. R.Bolt, Put that there,:Voice and gesture at the graphics interface. In Proc.SIGGRAPH, 1980.

6. T.Schlomer, B. Poppinga, N. Henze and S. Boll, Gesture recognition with a Wii controller, In Proceedings of the second international conference on Tangible and embedded interaction, 2008, pp. 11 14. Available from: ACM Portal: ACM Digital Library.

7. Cootes, A.Hill, C.J.Taylor and J. Haslam, 1994, The Use of Active Shape Models For Locating Structures in Medical Images. Image and Vision Computing, Vol.16, No.6, July [1994], pp. 355 366.

8. M.A.Isard and A.Blake, Visual tracking by stochastic propagation of conditional density. In the 4th Proceedings European conference computer vision, pp.343 356, 15 April [1996], Cambridge, England.

9. G.Kitawaga, Monte Carlo Filter and Smoother for the Non Gaussian Nonlinear State Space Models. Journal of Computational and Graphical Statistics, Vol. 5, No. 1, 1996, pp.1 25.

10. Hojoon Park, A Method for Controlling Mouse Movement using a Real Time Camera 2008.

11. Chu Feng Lien, Portal Vision Based HCI A Real Time Hand Mouse System on the Handheld Devices.

12. A. Erdem, E. Yardimci, Y. Atalay, V. Cetin, Computer vision based mouse, A. E. Acoustics, Speech, and Signal Processing. Paper presented at the proceedings, (ICASS). IEEE International Conference, 2002.

13. Robertson P., Laddaga R., Van Kleek M., Virtual mouse vision based interface, In the Proceedings of the nineth international conference on intelligent user interfaces, pp. 177 183. Available from: ACM Portal: ACM Digital Library. [January 2004]

14. Shahzad Malik, Real-time Hand Tracking and Finger Tracking for Interaction,CSC2503F Project Report. [18 December 2003].

15. Nasser H. Dardas et al, Hand Gesture Interaction with a 3D Virtual Environment, The Research Bulletin of Jorden, ACM , ISSN: 2078 – 7952 , Vol II(III) , Page – 86.

16. Azriel Rosfeld and Avinash C. kak, Digital Picture Processing, Academic Press, In. Orlando,FL,USA, 1982.

17. Olivier Faugeras, Three Dimensional Computer Vision, A Geometric viewpoint, MIT press, 1993.