Gesture Sensing Device

Gesture Sensing Device

Swanand Nabar

Electronics Department, University of Mumbai

Renuka Deshpande

Electronics Department, University of Mumbai

Abstract

A Gesture Sensing device is precisely a device which will bridge the gap between the physical world and digital world. Today to do any form of computing we have to make use of certain input devices and output devices with which we can communicate with the computer and perform various work. But this device removes this wall of physical objects. Any work like dialling a phone number, can be easily done using this device without any keypad to enter numbers or any LCD to display the dialled number. Using this device we can project the display anywhere we want and by just doing the gesture of dialling a number, the number actually gets dialled. This Device will hence give a new revolution in the Technological World.

1. Introduction

   Gesture Sensing Device as the name suggests will sense our gestures and will give us a response. The device can sense many gestures by image recognition, colour recognition and image processing and will give us a very intuitive environment to play with the digital world. The device will be more helpful for day to day work and will make the work flow easier. With the help of our fingers and some colour markers we can make some gestures, the device tracks the gestures and our responses to show us where our hand is actually moving. In the following paper we have given a broad idea of the device construction and working .This device can make us free from the physical limitations of the computing world.

2. What is a gesture?

   A gesture is a form of non-verbal communication in which visible bodily actions communicate particular messages, either in place of speech or together and in Parallel with spoken words. Gestures include movement of the hands, face, or other parts of the body. Gestures differ from physical non-verbal communication that does not communicate specific messages, such as purely expressive displays, or displays of joint attention. Gestures allow individuals to communicate a variety of feelings and thoughts, from contempt and hostility to approval and affection, often together with body language in addition to words when they speak.

3. Gesture sensing examples

   1. Drawing on the Wall

      Using the hands and colour markers as shown in the figure we can draw various images on any wall. Here the camera tracks the movements of the colour markers on the fingers

   2. Clicking on Objects Displayed on Wall

      Then by using projector we can display the menu or any wall and click the objects. Here, as shown previously, the camera is sensing our gestures using image processing.

   3. Clicking Photos by making Gesture

      In this application we can make a gesture of clicking a photo and the device clicks the photo for you. In this the device senses the gesture and by using various algorithms of image processing it clicks the picture for here we need not depend on a camera to click the picture

   4. Dialling a Phone Number

      This is also an interesting example in which we use our palm to display the numbers and then click on the numbers using your fingers and directly click on the numbers to make a call. Here we dont need a keypad or a LCD.

   5. Playing a Game

      This example shows how this device uses a microphone and helps us to play games and do work on a piece of paper. The device tracks the sound of your hand wherever we touch on the paper

   6. Watching a live video

   7. Check your Weather for the Day

   8. Editing Documents

4. Construction of the Device: The Components

   1. Camera

      The camera is the key input device of the Sensing system. The camera acts as a digital eye of the system. It basically captures the scene the user is looking at. The video stream captured by the camera is passed to mobile computing device which does the appropriate computer vision computation. The camera helps to track down the finger movements of the user. A chip containing image processing algorithms is used to track down process the hand movements and records the input accordingly.

   2. Projector

      The projector is the key output device of the Sensing system. The projector visually augments surfaces, Walls and physical objects the user is interacting with by projecting digital information and graphical user interface. The mobile computing device provides the projector with the content to be projected. The projector unit used in prototype runs on a rechargeable battery. The major functions of the projector can be listed as:

      • Projects graphical user interface of the selected application onto surfaces or walls in front

      • Augments the physical objects the user interacting with by projecting real-time information, connected to the internet.

   3. Mirror

      The mirror reflects the projection coming out from the projector and thus helps in projecting onto the desired locations on walls or surfaces. The user manually can change the tilt of the mirror to change the location of the projection. For example in application where the user wants the projection on the ground instead of the surface in front, he can change the tilt of the mirror to change the projection. Thus, the mirror in the Sensing Device helps in overcoming the limitation of the limited projection space of the projector.

   4. Microphone

      The microphone is an optional component of the Sensing Device. It is required when using a paper as a computing interface. When the user wants to use a sheet of paper as an interactive surface, he or she clips the microphone to the paper. The microphone attached this way captures the sound signals of users touching the paper. This data is passed to computing device for processing. Later, combined with the tracking information about users finger, the system is able to identify precise touch events device and those images which are in use are saved rest are discarded.

   5. Mobile computing device

      The Sixth Sense system uses a mobile computing device in users pocket as the processing device. The software program enabling all the features of the system runs on this computing device. This device can be a mobile phone or a small laptop computer. The camera, projector and the microphone are connected to this device using wired or wireless connection. The detail of the software program that runs on this device is provided in next section. The mobile computing device is also connected to the Internet via 3G network or wireless connection.

5. Working of the device

   • The device has a camera which continuously captures the images of all the gestures happening around the device.

   • We wear colour markers on our fingers. These colour markers have specific grey levels.

   • When the images of the gestures are taken then the computing device through various algorithms searches for these grey levels in the images taken by the Camera.

   • The co- ordinates of the specific grey levels searched are been processed by various algorithms and hence the Gesture is been sensed.

   • This working is a simple explanation of the working of the device. Some algorithms are been made whereas some are under process.

   • Thus by using Digital Image processing these devices can be developed.

   • The computing Device uses a microcontroller an some memory. The memory stores the images taken by the camera.

   • Captures the scene in front and objects the user is interacting with (used in object reorganization and tracking).

   • Takes a photo of the scene in front when the user performs a framing gesture.

   • Captures the scene of projected interface (used to correct the alignment, placement and look and feel of the projected interface component).

6. Application

   • The Device can be used by Deaf and Dumb people to speak out.

   • The device can be used by physically challenged people to interact with the world

   • People who have Dyslexia can interact and learn easily.

   • This device can be used to reduce crimes or capture a crime scene without the criminals knowing about it, just by gestures.

   • In the world of fast sharing technology as this device is connected to the internet the photos or videos taken by gestures can directly be shared on a social networking site.

   • This device can help this world come closer by reducing the language barrier between people from different regions of the world.

7. Conclusion

   This device helps us to disconnect from the machine and interact with the world directly and yet somehow be connected to the machines. The achievements of this device are not completed without the technology and technology includes the use of different machines, but its limited to a few objects. The great part about this device is that we are not a slave to these devices rather we can control these devices by our free will.

8. References

1. P. Mistry, P. Maes. SixthSense A Wearable Gestural Interface. In the Proceedings of SIGGRAPH Asia 2009,

   Sketch. Yokohama, Japan. 2009

2. P. Mistry. The Thrilling Potential of SixthSense. TED talk in Wonders session. TEDIndia 2009. Mysore, India

2009