Gesture Controlled IR Mouse Pointer

Gesture Controlled IR Mouse Pointer

Miss. Sushma V. Achari, Prof. Suneeta V. Budihal.

Department of Electronics and communication, B V B College of Engineering and Technology, Hubli. Karnataka India.

Abstract – Currently there exists an advanced and natural form of interaction, named gesture-based interaction. It has been applied in some fields successfully, especially in the form of game applications. Although people are filled with wonder at the new interaction, it has not replaced the traditional interaction, like keyboard and mouse. The intension behind this work is to develop gesture controlled computer mouse. The gesture controlled IR mouse controls computer pointer by object movement. This mouse is built with the help of recently developed compact Mbed ARM cortex M3 core and IR Distance sensors. This project is been divided into 2 major parts:

1. Hardware part: Capturing object with the help of IR distance sensors and sending captured data serially to PC with the help of Mbed microcontroller.

2. Software part: Programming in c# for controlling computer pointer on screen with the help of acquired serial data from Mbed microcontroller.

   For palm gesture movement detection, the project uses the Distance sensor, the sensor data is sent through serial port to pc where secondary C# program controls movements of pointer on screen.

   Keywords – C#, Gesture, IR sensor, MBED, mouse pointer, Serial communication.

   1. INTRODUCTION

      In current-day society, with the development of computer science, the software and hardware are highly advanced. But the form of interaction between person and machine is developing slowly and even become an obstacle to the inherent technological development. The wide adoption of touch-enabled phones and multi-touch slates shows peoples increasing affection for mouse-less interfaces, often referred to in the media as Natural User Interfaces (NUIs) [8, 9]. Therefore, a great number of researchers start to divert their attention from computer screen or programming language to human-computer interaction (HCI). The research field of HCI has raised much attention and interest towards this. However, several issues also present. This abstract is not enough to fully express peoples intentions [10, 11].

      There is no doubt that gestures play an important role in interaction between people and machine. There have been varied approaches to handle gesture recognition [2], ranging from mathematical models based on hidden Markov chains

      [3] to tools or approaches based on soft computing [4]. A gesture is a motion of the body and covey lot more information. Waving goodbye is one example. The most importance of gesture is that it is a natural way. Gesture recognition has wide-ranging applications [1]. People usually use a broad range of gestures naturally to accompany speech.

      Common interaction, like keyboard and mouse, is almost totally free of gestures, and depend on direct manipulation. Gesture-based interaction provides a new way for people to interact with devices and shows great breakthrough in history of human computer interaction. In this technology, devices can get visual input and recognize peoples gesture without touching keys or screens. Therefore, in the field of human computer interaction based on user-centered theory, gesture- based interaction has received great attention around the world and even is considered as the trend of future.

      The Gesture controlled computer pointer is an input device to a computer. Every computer is controlled generally in the same manner you point, You click, you use keyboard shortcuts. And most of companies are looking to change interaction with computers. Making use of new motion sensing technology aimed to replace typing and mouse-clicking with some of the gestures and movements used in everyday life.

      Bending to the lack of good appropriate input devices, people with disabilities often encounter obstacles when using computers. Several researchers have analyzed the ability of healthy subjects to use a computer mouse [5][7]. Currently, keyboard and mouse are the common input devices. Due to the increasing popularity of the Microsoft Windows interface computer mouse has become even more important. Therefore, it is extremely urgent to invent a simple and effective mouse system for people to operate their computers.

   2. METHODOLOGY 2.1.Hardware implementation:

      Before going deep into projects hardware concept lets

      see some basic details which should be known to achieve aim of the project.

      As the title indicates hardware part includes IR sensors and Mbed microcontroller.

      The hardware implementation of this project is quite simple where all credit for simplistic implementation role goes to major hardware part of this project that is Mbed microcontroller. How this microcontroller does make implementation easy? Answer is, In Mbed microcontroller programming is done in simple C or C++ languages, it gives rapid prototyping for all microcontroller based applications, Ethernet, USB and 32-bit ARM® Cortex-M3 based designs and It is designed for prototyping all sorts of devices, especially those including Ethernet, USB, and FLASH memory etc. Mbed microcontroller has 40 pins each pin is made for different functionality. In this way Mbed microcontroller is different and easy compared to other

      microcontroller. Thus making use of the most of functionalities of Mbed microcontroller project could come out with great product.

      Another major hardware part of project is IR sensors. This sensor also called obstacle sensor, light sensor, fire sensor. Because this sensor senses obstacle, light, fire. The project uses this sensor only because it costs very less as compared to other sensors. But this could make disadvantage for this project, as it senses other than object. But this project could be implemented with the help of other sensor which senses particularly object but it costs more, but which could be very advantageous. As module obtained by this project couldnt be operated in all places. Hence module obtained by this project works accurate only in place with less light, no fire present.

      This versatile sensor module works on the principle of Reflection of Unmodulated Infrared Rays from the incident surface. A continuous beam of IR rays is emitted by the IR LED. Whenever a reflecting surface (white/obstacle) comes in front of the Receiver, these rays are reflected back and captured. Whenever an absorbing surface (Black/No Obstacle) comes in front of the Receiver, these rays are absorbed by surface and thus unable to be captured. A NPN Bipolar Transistor is used as a switch to provide the digital output. The analog input from the resistor photodiode potential divider is made externally available as analog output

      (A) on the module. When operating as Light or Fire sensor, the IR LED part remains unused and only the photodiode section is used because the light/fire source is itself generating the infrared rays. For different types of applications different calibration is recommended.

      The sensor module can also serve as an Analog Distance Sensor (low precision) with range from 0-22cm. i.e. 0cm=3.0V and so on up to 22cm=0.3V. The preset resistor (pot) is used to calibrate the sensitivity. High intensity LED indication is provided on-board for confirming Logic Output.

      Now coming to hardware setup of the project.

      Hardware setup is shown in the figure 2.2. Here two sensors are placed according to convenience which is facing upwards as shown in the figure 2.1.

      Figure.2.2: Hardware setup

      igure.2.1: Distance Sensors are Facing Upwards

      The two IR sensors are placed facing up and connected to Mbed microcontroller. And Mbed microcontroller is connected to personal computer through USB cable which is provided with the microcontroller.

      Mbed gets required power from computer as it is connected to USB port of PC. And sensors get supply voltage by pin number 40(Vout) provided in Mbed board which supplies 3.3V supply otherwise external power supply is given.

      As the sensors used are both analog and digital, it senses analog voltage values of hand or obstacle when hand is waved over sensor. The value of the analog sensors (assuming a static voltage of 0-3.3V) will be set in this program to have a value 0-1 (represented as a float value). Thus, in order to get a proper distance for a computer, we will multiply the values by 100, changing our applicable range to 0-100. Now these values are given to personal computer serially. Now secondary part of the project i.e. software implementation takes care of pointer controlling process through C# programming.

      Points to be remembered while working with MBED Microcontroller

      1. You have to make sure that there is a Serial Port for the microcontroller that it is not in use.

      2. You must open the Serial Port and match configuration setting to the Microcontroller

      3. You can then send any character you want on that port.

      4. Remember to close the port.

         In this explanation part the instructions on how to get communication between a computer and a microcontroller will be laid out.

         Algorithm to sense the Gestures from Mbed

         1. Check if sensor2>= threshold.

            • If sensor 1>= threshold & sensor 2>= threshold.

              Then, sensor 1 & sensor2 is detected.

            • If sensor3>= threshold & sensor1<= threshold.

              Then, sensor 3 & sensor2 is detected

            • If none of conditions meet. Then only sensor2 is detected.

         2. Check if sensor1>= threshold Then sensor1 only is detected.

         3. Check if sensor3>= threshold Then sensor3 only is detected.

         4. If none of the conditions in step 1, 2, 3 are met. Then no inputs are detected.

      2.2 Software implementation:

      Before talking about software implementation part of the project, serial communication process should be known so that programming is done according to project requirements.

      When we think about serial communication we get a flash of UART. This application note will provide details on how to get simple communications between a computer and a microcontroller using UART (Universal Asynchronous Receiver/Transmitter). UART is piece of hardware which translates data between parallel and serial communication mediums. Which require lot of connections and procedure. But this project makes use of USB to UART driver installed in personal computer which makes USB port act as serial port instead of making use of additional COM port connection. This helps in situation when application is running on laptops which are normally not provided with the COM port.

      Here how to set up a simple interface between a computer and a microcontroller by utilizing Microsoft Visual Studio, and the Mbed microcontroller is discussed.

      Hyper terminal interface module for sending serial data to pc: Before starting with actual task some prerequisite experiments are done in order to get familiar with the sources available like MBED board. Therefore prerequisite experiment of serial data transfer is done and checked the results on hyper terminal. Serial is generic protocol used by computers & electronic modules to send or receive control information or data. Serial link has 2 channels. i.e., transmitter & receiver. Here serial connection is done between Mbed USB ports to host pc. This easily allows us to communicate between Mbed and PC.

      This project makes use of GUI created in Visual Studio 2010 by programming in C# instead of using hyper terminal for checking values sent serially from hardware. Because there is no need of making settings for each connection. C# is a modern, general-purpose, object-oriented programming language developed by Microsoft and approved by ECMA and ISO. Once the data is received in personal computer secondary c# programming in personal computer controls cursor movement on computer screen.

      For reception of serial data from Mbed, GUI is created in C#. The GUI is made up of button and text box. Button is named as, open port so when the button is pressed, it opens the serial port and serial data starts displaying on text box. And as there are two separate data coming from two different sensors so one value is used for left and right movement and other data is used for top and bottom movement. And all these movement is achieved by programming in C#.

   3. RESULTS AND DISCUSSION

      Figure 2.3: Snap of Sample output data from sensors received in PC and display on GUI.

      Figure 2.3 shows the snap of output, which indicates analog data received in PC. The programming is done for reception of serial data and moving the cursor on the computer screen based on obtained data in GUI.

   4. CONCLUSION

      As per discussion, sensing the hand gestures and sending it to the PC through MBED board is done successfully, and also reading the sensed data coming from MBED board is displayed on GUI created using c# programming in PC is done which is as shown in Figure 2.3. The data obtained in PC is made use effectively for respective movements of cursor on screen using secondary C# program. Now cursor moves successfully left, right, top and bottom.

   5. FUTURE WORK

To enhance the features of the mouse there are many more functions which can be added to the design, such as invoking scroll, left and right click options.

As module doesnt need to be ergonomic it can be added with lot many options making use of functions provided in Mbed board, such as Ethernet, PWM, ADC etc. thus making use of Ethernet pin, internet option can be provided in mouse. Even hot function keys like volume, Numlock many more functions can be added to mouse and come up with a new and advanced mouse.

REFERENCES

1. C. L. LISETTI AND D. J. SCHIANO, AUTOMATIC CLASSIFICATION OF SINGLE FACIAL IMAGES, PRAGMATICS COGN., VOL. 8, PP. 185235, 2000.

2. V. I. Pavlovic, R. Sharma, and T. S. Huang, Visual interpretation of hand gestures for human computer interaction, IEEE Trans. Pattern Anal. Mach. Intell., vol.19, no. 7, pp. 677695, Jul. 1997.

3. L. R. Rabiner, A tutorial on hidden Markov models and selected applications in speech recognition, Proc.IEEE, vol. 77, no. 2, pp. 257285, Feb. 1989.

4. S. Mitra and T. Acharya, Data Mining: Multimedia, Soft Computing, and Bioinformatics. New York: Wiley, 2003.

5. N. Walker, D. E. Meyer, and J. B. Smelcer, Spatial and temporal characteristics of rapid cursor-positioning movements with electromechanical mice in humancomputer interaction, Hum. Factors, vol. 35, no. 3, pp. 431458, Sep. 1993.

6. J. G. Phillips, T. J. Triggs, and J. W. Meehan, Cursor orientation and computer screen positioning movements, Hum. Factors, vol. 43, no. 3, pp. 435441, 2001.

7. J. G. Phillips and T. J. Triggs, Characteristics of cursor trajectories controlled by the computer mouse, Ergonomics, vol. 4, no. 5, pp. 527 536 Apr. 2001.

8. G. Goth, Brave NUI world, CACM, vol. 54, no. 12, pp. 1416, 2011.

9. D. Wigdor and D. Wixon, Brave NUI World: Designing Natural User Interfaces for Touch and Gesture, Morgan Kaufmann, 2011.

10. D. Frohlich, Direct Manipulation and Other Lessons. In M. Helander,

    T. K. Landauer, and P. V. Prabhu, editors, Handbook of Human- Computer Interaction, pp. 463488, 1997.

11. D.M. Frohlich, The History and Future of Direct Manipulation, Behaviour & Information Technology,vol. 12, no. 6, pp. 315329, 1993.