Invisible Mouse : Interacting with Computer Systems without Physical Mouse

Invisible Mouse : Interacting with Computer Systems without Physical Mouse

This method removes the requirement of having a

Nikhil Srivastava

B.Tech ,Electronics &Communication Engineering Galgotias College of Engineering &Technology, Greater Noida

ABSTRACT

This paper proposes an effective and robust invisible computer mouse based on image processing using MATLAB and IR LED Array .

An invisible computer mouse consists of two stages. First , based on the image processing using MATLAB in which labeling of blobs pertaining to each finger is detected by illuminating hand on table by infrared source of light , adjacent to users hand, attached with computer . Secondly, the dynamic position of blob is translated into cursor movement on computer screen. At the same time we can perform right click, left click and such related operations by finger movements as same as movements done on classical hardware mouse.

The invisible computer mouse described in this paper is particularly advantageous for Human- Computer interaction (HCI) ; while still allowing computer users to interact with their computers in a way with which they were familiar with.

General Terms

Human-Computer Interface, image processing, invisible mouse, matlab.

1. INTRODUCTION

   In the artificial Intelligence field, scientists are trying to develop some wise methods of vision-based interaction systems that reduces cost, hardware and increases efficiency. Virtual mouse as input is one of the desirable modes in HCI [3] [1].

   physical mouse altogether but still provides the intuitive interaction of a physical mouse that we are familiar with. It consists of an Infrared (IR) LED array and an Infrared Camera [4] . Both IR Source and IR camera are embedded in the computer. The IR LED array creates a plane of IR light just above the surface the computer sits on. The user cups their hand, as if a physical mouse was present underneath, and the IR LED array lights up the hand which is in contact with the surface. The IR camera detects those bright IR blobs using computer vision with MATLAB. The change in the position and arrangements of these blobs are interpreted as mouse cursor movement and mouse clicks. As the user moves their hand the cursor on screen moves accordingly. When the user taps their index finger, the size of the blob changes and the camera recognizes the intended mouse click.

   Fig 1: Experiment Setup and devices layout

2. METHOD

   1. Image acquisition

      The IR Camera is automatically triggered by MATLAB commands and starts taking snaps. The snaps are of 640 X 480 pixels at frame rate of 5 frames/sec and stored as RGB form of images.

      1. Blob Recognition

         The input RGB image is converted to binary image by thresholding and later by noise removal .Each blob generated in binary image is labeled and centroid of respective blob is stored [2].

         1. IR camera detects those bright IR blobs in RGB image

      Fig 2 :(b) Detected blobs in binary image

3. Interpretation of Mouse Commands

   1. Dynamic cursor motion

      The index finger movement labeled as blob2 in FIGURE 4 has a dynamic centroid that is mapped into its computer coordinates of 1366X768 pixels on computer screen by using a multiplier and JAVA public class Robot .The purpose of Robot is to facilitate automated testing of Java platform implementations in MATLAB.

      Fig 3: Mapping of space coordinates

   2. Right click

      A left click is said to occur when a new blob or a blob in next frame is between blob2 and origin (FIGURE 4) .This condition is recognized as the intended mouse click or right-click.

   3. Left click

      A left click is said to occur when a new blob or a blob in next frame is between blob1 and blob3 (FIGURE 4) .This condition is recognized as the intended mouse click or left-click .

      Fig 4: Algorithm Flowchart

4. ANALYSIS AND EXPERIMENTAL RESULTS

   Two types of tests were performed. The first was to determine the accuracy of algorithm at different light intensities . Second one was performed on varying surface types, ranging from rough surface to smooth surface. Tests were performed with a 640X480 resolution camera .The camera was a extended IR camera and the computer had a

   2.16 GHz Intel Core i3 processor with 2 gigabytes of RAM.

   1. Varying Light Conditions

      Day light conditions add high intensities Infrared Spectrum to experiment region that creates multiple white blobs in its binary image captured from IR camera. Hence, indoor light conditions act as ambient working conditions for this system to work.

   2. Varying Surface Conditions

   With presence of rough surface in plane of IR emitter, noise is produced that appears in binary image as blob4 ( Figure 5) and is ruled out; as its centroid s X- coordinates is maximum than other blobs.

   Fig 5 : RGB and final Binary Image

5. CONCLUSION AND FUTURE WORK

   The prototype for this project was built for around $5 USD . The system has been implemented in MATLAB environment using MATLAB Image Processing Toolbox. Using the pointer, moving the cursor and the simulating the mouse click events also worked well. However, system has some disadvantages such as; being invariant to illumination up to some scale .

   My upcoming project is based on this sytem in determining touch ;using IR emitter and IR camera and mini projector mounted vertically attached to computer that translates touch gesture into intended click operation on computer screen inspired from LUMINAR.[5]

   I will be working on improvements to the recognition object and tracking algorithms. This may in the future lead to more complex gesture recognition than is possible at present, and could ultimately give us number of advantages over a physical mouse, since the number of functions handled by a physical mouse is limited.

6. REFERENCES

DUAN-DUAN YANG, LIAN-WEN JIN, JUN-XUN YIN, LI-XIN ZHEN, JIAN-CHENG HUANG, An

Effective Robust Fingertip Detection Method For Finger Writing Character Recognition System , Proceedings of the Fourth International Conference on Machine Learning and Cybernetics, Guangzhou, 18-21 August 2005

1. WebReference:SixthSense

   http://www.pranavmistry.com/projects/sixthsense/

2. Web Reference : Camera mouse, January 2010.

   http://www.cameramouse.org.

   Web Reference: Mouseless

   http://www.pranavmistry.com/projects/mouseless/

3. Web Reference: Natan Linder, LUMINAR, http://fluid.media.mit.edu/people/natan/current/lumina r.html