A Overview On Designing Of Hands Free Mouse Pointer For Motor Impairment People Using Motion Tracking And Speech Recognition

A Overview On Designing Of Hands Free Mouse Pointer For Motor Impairment People Using Motion Tracking And Speech Recognition

Priyanka G. Jaiswal, Prof. Pragati Patil, Prof. Girish Agrawal

AGPCE, Nagpur (M.H.), INDIA

ABSTRACT

Information technology (IT) is changing the way we perform work and providing society with a multitude of entertainment options. Unfortunately, in the past designers of many software systems have not considered the disabled as active users of technology, and thus this significant part of the world population has often been neglected. The intention of this paper is to provide an overview on the subject of Human- Computer Interaction and it describes an alternative method to the traditional mouse input device.

Due to recent increase of computer power and decrease of camera cost, it has become very common to see a camera on top of a computer monitor. The overview of HCI includes the audio and video based interaction between a computer and a human, which is basically used for people with motor impairments. Here, we are Introducing Camera-based mouse replacement system which allows people with motor impairments to control the mouse pointer with head or nose movements if they are unable to use their hands. The system tracks the computer users movements with a video camera and translates them into the movements of the mouse pointer on the screen with the help of thresholding. This system can be most beneficial to those who retain the ability to control their head movements, but do not have control of their extremities, for students and small children for playing games and adults with degenerative condition.

Index Terms: Adaptive User Interfaces, Camera Mouse, HCI (human Computer Interaction), Microphone, Video-Based Interfaces,.

1. INTRODUCTION

   Persons with disabilities are often unable to use computers. This is because they are either unable to find a suitable means of interaction or they simply cannot afford commercial solutions. In study also found that available solutions do not promote the individuals sense of independence, as

   they require a third party to attach markers at various points of their anatomy. This project work addresses these shortcomings by providing a non- intrusive, reliable, inexpensive and robust visual tracking system. It allows persons, who may have disabilities ranging from not being able to use their hands to severe cases where the person is only able to move their head, to navigate and manipulate the graphical user interface using head movements and speech.

   This System is used for user with motor impairments people who cannot use their hands to operate a computer mouse also they are unable to use the shortcuts of keyboard to operate the system because of disability, but wish to operate a computer. This system is basically working under two media one is audio media and another is video media. Video media will track the users head or facial feature positions with the help of web camera to control mouse pointer movement on the screen. Here mouse pointer movements are consisting of left click, right click and double click of mouse buttons which is possible due to head movements. Head or facial feature positions may be detected by camera-based systems such as Camera Mouse. Motion of the head or tracked feature in the video frame is typically scaled by a constant factor and transferred to control the mouse pointer. This resulted in constricted motion of the mouse pointer when mouse pointer control functions with constant scale factors were used. With an adaptive function, the user is able to move the mouse pointer to all positions on the screen with head movements that are most comfortable.

   1. OVERVIEW OF HCI

      Utilizing computers had always begged the question of interfacing. The methods by which human has been interacting with computers has travelled a long way. The journey still continues and new designs of technologies and systems appear more and more every day and the research in this area has been growing very fast in the last few decades.

      The growth in Human-Computer Interaction (HCI) field has not only been in quality of interaction, it has also experienced different branching in its history. Instead of designing regular interfaces, the different research branches

      have had different focus on the concepts of multimodality rather than unimodality, intelligent adaptive interfaces rather than command/action based ones, and finally active rather than passive interfaces.

   2. NEED OF ASSISTIVE TECHNOLOGIES

      Recognition of the need for technologies and guidelines for providing accessibility to electronic resources for users with disabilities has been increasing in recent years. A majority of persons with disabilities can now lead more independent lives in their communities, attend regular schools, and seek professional careers more than ever before in history. Assistive technology providers are changing their focus from people with disabilities as requiring treatment and intervention, to a view of the person with a disability and the minimization of obstacles to living in the community and participating in the workforce. Assistive technologies have been an important key to successful community participation. Following are the applications where this system can be used.

   3. RESEARCH MOTIVATION AND RELATED WORK

   algorithm relies on detecting specific features such as the eyes or nose to follow head movements across multiple frames.

   There are also various commercial mouse alternatives available today. NaturalPoint(NaturalPoint, 2006) markets several head-tracking-based mouse alternatives on their web site. While the benefits are real, these devices still require the user to attach markers either to the head or glasses. Other systems use infrared emitters that are attached to the users glasses, head-band, or cap. Some systems, for example the Quick Glance system by EyeTech Digital Systems (EyeTech, 2006), place the transmitter over the monitor and use an infrared reflector that is attached to the users forehead or glasses. Mouse clicks are generated with a physical switch or a software interface.

2. PROPOSED SYSTEM

   1. PROBLEM DEFINITION

      The proposed system is categorized into two part according to functionality including Mouse movement with the help of head tracking which is video based and another is mouse movement with speech recognition is audio based system.

      Persons with disabilities are often unable to use computers. This is because they are either unable to find a suitable means of interaction or they simply cannot afford commercial solutions. After studying many research paper it is also found that available solutions do not promote the individuals sense of independence, as they require a third party to attach markers at various points of their anatomy. This system addresses these shortcomings by providing a non-intrusive, reliable, inexpensive and robust visual tracking system. It allows persons, who may have disabilities ranging from not being able to use their hands to severe cases where the person is only able to move their head, to navigate and manipulate the graphical user interface using head movements and speech.

      Research into assistive technologies is ongoing and in (Evans et al., 2000), the authors describe a head-mounted infrared-emitting control system that is a relative pointing device and acts like a joystick rather than a mouse. In (Chen et al., 1999) a system containing an infrared transmitter was described. The transmitter was ounted on to the users eyeglasses, along with a set of infrared receiving modules that substitute the keys of a computer keyboard, and a tongue-touch panel to activate the infrared beam. In (tyabi etal., 2006) the authors describe a system for translating a users motion to mouse movements. Their tracking

      The tracking of head movements with a web camera provides a simple solution to the motion tracking of mouse buttons. The proposed system utilizes off-the-shelf hardware and software components to make the system easily available to the greatest possible number of users by performing all the functions of mouse including left click, right click, double click and mouse move which can be achieved by the head movements.

      The proposed system utilizes LPC, which is used for extracting features of voice signals. By issuing the relevant voice command, the user may at any time perform common tasks, such as left click, right click, mouse move and double clicks which is a great challenge for people with reduced dexterity and motor control in their hands. This system not only enables the user to execute single- or double-clicks, but also more complex operations. For example, opening of any folder or any application by double clicking on it..

      Voice commands used to activate buttons are:

      • Click single left click

      • Double double left click

      • Right single right click

      • Move mouse movement

        The number of commands is limited, providing good recognition accuracy, and the commands themselves may be changed to suit the

        individual user. There is scope to extend the set of commands for devices offering more buttons.

        The MATLAB Framework is chosen as a development platform, as it offers a good level of interoperability with the functionality of Image processing parts.

   2. FEATURES

      There are various features of Adaptive mouse replacement system it allow people with motor impairment to control the mouse pointer with head movement and with speech command as which uses the assistive technology; following are the highlighted features of system.

      1. Adults with Degenerative Conditions

         It will be useful for the people those are generally adults who have spent much of their lives without disabilities. Some of them are very familiar with technology and are accustomed to working with computers. Some of these people have sufficient motor control so that they remain mobile, e.g., controlling a wheelchair, by interacting with a computer via a mouse or other input device. Residents with the ability to speak may use speech recognition software to control computer software and dictate emails.

      2. Adults with Stable Conditions

         For such people whose condition has been stable for a long period of time. This user has been especially helpful in understanding the challenges we face in developing human-computer interaction systems for people with severe motion impairments. Also for such people those who are stable but not having any computer knowledge, such people can also operate this system such as to view photos or to open any existing folder.

      3. Students and Small Children

         It is also useful for such children those who need occupational therapy, provides education and therapy for students with multiple disabilities. Some students have cognitive disabilities in addition to physical limitations. Some of the young children may be learning about their own motor control as they use the technology. When using the Camera Mouse, students may learn that their head movements correlate with the movements of the pointer on the screen. And with this excitement it will make some improvements to their condition.

      4. Voice-controlled human-computer interface

         A voice-controlled human-computer interface has been designed that enables severely handicapped individuals to operate a computer.

      5. User can fully manipulate the computer via voice commands

         The user interface, in the industrial design field of humanmachine interaction, is the space where interaction between humans and machines occurs.

         The goal of interaction between a human and a machine at the user interface is effective operation and control of the machine, and feedback from the machine which aids the operator in making operational decisions

      6. Ideal method for disabled persons who cannot use conventional computer interfaces such as keyboard, mouse.

      In human-computer interaction, computer accessibility (also known as Accessible computing) refers to the accessibility of a computer system to all people, regardless of disability or severity of impairment. It is largely a software concern; when software, hardware, or a combination of hardware and software, is used to enable use of a computer by a person with a disability or impairment, this is known as Assistive Technology.

   3. GOALS

      • To develop a non-intrusive, reliable and inexpensive system that adds to a disabled persons independence and is inherently adaptable to many different circumstances.

      • To develop a robust visual tracking system that uses a modified Lucas-Kanade optical flow algorithm to track head movements, without needing to locate any specific features of the face and using linear predictive coding for the speech

      • The proposed system will allow persons, who may have disabilities ranging from not being able to use their hands to severe cases where the person is only able to move their head, to navigate and manipulate the graphical user interface of the Microsoft Windows operating system using head movements and speech.

   4. OBJECTIVE AND SCOPE

   1. Technology supporting disabled Students There is a growing trend for students with a variety of impairments and special needs to be integrated into mainstream education. For these students to have the same experience as their non- disabled peers requires classroom material, such as worksheets, homework assignments and assessment and examination papers, as well as books, to be made available in alternative, accessible formats. Responsible bodies, including local authorities, national government and other relevant agencies, are obliged to consider the needs of all disabled pupils, and should therefore make provision of accessible learning resources and services for all pupils who have literacy support needs.

   2. Motivating mobility: interactive systems to promote physical activity and leisure for people with limited mobility

      Disability, illness and/or increasing infirmity associated with older age can all contribute to people becoming less mobile. The reasons for reduced mobility are complex and vary with each individual, but may include fear, lack of interest or limited motivation. Researchers from the Universities of Nottingham, Southampton, Dundee, Sheffield Hallam and Sussex have proposed that interactive technologies and communication methods could be combined to provide the encouragement to disabled people needed to promote greater physical activity.

   3. Living in the moment: developing an interactive multimedia activity system for elderly people with dementia

      Dementia progressively undermines an individuals cognitive function, making daily living activities and social situations very difficult. The pervasive short-term memory problem makes challenges of even the simplest tasks as people continuously forget what they are doing, what has just been said and that they have started tasks, such as boiling the kettle. Consequently, dementia is disempowering as relatives and professional careers step in and gradually take over all aspects of a persons life.

   4. Improving computer interaction for older users: an investigation of dynamic on-screen targets

      One problem for some older people is difficulty using a mouse to position the cursor and select targets on a computer screen. So tis project will helpful for such persons.

   5. Improving online accessibility

   The internet has opened up a wealth of new business and leisure opportunities, from using email to keep in touch with friends and family in faraway places to booking flights and making purchases online. For older people and disabled people online technology can make it much easier to participate in the wider world, to find information and to overcome geographic isolation, but only if online facilities are designed to take account of their specific needs.

3. SYSTEM OVERVIEW

The System which we have implemented consisting of Web camera and microphone now a days which is easily available with monitor and microphone, which fully emulates the functionality normally associated with a mouse device. This section is providing a short overview of the image-processing algorithms and speech- interaction technologies on which the system is based on.

The system consists of two signal- processing units and interpretation logic (As shown in Figure). Image processing algorithms can be applied to the video stream to detect the users face and follow tracking points to determine head movements. The audio stream is analyzed by the speech recognition engine to determine relevant voice commands. The interpretation logic in turn receives relevant parameters from the signal-processing units and translates these into on-screen actions by the mouse pointer.

VIDEO STREAM AUDIO

Image Audio

Processing

Head feature Tracking

Speech Analysis

INTERPRET ER

CONVERTED TO MOUSE POINTER MOVEMENT

Figure 4.1: Flowchart of Interfacing of

So, from the above figure which shows the Processing of system is consisting of mainly two parts:

1. Head Tracking

2. Speech recognition

1. EAD TRACKING

   Head tracking system is based on Threshold level algorithm which is used to

   Head tracking is consisting of following image processing steps:

   1. Thresholding

   2. Segmentation

   3. Filtrating of Noise

   4. Gray Scale Image to Binary Scale Image

   5. Removing All the Small Areas

   6. Removing All the Small Areas

1. PEECH RECOGNITION

   For humans, speech represents the most natural way to communicate, and human-Computer interaction is no exception. If an application can be controlled by voice commands, its features can be opened up to users otherwise unable to use them. Although the idea of using speech recognition as a human-computer interaction method is not new, there is still a distinct lack of speech interaction in todays software market. It can only be of advantage to software vendors to produce more applications that feature a speech interaction component, either as an extension to existing functions, or as an alternative input method.

   Speech recognition technology has progressed to a level, where users can reliably control certain actions on the computer using voice commands. The latest release of the Microsoft Windows operating system, Windows Vista, features much improved speech recognition capabilities compared to its predecessor. Using speech recognition in the operating system, the user can dictate emails and documents, and use voice to control programs and navigate Web sites.

   Linear predictive coding (LPC) is a method for signal source modelling in speech signal processing. It is often used by linguists as a formant extraction tool. It has wide application in other areas. LPC analysis is usually most appropriate for modeling vowels which are periodic, except nasalized vowels. LPC is based on the source-filter model of speech signal.

   Following technique is used to implement speech recognition.

   • LINEAR PREDICTIVE CODING

   Linear predictive coding (LPC) is a method for signal source modelling in speech signal processing. It is often used by linguists as a

   formant extraction tool. It has wide application in other areas. LPC analysis is usually most appropriate for modeling vowels which are periodic, except nasalized vowels. LPC is based on the source-filter model of speech signal.

   Principles

   1. First, the observation of input and output sequences produces a model, with a number of poles, or formants.

   2. A resulting set of coefficients can then describe the behaviour of a system which is not known yet. It is used for predicting a sample. This set of coefficients is an all- pole model, a simplified version of the acoustic model of the speech production system.

   3. The analysis then estimates the values of a discrete-time signal as a linear function of previous samples. The spectral envelope is represented in a compressed form, using the information of the linear predictive model.

REFERENCES

1] Priyanka G. Jaiswal, Pragati patil, Parul Bhanarkar computer science department presented in international conference on Electrical, Electronics & Computer science (ICEECS-2012) on Adaptive mouse replacement for person with disabilities using human computer interface.

2] IEEE Transactions On Neural Systems And Rehabilitation Engineering, Vol. 10, No. 1, March 2002, Fast, Reliable Head Tracking Under Varying Illumination: An Approach Based On Registration Of Texture-Mapped 3d Models Marco La Cascia, Stan Sclaroff, Member, Ieee, And Vassilis Athitsos

3] IEEE Transactions On Pattern Analysis And Machine Intelligence, Vol. 22, No. 4, April 2000 Algorithms For Threshold Level Selection And Decoder Logic Design In Decision-Aided Symbol-By- Symbol Data Receivers For Magnetic Recording Applications

4] Magnetics, IEEE Transactions On (Volume:31, Issue:5 ) Fast, Reliable Head Tracking Under Varying Illumination: An Approach Based On Registration Of Texture-Mapped 3d

Models Marco La Cascia, Stan Sclaroff, Member, Ieee, And Vassilis Athitsos

5] IEEE Transactions On Pattern Analysis And Machine Intelligence, Vol. 22, No. 4, April 2000, Fast, Reliable Head Tracking Under Varying Illumination: An Approach Based On Registration Of Texture- Mapped 3d Models Marco La Cascia, Stan Sclaroff, Member, Ieee, And Vassilis Athitsos

6] IEEE transaction on Neural system and rehabilation engineering, vol. 10 1,march 2002 The camera Mouse: visual tracking of Body features to provide computer access for people with severe disabilities.

7] Evans, D. G., Drew, R., & Blenkhorn, P. (2000). Controlling mouse pointer position using an infrared head-operated joystick. Rehabilitation Engineering, IEEE Transactions on [see also IEEE Trans. on Neural Systems and Rehabilitation], Vol. 8, No. 1, March 2000, 107-117, ISSN:

1063-6528 EyeTech Digital Systems Inc.

8] A Method For Controlling Mouse Movement Using A Real-Time Camera Hojoon Park Department Of Computer Science Brown University, Providence, RI, USA

9] Motion-Tracking And Speech Recognition For Hands-Free Mouse-Pointer Manipulation Frank Loewenich And Frederic Maire Queensland University Of Technology Australia

10] Hands-Free Mouse-Pointer Manipulation Using Motion-Tracking And Speech Recognition, By Frank Loewenich Australia

11] Special Issue Of International Journal Of Computer Applications (0975 8887) On Optimization And On-Chip Communication, No.9. Feb.2012, Ww.Ijcaonline.Org, Speaker Independent Recognition System With Mouse Movementsby R.L.K.Venkateswarlu, R. Vasantha Kumari, A.K.V.Nagayya

12] Fakhreddine Karray, Milad Alemzadeh, Jamil Abou Saleh And Mo Nours Arab Pattern Analysis And Machine Intelligence Lab., Department Of Electrical And Computer Engineering University Of Waterloo, Waterloo,

Canada Human-Computer Interaction: Overview On State-Of-The-Art

13] John J. Magee, Samuel Epstein, Eric S. Missimer, Christopher Kwan And Margrit Betke Computer Science Department, Boston University 111 Cummington St., Boston, Ma 02215 UsaAdaptive Muse- Replacement Interface Control Functions For Users With Disabilities

14] Dmitry O. Gorodnichy*, Gerhard Roth, Institute For Information Technology, National Research Council Of Canada, M-

50 Montreal Rd, Ottawa. Ont., Canada K1a 0r6 Received 31 July 2003; Received In Revised Form 2 December 2003; Accepted 22 March 2004 Nouse Use Your Nose As A Mouse Perceptual Vision Technology For Hands-Free Games And Interfaces

15] Kesari Verma, Aniruddha S. Thoke, Pritam Singh, International Journal Of Computer And Information Engineering 6 2012 Neural Network Based Approach For Face

16] Detection Cum Face Recognition Phillip Ian Wilson Texas A&M University Corpus Christi 6300 Ocean Dr., Corpus Christi, Tx 78412 361-877-9062, Facial Feature Detection Using Haar Classifiers*