Eye Gaze Controlled Wheelchair

Eye Gaze Controlled Wheelchair

Ms. Anagha Dwajan B

Student,

Department of Electronics and Instrumentation Engineering,

JSS Academy of Technical Education Bangalore, India

Ms. Bhavani S

Student,

Department of Electronics and Instrumentation Engineering,

JSS Academy of Technical Education Bangalore, India

Mrs. Sowmya M S

Asst. Prof., Department of Electronics and Instrumentation Engineering,

JSS Academy of Technical Education Bangalore, India

Ms. Priya Reddy N

Student,

Department of Electronics and Instrumentation Engineering,

JSS Academy of Technical Education Bangalore, India

Ms. Usha M R

Student,

Department of Electronics and Instrumentation Engineering, JSS Academy of Technical Education

Bengaluru, India

AbstractEye gaze wheelchair is a unique technique executed mainly for the disabled persons who are fully paralyzed. In this system manual control of wheelchair is being replaced be automatic control i.e. controlled by the eyeball movement, so that the patients feel free and less or no difficulty in their movements. Continuous image is captured with the help of webcam which further undergoes several image processing techniques, to detect the position of eye pupil Haar cascade algorithm is being implemented with the resultant of the image processing technique wheelchair moves accordingly. DC motor is mounted to the wheels for easy motion of the wheelchair. The ultrasonic sensor is mounted to the wheelchair so that it detects any obstacles in the path of its movements and wheelchair stops movement as per sensor command.

KeywordsImage processing technique, Haar cascade Algorithm, Eye gaze, Ultrasonic sensor;

1. INTRODUCTION

   The Wheelchair is a vulnerable system used by aged and physically disabled people. By considering their requirements there are already many such systems in the market yet this our work is quite simpler and easy to use. Those complex control systems may be difficult to operate for ill persons. Hence eye gaze wheelchair provides the freedom to make their life easy and more convenient. This Eye gaze wheelchair does not require much energy and manpower to operate. The Camera captures the picture of an eye pupil of the patient on the wheelchair and sends these as input command for the DC motor which in turns moves the wheelchair accordingly through GPIO pins .The DC motor is used to perform the different operations such as to move the wheelchair left, right, forward, backward and stop. The advance level of Image Processing open computer vision library is used for Face and Eye detection. In general, there are many algorithms and applications for detecting these eye and face detection. One of

   them is Haar cascade like features detection algorithm used to detect single or multiple faces and detection of both eyes, to detect the exact eye pupil and locate its middle point. For automatically finding of Eye pupil and tracking eye pupil many computer vision libraries of Image processing are used like object detection, motion detection, image color conversion, edge detection, pattern matching, etc. But they have their restrictions.

   In Electrocardiograms, Electroencephalography and Electrooculography sensor-based eye pupil detection technique, voltage variation-based output assumed to decide the location of the pupil. As the peoples face and the eyes distance may differ for each person but presetting the input will result in the faulty location of the eye pupil. Similarly, in the head movement-based system, for totally disabled persons it becomes quite complicated for them to move their head frequently. Whereas, voice-activated power wheelchair operates well when the patient can speak fluently and command the direction they have to move but if there is any disturbances like heavy noise the system cant detect their voice properly even the patient who are using it cannot shout at the instance as they are ill. Hence under these circumstances this usage goes in ruin. The infrared reflection-based eye pupil detection system provides accurate detection of the eye pupil, middle location, and also a system that can track the eye movement. But the infrared radiations may affect the eye and user may lose their eye visibility.

   The Raspberry pi board is used to perform the control of the complete system operation. DIP-based output signal is sent to the Raspberry pi board. The Raspberry pi collects the data and analyses it. Raspberry pi sends the control signal to the motor driving circuit based on the location of the eye pupil. The motor driving circuit will decide to operate on the motor like run the motor in clock vice direction, antilock vice direction

   and even stop the motor. The wheelchair has two motors fixed for the movements of the wheel respective to the commands detected by the camera. The Ultrasonic sensor is also mounted on the wheelchair for the detection of any obstacle. If the sensor gets the obstacle close to the wheelchair, it will indicate raspberry pi and send the signal to the motor driving circuit to stop the motor. Here, an effective camera image-based eye pupil detection and tracking system is introduced. This is efficient as well as cost-effective system. In this the persons face, Eye and eye pupil is detected with less delay time. It is a unique Eye tracking technique, it even records the real time picture, identifies the presence of human face , eye location , eye pupil and performs several operations of basic image processing like the color image to grey conversion, filtering, threshold, pattern matching, noise reduction and circle detection.

2. LITERATURE SURVEY

   Smart wheelchair purely based on eye tracking is designed especially for people with locomotory disabled with additional remote control for simple electrical device to turn ON and OFF and to communicate via message with care taker or any other family persons. Here motion of the eye pupil is used as the control cursor on the raspberry PI where the image captured by webcam is processed using open CV customised image processing to drive in the direction of eyeball movement and also control module controls the manual control of the wheelchair for equipped appliance like small electrical switch and messages is being sent via smartphone to the caretaker [1]. In this paper for a person with propeller debilitated due to diseases like ALS (Amyotrophic Lateral Sclerosis). a Novel system is being proposed where the wheelchair is controlled through eye-gaze and in an unknown environment it provides a continuous, real time navigation or target identification, path planning and navigation based on novel algorithm. A novel N- cell grid-based graphical user interface to input and output specification and a calibrating method to minimise the calibration overhead which is caused in eye tracking system. This proposed system is not only beneficial for only Amyotrophic Lateral Sclerosis (ALS) population but also for

   many paralysed population [2].

   In this paper electrical wheelchair with joystick is replaced by electric wheelchair with Human computer interface (HCIs) instead of joystick which is used for manual control of wheelchair. Fuzzy set theory is being used for gaze detection and for recognition of unknown environment where the passenger tends to move. By making use of Laser Range Finder (LRF) technology the direction of motion and speed of wheelchair is calculated depending upon the motion of the eye gaze, the wheelchair automatically moves in that particular direction as that of passengers gaze by avoiding the obstacles in real time in an unknown envronment [3].

   In this paper motion of wheelchair is fully controlled by eye ball movement and eye blinks which uses deep convolutional neural network learning for classification which performs better than that of traditional image processing technique. In this system image of either one eye is captured where the image is divided into four frames which are send to the classifier to obtain the direction of motion. For the wheelchair

   to move the passenger has to look in the particular direction for particular desired amount of time assuming it to be 2 seconds then the wheelchair starts to move or it stops the movement when the passenger closes his eye technically when 16 consecutive frame of image is generated with closed eye lid the movement of wheelchair stops, convolution neural network assumes the input to be images which carries the related information for movement of direction of the wheelchair [4].

   In this paper electric wheelchair which is operated by hand flexibility is replaced by eye movements that is manually operated module is replaced by eye tracker of Tobii which converts the eye movement signals of patients into coordinates of gazing points. To reduce the range of fluctuation of gazing point and to identify the controlled area Kalman filter algorithm is implemented. Tobii X120 is the eye tracker which is used to collect tracking data of eye where it uses video camera and infrared light source which identifies the position of eye pupil by corneal reflection data, it determines the 2- Dimensional coordinates of eye gazing point on the screen to identify pupil position and signals is sent to computer via network port which transform eye tracking information into wheelchair movement control signal [5].

   Smart eye tracking system is designed for people with disabilities and elderly people. Wheelchair movement is mainly based on the eye movement the camera is mounted on the person head which captures the image of the eye and process the image by digital image processing with open source Computer vision (CV). The captured image is transmitted to Raspberry pi microcontroller to drive the wheel chair. The blink is regarded as the enter button on the keyboard to start the movement of the wheelchair. Two servo motor is used to move wheelchair in 2-Dimension and also to automatically stop the movements when there is any obstacles and change the direction of the wheelchair movement [6].

   In this paper assistance for neuromotor disabled person is eliminated by implementing the novel technique for eye controlled based wheelchair which is more independent and cost-effective system rather than the traditional electrical wheelchair. Here the experimental setup is made where a webcam is being mounted Infront of the patient to capture the image of any one of the eyes which is highly precise for tracking the position of pupil where image processing technique is implemented for determination of accurate pupil position. According to the position of pupil determined the motors which is attached to the wheels of wheelchaired directed to move according to that desired direction if there is any presence of obstacles in the direction of motion ultrasonic sensor senses the obstacles and stops the movement or changes the direction of movement automatically. Raspberry PI is used as the control unit which allows the system to access the control [7].

   A dummy wheelchair is controlled via iris movement using digital image processing in MATLAB. The whole system is divides into two parts, First the eye image is captured continuously by using web camera which is mounted on the helmet the captured image is being processed by using digital image processing where the edge detection technique is being incorporated to calculate the position of iris and also to calculate the distance to be moved in that particular direction. Motor is being incorporated to control the movement Arduino

   micro controller receives signal from relays via serial communication and generates the appropriate signal to L293D motor driver to run the motor in particular direction which in turn starts the movement of the dummy wheelchair to move in that desired direction without any delay and if there is any obstacle in between the infrared sensors present in the system senses the obstacle and send message to the microcontroller which in turn sends the message to the motor to stop the movement or to change the direction of motion [8].

   In this paper for people suffering from diseases like SMA (Spinal Muscular Atrophy) Cerebral Palsy etc., to achieve their daily life tasks easily uses WMRA (Wheelchair Mounted Robotic Arms) with high degree of freedom which is expensive and bulky to overcome this a simple robotic arm with four degree of freedom for specific application, affective and easy arm interaction and control via human machine interface is designed which is controlled by an simple embedded platform. Visual camera captures the surrounding environment which will be displayed on the mounted screen where the person select the object of operation the robotic arm automatically reaches the object by calculating its position from image being captured , distance between the objects and the environment is calculated by the end effector which uses different types of sensors to detect. The whole system is mainly implemented by visual servoing algorithm and computer visual algorithm for detecting the position of object, Human Machine Interface (HMI) based on LCD3.511 touch screen is designed to minimize the number of touches which controls the full system [9].

   In this paper EPW (Electrical Powered Wheelchair) is controlled by eye tracker system which is based on fuzzy logic controller is presented to improve the level of perfection in intelligent wheelchair or electric powered wheelchair (EPW). By using Viola and Jones technique region of each eye is extracted which is treated as an individual image, the captured image are converted to gray scale to binary images which consists of constant threshold which is not appropriate for each and every luminance condition to calculate the online threshold for binary image adaptive threshold is adopted, by calculating the online threshold the pupil position of user is being detected. Depending upon the calculation of black pixel present in online threshold pupil position is detected and considered as right, left, up and downward , pixel fusion method is used to obtain the accurate movement of electrical powered wheelchair (EPW) from both eye image by calculating the sum of both eyes image pixels in same position, Mamdani-type fuzzy logic controller is used to extract the pupil position which determines the direction of electrical powered wheelchair (EPW) movement [10].

   A power assisting control system with driving assistance using monocular fish-eye vision for electric wheelchair is implemented for highly disabled persons. The fish eye camera is mounted on the front part of the robotic wheelchair to capture the image of eye pupil and direct drive motor for wheel actuators. RT-pre-empt patched Linux PC and image processing PC are placed under the seat for motor controller. The distance that has to be travelled is calculated based on the optical flow vector (OFV), depending on the optical flow vector calculated from the floor area in the fish eye image the

   electric powered wheelchair moves from one particular area to another [11].

   In this paper OPCM (Optical Preview Control Model) is used for the analysis of driving task of the powered wheelchair is proposed which mainly focuses on the influence od driving speed on the drivers visible distance which is measured via eye tracker system. The essential element of OPCM (Optical Preview Control Model) is the number of trajectory samples. Here a case study is being done where a eye tracking system is used to calculate the visual distance by knowing the speed of the powered wheelchair time trajectory of that particular case is calculated by considering different conditions, trajectory of many cases is being calculated and the utput of all cases trajectory is being compared to the standard output of OPCM (Optical Preview Control Model) where in the compared result shows OPCM 9Optical Preview Control Model) trajectory matches the experimental cases results. In one particular case there is a tracking error which is caused by variation of speed which cannot be recorded by simple tracker hence drop-outs occurs for a particular visible distance which provides the assumption of minimum threshold to get good tracking OPCM (Optical Preview Control Model) performance [12].

   In this paper performance of intelligent wheelchair motion control using eye ball movement signals acquired via electrooculography (EOG) based on fuzzy logic control algorithm is being analysed. The input signal electrooculography (EOG) is obtained from Ag/Agcl electrode which is conditioned by g.USBamp amplifier, obtained signal is being feed to the wheelchair where the eye movement is decided as the direction of wheelchair which is fully controlled by fuzzy logic controller which decides the plant input for the desired performance. Fuzzy logic controller divides into two categories where the wheelchair movement is controlled based on the degree of rotation of eye movement this is done by fuzzy classifier (FC), motion of wheelchair is controlled by fuzzy- proportional derivative (FPD) which controls the angular speed of motor for a linear and steering motion [13].

3. METHODOLOGY

   1. System Overview

      The basic fundamental of the proposed system is eye tracking and detection of eye movement. To detect the location of eye pupil Haar cascade algorithm is used this technique comprises of several stages which are implemented to find the eye movement and also for face and eye detection, color switching, object tracking, Hough transform, edge detection and motion detection.

      Initially system captures image by making use of webcam. First step is the algorithm accurately detects the face of the user, if there is a greater number of faces it displays the error in runtime. According to the algorithm the system represents the user face in specific area of indicated image. Several process of image processing techniques is performed for eye pupil tracking. Fig.1. shows the process of the system.

      Fig .1. System Process

      Primarily camera module captures image, face is detected using Haar cascade algorithm, once the face is detected it detects eye inside the face region rectangular boxes are drawn across eye once it is detected. Now detection of eye pupil and definition of pupil center point is need for doing this several image processing techniques like color conversion/switching, edge detection, blur image, filtering, Hough transform and thresholding is implemented, for detecting circular shapes Haugh transform is implemented, with the help of webcam images are captured and transferred to Raspberry pi perorally OpenCV library is installed to Raspberry pi which supports the image processing technique.

   2. System Implementation

      The working of the system is purely based on real-time DAC (Data Acquisition) operating system. Low power consuming Arduino UNO is used here. Arduino UNO is a open source microcontroller based on microcontroller ATmega328P. Arduino is equipped with digital and analog (I/O) Input/output pins set which can be interfaced to variety of expansion boards and other electrical circuits. The board consists of 14 digital, 6 analog I/O pins. Power supply, ultrasonic sensor, LCD display, DC motor all are connected to the Arduino board.

      Camera module is placed in front of the users eye, distance between the eye and the fixed camera device is the main important part distance should be 10-14cm. camera captures the image of the user face to find the exact location of the pupil after detection of the location of the eye pupil, system algorithm measures the average values of center from the eye corner, which gives exact information of the eye ball movement. Motor driving IC is controlled by using two- channel relay board and for power supply batter is used motor driving circuit is integrated to Arduino which is connected to Raspberry Pi, which helps to operate the system. Camera module and Raspberry Pi is directly connected to each other for continuous image capturing. The system generates the command signal to enable GPIO pins and to perform left, right, forward, backward and cease operation.

      Various steps are involved for implementing the project and they are listed below:

      1. Raspberry Pi operating system installation: first the operating system is copied to micro SD card, which is later placed inside Raspberry Pi. Raspbian image file is booted by using WIN32disk imager software. By placing memory card inside Raspberry Pi board

         WIN32 can access the Raspbian OS (Operating System) directly without any booting.

      2. System Algorithm: In this system we use OpenCV (open Computer Vision) library for image processing technique. The Open computer vision (OpenCV) library plays vital role for giving the knowledge based on image processing technique. Where it is novel algorithm which is used for execution of the system and for performing operations.

         To obtain the center point in pupil of the eye various steps is being followed:

         1. Face detection and Eye detection: for detection of eye and face (OpenCV) Open Computer Vision is being used. For both eye and face detection Haar cascade algorithm is used. Camera modules detects the uses face once the face is detected it finds the location of eye and it marks the eye region by making use of Haar Cascade algorithm. Both eyes are detected based on distance.

         2. RGB to Gray conversion: here image color is conventioned to cut down delay time in the system. The size of the image frame should be low as possible because the processor cannot process the frames in a runtime condition. Hence, we use RGB to Gray conversion to convert the color imaged to Gray image.

         3. Features detection and blurring image: to blur the image Gaussian blur filter is used, which detects the perfect edges of the specific area of the cropped image. For finding the special pattern on an image which is to be unique, based on the pattern made features is implemented.

         4. Edge detection: corner edge detection and canny edge detection algorithm are being applied to determine soft edges in an image. It allows easy recognition of circle/rectangle presented in the image in order to set a proper threshold value.

         5. Hough transform: circle Hough transform method is used for drawing circle on an eye pupil which is a result of edge detection. Images are continuously captured according to the eye movements. Circle Hough transform detects the eye pupil movement and draw the circle.

         6. Eye tracking: for tracking movement of eye projection function algorithm is being used where coordinate system points the center point location of the eye.

            .

            For representation of movement of eye in right or left direction and to detect average points of location of eye pupil, two directional graphs based on horizontal and vertical axis is implemented. Relationship between projection point and eye center is shown by projection function graph.

            Fig .2. Coordinate system with eye position

            The direction of movement of eye is indicated by X and Y scale points. The eye pupil position at (A0, B0) points is:

            And

            A0=(A1+A2)

            B0=(B1+B2)

            A1, A2=eye pupil corner point in X-direction B1, B2=eye pupil corner point in Y-direction

            Coordinate points (A0, B0) =Actual eye location.

      3. Interfacing peripherals: here for video capturing USB

         2.0 webcam is used and for obstacle detection ultrasonic sensor is being used. 680*480 pixels is the pixel resolution value for transferring the image. For taking picture of the testing and experimental result VLC video player is used. Results acquired by the system are based on the image processing, value of eye pupil center and value of signal is sent to motor driving circuit for motion of wheelchair. Ultrasonic sensor used for detection of obstacle successfully measures the distance between the obstacle and the wheelchair. When ever the object is very near to the wheelchair objected is detected and when ever it crosses the minimum threshold distance value, brake is applied to shutdown the movement of the wheelchair.

   3. System Flowchart

   Initially region of interest of eye is being cropped where all the possible circle is presented on that particular area is being detected. Later it detects the eye ball once the eye ball is successfully detected we apply corner detection method to find out corners where center point is the average of the distance between eye circle center point and the center point.

   Fig .3. Flowchart.

   Distance is being accordingly to the movement of eye pupil where the minimum distance represents eye pupil moved to left position and the maximum distance represents eye pupil moved to right direction, when there is no movement of eye its is in middle position when the movement of eye is in left the left motor which is mounted to wheels of wheelchair run and vice versa. Wheelchair is stopped when the eye is closed for about 3 seconds. The process is started by continuous image capturing with the help of webcam, Raspbian system processes, image capturing where high pixel rates image capturing cameras is being used.

4. RESULT

   To test the system experimental setup is being done where small chair is being attached to the wheel which is connected to the DC motor to run in different direction. According to the experiment being done webcam captures the image, the captured image is then sent to Raspberry PI module where the image processing technique takes place. The entire system

   acquires the resulted image processing data and based on the center eye pupil value signal is sent to motor driving circuit for chair movement. Ultrasonic sensor is attached to the chair to sense the obstacles when there is any obstacle the chair slightly turns to left and then it moves forward. Figure 4 shows the different positions of chair based on eye movement.

   To check the proper working of the model first the chair is placed in fixed position once the webcam captures the eye image, and image processing is being done. Based on the resultant of the image processing technique, the motor mounted to the wheels does not shows any movement and there is no change in position of the chair.

   Fig .4.1. Chair in Fixed position

   When the resultant of the image processing technique is of minimum value the motor mounted to the wheels of chair gradually turns to the left side and chair starts to move in the left direction of motion.

   Fig. 4.2. Chair moving in left direction.

   When the resultant of the image processing technique is of maximum value the motor mounted to the wheels gradually turns to the right side and the chair starts to move in the right direction of motion.

   Fig. 4.3. Chair moving in right direction.

   When the resultant of image processing technique is less than the set threshold value the motor mounted to the wheels of chair remains in the same position and tends to move in the forward direction of motion.

   Fig. 4.4. Chair moving in forward direction

   When the resultant of image processing technique is high than the set threshold value the motor mounted to the wheels of wheel chair remains in same position and tends to move in a backward direction.

   Fig .4.5. Chair moving in backward direction.

   Ultrasonic sensor is being attached to the chair whenever the sensor senses the obstacles it measures the distance between the obstacle and the chair and sends the signal to the microcontroller which in turn commands the motor mounted to the wheel to stop/shutdown the movement of the wheels.

5. CONCLUSION

The notion of eye gaze wheelchair is not only regarded as the alternative source but also its is helpful for physically disabled persons to lead their life independently. The prototype wheelchair is constructed and tested successfully main objective of implementing the eye gaze wheelchair is to call attention for the digital image processing technique features. Even though our concept is most helpful for the physically disabled persons, it does have some drawbacks like:

• System has a delay of about 4 seconds due to which there is a delay in wheel movements.

• It is very difficult to track eye pupil during dark light sight.

Except these issues the system works flawlessly.

REFERENCES

1. N Wanluk, S Visitsattapongse, A Juhong (2016). Smart wheelchair based on eye tracking. 2016 9th Biomedical Engineering International Conference (BMEiCON). doi: 10.1109/BMEiCON.2016.7859594

2. MA Eid, N Giakoumidis, A El Saddik(2016). A novel eye-gaze- controlled wheelchair system for navigating unknown environments: case study with a person with ALS. IEEE Access ( Volume: 4 ). doi: 10.1109/ACCESS.2016.2520093

3. A Ishizuka, A Yorozu(2017). Motion control of a powered wheelchair using eye gaze in unknown environments. 2017 11th Asian Control Conference (ASCC). doi: 10.1109/ASCC.2017.8287148

4. A Rajesh, M Mantur (2017). Eyeball gesture controlled automatic wheelchair using deep learning. 2017 IEEE Region 10 Humanitarian Technology Conference (R10-HTC). doi: 10.1109/R10-

   HTC.2017.8288981

5. D Bai, Z Liu, Q Hu, J Yang, G Yang, C Ni(2016). Design of an eye movement-controlled wheelchair using Kalman filter algorithm. 2016 IEEE International Conference on Information and Automation (ICIA). doi: 10.1109/ICInfA.2016.7832085

6. Aniwat Juhong, T Treebupachatsakul, C Pintavirooj (2018). Smart eye- tracking system. 2018 International Workshop on Advanced Image Technology (IWAIT). doi: 10.1109/IWAIT.2018.8369701

7. SN Patel, V Prakash (2015). Autonomous camera based eye controlled wheelchair system using raspberry-pi. 2015 International Conference on Innovations in Information, Embedded and Communication Systems (ICIIECS). doi: 10.1109/ICIIECS.2015.7192876

8. P Arora, A Sharma, AS Soni(2015). Control of wheelchair dummy for differently abled patients via iris movement using image processing in MATLAB, 2015 Annual IEEE India Conference (INDICON). doi: 10.1109/INDICON.2015.7443610

9. A Palla, L Sarti, A Frigerio(2016). Embedded implementation of an eye- in-hand visual servoing control for a wheelchair mounted robotic arm. IEEE Symposium on Computers and Communication (ISCC). doi: 10.1109/ISCC.2016.7543753

10. FB Taher, NB Amor, M Jallouli (2015). An extended eye movement tracker system for an electric wheelchair movement control.2015 IEEE/ACS 12th International Conference of Computer Systems and Applications (AICCSA). doi: 10.1109/AICCSA.2015.7507101

11. N Oda (2018). Development of Power Assist Wheelchair Control with Driving Assistance using Fish-eye Vision. 12th France-Japan and 10th Europe-Asia Congress on Mechatronics. doi: 10.1109/MECATRONICS.2018.8495884

12. H Zatla, A Hadj-Abdelkader, Y Morere (2015). OPCM model application on a 3D simulator for powered wheelchair. 2015 International Conference on Virtual Rehabilitation (ICVR).

    doi: 10.1109/ICVR.2015.7358611

13. NMM Noor, S Ahmad (2015). Performance analysis of an electrooculography-based on intelligent wheelchair motion control. 10th Asian Control Conference (ASCC). doi: 10.1109/ASCC.2015.7244813