Eye Blink Detection Method for disabled: Assisting System for Paralyzed

Eye Blink Detection Method for disabled: Assisting System for Paralyzed

Meghana S1, Monica R 2,Nayana K, Neha S S4

Computer Science and Engineering

Vivekananda Institute of Technology, Bangalore-74, Karnataka, India

Abstract – We represent a real time method based on some video and image processing algorithms for eye blink detection. The motivation of this research is the need of disabling who cannot communicate with human. A Haar Cascade Classifier is applied for face and eye detection for getting eye and facial axis information. In addition, the same classifier is used based on Haar- like features to find out the relationship between the eyes and the facial axis for positioning the eyes. An efficient eye tracking method is proposed which uses the position of detected face. Finally, an eye blinking detection based on eyelids state (close or open) is used for controlling android mobile phones. The method is used with and without smoothing filter to show the improvement of detection

accuracy. The application is used in real time for studying the effect of light and distance between the eyes and the mobile device in order to evaluate the accuracy detection and overall accuracy of the system. Test results show that our proposed method provides a 98% overall accuracy and 100% detection accuracy for a distance of 35 cm and an artificial light.

1. INTRODUCTION

   Motor neuron disease (MND) is a medical condition where the motor neurons of the patient are paralyzed and is it incurable. It also leads to weakness of muscles with respect to hand, feet or voice. Because of this, the patient cannot perform his voluntary actions and it is very difficult for patients to express is needs.Tetraplegia is also one such condition where people cannot move parts below their neck. In this electronic era, solutions for patients with above mentioned diseases are found, one such innovation is the proposed system explained throughout. The proposed system can be

   used to control and communicate with other people through eye blinks. In the recent years due to the rapid advancement in the technology there has been a great demand of human-computer or human- mobile interaction (HCI or HMI). Eye blink is a quick action of closing and opening of the eyelids. Blink detection is an important enabling component in various domains such as human-computer interaction, mobile interaction, health care, and driving safety. For example, blink has been used as an input modality for people with disabilities to interact with computers and mobile phones.

   The proposed system detects the voluntary blinks of the patient and accordingly sends the message about therequirement to the care taker and also gives the voice output via call to the caretaker. .System uses an inbuilt camera to capture the video of the patient and with the help of facial landmark algorithm it identifies the face and eyes of the patient. Thesystem then slides a bunch of images one after the other on the screen and the patient can choose to blink over the image he wants ,just to convey message of his desires. The system identifies the blink with help of eye aspect ratio and then sends a message to the care taker of what the patient wants and also the system initiates a call to the care taker where in a voice is audible saying what the patient wants.

   In these days electronic devices are improving day by day and their demand is also improving. Smart phones, tablets are example of this. The system detects the eye blink and differentiates between an intentional long blink and a normal eye blink. Tetraplegia is a condition where people cannot move parts below neck. The proposed system can be used to control and Communicate with other people. In the recent years due to the rapid advancement in the technology there has been a great demand of human computer or mobile interaction (HCI or HMI). Eye blink is a quick action of closing and opening of the eyelids. Blink detection is an important enabling component in

   various domains such as human computer interaction, mobile interaction, health care, and driving safety. For example, blink has been used as an input modality for people with disabilities to interact with computers and mobile phones.

   In Viola the chain of single-feature filters, Haar Cascade Classifier for identifying sub-region image is used. With the fast calculation of integral image technique, it can work in real time. Eye tracking provides an almost

   seamless form of interaction with the modern graphical user

   interface, representing the fastest non-invasive method of measuring user interest and attention. While the mouse, keyboard, and other touch-based interfaces have long reigned as the primary mediums associated with the field of human computer interaction, as advances continue to improve the cost and accuracy of eye tracking systems, they stand poised to contend for this role. An open and close eye template for blink pattern decisions based on correlation measurement is used. The method was specifically useful for people with severely paralyzed. A real-time eye blinking detection was proposed based on SIFT feature tracking with GPU based implementation.

   An efficient method is proposed based on image processing techniques for detecting human eye blinks and generating inter-eye-blink intervals. A Haar Cascade Classifier and Camshaft algorithms for face tracking and consequently are applied for getting facial axis information. The algorithm results show that the proposed method can work efficiently in real-time applications.

   Adaptive Haar Cascade Classifier from a cascade of boosted classifiers based on Haar-like features using the relationship between the eyes and the facial axis applied for positioning the eyes. The algorithm results show that the proposed method can work efficiently in real-time applications. An EyePhone application which is developed in is a system that capable of driving mobile applications/functions using only the users eyes movement and actions (e.g. Wink). EyePhone tracks the users eye movement across the phones display using the camera mounted on the front of the phone.

   The results indicate that EyePhone is a promising approach to driving mobile applications in a hand- free manner. An efficient eye tracking system is presented in having a feature of blink detection for controlling an interface that provides an alternative way of Communication for the people who are suffering from severe physical disabilities the

   proposed system uses pupil portion for tracking the movement of eyes.

   Paralyzed people lack the ability to control muscle function in one or more muscle group. The condition can be caused by strokes, ALS, multiple sclerosis, and many other diseases. Locked in Syndrome (LIS) is a form of paralysis where a patient has lost in control of nearly all voluntary muscles. These people are unable to control any part of their body, besides eye moment and blinking. Due to their condition these people are unable to talk, text, and communicate in general. Even through people that have LIS are cognitively aware, their thoughts and ideas are locked inside of them. These people depend on eye blinks to communicate. They rely on nurse and caretakers to interpret and decode their blinking. Whenever patients do not have a person to read their Eyeblink, they have no means of self- expression.

   Blink To Speak offers a form of independence to paralyzed people. The software platform converts eye blinks to Speak. Every feature of the software can be controlled by eye movement. Thus, the software can be independently operated by paralyzed people. Using the sofware, patients can record messages, recite those messages aloud, and send the messages to others.

   The software can be run on any low-end computer, from a Raspberry Pi to an IBM ThinkPad. The software uses computer vision and Haar cascades to detect eye blinking and convert the motion into text. The program uses language modelling to predict the next words that the user might blink. The software can be easily customized for each patient as well. Blink to Text is free open source software. It is distributed under the MIT Permissive Free Software License.

2. PROBLEM STATEMENT

   As said earlier the patients lose the ability to speak and write they can only contact the outside world through human-computer interaction; e.g. controlling brain waves or tracking eye movements. Currently, brain wave controlling devices need to be worn by users, so they are not convenient for people to use. There exists eye-motion based software which enables the MND patients to write in the computer by using their eye functions only. When they are away from PC and lie on the bed, they cannot communicate with care providers. With the goal of helping MND patients on the bed to call for other people with a simple and easy approach this research aims to develop a real time video processing system, which can successfully research

   aims to develop a detect the eye blinks regardless the head directions, day or night.

3. PROPOSED SYSTEM

The proposed project aims to bring out a solution for the paralyzed people without any harm to their body externally or internally. It overweighs the previously developed prototypes in this field because none of the components are in direct contact with the patients body hence it definitely will prove to be safer.

Cost effective: The main objective of developing algorithm of a real time video Oculography system is that to provide cost effective for those people who cannot afford. The existing technique for such patients to communicate is too costly. Thus, it is necessary to design a system which is affordable to common people which includes cost effective components for designing.

Fast: There are few algorithms which are developed for video Oculography system for communication. The main objective of this project is to develop an algorithm which is extremely fast compared to the existing ones.

Accuracy: The main objective of this project is to develop an algorithm which is more accurate compared to the existing ones.

that can afford the technology. BlinkToSpeak focuses on a different demographic that are often ignored. BlinkToSpeak is free and open source. The software runs on wide variety of low-end computers. The only required peripheral is a basic webcam. Not only is this software accessible to paralyzed people, but paralyzed people of almost all financial classes as well.

V. SYSTEM ARCHITECTURE

IV. OBJECTIVES

1. Allow paralysis victims to communicate independently.

   Many paralysis victims already use eye blinks as a form of communication. It is common for nurses and caretakers to read a patients eye blinks and decode the pattern. The ALS association even offers a communication guide that relies on eye blinks. Blink To Speak automates this task. The software reads a persons eye blinks and converts them into text. A key feature of the software is that it can be started, paused, and operated entirely with eye blinks. This allows patients to record their thoughts with complete independence. No nurses or caretakers are required to help patients express themselves. Not only does this reduce the financial burden on paralysis patients, but this form of independence can be morally uplifting as well.

2. Be accessible to people with financial constraints.

Many companies are developing technologies that are controlled by eye movement. These technologies rely on expensive hardware to track a user`s eyes. While these devices can absolutely help LIS victims, they are only available to people

Figure 1: System Architecture

1. Capturing the frame from the video using the systems camera initializes the execution of the proposed system.

2. The frames are converted into gray scale image and we are making zero and ones combination of white and black pixels.

3. We are considering 68 point pixel for face and 6 point pixel for eye lids.

4. The Face Detection is done using haar cascade Algorithm processes on the captured video frames which is converted as gray scale image to give out the rectangular boxed face.

5. The output from Face Detection Algorithm then gets processed using AdaBoost Classifier to detect the eye region in theiface.

6. Eye detected will be sent to check whether the eye is closed or open.

7. When the eyes are open the virtual keyboard numbers are being played sequence until the eyes areclosed.

8. Once eye is closed then the particular need of the patient in that number is given as a output for care takers.

9. The outputs are in the form of image display, text alerting, audio played and message sent to the concerned person.

   Figure 3: Gray Scale Image

   1. MODULES

      1. Camera and frame capturing

      2. Face Detection

      3. Eye detection

      4. Eyetracking

      5. Eye blinking

1. Camera and frame capturing

   The first step of the proposed system is the initialization. After taking a short video of the participants face using the front camera of the device used which is more likely a laptop. A process Frame method will be usedto create the frames from the captured video. Afterwards the coloured frames will be converted to gray scale frames by extracting only the luminance component.

   The luminosity method is a more sophisticated version of the average method. It also averages the values, but it forms a weighted average to account for human perception. Were more sensitive to green than other colours, so green is weighted most heavily. The formula for luminosity is 0.21 R +

   0.72 G + 0.07 B. The luminosity method works best overall.

   Figure 2: Original Image

2. Face Detection

   Figure 4: Face Detecting

   The Haar classifier is used in algorithm for face detection. . Haar classifier rapidly detects any object, based on detected featureinot pixels, like facial feature. However, the area of the image being analysed for a facial feature needs to be regionalized to the location with the highest probability of containing the feature. By regionalizing the detection area, false positives are eliminated. As the result face is detected and marked with rectangle box and will be used later to approximate an axis of the eyes for eye detection step.

3. Eye Detection

   Figure 4: Eye Detection

   To detect the eye, first, the Haar cascade classifier should be trained, in order to train the classifiers, the AdaBoost algorithm and Haar feature algorithms must be implemented, two set of images are needed. One set contains an image or scene that does not

   contain the object. The EBCM used all detected elements of Haar Cascade Classifier, and the result show the detected eye in rectangle box.

   AdaBoost algorithm is used to train node classifiers on a Haar-like feature set to improve the generalization ability of the node classifier. Consequently, the face detection performance of the face detector is improved. Experimental results have proved that the proposed algorithm can significantly reduce the number of weak classifiers, increase the detection speed, and slightly raise the detection accuracy as well.

4. Eye Tracking

   Figure 5: Parts of Eye

   The corneal-reflection and pupil-centre are the two eyes parts that are the most important parts to extract the features that will be used in EBCM method. These features help us in tracking the eyes movement. By identifying the center of the pupil and the loation of the corneal reflection, the vector between them is measured. Besides, with further trigonometric calculations, point-of-regard can be found. The EBCM method succeeded in making the face and the eye's pupil moved together in the same direction synchronously and with the same direction. Let suppose that X is the human face which has been detected , P1 and P2 are two points related to the left eye, and they are moving synchronously with the movement of X.

5. Eye Blinking

Eye blinking and movement can be detected with relatively high reliability by unobtrusive techniques. Though, there are few techniques discovered for the active scene where the face and the camera device move independently, and the eye moves freely in every direction independently of the face. Although care must be taken, that eye- gaze tracking data is used in a sensible way, since the nature of human eye movements is a combination of several voluntary and involuntary cognitive processes.

EYE ASPECT RATIO (EAR)

We can apply facial landmark detection to localize important regions of the face, including eyes, eyebrows, nose, ears, and mouth. This also implies that we can extract specific facial structures by knowing the indexes of the particular face parts

In terms of blink detection, we are only interested in two sets of facial structuresthe eyes. Each eye is represented by 6 (x, y)-coordinates, starting at the left- corner of the eye (as if you were looking at the person), and then working clockwise around the remainder of the region. Based on this image, we should take away on key point. There is a relation between the width and the height of these coordinates Where p1, , p6 are 2D facial landmark locations.

EAR = || p2 – p6 || + || p3 p|| / 2 || p1- p4 ||

Figure 6: Eye Aspect Ratio

System Implementation uses the structure created during architectural design and the results of system analysis to construct system elements that meet the stakeholder requirements and system requirements developed in the early life cycle phases. These system elements are then integrated to form intermediate aggregates and finally the complete system of-interest. Implementation is the process that actually yields the lowest-level system elements in the system hierarchy (system breakdown structure). System elements are made, bought, or reused. Production involves the hardware fabrication processes of forming, removing, joining, and finishing, the software realization processes of coding and testing, or the operational procedures development processes for operators' roles. If implementation involves a production process, a manufacturing system which uses the established technical and management processes may be required. The purpose of the implementation process is to design and create (or fabricate) a system element conforming to that elements design properties and/or requirements. The element is constructed employing appropriate

technologies and industry practices. This process bridges the system definition processes and the integration process.

1. CONCLUSION

   Eye blink detection is a very challenge problem for controlling mobile phones in a real time application. This is due to the movement of eyes and the variation of light for different distances from the mobile camera. The proposed method provides 8% of accuracy improvement for eye detection and blinking. When an artificial light is used the overall and detection accuracy are 98% and 100% respectively for a distance equal to 35 cm. Each frame takes an average of 71 ms for time execution which is very efficient for real time application.

2. FUTURE WORK

   As far as the future of this system is concerned, an alarm can be set when the caretaker miss to attend the call or miss to view the message. This alarm will alert the caretaker and he can respond to it immediately. Another improvisation can be of setting the IOT devices. An IOT device can be set in a way such that the patient is able to operate light switch and regulate the fan with the help of blinks which will reduce the work of caretaker and also patient feels independent.

3. REFERENCES

1. Dunya Goz Hastanesi. Research on Blink Detection and Eye Tracking for Eye Localization Information[D]. Wuhan University of Technology, 2018.

2. T. N. Bhaskar, Y. V. Venkatesh, etc. Improved Haar-Cascade Algorithm Combined with Kalman Filter for Moving Target Tracking[J]. Computer Measurement & Control, 2017, 25(3): 209-212

3. Ahmad Hammoud, Daniel Bourget, etc. Algorithm of Moving Object Tracking Based on Video Images Sequence[J]. Journal of Jilin University, 2017, 35(3): 347-353.

4. Veena N. Hegde, Ramya S. Ullagaddimath, Comparison of eye tracking, electrooculography and an auditory brain computer interface for binary communication: a case study with a participant in the locked-in state, (2015).

5. Ioana Bacivarov, Research on Eye Monitored Device for disable People. Master's degree thesis of Zhengzhou University, 2015

6. Oualla, M., Sadiq, A. and Mbarki, S. 2015. Comarative study of the methods using Haarlike feature. International Journal of Engineering Sciences & Research Technology.

7. Vibodha Yasas Sri Bandara, Prof. Asiri Nanayakkara, et al. High-Speed Tracking with Kernelized Correlation Filters[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2014, 37(3):583- 596.

8. Asfand Ateem and T. Bock, Statistical Models of Appearance for Eye Tracking and EyeBlink Detection and Measurement, Proceeding of 31st International Sysposium on Automation and Robotics in Construction and Mining (ISARC), Sydney, Australia, (2014).

9. Nayel Al-Zubi, Z. Murtaza and Ali Shah," Differentiation of signals generated by eye blinks and mouth clenching in a portable brain computer interface system., Robotic and Emerging Allied Technology in Engineering (2014).

10. WANG Liting, DING Xiaoqing, FANG Chi and A. Likhachev, Eye Blink Detection for the Implantable System for Functional Restoration of Orbicularis Oculi Muscle, IEEE Conference on Robotics and Automation, (2014)

11. Atish Udayashankar, Amit R Kowshik, Chandramouli S Based on Multi-Feature Fusion Combines Kalman Prediction Algorithm for Real- Time Visual Tracking[J]. Computer Applications, 2014

12. Leo Pauly, Deepa Sankar, Automatic Motion Control of Powered Wheel Chair by the Movements of Eye Blink, in Proceedings of the 21st International Conference on Pattern Recognition (ICPR '12), pp. 24562459, November 2012.

13. Ippei Torii ,Kaoruko Ohtani ,Takahito Niwa

    ,Naohiro Ishii, Unsupervised Eye Blink Artifact Identification in Electroencephalogram in Proceedings of the 20th International Conference on Pattern Recognition (ICPR '10), pp. 38503853.

14. Danylo Batulin, Home Automation Using SSVEP & Eye- Blink Detection Based BrainComputer Interface, in Proceedings of the 21st International Conference on Pattern Recognition (ICPR '12), pp. 441444, November 2012.

.