Drowsy Driver Detection System using Deep Learning

Drowsy Driver Detection System using Deep Learning

Revanth S, Roshni Dash, S R Pooja, and Sthavir G SoroffDepartment of Computer Science And Engineering

B.M.S. College of Engineering Bengaluru, Karnataka, India

Mr. Basavaraj Jakkali

Department of Computer Science And Engineering Visveswaraya Technological University, Belgaum Bengaluru, Karnataka, India

Abstract Drowsiness interferes with people's daily lives and can cause ineffective work or car accidents due to insomnia, which can cause harm to persons and property. Driver drowsiness detection systems continuously monitor the driver's behaviour and inform them if they exhibit signs of tiredness in an effort to prevent accidents brought on by driver fatigue. In order to identify whether a driver is drowsy or not, these systems combine a variety of computer vision techniques, such as Histogram of Oriented Gradients (HOG) for face detection and landmark extraction, and eye aspect ratio calculation. The technology operates by taking pictures of the driver's face in real time and analysing the features that can be extracted to gauge their level of awareness. The driver may receive a warning if the system notices signs of drowsiness, such as a visual, audio, or haptic warning, to assist them stay awake and alert while driving. In order to obtain a feature vector from the face that captures its distinct textures and contours, HOG is utilized for face detection. It is possible to accurately and robustly identify driver tiredness in real-time using HOG and a machine learning algorithm, enhancing road safety and lowering the probability of accidents.

1. INTRODUCTION

   The goal of the Drowsy Driver Detection system is to monitor driver behaviour, identify sleepiness, and send out timely alerts to protect both the driver and other road users. Technological developments have made it possible to develop creative solutions to this problem. The Drowsy Driver Detection System is a sophisticated computer program created to track and examine driver behaviour and physiological indications in order to identify signs of sleepiness. This device offers in-the- moment monitoring and prompt alerts to stop accidents brought on by driver weariness. In this paper, well go into the underlying ideas of a drowsy driver detection system and examine its various parts, functions and supporting technology. It uses cutting-edge sensors and sophisticated algorithms to continuously analyse different driver-specific characteristics, including eye movements and facial expressions. The eye tracking technology, which records and examines the driver's eye movements, is one of the main parts of this system. With the aid of this technology, it is possible to spot key signs of drowsiness such prolonged eye closures, frequent blinking, and eye gaze patterns that indicate

   inattention. Additionally, the system uses facial expression analysis to look for indicators of exhaustion and drowsiness. The technology can recognise tiny variations in face expressions including drooping eyelids, yawning, and changes in facial muscle movements thanks to sophisticated picture processing capabilities. In order to get the drivers attention, the system launches an alert mechanism when it notices indicators of sleepiness. These notifications urge the driver to act right away, such as stopping for a rest or handing the wheel to a co- driver. This technology offers a proactive strategy to address driver weariness, decrease accidents, and save lives by utilising advanced sensors.

2. PROBLEM STATEMENT

   The challenge is to create an effective and precise drowsy driver detection system that can accurately track a driver's level of alertness in real-time and issue prompt warnings to avert potential accidents brought on by weariness or sleepiness.

   Detection Accuracy- The system's ability to identify driver weariness or drowsiness must be highly accurate. It ought to be able to distinguish between normal levels of alertness and drowsiness, accurately spotting the first indications of exhaustion to avert mishaps.

   Real-time Monitoring- The driver's actions and physiological indications should be continuously and in-the-moment monitored by the system. It should be able to examine multiple types of data, including facial expressions and eye movements. Adaptibility to different conditions- The system should be able to adjust to various driving circumstances, such as changing weather, road kinds, and lighting conditions.

   User-friendly interface- Drivers should be able to grasp and use the system with ease thanks to its user-friendly interface. It should be transparent about the driver's state of attentiveness, suggested actions, and any potential concerns related to intoxication.

3. LITERATURE SURVEY

   1. Background

      Drowsy driver detection system using deep learning has been a topic of interest for researchers for a long time. Various methods have been proposed for drowsy driver detection. It was found that face detecting with facial landmarks is the most effective method.

      The system works on the frames captured by the web-cam on the computer machine or built-in camera on a laptop. By creating the video capture object, the system will capture video using the webcam in real-time. The camera should be positioned in a way so that it can see the users face in the right positions.

      The objective of this paper is to spot and sound an alarm for any probable tiredness and improper head positions of a person utilising the camera.

   2. Literature Survey

   In the previously proposed system by P. Inthanon and S. Mungsing, Detection of Drowsiness from Facial Images in Real- Time Video Media using Nvidia Jetson Nano they have used OpenCV for image processing and for detecting and extracting features from images, such as corners, key points and edges. In our system we will be using MediaPipe which provides a higher-level framework with built-in components and pipelines, making it more suitable for real-time applications. In this system the Eye Aspect Ratio is calculated taking the Euclidean distance between the vertical landmarks of the eye, the horizontal landmarks of the eye and the landmarks defining the eye width. The EAR is interpreted as a sign of closed or partially closed eyes, which may suggest tiredness or distraction, if it drops below a specific threshold. The Mouth Aspect Ratio is calculated by taking the Euclidean distance between the upper and lower lip, between the corners of the mouth and between the landmarks defining the mouth width. The MAR indicates that the mouth is open if it rises beyond a particular threshold, which is helpful for spotting yawning or other mouth-related behaviours. If the MAR in the provided code rises beyond a particular threshold, an alarm that denotes an open mouth is triggered.

   Infrared cameras are used due to their capacity to record thermal data. Thermal imaging allows an infrared camera to identify and follow the driver's face. The camera can function in dim light or total darkness because it senses the heat that the face emits. This eliminates the need for extra lighting while enabling continuous driver face monitoring. But finding an infrared camera is challenging. Without this type of camera, it is challenging to effectively identify face gestures in dimly lit areas.

   Additionally, since the eyes and mouth are necessary for indicating tiredness, it is impossible to identify it when they are covered. To overcome this our proposed system takes into consideration the head movements of the driver. The system ca recognise abnormal or typical head movements linked to tiredness by examining the driver's head position, which includes the rotation and tilt of the head. One sign of exhaustion or loss of awareness might be a drooping tilted head.

4. PROPOSED SYSTEM

   1. Overview

      The proposed system has the following components:

      A number of variables and objects, including the Mediapipe FaceMesh solution, drawing requirements, and flags to track head movements are initialised. The collected frame is scaled, transformed to RGB colour space, and then processed for face detection as part of the pre-processing stage.

      The system extracts facial landmarks and detects faces in the frame using the dlib library. It determines whether the eyes are closed or blinking by analysing the eye aspect ratio, which is computed using landmarks. The eyes must be closed for a specific amount of time before an alert is sent. The mouth aspect ratio is computed using landmarks to determine whether the mouth is open. A particular amount of time that the mouth is open results in an alert being sent.

      Facial landmarks are extracted using the Mediapipe FaceMesh solution, and the rotation angles of the head are determined using these landmarks. To detect whether the head is looking left, right, up, or down, the system looks for specified angles. Long-term head postures cause alerts to go off. Results are visualised using visual indicators and alerts for recognised face characteristics and head movements.

   2. Models

   The Mediapipe library is used for Facemesh detection and tracking. It is utilized in the following steps:

   -For the lost detection and tracking confidence the Mediapipe FaceMesh solution is created with a set of predetermined parameters.

   -The FaceMesh solution is used to process the input image in order to find and follow face meshes.

   -In order to determine head movement, additional analysis is performed using the collected landmarks.

   -The rotation and angles of the head are computed using the 2D and 3D coordinates of the landmarks on the face.

   -Mediapipe is also used to add landmarks and outlines to the image for visual representation.

   The Dlib library is used for face detection and facial landmark extraction. It is utilised in the following steps:

   -The pretrained file is used to load the Dlib shape predictor model.

   -The source image is converted to grayscale.

   -Faces are found in the grayscale image using the Dlib face detector.

   -The aspect ratios of the mouth and eyes are determined using the facial landmarks.

   -On the image, the mouth and eye contours are drawn.

   -Based on the determined aspect ratios and head movements, additional analysis is carried out.

   -The outcomes are shown on the image, along with alerts for unusual head motions.

   Overall, dlib provides the functionality for face detection, landmark extraction, and subsequent analysis of facial features.

   ALGORITHM

   # Import the required libraries

   # Set up MediaPipe FaceMesh

   # Set up constants and variables

   # Set up other components

   # Set up video capture while True:

   # Read and preprocess the video frame

   # Run face detection and landmark extraction

   # Analyse head movement for subject in subjects:

   # Iterate through the detected facial landmarks

   # Calculate head movement angles

   #Apply thresholding or rules to determine head movements

   # Visualize head movement

   # Play audio alert

   # Display the frame

   # Check for termination

   # Clean up

5. WORK DONE AND ANALYSIS Here are the key components-

   -Face Detection: To find the driver's face in the video stream or

   camera frames, the system employs a face detector (such as Dlib's face detector). This guarantees that the driver's face will be examined during the future study.

   -Facial Landmark Detection: After the face has been located, the system uses Dlib's shape predictor to locate certain facial landmarks like the eyes, mouth, and nose. These landmarks reveal details regarding the driver's eye blink, EAR, MAR, and head movements.

   -Eye Aspect Ratio (EAR): By monitoring the ratio of distances between specified landmarks on the driver's eyes, the system determines the Eye Aspect Ratio (EAR). Eye closures that signify tiredness or exhaustion are picked up by the EAR. The driver's eyes are closed or partially closed, which indicates an alert if the EAR drops below a predetermined threshold.

   -Mouth Aspect Ratio (MAR): The device measures the ratio of distances between particular locations on the driver's mouth to determine the Mouth Aspect Ratio (MAR). The mouth openings that signify yawning or preoccupation are picked up by the MAR. The MAR indicates that the driver's mouth is open and raises a warning if it rises above a predetermined threshold.

   -Head Movements: Utilising 3D projection techniques, the system keeps track of the driver's head's rotation angles. It can recognise head movements such as left, right, up, and down by examining the rotation angles. Unusual or erratic head motions could be a sign of tiredness or distraction.

   -Alert System: The technology sends out an alarm to the driver to get their attention when it finds indications of distraction, tiredness, or strange head movements. The alarm might be a sound that is played using a tool like Pygame. mixer, for example.

   -Eye Blinking: This system gives an alert message if the eye is opened for more than some threshold value, by using eye land marks.

   -Real-Time Processing: The system continuously monitors the driver's status by processing video frames or camera information in real-time. It examines each frame to find head movements, mouth openings, and eye closures, enabling prompt alerts and action.

   -User Interface: The system may include a user interface that shows the video feed together with visual cues for detected head movements, mouth openings, and eye closures. For the driver's improved knowledge and comprehension of their status, it can also display alerts and notifications.

   -Integration: The suggested solution can be employed independently of or in conjunction with already installed driver aid technologies in vehicles. To improve driver safety and lower the risk of accidents brought on by driver fatigue or distraction, it can complement existing safety features like lane departure warning systems or driver attention monitoring systems.

   -Customization and Optimization: Based on particular needs and scenarios, the system can be modified and optimised. You can change the threshold values for the EAR, MAR, and head movement angles to suit different people or driving situations. Models can be trained using machine learning techniques for increased adaptability and accuracy.

   Fig 1. Looking towards the right

   Fig 2. Looking towards the left

   Fig 3. Looking up

   Fig 4. Looking Down

   Fig 5. Looking Forward

6. CONCLUSIONS

In conclusion, the creation of a system to identify drowsy drivers using deep learning methods is a viable answer to a serious problem with road safety. It uses computer vision techniques to examine the driver's facial features, paying close attention to the motions of the eyes and mouth to spot indicators of drowsiness. The system successfully recognises and tracks facial landmarks in real-time by utilising the Dlib package for pattern detection and shape prediction. To detect how wide open the driver's eyes are, the system uses Eye Aspect Ratio (EAR). It accomplishes this by calculating the Euclidean distances between particular eye area landmarks. Additionally, using the same methodology as EAR, the Mouth Aspect Ratio (MAR) is calculated to evaluate the degree of mouth opening. The system emits alert messages in the form of audio when the EAR and MAR surpass a specific threshold to indicate drowsiness. It attempts to improve road safety and lower the probability of accidents brought on by driver weariness by instantly alerting the driver when indicators of sleepiness are discovered.

REFERENCES

[1] Phanikrishna B, V., & chinara, S. (2020). Time Domain Parameters as a feature for single-channel EEG-based drowsiness detection method. 2020 IEEE International Students Conference on Electrical,Electronics and Computer Science (SCEECS).doi:10.1109/sceecs48394.2020.61

[2] Inthanon,P., &Mungsing, S. (2020).Detection of Drowsiness from Facial Images in Real- Time Video Media using Nvidia Jetson Nano.2020 17th International Conference on Electrical Engineering/Electronics, Computer, Telecommunications and Information Technology (ECTI-

CON).doi:10.1109/ecti-con49241.2020.9158235

[3] Baek, J. W., Han, B.-G., Kim, K.-J., Chung, Y.-S., & Lee, S.-I. (2018). Real-Time

Drowsiness Detection Algorithm for Driver State Monitoring Systems. 2018 Tenth International Conference on Ubiquitous and Future Networks (ICUFN).doi:10.1109/icufn.2018.8436988

[4] Suhaiman, A. A., May, Z., & A.Rahman, N. A. (2020). Development of an intelligent drowsiness detection system for drivers using image processing technique. 2020 IEEE Student Conference on Research and Development (SCOReD).doi:10.1109/scored50371.2020.9250948

[5] Garg, H. (2020). Drowsiness Detection of a Driver using Conventional Computer Vision Application. 2020 International Conference on Power Electronics & IoT Applications in Renewable Energy and Its Control (PARC).doi:10.1109/parc49193.2020.236556

[6] Lee, K. H., Kim, W., Choi, H. K., & Tae Jang, B. (2019). A Study on Feature Extraction Methods Used to Estimate a Drivers Level of Drowsiness. 2019 21st International Conference on Advanced Communication Technology (ICACT).doi:10.23919/icact.2019.870192

[7] Fouzia, Roopalakshmi, R., Rathod, J. A., Shetty, A. S., & Supriya, K. (2018). Driver Drowsiness Detection System Based on Visual Features. 2018 Second International Conference on Inventive Communication and Computational Technologies (ICICCT).doi:10.1109/icicct.2018.847320

[8] R. Jabbar, M. Shinoy, M. Kharbeche, K. Al-Khalifa, M. Krichen and K. Barkaoui, "Driver Drowsiness Detection Model Using Convolutional Neural Networks Techniques for Android Application," 2020 IEEE International Conference on Informatics, IoT, and Enabling Technologies (ICIoT), 2020, pp. 237-242, doi: 10.1109/ICIoT48696.2020.9089484.

[9] M. Ngxande, J. -R. Tapamo and M. Burke, "Detecting inter-sectional accuracy differences in driver drowsiness detection algorithms," 2020 International WISAUPEC/RobMech/PRASA Conference, 2020, pp. 1-6, doi: 10.1109/SAUPEC/RobMech/PRASA48453.2020.9041105.

[10] Gwak, Jongseong, Akinari Hirao, and Motoki Shino. 2020. "An Investigation of Early Detection of Driver Drowsiness Using Ensemble Machine Learning Based on Hybrid Sensing" Applied Sciences 10, no. 8: 2890.