Scientific research project

Abstract

Drowsiness is a common problem that many drivers encounter due to long working hours, lack of sleep, and tiredness. Tired drivers are as dangerous as drunk drivers because they have slower reaction times and suffer from reduced attention, awareness, and ability to control their vehicles. Drowsy driving causes many traffic accidents, especially fatal crashes. Therefore, the best way to prevent accidents involving drowsiness is to alert the drivers ahead of time. The accuracy of the drowsiness prediction reduces if the studies only focus on facial landmarks, ignoring other fatigue features such as tilting head, blinking, and yawning. To solve these problems, we propose an approach to detect driver drowsiness efficiently and accurately using IoT and deep neural networks improved from LSTM, VGG16, InceptionV3, and DenseNet. The use of transfer learning technique combined with multiple drowsiness signs is to improve the accuracy of the drowsiness detection in various driving conditions. The time-varying factor is also taken into consideration in the models developed from LSTM and DenseNet. When the driver's fatigue is detected, the IoT module emits a warning message along with a sound through a Jetson Nano monitoring system. The experimental results demonstrate that our approach using deep neural networks can achieve high accuracy of up to 98%. Notably, this approach has also been verified in cases with/without wearing a mask and glasses. This has a practical meaning in the Covid-19 pandemic situation when everyone needs to comply with the wearing of masks in public places.

Introduction

Feeling abnormally sleepy or tired during the day is commonly known as drowsiness. Drowsiness may lead to additional symptoms, such as forgetfulness or falling asleep at inappropriate times. This is a natural phenomenon in the human body that causes distraction and affects the lives of road users. According to statistics from the US National Highway Traffic Safety Administration, 50,000 injuries and nearly 800 deaths have been reported with 91,000 traffic accidents related to drowsiness [1]. According to the National Sleep Foundation, in 2005, 60% of drivers committed drowsy driving in the previous year [2] and an estimated of 6,400 people died annually in crashes involving drowsy driving [3]. The Foundation for Traffic Safety reported that 21% of all fatal crashes involved a drowsy driver from 2009 to 2013 [4]. About 1/25 drivers admitted that they were drowsy driving in the last 30 days according to the Centers for Disease Control and Prevention [5]. In the first quarter of 2021, there were approximately 8,730 car accident fatalities in the United States [6]. The above alarming statistics have shown the necessity to implement a system for driver drowsiness monitoring and alerting, thereby preventing unfortunate traffic accidents from happening. Recently, many models have been developed for automatic drowsiness detection systems. The inputs to the systems are images obtained from a camera that will be used in detection models to conclude whether the driver is asleep. The general model of the system is shown in Fig. 1.

Figure 1. Model of the drowsiness detection system based on facial landmarks

Table 1. Illustration of drowsiness detection and prediction on a camera for four scenarios

Proposed Method

Figure 2. Proposed method for drowsiness detection with real-time image sequence analysis

1. Preprocessing: In step 1, images are extracted from videos related to drowsy driving from large datasets. The extraction rate of images from videos is 25 frames per second. We then perform face detection from the image dataset using SSD network (Single Shot MultiBox Detector) with a backbone like ResNet-10 [33]. It can detect faces at different angles in a fast computation time. We then normalize the face images to the size 224 x 224 to generate a face dataset. As a result, in this step, we obtain the dataset with many different features and signs of the face and head area. Finally, we proceed to divide this dataset into two sub-datasets of drowsy and non-drowsy states.

2. Feature extraction and training: The training dataset will be passed through the proposed deep neural networks in step 2 for feature extraction and training. We design and perfect the deep neural network models for drowsiness detection developed on LSTM, VGG-16, Inception-V3, and DenseNet by improving some of their layers. Details of the improvements are presented as follows.

Results

Figure 3. Illustration of drowsiness detection and predition on videos for four scenarios

Figure 4. Average accuracy and time of drowsiness prediction for four scenarios

Drowsiness is one of the main causes of accidents alongside with other causes like drunk driving, distractions, and so on. In order to overcome this issue, we construct a driver drowsiness detection system using deep learning combined with IoT to be able to detect, alert and potentially save a person’s life. For our system, a surveillance camera is used to capture the images of the driver’s activities and the entire system is incorporated using Jetson Nano. When the driver is drowsy, the IoT module emits a warning message along with impact of collision, location information, and a sound through a Jetson Nano monitoring system.

Fig 5. IoT based driver drowsiness detection system.

Conclusions

In this work, we address a drowsy driver alert system that has been developed based on multiple behavioral signs using IoT and deep learning techniques. We propose a deep learning based approach for drowsy detection and prediction of drivers by designing and perfecting four adaptive neural networks developed on LSTM, VGG16, InceptionV3, and DenseNet. We take advantage of the pre-trained networks, add some adaptive layers, and then apply the transfer learning approach to be able to appropriate to our research. This helps to shorten training time, avoid over-fitting, and improve the accuracy of drowsiness prediction. The proposed networks analyze the driver's signs of drowsiness and learn all the characteristics of the drowsy state. They take advantage of the deep learning neural networks to extract all drowsiness features to detect and predict the state of drowsiness accurately. In addition, we perform the experiments on four scenarios. Experimental results show that the training accuracy is up to 98% and these scenarios are feasible and suitable for the development of drowsiness warning applications. Moreover, we provide the test accuracy in the cases of with/without wearing a mask and glasses that have not been provided by previous studies. We also make a comparison of our methods and recent methods. It shows that the proposed networks could be advantageous because they can be trained faster and more efficiently, especially with limited hardware.

References