Human Fall Detection using Deep Learning and IoT

Human Fall Detection using Deep Learning and IoT

Sreelakshmi K. S.*

*M.Tech,

Department of Electronics & Communication Engineering, Vidya Academy of Science & Technology,Kerala, India

Sojan Francis P**

**Asst. Professor,

Department of Electronics & Communication Engineering, Vidya Academy of Science & Technology, Kerala, India

AbstractIn recent years, technologies like Internet of Things (IoT) and mobile communications have been developed to collect data about people and the environment, which is then used for a variety of applications. In remote monitoring of elderly and disabled people living in smart homes is extremely challenging due to probable accidents which might occur by daily activities such as falls. Fall is considered as a major reason because it may cause death in case of post-traumatic complication for aged people. As a result, early detection of elderly people falling in smart homes is required to increase the persons survival rate. Nowadays, the arrival of artificial intelligence (AI), IoT, wearables, smartphones, etc. makes it ease to design fall detection identification systems for smart homecare. A consistent fall detection will help to find the human fall at that moment and it can lead to provide timely treatment for that person. Existing wearable devices may cause user discomfort. In this aspect, the proposed project presents a fall detection system using a camera, which monitor the person activities and fall event. Proposed system introduces an IoT enabled elderly fall detection model using deep convolutional neural network based Inception v3 model (CNN-Inception v3) for smart homecare. Multiple cameras fall dataset and UR fall dataset are used applied to get the proposed model.

Index TermsSmart homecare, fall detection, artificial intelli- gence, elderly people, deep learning.

1. INTRODUCTION

   Falls are a main risk factor of injury for old aged people and it is a significant barrier to seniors independent living. They are a leading causes of injury-related hospitalizations in people who aged 65 years and above. According to the previous statistical results, at least one-third of people aged 65 and up fall one or more times per year [1]. After a fall incident occurred, an injured elderly person may be left on the ground for several hours or even days. Frequently, the individual might not be able to rise up with no assistance or support and might require immediate medical consideration. Also there is a fact that, fear of fall is generated or associated with the fall event. So especially for elder people who have experienced falls earlier, definitely will have a tendency to avoid from doing daily physical activities. It create a negative feelings of helplessness in their minds if no one is there. For preventing the serious consequences of this fall, continuous orconsistent fall detection is required.

   Human fall detection system observe and classifies daily

   life activities of human to identify an unintentional fall. To distin- guish human falls, various sensors and techniques have been used to classify daily activities. Researchers have classified fall detection systems into three categories based on cameras, wearable devices, and ambiance sensors. Among the wearable devices, accelerometer is the most widely used method to realize a fall. It uses the measure of the acceleration of the body to classify falls. Clifford et al patented a human body fall detection system using accelerometer, a processor and a wireless transmitter . The processor uses accelerometer mea- surements to determine if the person with wearing the device is falling and there is a non-movement phase followed by the fall. The generated response is then remotely transmitted to a signal receiver by a wireless transmitter [2].

   There are research being undertaken to determine human fall using the posture movements. Body orientation as posture movement is used to detect a fall using either posture sensors or multiple accelerometers. Kaluza et al presented a posture based fall detection algorithm using the ideology of recon- struction of an objects posture. The posture reconstructed in a 3D plane by locating the wireless tags which were placed on body parts (sewn on clothes) such as shoulders, ankles knees, wrists elbows and hips. Some tags are also placed at specific positions such as bed, chair, sofa, table to identify some postures such as lying on bed or sitting on chair.. The fall detection algorithms use acceleration thresholds along with velocity profiles. Acceleration is derived from the movements of the tags. Acceleration and accurate velocity calculation is subject to the tags localization precision [3]. Kangas et al used a waist worn tri-axial accelerometer, transceiver and microcontroller to develop a new fall detector prototype based on fall associated impact and end posture [4].

   Later, Li et al presented a novel fall detection system using both accelerometer and gyroscopes. By using two tri-axial accelerometer at separate body locations they are able to recognize four kinds of static postures: standing, bending, sitting and lying. Motions between these static postures are considered as dynamic transitions and if the transition before lying posture is not intentional, a fall is detected. Whether motion transitions are intentional or not is determined by the linear acceleration and angular velocity measurements [5].

2. RELATED WORKS

This section refers existing models related to human fall detection system. Faisal Hussain and Azam proposed a system based on an activity-aware fall detection and recognition based on wearable sensors [6]. Wearable sensors and systems, as technology advances, provide a valuable way to continuously monitor the elderly in order to detect any fall incidents that may occur. The majority of these wearable fall monitoring systems are only concerned with detecting a fall incident. However, in order to avoid the risk of a future fall, it is also necessary to understand the cause of a fall incident. The proposed schemes performance is investigated through a series of experiments using three machine learning algorithms: K- Nearest Neighbors, Support Vector Machine, and Random Forest.[6]. The proposed methodology achieved the highest accuracy for fall detection, i.e.,

99.80 percentage, using K- Nearest Neighbors(KNN) classifier, whereas the highest ac- curacy achieved in recognizing different falling activities is

96.82 percentage using Random Forest classifier.

To get more feasible model, an energy-efficient fall detection method was proposed by Liu and Dong based on FD-DNN for elderly people [7]. It is a fall detection module is an important component of community-based care for the elderly to reduce their health risk. It necessitates the detection accuracy while also conserving energy. To meet the above requirements, an energy- efficient sensing module-integrated sensor that can sense and cache human activity data in sleep mode was developed., and an interrupt-driven algorithm is proposed to transmit the data to a server integrated with ZigBee [7]. Second, on the server, a deep neural network for fall detection (FD-DNN) is carefully designed to detect falls accurately.

The FD-DNN algorithm, which combines convolutional neural networks (CNN) and long short-term memory (LSTM), was tested on both online and offline datasets. The experimental result shows that it takes advantage of CNN and LSTM, and achieved 99.17% fall detection accuracy, while its specificity and sensitivity are 99.94% and 94.09%, respectively. Mean- while, it has the charcteristics of low power consumption.

1. Carletti and A. Greco proposed a smartphone-based system for detecting falls using anomaly detection[8]. According to the World Health Organization, falls are a serious medical and financial problem; they are the second leading cause of unintentional injury death, after traffic accidents. As a result, there has been a significant increase in interest in developing fall detection systems in recent years. Although the overall architecture of such systems in terms of the basic components are consolidated, defining an effective method to detect falls is a difficult problem due to several difficulties that arise when the system is required to work in the real world.

   A recent research trend is focused on the realisation of fall detection systems that run directly on a smartphone, avoiding the inconvenience of purchasing and

   transmitting additional devices [8]. Here presents a novel smartphone-based fall detection system that treats falls as anomalies in comparison to a model of normal activities. Our method is compared to other very recent approaches in the state of the art, and it is shown to work on a smartphone placed in a trouser pocket. This result is supported by both the achieved accuracy and the hardware resources required.

   later Jin Zhang and Cheng Wu proposed a model for Human Fall Detection Based on Body Posture Spatio- Temporal Evo- lution.[9] Generally the vision-driven fall event detection has the huge advantage of being non- invasive. However, in actual scenes, the fall behaviour is diverse, resulting in high detection instability. The paper proposed a new model of human posture representation of fall behaviour, called the five-point inverted pendulum model, based on a study of the stability of human body dynamics, and employs an improved two-branch multi- stage convolutional neural network (M-CNN) to extract and establish the inverted pendulum structure of human posture in real-world complex scenes [9]. Create a spatio-temporal evolution map of human posture movement using multimedia analytics to observe time series changes in human inverted pendulum structure. Finally, using the combined results of computer vision and multimedia analytics, we reveal the visual characteristics of the spatio-temporal evolution of human pos- ture in a potentially unstable state, as well as investigate two key features of human fall behaviour: motion rotational energy and generalized force of motion. The experimental results in actual scenes demonstrates that the method has strong robustness, wide universality, and high detection accuracy.

   Chalavadi Vishnu and colleagues constructed a fall motion mixture model(FMMM) for human fall detection in surveil- lance video mentioned in figure 1, using fall motion vector modeling [10]. But here some fall and non-fall events cannot be separate due to some visual cues of videos.

   Fig. 1. Vision Based Systems [11].

   III. PROPOSED SYSTEM

   In this section, the design and methodology of the pro- posed system is explained. Initially have a separate model description on inception CNN [12]. There is a research on RNN application for IoT for deep

   learning [13] The proposed system explains a model for human fall detection system via a smartphone notification is send to the caretaker of the elderly. If the input is a video of human action is related to fall then , at that moment the model identifies and predicts the possibilityof fall detection of human.

   The model proposed in this design includes a CNN- RNN based video classification deep learning algorithm. This model is a base structure for the real time implementation like figure

   1. For model building, initially a set of videos inserted into the model classified as fall and Non-fall videos. Within that deep learning process, a number processes are done and the model is created according to the objective. In testing process, insert a video and process the input video within the model. If the video is a human fall then a smart phone notification will sent via node mcu module. IoT system includes the interfacing of the proposed model with the smartphone wireless notification. The model building takes place with a set of dataset of fall events and Non-fall events.

      1. Block Diagram

         Fig. 2. Functional Block Diagram

         According to figure 2, initially a video camera intro- duces as a starting element which captures human presence. This crucial part of capturing videos goes to further processing procedure. The Video frames passes into a feature extraction by a CNN based inception v3 model to generate useful set of feature vectors or feature descriptors. Detection of the occurrence of fall and non-fall events done by a RNN based classification model. A non-fall event detected and it is represented as class 0, no alarm will be transmitted and the event occurrence is discarded. A fall event detected and it is represented as class 1, an alarm is transmitted to enables the smartphone to generate an alert to the caretakers on the occurrence of fall event. Validate the fall detection performance of the proposed model on UR fall detection (URFD) and multiple cameras fall dataset(MCFD).

      2. Methodology

         The overall working process of the proposed model is a video based human fall detection by IoT and deep learning using CNN-RNN architecture represented in figure 3. It in- volves different subprocesses and they are given below:

         • Data collection- UR fall detection (URFD) [14] and multiple cameras fall dataset(MCFD) [15] are te main two different set of dataset collected for the

           proposed system modeling.

         • Set Up- Datasets separated as train and test. And then undergoes different preprocessing stages.

         • Define Hyperparameters- Hyperparameters defines some variables such as IMG SIZE is the height and width of the image that we extract from the video. As neural network needs images to be of same size.

         • Data Preparation- The data downloaded is saved in a folder. The folder contains videos of different lengths. Then classified as two files train.csv and test.csv which only have location of the video in one column and label of video in the second column.

           The downloaded videos can be grouped under the name of video name and mention the corresponding activity or status included in the videos in the group tag.

         • Feature extraction- CNN based inception v3 model isused to extract features.

         • The sequence model- RNN sequence model undergoesclassification process with the fall and Non-fall events.

         • Inference- After model building, testing of model is takesplace here.

           In the proposed system primarily, the input videos are cap- tured and are stored into a file folder for additional processing where the proposed model gets executed. The video frames are then split and pre- processed. In three major levels such as resizing, augmentation, and normalization is used to improve the quality of video frames. Afterwards, the features from the video frames are extracted to derive useful feature vectors using Inception v3 model. Here in this process the conversion of feature vectors to corresponding array of feature values takes place. Subsequently, the feature values are fed into the RNN based sequence classifier model to detect the occurrenceof falls.

           Fig. 3. The working process of proposed model

           Based on the trained model, the subsequent actions will be performed. The following actions are taken based on the classification outcome value:

         • When an event is detected as a fall incident and is denoted as label 0, a notification is transmitted to

      the patient device from where the caretaker can be notified automatically if the fall was not excluded from the application by the monitored person.

      • When an event is detected as non-fall incident and is represented as label 1, no alarm will be transmitted and the event occurrence is discarded.

   /ol>

   1. RESULTS AND DISCUSSION

      In this section, the output and the different analysis carried out for the proposed system were discussed.

      1. Model Performance Analysis

         The CNN-RNN model trained and observe the trained accuracy of the model. Depends on the epoch value the trained accuracy formed to plot the model performance analysis. It is the graph plotting of trained accuracy vs epochs. For epoch value 15, accuracy obtained and plotted in the figure 4.

         Fig. 4. Model Performance Analysis

      2. Inference

         Deep learning inference is the process of using a trained CNN-RNN model to make predictions against previously unseen data. As explained above, the training process actually involves inference, because each time an image is fed into the CNN-RNN model during training, the deep neural network attempts to classify it.

         Here inference applies knowledge from a trained neural net- work model and a uses it to infer a result. So, here a new unknown data set(test data) is input through a trained neural network, it outputs a prediction based on predictive accuracy of the neural network. If the test data is a fall event then a smartphone notification sent to blink app and it represented as seen in the figure 5.

         Fig. 5. fall notification on blink application

   2. CONCLUSION

   This paper has designed a new CNN-RNN combined video classification model to detect fall events in smart homecare of elderly people. The CNN-RNN model allows IoT devices and intelligent deep learning algorithms to detect the occurrence of falls in the smart home. Once the fall is identified, an immediate notification is sent to the caretakers. There by proper timely treatment can be provided to the person who undergoes the fall event. The Inception v3 model was chosen using a convolutional neural network and RNN sequence algo- rithm helps to considerably improve the overall fall detection performance. An small set of simulations is carried out on UR fall detection dataset and multiple cameras fall dataset. Since different human have different falling behavior the model will provide less efficiency. Need more dataset and classification procedure.

   In future, the fall detection performance of the proposed model can be improved by the use of advanced deep learning models for classification process. Besides, scalable and robust versions of the model can be developed to assist real time fall detection events from low-quality videos. More number of collected data related to fall and Non fall events creates more accurate model.

   REFERENCES

   [1] Heinrich, S., Rapp, K., Rissmann, U., Becker, C., Konig, H.-H.: Cost of falls in old age: a systematic review. Osteoporosis Int. 21, 891-902 (2010)
   [2] Clifford, M.A., R.L. Borras, and L. Gomez, System and method forhuman body fall detection. Google Patents, 2007.
   [3] E. J. Curry, K. Ke, M. T. Chorsi, K. S. Wrobel, A. N. Miller, A. Patel, et al., Biodegradable piezoelectric force sensor, Proceedings of the National Academy of Sciences, vol. 115, no. 5, pp. 909-914, January 2018.
   [4] Kangas, M., et al., Sensitivity and specificity of fall detection in people aged 40 years and over. Gait & posture, IEEE, 29(4): p. 571-574. 2009.
   [5] Li, Q., et al. Accurate, fast fall detection using gyroscopes and accelerometer-derived posture information. in Wearable and Implantable Body Sensor Networks Sixth International Workshop on IEEE, 2009.
   [6] Faisal Hussain and M. A. Azam, Activity-Aware Fall Detection and Recognition based on Wearable Sensors ,IEEE, Feb.2019.
   [7] L.Liu and R.Dong, An energy-efficient fall detection method based on FD-DNN for elderly people, International Journal of Advanced Robotic Systems, vol. 11, 2019.
   [8] V. Carletti and A. Greco, a smartphone-based system for detecting falls using anomaly detection, ICIAP pp 490-499, 2019.
   [9] Jin Zhang, ORCID,Cheng Wu, and Yiming Wang Human Fall Detection Based on Body Posture Spatio-Temporal Evolution, Intelligent Sensing Techniques in Ambient Intelligence) (ISTAI), 2020.
   [10] Chalavadi Vishnu, Rajeshreddy Datla, Debaditya Roy, Sobhan Babu and

   1. Krishna Mohan Human Fall Detection in Surveillance Videos Using Fall Motion Vector Modeling, IEEE Sensors Journal, Volume: 21 Issue: 15, 2021.

   [11] https://www.mdpi.com/robotics/robotics-09-

   00055/article deploy/html/images/robotics-09-00055-g002.png [12] Shahna Sherin P, Anju.J Human Fall Detection using

   Convolutional Neural Network, IJERT, Volume 08, Issue 06 , june 2019.

   [13] Mohammed Farsi Application of ensemble RNN deep neural network to the fall detection through IoT environment, Alexandria Engineering Journal Volume 60, Issue 1, Pages 199-

   211, February 2021,.

   [14][Online]. Available: http://fenix.univ.rzeszow.pl/ mkepski/ds/uf.html The First 18 Years of Basic Research and Its Clinical Application, Clinical orthopaedics and related research, vol. 242, pp. 12-25, 1989.
   [15] E. Auvinet, C. Rougier, J. Meunier, A. St-Arnaud, and J. Rousseau, Multiple cameras fall dataset, DIRO-Universit de Montral, Montreal, QC, Canada, Tech. Rep. 1350, Jul. 2010.