Face Recognition Technology based Smart Doorbell System using Python’s OpenCV library

Face Recognition Technology based Smart Doorbell System using Python’s OpenCV library

Shweta Malve

Department of Electronics & Telecommunication KKWIEER Nashik

Maharashtra, India

Dr. S. S. Morade

Department of Electronics & Telecommunication KKWIEER Nashik

Maharashtra, India

Abstract In recent years, face recognition technology has proved to be the most popular method of contactless biometric system, outstanding the ones based on iris, voice and gait recognition. Moreover, computers can now outperform humans in terms of many face recognition tasks particularly when it comes to searching of faces in larger databases, capturing and storing minute facial details in as minimal format as possible. A system with the ability to detect and recognize faces has many potential outcomes not just in crowded places but in home security systems as well. One such application of automating the household doorbell can not only solve the security issues but also offer extra flexibility to smart house control by recognizing the person at doorstep and announcing the name. We propose a system by making use of Raspberry-Pi with ARMv8 Cortex-A53 as the core. The project aims at porting the Open Source Computer Vision (OpenCV) library to the Raspberry-Pi board and using its pre-trained classifier Haar-Cascade and recognizer Local Binary Pattern Histogram (LBPH) for face detection and recognition purpose. Furthermore, using the Raspberry-pi accelerated command line media player, name of the person will be announced and faces unknown to the database will be captured and stored. This low-cost and low-power system will indeed prove a boon for persons with disabilities.

Keywords Face recognition, biometric, raspberry-pi, OpenCV, classifiers, e-speak synthesizer

1. INTRODUCTION

   Biometric systems brought in an added layer to security to networks, applications, personal computers and physical facilities wherein the binary code of physical or biological characteristic of a persons identity is compared to the binary code electronically stored in the system. Fingerprint, palm print, hand geometry and signature fall into the category of contact based biometric systems. Face, iris, voice and gait fall into those of contactless biometric systems. Today, the mostly widely used biometric system is that of fingerprint where people just punch in their thumbs and the authentication is done. Signature biometric is highly used in government organizations. But, the problem encountered in these systems are that of change in hand geometry. Our hands have ridges, valleys, contours, principal lines and wrinkles which keep on changing to some extent during a lifetime. Also, environmental factors like dry weather, muddy or smudged hands, soaky hands can cause a great problem in authentication. So, people slowly started switching to contactless biometric systems wherein no physical contact with the system is made. Iris, voice and gait recognition have got their own pros and cons. But, in this paper we choose to go for the most researched domain face recognition. Face recognition domain has swiftly made an entry into the real

   world and has proved to be the most successful and bang on technology which is no more just in the world of science fiction. People can actually use it as a surveillance system, criminal identification, identity verification access or attendance system, home automation and much more. From 2D algorithms like PCA, Eigen faces, Fisherfaces, LDA, IDA to SVM, CNN and now deep learning, ML and artificial intelligence, face recognition technology has evolved immensely. A very big thanks to the upcoming and well- developed technologies related to computer science where we can obtain considerable features for identification purpose.

   In this paper, face recognition process is initiated by pressing the doorbell. This will turn on the integrated camera and capture images. The image captured will be compared with the one stored in the backend database. On matching, the name of the person at the doorstep will be announced. In case, the face is not present in the database, it will be stored newly. Comparing to the old traditional doorbell, this improvised one notifies us with the person at the door. This will be helpful for persons with disabilities particularly the blind people. The system design is discussed in Section 3 of the paper. Section 4 and 5 are dedicated for corresponding implementation architecture and experimental results.

2. NEED OF THE SYSTEM

   Flawless identication of individuals is a very a basic societal requirement. Just a few decades ago when technology was not at its peak, people in small tribes and villages knew everyone and recognized everyone else, just because there were manual face-to-face interactions between them. One could easily detect a stranger or identify a potential breach of security. In todays larger world where the society norms are getting more rigid, it isnt that easy. In todays age where interactions take place electronically, security becomes a crucial factor where a persons individual identity is something which cannot be compromised. Until now, magnetic swiping cards or passwords were used. But lately, these arent considered to be secure as they can be lost or easily misused by some other person. So, the solution to this was Biometric systems. In surveillance and monitoring systems, say a public place, people cannot be asked to punch in their thumbs or cards always. So, something that wont require physical touch and easy to recognize an individual same up, which is nothing but the face recognition technology. In this paper we are trying to build a system for the disabled persons. The main aim of the system is to:

   1. Notify the person in house about the visitor at the doorstep by giving a sound notification.

   2. Store the image of an unknown person.

   3. Safety and Security.

3. DESIGN CONCEPT & PROPOSED SYSTEM

   The main goal of this work is to create a smart doorbell system mainly based on human face identification. Fig.1. shows the steps involves in the face recognition method. A flowchart of the control procedures is shown in Fig.3.

   Fig.1. Steps involved in Face Recognition method

   The face recognition method is broadly classified into 3 distinct phases: Face Detection & Data Gathering, Face Extraction & Face Recognition.

   1. Face detection

      Face Detection has a very crucial aspect wherein the inputs can either be collected from surveillance videos, objects tracking, expression analysis and template matching. This phase itself has got 3 aspects to it namely pre-processing, classification and localization. Initially, in the pre-processing stage before the images are fed to the network, it is converted from RGB to Grayscale because it is easy to detect faces in grayscale. After that, the images are manipulated by resizing, cropping, blurring and sharpening. Regions that are extracted are still worked on factors like the lighting conditions, resolution, levels of zoom and orientation to tell us whether it is a face (positive) or non-face (negative) image. Note that, only the frontal view of the face is obtained in positive examples. Classification is nothing but segmentation wherein multiple objects are combined in a single image so that it is easy for the classifier to detect the objects and faces in an image. Localization step is used to search for faces in an image and if present, draw a bounding box around the region of interest (face).

      Fig.2. Haar-like features for face detection

      Fig.3. Flowchart of Sstem Procedures

      Real time face detection based on Haar-Cascade is used in this project. Fig.2. shows us the feature selection or feature extraction for Lisas image where the window is moved on the edges, line and center for detecting distance between eyes, nose, mouth etc. Next, we have to give the coordinates of x,y,w,h to draw a rectangle around the face. This is how the face is detected elsewhere in the image. Fig.4. shows us how the data is gathered from RPi camera.

      Fig.4. Data Gathered from Raspberry-Pi Camera

   2. Face Extraction

      Human-faces are extracted after detecting them from an image and stored to reduce dimensions and noise. The face image is taken, extracted and transformed into a vector with fixed dimension with space vector containing points and locations in an XML or PCD file. Both of them are considered to be training files. Fig.5. shows training the recognizer with features extracted from an image and that being converted into a trainer file so that minimum space is required for storing the facial features.

      at our end. OpenCV library is a must to support real-time computer vision applications. Haar-Cascade which is a pre- trained classifier in OpenCV library in Python was used for face detection purpose. Local Binary Pattern Histogram (LBPH) is used for face recognition purpose. Fig.7. shows us the software approach for the system.

      Fig.5. Features extracted fed to the recognizer

   3. Face Recognition

   The process of face recognition is often confused with that of face detection. After preparing the training data file, its time

   B. Hardware

   Fig.7. Software approach

   to match the stored data and the input image. Numerous face recognition algorithms are available such as Eigen faces, PCA, LBP, IDA, Fisherfaces etc. to extract the image. In this system, LBPH (Local Binary Pattern Histogram) method is used to perform face recognition. The Eigen faces and Fisherfaces determine the most dominant features of the face in the training set and more time consuming methods as compared to the LBPH, however LBPH figure out all the faces in training set individually [3]. LBPH is readily available in the OpenCV library. It represents faces extracted in a simple manner by monotonic grayscale transformations which reduces the processing time indeed. Fig.6. shows us the face recognition technique wherein a new image is fed to the system, detected first, then compared with the existing database and matched with 56% probability. So, face recognition is not just detecting but verifying the faces.

   As this work is based on Raspberry Pi 3 model B, Raspbian operating system is opted to be installed. After setting up the RPi environment, we install the libraries necessary for setting up OpenCV on it. Dlib, Face_Recognition, Imutils and Pillow libraries were installed. Thonny Python IDE which is bundled with the latest of the Raspbian PIXEL operating system is used for coding. The same software code was downloaded on the hardware. This project only entails a few hard devices: memory card, push button and a Raspberry-pi camera. Fig.8. shows the systems hardware approach.

   Fig. 8. Hardware approach

   V. TEST RESULTS

   1. Software

      Fig.6. Recognizing face from input image and trained classifier

4. IMPLEMENTATION

The project has got two facets to it : Software and Hardware. Let us have a look at each approach.

1. Software

   To start with we first adopted the software approach for this project. We installed the latest version of Python and Visual Studio which works as an integrated development environment (IDE) in our system. PIP, CMake, Pillow and Face_Recognition are some of the essential libraries installed

   Fig.9. shows us that single faces which are present in the backend database can be identified with name and those not present in the database are shown as unknown. Fig.10. shows us that two faces when shown simultaneously can be detected and recognized.

   Fig. 9. Single face identification

   Fig. 10. Two face identification simultaneously

2. Hardware

Fig.11. shows us the training images that were fed to the database. Also, we can see the training datasets of the respective images. For a single image fed in the database, the trainer gave us 30 training images. Fig.12. shows us the hardware setup wherein the RPi camera, doorbell, USB power supply is used. Headphones were used to hear the audio of the name of the visitor arrived at the door since we did not have speakers. In Fig.13. as the doorbell was pressed, the camera turned ON and the face infront of the camera was detected. Fig.14.shows us the terminal window view of face being detected and we could hear the name of the visitor on headphones.

Fig. 11. Training datasets and Training images

Fig. 12. Hardware setup

Fig. 13. Face detected and recognized

Fig. 14. Face detected shown in terminal window

VI. CONCLUSION & FUTURE SCOPE

This work is aimed to build a complete system for face recognition which is easy, low-power and cost-effective. Its utility is to be set as an alert for home visitors and provide information about the visitors for the persons with disabilities. This paper covers the analysis, design, development and testing results of the face recognition doorbell. In this project, face recognition system has been developed in order to study the potential application for home automation door security with real time response and better recognition rate. Among the other biometric techniques, face recognition approach offers one great advantage which is user friendliness.

There are some improvements to the system. Future work includes working on the environmental lighting conditions, training the classifier with more images and identifying people with masked faces as well. Further by adding a solenoid, this project can be integrated with a door-lock system too so that the door can be opened directly thus allowing a familiar person direct entry to home.

ACKNOWLEDGMENT

This project is a part of the final year Post-Graduate program Department of Electronics and Telecommunication, KKWIER, Nashik, Maharashtra, India. Grateful to the college faculty for providing assistance in every milestone of this project.

REFERENCES

1. Ayman Ben Thabet and Nidhal Ben Amor, Enhanced Smart Doorbell System Based On Face Recognition, 16th international conference on Sciences and Techniques of Automatic control & computer engineering

   – STA'2015, Monastir, Tunisia, December 21-23, 2015

2. Suraj Pawar, Vipul Kithani, Sagar Ahuja and Sunita Sahu, Smart Home Security using IoT and Face Recognition, 2018 Fourth International Conference on Computing Communication Control and Automation (ICCUBEA).

3. Vijay Kumar Sharma, Designing of Face Recognition System, Proceedings of the International Conference on Intelligent Computing and Control Systems (ICICCS 2019) IEEE Xplore Part Number: CFP19K34-ART; ISBN: 978-1-5386-8113-8.

4. Ciya James and David Nettikadan, Student Monitoring System for School Bus Using Facial Recognition, Proceedings of the Third International Conference on Trends in Electronics and Informatics (ICOEI 2019) IEEE Xplore Part Number: CFP19J32-ART; ISBN: 978- 1-5386-9439-8.

5. Alen Salihbai and Tihomir Orehovaki, DevelopmentofAndroid Application for Gender, Age and FaceRecognition UsingOpenCV, MIPRO 2019, May 20-24, 2019, Opatija Croatia.

6. Vivek Kishor Bhanse and Dr.M.D.Jaybhaye, Face Detection and tracking using Image processing on Raspberry Pi, Proceedings of the International Conference on Inventive Research in Computing Applications (ICIRCA 2018) IEEE Xplore Compliant Part Number:CFP18N67-ART; ISBN:978-1-5386-2456-2.

7. Nafis Mustakim, Noushad Hossain, Mohammad Mustafizur Rahman, Nadimul Islam, Zayed Hossain Sayem and Md. Asaduz Zaman Mamun Face Recognition System Based on Raspberry Pi Platfor, 1st International Conference on Advances in Science, Engineering and Robotics Technology 2019 (ICASERT 2019).