IoT based Assistive Smart Mirror with Human Emotion Recognition System

IoT based Assistive Smart Mirror with Human Emotion Recognition System

Lucky Tater Sunidhi Pranjale

Department of Computer Engineering, Department of Computer Engineering, Smt. Kashibai Navale College of Engineering Smt. Kashibai Navale College of Engineering,

Savitribai Phule Pune University, Savitribai Phule Pune University, Pune-41, India. Pune-41, India.

Siddhant Lade Aaryaneil Nimbalkar

Department of Computer Engineering, Department of Computer Engineering, Smt. Kashibai Navale College of Engineering, Smt. Kashibai Navale College of Engineering,

Savitribai Phule Pune University, Savitribai Phule Pune University, Pune, India. Pune-41, India.

P. N. Mahalle

Department of Computer Engineering, Smt. Kashibai Navale College of Engineering,

Savitribai Phule Pune University, Pune-41, India.

AbstractThe Internet changed our lives by associating us all the more effectively to data and others in the virtual world. Cell phones at that point became cell phones and from that point forward this idea has ejected and transformed into the Internet of Things, things which interface us to regular articles. There is no closure of articles that could be made "more brilliant", some being more fit to this than others. Mirrors, for instance, give a huge surface perfect to showing data and collaborating with. A great many people have reflected at home so the idea of a shrewd mirror that you can connect with is alluring and has been fantasized in numerous cutting-edge motion pictures. Brilliant mirrors, for example, Magic Mirror and Home Mirror have as of late begun to be created by individuals in the Maker people group, with fluctuating degrees of intelligence. The objective of the undertaking was to make a Smart Mirror gadget that individuals could collaborate with yet in addition to additionally build up the innovation so it would allow you to introduce and build up your own applications for it. The Smart Mirror was created in four months, beginning with the product lastly incorporating it with the equipment.

KeywordsInternet, shrewd, Raspberry Pi, Linux, mirror.

1. INTRODUCTION

   These days, everybody needs next-level simplicity in their life. They need everything to be accessible for them in the blink of an eye. IoT is an arrangement of interrelated figuring gadgets, mechanical and computerized machines, items, creatures or individuals that are given remarkable identifiers and the capacity to move information over a system without expected human-to-human or human-to-PC association. IoT makes a savvy home, which is an association of every computerized gadget required for correspondence Our saying behind this undertaking depends on the idea that we face the mirror when we

   prepare, so why not make the mirror keen and increasingly educational. Basic road for making a savvy reflect is to utilize a glass, LCD screen, edge to hold a glass and screen RPi with Python. This task is shaped with the idea to spare time. The face and emotion detection are carried out with the help of Viola Jones Algorithm. Commitments in this paper are

   • Develop a savvy reflect dependent on IoT that gives customized information like notes, camera, voice directions, data, for example, climate estimate, date, time and facial acknowledgment.

   • An effectively extensile edge of reference to coordinate shrewd apparatuses and give administration so as to make a savvy reflect.

2. MOTIVATION

   In this world anybody desires to be comfy in existence. Modern man has invented specific technology for his sake of life. The mirror become to appear to be an everyday reflect however might have a screen internal and you would be able to have interaction with it using voice instructions, hand gestures and smartphones. It has many advantages, which flawlessly fits in with IoT developments, and most importantly, it makes your lifestyles less difficult. For example, inside the residential zone, it can be used for recognizing people, learning and catering to their everyday behavior and desires, making it an important part of a smart mirror. In the car zone, smart mirrors are used as aspect view and rear-view mirrors for greater safety purposes and driving resource. Smart mirror gadgets can be powered via exceptional solutions, examples are digital glasses or earrings attempt ones. Another example is skin evaluation that may track and diagnose great pores and skin situations through digicam.

   Previous year, Amazon was granted a patent for a smart mirror that will enable buyers to try on clothes in the comfort of their home and then order online. Multiple retail chains and solution providers are testing different ways to engage their customers by providing more enjoyable and seamless shopping experience in the digital age. Many cosmetics retailers are cutting back on product costs by using virtual makeup display stands and smart mirrors so users can try a variety of products without using testers. Samsung Neon, a project with a simple goal: create completely original (i.e. not based on actors) digital avatars that converse and learn until theyre indistinguishable from humans. Neon is like a new kind of life, There are millions of species on our planet and we hope to add one more. Neons will be our friends, collaborators, and companions, continually learning, evolving, and forming memories from their interactions.

3. LITERATURE SURVEY

   In [1] portrays the structure and improvement of a keen mirror that speaks to an exquisite interface for looking data and furthermore utilized for cheat identification in a home situation. A brilliant mirror is a framework that capacities as a mirror with the extra ability of showing the date, time, current temperature, climate subtleties. To structure a shrewd mirror that gets online news and show it utilizing the Internet of things (IoT) hardware and to identify criminal when no one is in home.

   In [2] enables clients to get to data and furthermore control the lights in the house. Pertinent data can be followed, for example, time and date, climate, cautioning, traffic, and area map. The framework applies Sonus innovation as a medium of connection among individuals and frameworks. In this way, clients need to give directions to the framework orally to secure the framework's reaction. The methodological methodology utilized in this task is The Developmental Prototyping which assembles all prerequisites and plans the framework in a snappy technique

   In [3] defines the nuts and bolts, equipment, and usage of the ultrasonic sensor utilizing a raspberry pi board. This paper is essentially founded on estimating the separation of an article by the utilization of the ultrasonic sensor. For this reason, raspberry pi is utilized. At first, this was finished by utilizing chips, microcontrollers and so forth. The raspberry pi board is versatile. This paper portrays that we can gauge the separation of an item by the utilization of raspberry pi by composing a basic code in python language. A similar strategy can be utilized to locate the normal of the separation of an article in the event that we move the item. The raspberry pi can likewise be utilized for the web of things like controlling the gadgets by remote access.

   In [4] depicts an AI approach for visual object discovery which is equipped for handling pictures very quickly and accomplishing high discovery rates. The work is recognized by three key commitments. The first is the presentation of another picture portrayal called the Necessary Image" which permits the highlights utilized by

   our locator to be figure rapidly. The second is a learning calculation, in view of AdaBoost, which chooses a modest number of basic visual highlights from a bigger set and yields amazingly productive classifiers. The third commitment is a strategy for consolidating progressively increasingly complex classifiers in a "course" which permits foundation districts of the picture to be immediately disposed of while spending more calculation on promising item like areas.

   In [5] The proposed strategy accomplishes a quick and vigorous facial element extraction dependent on continuously applying channels to the picture. The proposed strategy executes the continuous feeling acknowledgment from the facial picture utilizing three steps face identification, highlights extraction and classifier for arrangement of feelings. The proposed technique employments raspberry pi for actualizing feelings acknowledgment.

   In [6] The smart mirror also controls home appliances with very less human intervention using a mobile application. For controlling home appliances. The mobile needs to be paired with the smart mirror successfully. It describes about the monitoring and controlling of house base devices with mirror. To ease the human task and develop interaction between people and system.

   In [7] Though applications of IoT are diverse, but this research paper helps in using IoT for making life easier. The mirror has the ability to display Date and time, news updates, weather conditions, to-do lists, reminders, traffic conditions etc. The machine required for computing result is a Raspberry Pi which does not require large

4. GAP ANALYSIS

   The Smart Mirrors that are already in the market have features like give information about date, time, weather forecast, news. Some of the mirrors have features like facial recognition. The idea behind this IoT based Assistive Smart Mirror is to make the mirror more interactive so that it is very easy to use. Also, voice commands are added for better experience and use, the mirror detects human emotions and displays it on the screen or through voice commands. One of the advantages is the use of Proximity Sensors (PIR), so that there is less use of power. If there is no moment in front of the mirror for at least 60 seconds then it goes to standby mode, which lessens power consumption.

   Table 1. Gap Analysis

   Parameters	[1]	[2]	[3]	[4]
   Technology	Raspberry Pi 3	Raspberry Pi 2	Raspberry Pi 2	Raspberry Pi
   Language	Python	Python	Python	Python
   Map	No	Yes	No	No
   Weather	Yes	Yes	No	No
   News	No	No	No	No
   Voice Commands	No	Yes	No	No
   Gestures	No	No	No	No
   Reminder	No	Yes	No	No
   Date &Time	Yes	Yes	No	No
   Human Emotion Detection	No	No	No	Yes

5. PROPOSED SYSTEM

   In the proposed system our aim is to provide the user with an interactive interface which can be used conveniently in home environment or office spaces. The system includes a Raspberry Pi 3, monitor, webcam and various different sensors. Users can access personal notes, reminders, calendars, news and stock updates, weather information, traffic updates etc. and use this data as and when required.

   Fig. 2 Haar Features

   1. Working

      Fig. 1 System Overview

      2. Creating an Integral Image

      For faster calculations during the matching of Haar Features the normal image is not useful. Hence, we create a digital form of the image which is used to perform all the calculations faster in the Viola Jones Framework.

      The system uses a proximity sensor which allows it to understand whether there is an object in front of it or not. Once the object is detected by the proximity sensor the Raspberry Pi 3, which is in stand- by mode, will be activated. The Raspberry Pi 3 is connected to the Monitor using a HDMI cable and the webcam is attached using a universal serial bus.

   2. Face and Emotion Detection

      The system uses the Viola Jones model of facial detection, in this model the various features of the face are considered and theses features are cross checked with a set of pre- defined rules and then the image in consideration is checked using these rules. Emotion detection is used to keep the record of the users daily emotions and draw a pattern from these findings which can be used to create an emotional profile of the user.

      Viola Jones Algorithm:

      The Viola Jones Algorithm works in four stages-

      1. Haar Features Selection

      All human faces share similar properties, these irregularities can be matched using Haar Features.

      Fig. 3 Integral Image

      1. AdaBoost Training

         The object detection framework uses a type of the learning algorithm AdaBoost which selects the best features and also trains the classifier that use these features.

         The algorithm is executed for an arbitrary number of rounds, I.

         For i = 1I

         1. Normalize the weights as follows so that wi,l is a probability distribution:

         2. For each feature j, train a classifier hj which is restricted to using a single feature. The classifiers error rate is evaluated with respect to wi,l:

         3. Choose the classifier, hi, with lowest error i. Update the weights:

            The final classifier is:

      2. Cascading Classifiers

      In cascading, each stage consists of a strong classifier, so all the features are grouped into several stages where each stage has certain number of features.

      Fig. 4 Cascading Classifiers

   3. Mobile Application

      The system also includes a mobile application which is used to update information on the mirror remotely by particular users. The data updated in the mobile application is sent to the database which is commonly accessed by the Smart Mirror as well as the mobile application. This way the users can update notes, set reminders and alarms which can then be displayed on the Smart Mirror.

   4. Display

   As the display is also supposed to be a mirror, we use a two-way mirror on top of the display for the purpose of dual functionality. The display and the two-way mirror are encased in a wooden frame which holds them together. This way when the Smart Mirrors display is switched off it can act as a normal mirror and can be used as a Smart Mirror once the display is switched on.

   Hardware Architecture: –

   The proposed system consists hardware equipment such as LCD display screen, Infrared sensor, Camera, Microphone, Speakers, Raspberry Pi, Connecting wires, Power supply (+5V), Temperature/Humid Sensor.

   Fig. 5 Hardware Architecture

   Software Architecture: –

   The proposed system consists software equipment such as Android Studio Software, any operating System which supports Android studio, Android Software Development kit, Python

   Fig. 6 Software Architecture

6. CONCLUSION AND FUTURE WORK

The Smart Mirror as a consequence accomplishes this, by in spite being a mirror without all of the technology inside it, making it very approachable to apply and integrating seamlessly into our lives. The Smart Mirror has scope in the subject of IoT and home automation. The Smart Mrror can be associated with the home automation, cell phones, and numerous others gadgets. Which can extend the functionality of the mirror. The facial recognition technology used can be future optimized as a method of protection. Adding security method that no one can attempt to get admission to sensitive statistics that maybe displayed in your mirror through the use of APIs. We believe that the destiny of the home can be a brilliantly connected atmosphere of smart generation technology designed to make your existence easier, greater enjoyable, and efficient. Obviously, there are a ton of possibilities in the home for technology integration but a mirror is one of the great locations to begin.

The scope of this proposed system is to increase the efficiency and cost for the improvement of a Smart Mirror

to reduce and probably eliminate the want for the user to make time for each day morning or night routine to check their PC, tablets, or cellphone for the facts they need. The mirror will offer the records with little to no effort from the person with the purpose of now not being a burden that he or she have to preserve. The mirror wouldnt be every other activity, alternatively an enhancement to the already common use of mirrors in maximum modern-day bathrooms. The mirror will do the thinking for the person. First, it will turn on and off by itself. Then, it will update with the users calendar agenda, to-do lists, Twitter, news, weather and other services. The data wouldnt be displayed on the users reflection but rather would be displayed on the edges of the mirror to still allow use of the actual mirror. The mirror provides common data that majority of the people check on their smartphones or tablets which includes weather, news, Twitter and other services. This allows the users to read, think, and plan their day while getting ready in the morning or night.

REFERENCES

1. Lakshmi N M, Chandana M S, Ishwarya P, Nagarur Meena, Rajendra R Patil, 2018, IoT based Smart Mirror using Raspberry Pi, INTERNATIONAL JOURNAL OF ENGINEERING RESEARCH & TECHNOLOGY (IJERT) NCESC 2018 (Volume 6 Issue 13)

2. Yusri, Muhammad & Kasim, Shahreen & Hassan, Rohayanti & Abdullah, Zubaile & Ruslai, Husni & Jahidin, Kamaruzzaman & Arshad, Mohammad. (2017). Smart mirror for smart life. 1-5. 10.1109/ICT-ISPC.2017.8075339.

3. Charulata Ingle, Measuring Conceptual Object Distance through Sensors using Raspberry pi, International Journal of Innovative Research in Computer and Communication Engineering, Vol. 5, Issue 6, June 2017.

4. Viola, Paul & Jones, Michael. (2001). Rapid Object Detection using a Boosted Cascade of Simple Features. IEEE Conf Comput Vis Pattern Recognit. 1. I-511. 10.1109/CVPR.2001.990517.

5. Priyanka, T. R. Revanth kumar Real-time Facial Expression Recognition System using Raspberry Pi, International Journal of Advanced Engineering Research and Science (IJAERS) Special Issue-4,2017.

6. Mohammed Ghazal, Tara al Hadithy, Yyasmina al Khalil, Muhammad Akhamal and Hassan Hajjdiab, a Mobile Programmable Smart Mirror for ambient IoT environment , in 5th Internation conference on future IoT and cloud workshop, 2017.

7. Jane jose, Raghav Chakravarthy, Jait Jacob, Mir Masood Ali, Sonia Maria Dsouza, Home Automated Smart mirror as IoT Implementation, International journal of advanced Research trends in engineering and technology, February:2017.

8. Muhammad Muizzudeen Yusri, Shahreen Kasim1, Rohayanti Hassan, Zubaile AbdullapHusni Rusla, Kamaruzzaman Jahidin, Mohammad Syafwan Arshad Smart mirror for smart life, 2017 6th ICT International Student Project Conference (ICT-ISPC),23- 24 May 2017.

9. Sun Yong, Geng Liqing, Dan Ke ,2018 Design of a smart mirror using Raspberry Pi , International Conference on Intelligent Transportation, Big Data & Smart City

10. Guo Liang Ye Aimin Lin Tao Fan Ruixiang "Design Of Remote Real-Time Measuring System Of Temperature And Humidity Based on Raspberry Pi and Java Language" Computer Measurement &Control vol. 25 no. 6 pp. 4-8 2017.