Magic Mirror-An Intelligent Tool for Elderly People

Magic Mirror-An Intelligent Tool for Elderly People

Magic Mirror-An Intelligent Tool for Elderly People

Prof. Keerthi Prasad G Assistant Professor Dept. of ISE, GMIT

Davangere, Karnataka

Dixit R Jain

Student

Dept. of ISE, GMIT Davangere, Karnataka

Krishna Chaitanya S Holla

Student

Dept. of ISE, GMIT Davangere, Karnataka

Rashmi M G

Student

Dept. of ISE, GMIT Davangere, Karnataka

Vanitha P

Student

Dept. of ISE, GMIT Davangere, Karnataka

Abstract:- The world of Internet transformed our lives by connecting us more conveniently to information and other people in the virtual world. Mobile phones then became smart-phones and since then this concept has erupted and morphed into the Internet of Things, things which connect us to everyday objects. There is no end of objects that could be made Smarter, some being more suited to this than others. Mirrors, for example, provide a large surface which is ideal for displaying information and interacting with. Most people have mirrors at home so the concept of a smart mirror that we can interact with is attractive. The device was to go beyond an ordinary mirror. The device was to look like a regular mirror but would have a screen inside and you would be able to interact with it. The main features would be showing basic weather, location-based time information, reminders etc.

General Terms:- Magic Mirror, Raspberry Pi 3b, Amazon Voice Services, Artificial Intelligence, Cloud Computing, Face recognition, Amazon Lambda Servies, IOT Home Automation, OpenCV, Amazon SQS service

Keywords:- Smart Mirror, Magic Mirror, Home Automation, IoT, Alexa Voice Assistant, Alexa Skill, AWS IoT sdk, Amazon Lambda Services, Face recognition

1. INTRODUCTION

   This paper concentrates on design and development of a smart mirror that represents an elegant interface for glancing information for multiple people in a home environment powered with the interaction provided by Amazon Voice services. Face-recognition based authentication is used to detect the user. It provides a web-based interface to access data feeds and other services. It is interdisciplinary work comprising of domain knowledge of Artificial Intelligence, Image Processing, Internet of Things, Cloud Platform and home automation.

   1.1 The Problem

   In our rapidly developing world, information is always right at your fingertips – on your phone, on your

   computer, maybe even on your watch. Staying connected with new information is both important for entertainment and daily life. With such a variety of options, there is difficulty in following all of your data streams. Often, during your day, you may end up in a position where it is inconvenient, or even impossible, to take out your phone or computer and check the newest update.

   You cannot commit to a slower interaction. You need a display to glance at, with the information you need ready to go. However, aesthetics is just as important as displaying information. Keeping an extra computer in your bathroom or hall would be inconvenient and would not fit well with the look of a modern room. A sleek, simple display, easy for an average consumer, is a necessity in todays world. Also, people find it friendly and easy to use when interaction is involved.

   1. Related Work

      Below we briefly comment on some related research in this direction. The Smart Mirror contains some devices equipped with a touch screen or TV enhanced externals devices. However, most of them support entertainment and some interactive tasks. The work has been reviewed as follows:

      • Mirror 2.0 [1] combines the advantages of a smartphone and a mirror. It contains an LCD display positioned behind the glass. It provides news and weather information and it allows the playback of both videos and

        music.

      • Smart Washbasin [2] displays different information in a washbasin mirror such as mails, weather forecast, the water temperature and pressure, the calendar and the users

      weight measured through a built-in-scale in the base portion. The device consists of an Android Tablet that displays the widgets on the basin mirror, made with a semi- reflecting glass put on top of an LCD display. It is possible to control it

      without touching the screen surface, since it is equipped with proximity sensors able to track the hands position and motion.

      – NEOD Framed Mirror TV [3] is a standard LCD

      screen (up to 50 inches), covered by a mirror, and specifically designed for the screen. The screen provides some TV

      functionalities, but it does not provide more interactive

      features.

      – Smart Mirror for home environment [4] allows to control all the smart devices at home. It relays on face

      recognition for authenticating the user and displays

      personalized information (news, mail, messages etc.). The system exploits a touch screen monitor and two webcams, one

      for the face recognition and one for the home surveillance.

      • Multi Display in Black Mirror [5] by Toshiba is a prototype that combines the functionalities of a tablet together

        with the reflecting surface of a mirror. It provides two

        configurations taking into account two different home environments: the bathroom and the kitchen. Considering the bathroom, the prototype provides useful information for the beginning of the day such as the weather forecast and fitness information coming from personal devices. In the kitchen, the setting includes a camera allowing the user to interact through gestures while preparing recipes and controlling the appliances.

        • The Reveal Project [6], created in the New York Times research and development, consists of an LCD Display covered by a mirror glass. The device exploits a Microsoft Kinect for tracking users movements in real-time. It

          visualizes different information on its surface (calendar, mail, news, online shopping websites, instant messenger etc.). In addition, it responds to vocal commands. A peculiar feature is the medicine box scanner, which allows the user to buy

          medicines recognizing their packages.

        • Cybertecture Mirror [7], is a complete PC contained into a 37 inches mirror, equipped with a 32 inches LCD screen. Through a smartphone application, the user accesses

      different information overlaid on the reflected image. The interface allows to visualize instant messages, the calendar,

      the mailbox, and the weather forecast. In addition, it provides

      information on the users physical state. Indeed, the device

      provides a set of external wireless sensors that allow to

      measure the users weight, fat, muscle and bone mass.

      – Interactive Mirror [8] by Panasonic seems to be an ordinary mirror: neither camera nor the other sensors suggest

      the features of a smart object. Once the user sits down in front of it, the mirror displays an enlarged frame for her face, together with menus for accessing different functionalities. The system analyses the face hydration, wrinkles and other

      details in order to recommend products and treatments to take care of her skin (e.g., to make it softer etc.), to slow ageing and so on. The mirror supports the user in buying such products. In addition, it provides make-up style previews,

      simulating lighting and ambient conditions (e.g., at home, outdoor, shopping center, etc.).

      • Connected Store Demo [9] by eBay and Rebecca Minkoff provides interactive experiences in both the store showcase and in the fitting room. Once finished, the shopper

        prepares the fitting room with all the items. Inside the fitting room, the user exploits the mirror for looking for other items and/or providing feedback. In addition, she may select some of them for buying.

      • Brushing Teeth Mirror [10] displays the information collected by a smart brush about inflammations or infections of the teeth and gum.

      Medical Mirror [11] combines computer vision and signal

      processing technique for measuring the heart rate from the optical signal reflected of the face. The prototype consists of an LCD display with built-in camera and a two-way mirror fitted onto the frame. The smart mirror recognizes the presence of a user when she stands in front of it and, after about 15 seconds, it displays the heart rate below the users reflected image.

   2. Our Solution

   Our solution is an open platform for discrete display development. We offer an aesthetically pleasing mirror, with a hidden smart display underneath. With a generic display, the mirror can be built to any size so the information can be both in your face while showing you your face. Our product differs from the competition with an easy-to-use interface that is both simple for the average user and interaction provided by smart assistant Alexa from Amazon, which can be used to control different modules of the mirror as well as provide its basic services for user queries. A sleek display gives all levels of users a modern hub of technology for their personal daily interaction, one which both displays visually all the information you could need or want and operates with a simple interaction that you could fit into your daily routine.

   By creating a platform open to modification, developers will also be able to add new functionality at their own pace. This will allow our display to be a tailorable and adaptable platform. A web application provides the interface that the user sees and interacts with. Also, an online configurator will relieve the frustration and difficulty of personalizing your information, as well as allow streamlined development of new modules. Powered by a small computer, the smart mirror will have great potential for expansion by developers. Along with these functionalities, this system can be used to control home appliances like lights, fans and Doors etc. by giving voice. Our product will be a step in the future of IoT, connecting your daily mirror to your tech-savvy world.

2. SYSTEM DESIGN

   The system is designed to offer all the required services using various components. An Acrylic two mirror is the most important component used which provides a reflection layer on front surface and a transparent layer on the back surface. Beneath the mirror a standard LCD monitor is employed, which is the displaying component of our mirror. A System on Chip (SoC) microcontroller, Raspberry-Pi 3b is used that has computational capability and ability to control other devices with the help of a relay device. An USB microphone is connected to RPI board to receive input audio from the user. A speaker is utilized to to play audio feedback of Alexa assistant. Power module is used to supply AC power for the operation of smart mirror. On receiving voice command from the user, the system fetches the corresponding information from the cloud or executes any requested services like locking the door, turning on lights, showing CCTV footage of door and controlling fans etc. Systems interface is designed in

   modular fashion and it comprises of many modules like date, time, weather and many more. User may also get multimedia playback facility like playing music and videos.

   Figure 1 represents the block diagram of a Smart Mirror, which depicts the connection of all these components integrated into the system.

   Figure 1. Block-Diagram of Raspberry Pi and Alexa enabled smart mirror.

3. METHODOLOGY

3.1 Data Flow Diagram

Data flow diagram depicts the overall functioning of the smart mirror from the stage of being turned on till its shut down. The external elements are the devices connected to RPI 3B as a part of home automation. Initially when the smart mirror is powered on, it displays the default modules. Now the face recognition module gets activated and waits for the

Figure 2. Data flow in the Smart Mirror

user to stand in front it. Once the user comes in front of mirror, it detects the authorized user and loads his personalized modules. The voice recognition system is now ready for user queries or commands.

Upon receiving commands, the audio to text conversion takes place and necessary features from the converted text is extracted. Once the intent has been extracted, corresponding services will be executed by fetching data

from the cloud APIs. The response is produced in terms of text, which is then again transformed to audio and finally the results will be displayed and spoken out. After a cycle of execution, it again rolls back to face recognition stage.

3.1 Face Recognition Algorithm

OpenCV is a popular computer vision library started by Intel in 1999. The cross-platform library sets its focus on real-time image processing and includes patent-free implementations of the latest computer vision algorithms. In 2008 Willow Garage took over support and OpenCV

1. now comes with a programming interface to C, C++, Python and Android. OpenCV is released under a BSD license so it is used in academic projects and commercial products alike.

   OpenCV 2.4 now comes with the very new Face- Recognizer class for face recognition, so you can start experimenting with face recognition right away. The currently available algorithms are:

   1. Eigenfaces (see createEigenFaceRecognizer())

   2. Fisherfaces (see createFisherFaceRecognizer())

   3. LBHS (see createLBPHFaceRecognizer()) [12]

The Smart Mirror uses Local Binary Patterns Histograms algorithm. A more formal description of the LBP operator can be given as:

, with (xc, yc) as central pixel with intensity; and being the intensity of the neighboring pixel. S is the sign function defined as:

This description enables you to capture very fine- grained details in images. In fact, the authors were able to compete with state of the art results for texture classification shown in Figure

3. Soon after the operator was published it was noted, that a fixed neighborhood fails to encode details differing in scale. So, the operator was extended to use a variable neighborhood. The idea is to align an arbitrary number of neighbors on a circle with a variable radius, which enables to capture the following neighborhoods:

Figure 3. Texture Classification

For a given Point (xc, yc), the position of the neighbor (xp, yp), p P can be calculated by:

Where, R is the radius of the circle and P is the number of sample points.

The operator is an extension to the original LBP codes, so its sometimes called Extended LBP (also referred to as Circular LBP).

If a points coordinate on the circle

doesnt correspond to image coordinates, the point gets interpolated. Computer science has a bunch of clever interpolation schemes, the OpenCV implementation does a bilinear interpolation:

By definition the LBP operator is robust against monotonic gray scale transformations. We can easily verify this by looking at the LBP image of an artificially modified image.

Figure 4. Test Faces

1. Alexa Voice Command Processing

   Figure 5. Alexa Commands processing flow diagram

   Following is an instance which shows how Alexa command works:

   1. Initially the user says, Alexa, Turn on the lamp.

   2. Raspberry-Pi uses Alexa AVS package to capture the audio.

   3. Alexa AVS sends the audio to Alexa Skills to extract

      the intent.

   4. Alexa skills then sends the intent to AWS Lambda function.

   5. Lambda function passes the extracted intent to AWS

      IoT dk hosted in it.

   6. AWS IoT sdk sends this intent to AWS IoT sdk integrated in Magic Mirror package.

   7. Mirror on receiving intent, gives command to

      Raspberry Pi to execute scripts or to change modules.

   8. The scripts control different pins of the Relay device.

   9. On receiving signal, the device connected to particular pin in relay turns on (lamp).

4. COMPONENTS OVERVIEW

1. Hardware Components

   The following are the hardware components employed to implement this work:

   1. Two-way mirror: A special mirror known as a two- way mirror or observation mirror is used in this project.

      A two

      mirror is special as compared to an ordinary household mirror. Unlike a household mirror, the two- way mirror is not painted with an opaque color on the back, instead its left untouched. This gives the property of the mirror being

      reflective one side and transparent/translucent from the other. Hence the two-way mirror acts as mirror as long as there is no light send from the back of mirror.

   2. Raspberry-Pi 3B: Raspberry Pi is a credit-card sized computer manufactured and designed in the United Kingdom by the Raspberry Pi foundation with the intention of teaching basic computer science to school students and every other person interested in computer hardware, programming and DIY-Do-it Yourself projects.

   3. Webcam: A simple USB powered webcam is used to

      recognize peoples faces.

   4. LCD panel: An LCD panel placed behind the mirror is used to present to the user the desired interface.

   5. USB Microphone: One mode of interaction with the smart mirror is through microphones. Two microphones are used to power the voice recognition capabilities of the device. USB microphones has to be used because the

      Raspberry Pi does not have regular microphone input.

   6. Speakers: Speakers are used to support audio feedback from mirror and for multimedia functionality.

   7. Relay Device: Relay device is the hardware component which acts as switch to control devices.

2. Software Components

   The following are the software packages employed to implement our work:

   1. OpenCV: OpenCV (Open Source Computer Vision

      Library) is an open source computer vision and machine learning software library. OpenCV was built to provide a

      common infrastructure for computer vision applications

      and to accelerate the use of machine perception in the commercial products. Being a BSD-licensed product,

      OpenCV makes it easy for businesses to utilize and modify the code. The library has more than 2500 optimized algorithms, which includes a comprehensive set of both classic and state-of-the-art computer vision and machine

      learning algorithms. These algorithms can be used to detect and recognize faces, identify objects, classify human actions in videos, track camera movements, track moving objects etc. OpenCV has more than 47 thousand people of user community and estimated number of downloads exceeding 7 million. The library is used extensively in companies, research groups and by governmental bodies. It has C++, C, and Python, Java and MATLAB interfaces, and supports Windows, Linux, Android and Mac OS. OpenCV leans mostly towards real-time vision applications and takes advantage of MMX and SSE instructions when available. OpenCV is written natively in C++ and has a templated interface that works seamlessly with STL containers [13]. This project will utilize the facial recognition algorithm from the OpenCV library to recognize users.

   2. Raspbian OS: Raspbian is a free operating system based on Debian optimized for the Raspberry Pi hardware [14]. Raspbian comes with over 35,000 packages, pre- compiled software bundled in a nice format for easy installation on Raspberry Pi computer.

   3. NodeJS: NodeJS is a java-script engine for server- side applications. It comes included with Electron and we used it to launch processes to control things that are not available in web APIs such as the sensors and microphones

      for voice recognition. We also used it to access the filesystem and read the app files.

   4. Python: Python is a high-level, general purpose,

      interpreted programming language. It is very popular in the Raspberry Pi community and it has lots of support and libraries. We used it with the microphone to detect sounds and also for control of sensors.

   5. Electron: Electron allows for the use of standard web technologies to implement a front end. This allowed

      the use of HTML, CSS, and JavaScript to implement the entirety of the system. Using standard and familiar technologies was important for speed of implementation as well as future users who may wish to modify the system. Using unfamiliar technologies and programming languages would increase the barrier for entry and deter potential users who may not have as much of a technological background. We also used an Electron package called

      electron-boilerplate. This package provided a faster start to application development and allows for more integrated testing as well as the potential to build release executables

      for multiple systems.

      1. CONCLUSION

         The Smart Mirror thus accomplishes all set objectives, by still being a mirror with all the technology inside it, making it very approachable to use and integrating seamlessly into our lives. The Smart Mirror has scope in the field of IoT and home automation. The Smart Mirror can be connected to the home appliances, mobile devices, etc. which can expand the functionality of the mirror. The facial recognition technology used can be future enhanced as a means of security. Adding security means that no one can try to access sensitive data that maybe displayed on your mirror via the use of APIs. The artificial intelligence embedded using Amazon Voice Services in the form of Alexa enables the user to interact with and control other modules of the mirror. We believe that the future of the home will be a brilliantly connected ecosystem of smart technology designed to make your life easier, more enjoyable, and efficient. Obviously, there are a ton of opportunities in the home for technology integration but a mirror is one of the best places to start.

      2. REFERENCES

[1] R. Grynkofi, Mirror 2.0, http://bathroominnovation.com.au/finalists#Year2013 Accessed 2016-01-23

[2] C. Seraku, Smart Washbasin, http://smartwashbasin.seraku.co.jp/english/ Accessed 2017- 01-

23.

[3] NEOD: NEOD Framed Mirror TV. http://www.neod.org/

[4] M. A. Hossain, P. K. Atrey, A. El. Saddik, Smart Mirror for ambient home environment, 2007.

[5] Toshiba: Toshiba to Unveil Leading edge Technologies at CES 2014.

http://www.toshiba.co.jp/about/press/2014_01/pr0702.html.

[6] B. House, A. Lloyd, M. Zimbalist, Reveal Project, http://brianhouse.net/works/reveal/

[7] J. Law, Cybertecture Mirror, http://www.jameslawcybertecture.com/index.php?section

=Company

[8] Panasonic: The Future Mirror. http://youtube/- 2kc9GQYIE [9] Ebay: Rebecca Minkoff Connected Store Demo.

[10] T. Mullins, Briushing Teeth Mirror, http://wwwdesignboom.com/contest/view.php?contest_pk=36

&item_pk=44258&p=1.

[11] M. Z. Poh, D. McDuff, R. Picard, A medical mirror for non- contact health monitring, In ACM SIGGRAPH 2011 Emerging Technologies SIGGRAPH 11, New York, NY, USA, ACM (2011) 2:1-2:1.

[12] Ahonen, Timo, Abdenour Hadid, and Matti Pietikainen. Face description with local binary patterns: Application to face recognition. IEEE transactions on pattern analysis and machine intelligence 28.12 (2006): 20372041

[13] ABOUT OpenCV, Accessed May 04, 2016. http://opencv.org/about.html.

[14] "Home" FrontPage. Accessed May 07, 2016. https://www.raspbian.org