An Assistive System for Visually Impaired using Raspberry Pi

An Assistive System for Visually Impaired using Raspberry Pi

Isha S. Dubey

Electronics and Communication

SNDT University Usha Mittal Institute of Technology, Maharashtra

Jyotsna S. Verma

Electronics and Communication

SNDT University Usha Mittal Institute of Technology, Maharashtra

Ms. Arundhati Mehendale

Electronics and Communication

SNDT University Usha Mittal Institute of Technology, Maharashtra

AbstractThe paper briefs about a different combination of a reading machine (OCR), virtual assistant and Domotics system using Raspberry-Pi which will be a combination of a great system. This is a helpful aid for visually impaired people and people with disabilities. OCR stands for optical character recognition where it recognizes the present text and converts them into audio speech using pre and post processing with gTTS (Google Text to Speech). Google is used as its platform for virtual assistant which can be used in day to day life activities like checking mails, weather-forecast, news etc., further using Google Assistant, and python language we implement a voice based home automation. The major objective of this project is to help visually impaired by using various fields of technology. By just voice commands tasks such as reading of document, home automation and personal assistant can be achieved.

Index TermsOptical Character Recognition (OCR), gTTS, Virtual Assistant, Home Automation/Domotics, Raspberry-Pi and Python.

1. INTRODUCTION

   According to the World Health Organization (WHO), around 285 million people around the world are estimated to be visually impaired, out of which 90 percent live in developing countries [1]. Due to blindness visually impaired people are not able to read the paper which creates the major issue for blind population which are facing many problems, also people with disabilities are not able to control home appliance by their own which too leads to an another problem in itself.A major problem for a blind or visually impaired person (BVI) to interact with the world to share knowledge [2].This can be reduced by implementing a combination of three versatile technology i.e. ocr, personal assistant and home automation which is an great device for the people living their day to day life.In the Survey based on smart home system using Internet of things, a Smart central controller acts as an interfacing device between household devices and Internet server.The household devices will be connected to a switch module in such a way that when the switches change their state even the state of the devices change [8].

   Tesseract is an OCR engine that supports unicode and has the ability to recognize more than 100 languages. The tesseract library yields text from the image [5].Text extraction is conversion of the information/text which is carried out by Pytessseract-ocr which is a library and it consists of 2 major steps i.e. pre processing and post processing where

   Fig. 1. Connections of Raspberry Pi 3 B+

   extraction, segmentation and recognition takes place where the final information is received which is then converted into speech or audio output by using different speech engine such as gTTS, E-Speak though gTTs is more accurate than TTs.

   Additionally a Google based personal assistant is combined with system for upgrading the feature and providing basic service of day to day routine which works on enabling the API (Application Program Interface) of Google console. Google voice-hat acts as heart of the system which is used for making Google based assistant. Further Domotics i.e. Home automation system is proposed which uses Google assistant for its activation and deactivation of different

   appliances present at home by using voice commands which completed using python programing, we have also used a door sensor that uses Reed switch to detect if the door has been closed properly or not which ensure safety of the owner.

2. RELATED WORK

   In this paper, Personal and Intelligent Home Assistant to Control Devices Using Raspberry Pi , [1] authors explains about the designing and implementing this personal home assistant in same device, which can access internet and can control to home device.

   Interactive home automation System with Google assistant,

   1. author has explained about device focuses on controlling the devices using voice recognition and artificial intelligence by using API of Google.

      Voice Controlled Personal Assistant Using Raspberry Pi,

   2. where authors can explain about voice as an input to convert into text using a speech to text engine.

   An intelligent virtual assistant using raspberry pi , [4] implementation of a Voice Command System as an Intelligent Virtual Assistant (IVA) that can perform numerous tasks or services for an individual.

   Optical Character Recognition Robot , [5] where author explains about a web controlled robot that recognizes and converts textual messages placed in real world to the computer readable text files.

   IOT based home automation by using Personal Assistant,

   [6] explains about he smart devices and sensors in home automation help collect (or sense) the physical experience and convert it into information data.

   The author in Home Automation with Personal Assistant [7] explains about advance iot used for visually impaired.

3. SYSTEM REQUIREMENTS

   1. Operating System

      Raspbian OS is a Debian-based computer operating system for Raspberry Pi.

   2. Language

      Python is a well-defined object oriented programming lan- guage for general-purpose programming which wascreated by Guido van Rossum and rst was released in 1991. It has been designed in such a way that emphasizes code readability, notably using signicant whitespace. It provides a platform to constructs a clear programming stage on both

      small and large scales.The current python version which is used is 3.7.0 which was released on 27th June 2018.

   3. Library

      Tesseract-OCR, Pillow-PIL, gTTs, playsound, picamera, RPi.GPIO.

   4. IDE

      Python 3.5 has its own IDE (Integrating Development Environment) which was launched on 22nd of September 2014 which has various variety of packages like image processing, scale plotting, Text processing and other different packages which are present as default in it.

      Fig. 2. Flowchart of OCR

   5. Raspberry pi

      Raspberry Pi 3 B+ is a mini computer developed in UK which is credit card sized single board used as foundation for various core implementation of different projects.The system uses monitor(HDMI), keyboard, speakers which are connected with raspberry pi using GPIO pins and camera is interfaced with raspberry pi.

      Features:

      1. Cortex-A53 (ARMv8) 64-bit SoC @ 1.4GHz,Broadcom BCM2837B0.

      2. Bluetooth 4.2.

      3. Extended 40-pin GPIO header with 4 USB 2.0 ports. 4). high-quality 2.5A power supply.

   6. Pi-Camera

      The Raspberry Pi Camera Board can be plugged directly into the CSI(Camera Serial Interface)connector on the Rasp-berry Pi. It is capable for delivering a crystal clearimage

      Fig. 3. Block Diagram

      of 5MP resolution or having 1080p HD video recording at 30frames per seconds. The device itself is tiny which are around 25mm x 20mm x 9mm, and weighs just over 30 grams, which makes it perfect for various projects and other applications where size and weight matters. The sensor present in it has a native resolution of 5 megapixels, and has a fixed focus lens which is present onboard

   7. Voice Hat

      The Voice HAT cnnects the Raspberry Pi to the Google API (Application Program Interface) and Speech-to-Text ser-vice, which is allowing you to ask questions and get reply as its service. The AIY (Artificial Intelligence Yourself) voice HAT helps set up Google Assistant SDK which is useful for hot-word detection.The HAT includes on-board hardware to facilitate audio capture and playback, connectors for the dual microphone daughter board and speaker, GPIO breakouts to connect low-voltage components like micro-servos and sensors, and an optional barrel connector for dedicated power supply.

   8. Reed switch

   Switches are an electromechanical device [30] that is used to control flow of electricity. It is basically a device which is used for door sensor, when the door is open/close it indicates circuit break depending upon the type of configuration we choose (NO/NC). In our project we have used it in door sensor, so suppose when door is left open, we will get email that door is not closed properly.

4. SYSTEM ARCHITECTURE

   In this section the implementation of system is explained, which will be useful for visually impaired people and people with disability. The voice will be recognized then it will allow users to control appliances and also OCR to speech will beimplemented which act as audio machine for reading books,

   Fig. 4. Raspberry Pi with Google Assistance

   paper etc. the goggle assistant will help to search anything, play music as well as it will tell the location with date, day and time.For example, if an elder or disable person needs to turn on a light or switch off a fan, he/she can do it on his own without any physical difficulty with the help of smart home power management system [4].

   1. OCR module

      OCR is a task that consists of extracting text from images, hardcopy or scanned document so that it can be saved in editable soft copy, in other words, its like a TXT or DOC file from a scanned JPG of a printed or handwritten page. With the help of a digital camera OCR technology is widely used for data reading. They consist of step by step process these are as follows. OCR pre-processes images to improve the chances of successful recognition.The text is recognised by the Tesseract library in segments or parts. This is the character recognition part it is done by training and testing, it will compare the text with vector and find the closest

      match. Post processing will ensure that the recognised word makes sense grammar wise, spelling and accuracy.

   2. TTS module

      TTS is a technique that converts text into voice output. It will read aloud the text document, such as a help file, Web page, novels, product labels or instructions at a public place. In our project we will use gTTS i.e. Google Text-to-Speech engine since it has clear sound and is open source software. Other open source engines such as eSpeak, Wordtalk, etc. can also be used. When we combine OCR and TTS it can function as reader machine of the document that we present in front of the camera.

      Fig. 5. Setup of Raspberry Pi 3 B+

   3. Virtual assistant

      It is also called AI assistant or digital assistant performs task given by voice command by users. Virtual assistants are cloud-based programs that require internet connection and to perform task. There are also devices dedicated to providing virtual assistance. The most popular ones are available from Amazon, Google and Microsoft. It turns on when we say the wakeword of that device aloud. Google Assistant can come up with answers when you converse with it. OK Google or Hey, Google is the wakeword for Google AI Assistant.

   4. Home Automation

   Home automation is the process of controlling home appli- ances automatically using various control system techniques. The electrical and electronic appliances in the home such as fan, lights, outdoor lights, fire alarm, kitchen timer, etc., can be controlled using various control techniques. Which is also known as smart home, advantage of Smart Homes is comfort and convenience. There are various techniques to control appliances like using Cloud service, smart phones, remote, using IoT i.e. Internet of Things

5. IMPLEMENTATION

   Refer fig no. 4, 5 and 6, that is the implemented part of project that looks like as follows:

   fig no. 4 is setup of OS installation which will be shown on screen.

   fig no. 5 is our Raspberry Pi connected to Google Assistance, The box is our Assistant, components such as Voice HAT, Microphone, Speakers and Button are all enclosed in the box for it to carried around with ease.

   fig no. 6 is our output for Optical Character Recognition, The text printed on the paper in front of camera is detected and is shown in green color at top of screen in form of output.

6. FUTURE SCOPES AND APPLICATIONS

   OCR can be used in banking industry it is to handle cheques

   1. a handwritten cheque is scanned, and it is converted into digital text by which the signature is verified. It can handle a huge variety of scripts, from Arabic to Indian scripts to Japanese kanji.

      Further in future it can be used for other Indian languages and there is heavy demand for an OCR system which recognizes cursive scripts and manuscripts like Palm Leaves.Google assistant is used for scanning QR codes, alarm reminders, and notifies date and time. It has a wide scope in future in terms of artificial intelligence and controlling everything by just a word.

      By using Google assistant we can make path following robots, driverless car which will work on Google commands. It can also be used for making candy Dispenser system.The most commonapplications of home automationarelighting control,outdoor lawn irrigation controlling system, controlling kitchen appliances, andsecurity surveillance systems.This system can also be used for making a robot which would act as road guiding robot with an inbuilt stick in it which will convert the image to text to speech and accordingly notify about the obstacles around the person.

7. CONCLUSION

In this paper we have proposed a multifunctional system which act as an aid for enhancing the quality of ones life. This project is made on various platformslike python with its various packages which is programed and run on Raspberry Pi 3B+ model. By implementing this system we are performing image to text to speech conversion, Google based personal assistant with home automation. It was observed that personal assistants like Amazon Echo failed to provide the user with a sense of control as it sometimes remained unresponsive when given voice commands that are not valid and these personal assistant devices are expensive [7].

ACKNOWLEDGEMENT

It gives me immense pleasure to express my deep gratitude and sincere thanks to Dr. Shikha Nema, H.O.D. , and all faculty members from Department of Electronics and Com- munication, Usha Mittal Institute of Technology, SNDT, Juhu for their valuable and useful support and comments for making this workshop a successful event. Ild not forget to mention that their approach kept my working environment alive and their encouragement promoted me to do my task rigorously.

REFERENCES

1. S. Sonth and J. S. Kallimani, OCR based facilitator for the visually challenged, 2017 International Conference on Electrical, Electronics, Communication, Computer, and Optimization Techniques (ICEECCOT), Mysuru, 2017, pp. 1-7.

2. S. A. Sabab and M. H. Ashmafee, Blind Reader: An intelligent assistant for blind, 2016 19th International Conference on Computer and Infor-mation Technology (ICCIT), Dhaka, 2016, pp. 229-234.

   Fig. 6. OCR output

3. M. Owayjan, A. Hayek, H. Nassrallah and M. Eldor, Smart Assis-tive Navigation System for Blind and Visually Impaired Individuals, 2015 International Conference on Advances in Biomedical Engineering (ICABME), Beirut, 2015, pp. 162-165.

4. S. Rizvi, I. Sohail, M. M. Saleem, A. Irtaza, M. Zafar and . Syed, A Smart Home Appliances Power Management System for Handicapped, Elder and Blind People, 2018 4th International Conference on Computer and Information Sciences (ICCOINS), Kuala Lumpur, 2018, pp. 1-4.

5. A. G. Sareeka, K. Kirthika, M. R. Gowthame and V. Sucharitha, pseudo-Eye Mobility assistance for visually impaired using image recognition, 2018 2nd International Conference on Inventive Systems and Control (ICISC), Coimbatore, 2018, pp. 174-178.

6. M. Rajesh et al., Text recognition and face detection aid for visually impaired person using Raspberry PI, 2017 International Conference on Circuit ,Power and Computing Technologies (ICCPCT), Kollam, 2017, pp. 1-5.

7. V. Chayapathy, G. S. Anitha and B. Sharath, IOT based home automation by using personal assistant, 2017 International Conference on Smart Technologies for Smart Nation (SmartTechCon), Bangalore, 2017, pp. 385-389.

8. M. Dsouza, N. Wilfred, R. Pereira, T. Rayen and A. Telgote, Home automation using Internet of Things, 2017 International Conference on Energy, Communication, Data Analytics and Soft Computing (ICECDS), Chennai, 2017, pp. 559-561.

9. S. M. Felix, S. Kumar and A. Veeramuthu, A Smart Personal AI As-sistant for Visually Impaired People, 2018 2nd International Conference on Trends in Electronics and Informatics (ICOEI), Tirunelveli, 2018, pp. 1245-1250. doi: 10.1002/j.1538- 7305.1955.tb01474.x

10. G. Iannizzotto, L. L. Bello, A. Nucita and G. M. Grasso, A Vision and Speech Enabled, Customizable, Virtual Assistant for Smart Environ-ments, 2018 11th International Conference on Human System Interaction (HSI), Gdansk, 2018, pp. 50-56.

11. V. Kpuska and G. Bohouta, Next-generation of virtual personal assis-tants (Microsoft Cortana, Apple Siri, Amazon Alexa and Google Home), 2018 IEEE 8th Annual Computing and Communication Workshop and Conference (CCWC), Las Vegas, NV, 2018, pp. 99-103.

12. N. K. Sawant and S. Borkar, Devanagari Printed Text to Speech Conversion using OCR, 2018 2nd International Conference on I- SMAC (IoT in Social, Mobile, Analytics and Cloud) (I-SMAC)I- SMAC (IoT in Social, Mobile, Analytics and Cloud) (I-SMAC), 2018 2nd International Conference on, Palladam, India, 2018, pp. 504-507.

13. H. Rithika and B. N. Santhoshi, Image text to speech conversion in the desired language by translating with Raspberry Pi, 2016 IEEE International Conference on Computational Intelligence and Computing Research (ICCIC), Chennai, 2016, pp. 1-4.

14. N. K. Srivastava and S. Singh, Netra: Smart Hand Gloves Comprises Obstacle Detection, Object Identification OCR Text to Speech Converter for Blinds, 2018 5th IEEE Uttar Pradesh Section International Con-ference on Electrical, Electronics and Computer Engineering (UPCON), Gorakhpur, 2018, pp. 1-4.

15. S. Deshpande and R. Shriram, Real time text detection and recog- nition on hand held objects to assist blind people, 2016 International Conference on Automatic Control and Dynamic Optimization Techniques (ICACDOT), Pune, 2016, pp. 1020- 1024.

16. P. Harmo, T. Taipalus, J. Knuuttila, J. Vallet and A. Halme, Needs and solutions – home automation and service robots for the elderly and disabled, 2005 IEEE/RSJ International Conference on Intelligent Robots and Systems, Edmonton, Alta., 2005, pp. 3201- 3206.

17. J. Lee, B. W. Yogatama and H. Christian, Optical Character Recognition for Handwritten Mathematical Expressions in Educational Humanoid Robots, 2018 IEEE 8th International Conference on System Engineering and Technology (ICSET), Bandung, 2018, pp. 178-183.

18. A. KARTHIK, V. K. RAJA and S. PRABAKARAN, Voice Assistance for Visually Impaired People, 2018 International Conference on Com-munication, Computing and Internet of Things (IC3IoT), Chennai, India, 2018, pp. 465-468.

19. C. Peng and R. Chen, Voice recognition by Google Home and Rasp-berry Pi for smart socket control, 2018 Tenth International Conference on Advanced Computational Intelligence (ICACI), Xiamen, 2018, pp. 324-329.

20. Y. Lin, Y. Hsu and Y. Kao, A Smartphone-Based Chinese DAISY Dig-ital Talking Book Player for Visually Impaired, 2012 Sixth International Conference on Genetic and Evolutionary Computing, Kitakushu, 2012, pp. 296-299.

21. T. Parthornratt, D. Kitsawat, P. Putthapipat and P. Koronjaruwat, A Smart Home Automation Via Facebook Chatbot and Raspberry Pi, 2018 2nd International Conference on Engineering Innovation (ICEI), Bangkok, 2018, pp. 52-56.

22. D. Vasicek, J. Jalowiczor, L. Sevcik and M. Voznak, IoT Smart Home Concept, 2018 26th Telecommunications Forum (TELFOR), Belgrade, 2018, pp. 1-4. doi: 10.1109/TELFOR.2018.8612078.

23. S. Mohapatra, S. ham Rout, V. Tripathi, T. Saxena and Y. Karuna, Smart Walking Stick for Blind Integrated with SOS Navigation System, 2018 2nd International Conference on Trends in

    Electronics and Informatics (ICOEI), Tirunelveli, 2018, pp. 441- 447.

24. P. M. ee, S. Santra, S. Bhowmick, A. Paul, P. Chatterjee and A. Deyasi, Development of GUI for Text-to-Speech Recognition using Natural Language Processing, 2018 2nd International Conference on Electronics, Materials Engineering Nano- Technology (IEMENTech), Kolkata, 2018, pp. 1-4.

25. M. R. Lee, F. H. S. Chiu and C. Zhuo, Novel design of a social mobile robot for the blind disabilities, Proceedings of the 2013 IEEE/SICE International Symposium on System Integration, Kobe, 2013, pp. 161-166.

26. M. A. Kutlugn and Y. irin, A novel approach improvement framework for text to speech synthesis, 2018 26th Signal Processing and Com-munications Applications Conference (SIU), Izmir, 2018, pp. 1-4. doi: 10.1109/SIU.2018.8404828.

27. M. P. Agrawal and A. R. Gupta, Smart Stick for the Blind and Visually Impaired People, 2018 Second International Conference on Inventive Communication and Computational Technologies (ICICCT), Coimbatore, 2018, pp. 542-545.

28. A. Sen, K. Sen and J. Das, Ultrasonic Blind Stick for Completely Blind People to Avoid Any Kind of Obstacles, 2018 IEEE SENSORS, New Delhi, 2018, pp. 1-4. doi: 10.1109/ICSENS.2018.8589680.

29. Z. U. Din and W. David Lim, Home automation with smart robot featuring live video feed and remotely controlled switches, 2013 IEEE Conference on Sustainable Utilization and Development in Engineering and Technology (CSUDET), Selangor, 2013, pp. 1-6.

30. O. M. Hovgaard and G. E. Perreault, Development of reed switches and relays, in The Bell System Technical Journal, vol. 34, no. 2, pp. 309-332, March 1955.

31. N. Bourbakis, Automatic Image-to-Text-to-Voice Conversion for In-teractively Locating Objects in Home Environments, 2008 20th IEEE International Conference on Tools with Artificial Intelligence, Dayton, OH, 2008, pp. 49-55.