Smart Communication Framework for Blind , Deaf and Dumb

Smart Communication Framework for Blind , Deaf and Dumb

Authors, Savita B Patil , Jeevan Gowda B , Manjula A , Nagalakshmi B G Sri Krishna Institute of Technology, Bengaluru, Karnataka 560090

ABSTRACT – The capacity to talk, see, hear, and respond appropriately is one of the most priceless human gifts. Some people lack this. Creating a single compatible device for people with hearing, vision, and speech impairments is a difficult task, and communicating these people with the general public has always been the most difficult task. For the mute, deaf, and visually impaired, this suggested system integrates a new framework for innovative personal communication systems into a single adaptive device. With the help of her Tesseract OCR (Online Character Recognition) technology, this suggested gadget converts text to audio, enabling blind persons to read words online. To do. make it easier to read.Text-by-Voice or Speech-to-Text Conversion (STP) and the ability to read text or paragraphs to blind people, converting scanned images to text or text to language It can be used by scanning images with a Logitech camera or converting (TTS).

I . INTRODUCTION

Symbolic language is the most difficult form of communication between deaf and deaf people and the outside world, enabling improved and simpler lifestyles for fools, deaf and blind people. A proposed system was designed and developed. About 290 million people worldwide are visually impaired. Of the 290 million people, 60 million are blind, There are 255 million individuals who are blind. Only Braille can be read by blind persons. This ground-breaking tool was created to facilitate learning for blind individuals. This cutting-edge technology uses a camera to take pictures and then transforms the pictures into text and voice. With this device, even visually impaired people can easily read the characters. In this digital age, approximately 9.1 billion people are deaf.

These deaf, deaf and blind people face many problems in communicating with ordinary people in their daily life. Due to their disabilities, these people are not integrated into this digital world. Now they communicate with gestures using their knowledge of symbolic language. This is a carefully thought- out symbolic gesture, and each move has a distinct significance.Gestures are non-verbal communication skills that use the movements of hands, head, and other body parts. People with disabilities have a hard time communicating with ordinary people because ordinary people cannot understand symbolic language. Because of this, many stupid people use normal types of symbolic conversational language to get their messages and news out to the outside world, and they don't have a personalized symbolic vocabulary. As such, there are still many communication gaps between the deaf, deaf and blind and the world. Despite the large number of deaf-mute people, little research has been done to overcome communication barriers. Stupid people who communicate effectively and smoothly with each other. To overcome these barriers for the visually impaired and speech impaired, the proposed system was developed using a small credit card sized computer called the Raspberry-Pi 3 Model B. This proposed system provides technology for deaf, deaf, and blind people to communicate with each other, as well as ordinary people. For the visually impaired, scan the image with a Logitech webcam, convert it to text using Tesseract OCR, save the converted text to a folder in WordPad format, save the resulting text to E – Convert to speech using Speak technology. The converted text will also appear in WordPad.

. System Requirements :

1. Raspberry Pi

   The model we're using is the Raspberry Pi-3 Model B, a microcomputer. It has a Bluetooth version of 4.1, 10/100 Base Ethernet, IEEE standard 802.1x Wi-Fi, and offers a frequency range of 2.4 GHz.

   Fig 1 : Raspberry Pi-3 Model B Front view

2. Camera

   The webcam used in the suggested model is a Logitech C270. It has a built-in microphone, a 5-megapixel resolution, and background noise cancellation technology. It uses a XVGA video recording technology. 1024 by 768 pixels.

3. Tesseract OCR engine

   Tesseract OCR is a free Optical Character Recognition (OCR) software available for various operating systems. A process can be used to electronically extract text from images, and the converted text can be reused in various ways for document editing and text-to-speech conversion. OCR technology can be used to convert documents such as scanned paper, PDF files, and captured images into editable data. Tesseract is available for operating systems such as Windows, Linux, and Mac OS. Programmers can use the API to extract typed or printed text from images. The Tesseract OCR installation process consists of two parts: engine and language training data. Tesseract is available directly from your Linux distributor on Linux.

   operating system. The latest version of Tesseract OCR is 4.1.1.

4. Open C V

   Real-time computer vision is the primary emphasis of OpenCV, a cross-platform library and open-source tool with a range of programming capabilities. He was created by the Intel Research Centre, afterwards supported by Willow Garage, and is now being maintained by Itseez. The primary interface for OpenCV is written in the C++ computer language. Python, MATLAB, and Java are used for binding. Various operating systems, including Windows, Linux, MacOS, Open BSD, iOS, and Blackberry, can run Open CV. It may be utilised for many different things, including segmentation, mobile robots, item identity, detection of gestures, identification of faces, and more. It combines the programming languages Python, C++ API, and Open CV.

5. eSpeak

   eSpeak is Google's open source software. It is used for text-to-speech translation and supports many languages, mostly English. On platforms such as Windows and Linux, "formant synthesis" techniques are used in e- Speak. This method makes it simpler to provide numerous languages in a compact format. The converted speech is rather precise and clear, but it lacks the smoothness and naturalness of big synthesisers built from sound recordings of human voice.

6. VNC Viewer

VNC or Virtual Network Computing is a graphical desktop sharing system. It uses the Remote Frame Buffer (RFB) communication protocol. Mouse and keyboard events are sent from one computer to another. VNC is platform-independent software that can be accessed remotely as needed. Connect to a remote computer using VNC technology using the VNC server application. Inputs collected by VNC Viewer are sent to a VNC server for remote control access.

III. Implementation and Methodology Block Diagram

Block diagram of the proposed model. In this system, microphones and cameras are input devices. The microphone is used to capture audio input and the camera is used to capture images as input. Loudspeakers or headphones and monitors serve as output devices. Loudspeakers are used for audio output and monitors are used to display text and image output. A RaspberryPi-3 is used to control all these devices.

Fig 2 : Block diagram of the proposed system.

This proposed model behaves like a mobile phone-like wearable device that allows users to send and receive text messages. When users interact with others using this device, their input is through the microphone. The device converts the speech to text (STT) and the text is displayed on the device's LCD screenWhen an individual types a text message reply on the screen, the system convert it to speech using text-to-speech (TTS) and plays it back through the speaker.. The data flow diagram of the proposed system is shown below. The model has four options for choosing the desired transformation technique.

• Text-to-speech (TTS)

• Image-to-Speech or Text (ITST)

• Gesture to Speech (GTS)

• Text-to-speech (STT)

Fig 3 : Data flow diagram of the proposed model 1.

Text-to-Speech

What's Next Text-to-speech technology helps people with voice disabilities who can't speak normally like other people. Foolish people convey messages and information in text form that is converted into audio signals. The converted audio signal is output as output. Output is read by Google Synthesizer through headphones or speakers

Fig 4 : Text To Speech

1. Image reading using the camera

   This type of conversion is intended for visually impaired people who cannot see normal text. To assist these disabled people, the system is connected to a web camera that captures images with text, and the images are scanned using OpenCV image processing tools and image processing library packages.Scanned images are converted to text using Tesseract OCR technology and saved in a text file with a .txt extension. Paragraphs are rephrased into short sentences before saving. OCR uses adaptive linear thresholding techniques to convert a text image into a binary map image, which is converted into character outlines. The converted text is read aloud by E-Speak and the output can be heard through speakers or headphones.

   Fig 5 : Image To Speech

2. Speech-to-Text

   Fig 6. Speech-to-Text

   This conversion technology is designed to help people who are deaf or unable to identify voices. To help them, the proposed model is equipped with a speech-to-text converter. Converts the voices of the common people into text. A microphone is used to take audio as input and conversion to text format is done via an API.

3. Gesture control

Gesture conversion consists of converting gesture language to text or speech. This technique is used to help people with speech disabilities who use symbols to convey messages and information to others. An image of the displayed gesture is captured, scanned, cropped, and converted to a grayscale image to enhance features. To acquire a specific region of the form, a Gaussian threshold function is used to blur the scaled picture. Finger shape and angles are recognised. To determine the amount of flaws, count the number of angles that are less than 90 degrees.Based on the total number of errors, a message or text is determined and displayed on the device's

screen and output as audio output through the speaker.

Fig-7: Gesture Control

. CONCLUSION

Technology has played a very important role in our lives. We use it almost everywhere and all the time. The clear and rapid developments we discover every day prove to us that there is no point in giving up the obstacles in life and struggling. It offers important solutions. Our task is to use it appropriately to achieve a level of success that benefits individuals, society and the nation as a whole. Various prototype models are all developed into a single, easy-to-use device. The main advantage of the designed model is that it is a small and compact device that can be easily carried anywhere. It also supports hand gesture recognition with certain limitations. It can be further improved by implementing gesture recognition for numbers and letters. It can be further improved by implementing gesture recognition for numbers and letters. It can be extended to take a video as input, split it into frames, scan for text readings, and convert to text or audio format. This device is used to convert text/image to speech for the blind, speech to text for the deaf, and hand gestures to text for the deaf. .

V. REFERENCES

[1] M Delliraj, Design of Smart e-Tongue for the Physically Challenged People , 2013 International Conference on Recent Trends in Information Technology (ICRTIT), 2013.

[2] Fernando Ramirez-Garibay ,MyVox Device for the Communication Between People: Blind, Deaf, Deaf- Blind and Unimpaired., IEEE Global Humanitarian Technology Conference (GHTC 2014)

,2014.

[3] Pushpanjali Kumari, Pratibha Goel, S. R.

N. Reddy PiCam : IoT Based Wireless Alert System for Deaf and Hard of Hearing, 2015 International Conference on Advanced Computing and Communications (ADCOM),2015.

[4] R Suganya;T. Meeradevi ,Design

of a Communication Aid for Physically Challenged, 2015 2nd International Conference on Electronics and Communication Systems (ICECS),2015.

[5] L.Anusha;Y. Usha Devi

,Implementation Of Gesture Based Voice And Language Translator For Dumb People, 2016 International Conference on Communication and Electronics Systems

(ICCES),2016.

[6] B. Rajapandian;V. Harini;D. Raksha;V.

Sangeetha, A Novel Approach as an AID for Blind, Deaf And Dumb People, 2017 Third International Conference on Sensing, Signal Processing and Security (ICSSS) ,2017.

[6] Jinesh A Shah;Aashreen Raorane;Akash Ramani;Hitanshu Rami;Narendra Shekokar EYERIS: A Virtual Eye to Aid the Visually Impaired, 2020 3rd International Conference on Communication System, Computing and IT Applications (CSCITA),2020.

[7] Sartha Tambe,Yugchhaya Galphat,Nilesh Rijhwani, Aishwarya Goythale, Janhvi Patil,Analyzing and Enhancing Communication Platforms available for a Deaf-Blind user, 2020 IEEE International Symposium on Sustainable Energy, Signal Processing and Cyber Security (iSSSC) ,2020.