Two-Way Sign Language Converter for Speech-Impaired

Two-Way Sign Language Converter for Speech-Impaired

Ankita Shinde1, Raveena Dandona2

Department of Computer Engineering

Fr. Conceicao Rodrigues Institute of Technology, Vashi Navi Mumbai – 400706

Abstract – Considering the barriers faced by pupils of the speech impaired community we aim to introduce a tool which will bridge the communication gap and aid better interaction. In situations where a vocal individual is unskilled with sign language, the need for a sign-interpreter is inevitable in order to establish an interplay of expoundings. We propose a system that enables a two-way conversation between the speech-impaired and other vocal individuals. In this paper we present a prototype which is effective in two phases. In the first phase, the sign-language gestures are fed into the system in real-time through computer vision capabilities of the device. These gestures are then recognized with the help of our Deep Neural Network while the hand detection is crisped upon with an edge detection algorithm interpreting it in the text as well as audio format. The second phase accomplishes to convert audio into text and eventually displays relevant hand-gestures for the same. The system is capable of recognizing over 300 words gestured by the Indian Sign Language.

Key Words: Concurrent Neural Networks, Natural Language Processing, Machine Learning, Sign Language Converter, Computer Vision

1. INTRODUCTION

   The Sign Language is the primary mode of communication for speech-impaired. It enables them to communicate by gestures conveying their thoughts and participating in conversations. Many researchers have suggested possibilities of major linguistic challenges faced by the aforementioned community of people leading to inaccessibility of information, gaps in communication leading to uncompromisable situations of such people in the society. To put numbers in context, census 2011 revealed that of the existing 13.4 million people with disabilities in India in the employable age group of 15-59 years of which 9.9 (73.9 %) million were non-workers or marginal workers. This implied that only 26.1 % of the productive age group of the country was employed. People with hearing disabilities face many issues when it comes to employment. 31% of people feel they are treated differently because of their deafness, hearing loss and tinnitus while 33 % of people who are deaf avoid social situations because they find it difficult to communicate. 68 % of people with hearing loss feel isolated at work as a result of not being able to communicate.

   1. AIM & OBJECTIVE

      With advancements in computer science and technology it has become possible to develop an automatic interpreter which can be used to convert Speech to Text and can then

      further be translated to Sign Language Gestures that can be accessed globally. A government website www.indiansignlanguage.org has been launched for empowering the deaf and dumb, presenting a huge database of the Indian Sign Language (ISL) images. Considering the negligence in this particular domain we have aimed to develop our Two-Way Sign Language Converter for Speech-Impaired under the Data Artificial Intelligence discipline. The two phases of our system can be defined as:-

      1. Phase-I: Converting a stream of input hand gestures to their relevant semantic text as well as audio output in real time.

      2. Phase-II: Converting an audio input to text and eventually displaying the relevant hand gesture images.

   2. PROJECT OUTLINE

   Figure-1: UML diagram sign to speech for phase-1

   The first phase of this project is based on converting the real- time hand gestures to corresponding text and audio.

   Chart-1: Two-Way Sign Language Converter flow diagram

2. IMPLEMENTATION

   A dataset of 320 images was created. This dataset consisted of hand gestures for the alphabets a-z, digits 0-9 along with a few common words. All the images were flipped along the vertical axis with the purpose of obtaining a higher accuracy for both left and right handed users. The images were trained with a self-coded 7-layer concurrent neutral network.

   The accuracy achieved for Phase-II is 95%.

   We have developed a GUI but the system is still in its prototype phase. Further developments in the project include implementation of contours for the first phase to maximize the accuracy of the system and deploying the entire project as a user friendly mobile application.

   The future scope of this project is to

   • Deploy the project on cloud and create an API for using it.

   • Increase the vocabulary of our model.

   • Incorporate feedback mechanism to make the model more robust.

3. CONCLUSION

The project aimed on tackling a linguistic challenge faced by a significant number of population in India with respect to communication. The prototype is specifically designed for the speech-impaired and successfully demonstrates a solution to bridge the communication gap. The prototype can recognize 320 words and convert them to hand gestures with 100% accuracy. It is also capable of breaking up sentences and displaying appropriate hand gestures for a bunch of keywords in the sentence.

ACKNOWLEDGEMENT

We would like to express our deep gratitude towards Prof. Rakhi Kalantri, our project guide and also the project coordinator for the entire batch.

Figure-2: Output for PHASE-I ' Sign Language to Text'

Computer Vision techniques were deployed to enable the camera to capture real-time hand gestures. The image was converted to grey -scale. Canny edge detection algorithm was applied. A Google API was used to read out the detected text. An accuracy of around 85% was achieved for Phase-I.

The next phase dealt with converting speech to sign language. The voice was captured and converted into text using the speech recognition library. This text was later used to display the corresponding hand gestures.

Figure-3: Output for PHASE-II 'Speech to Hand Gestures'

REFERENCES

1. Ilan Steinberg, Tomer M. London, Dotan Di Castro, Hand Gesture Recognition in Images and Video, Technion-Israel Institute of

2. J. Jones. (1991, May 10). Networks (2nd ed.) R. Kurdyumov, P. Ho,

   J. Ng. (2011, December16) Sign Language Classification Using

3. Taner Arsan and Oguz Ulgen ,Sign Language Converter,International Journal of Computer Science Engineering Survey (IJCSES)

4. Adithya, V., Vinod, P. R., Gopalakrishnan, U. (2013). Artificial neural network-based method for Indian sign language recognition.

5. Warrier, K. S., Sahu, J. K., Halder, H., Koradiya, R., Raj, V. K. (2016). Software based sign language converter. 2016 International

6. Abdulla, D., Abdulla, S., Manaf, R., Jarndal, A. H. (2016). Design and implementation of a sign-to-speech/text system for deaf

7. Hasan, M., Sajib, T. H., Dey, M. (2016). A machine learning based approach for the detection and recognition of Bangla sign

8. Rajaganapathy, S., Aravind, B., Keerthana, B., Sivagami, M. (2015). Conversation of Sign Language to Speech with Human Gestures.

9. M.Suresh Anand, A.Kumaresan, An Integrated Two Way ISL (Indian Sign Language) Translation System A New Approach,