WARDAN: Wearable Assistive and Recognition Device using Android for Sign Language Translation

WARDAN: Wearable Assistive and Recognition Device using Android for Sign Language Translation

1Ishan Joshi, 2Harsh Rajguru, 3Lalit Mishra, 4Preet Jain

1, 2, 3 Student, 4Associate Professor Electronics and Communication Department

Shri Vaishnav Institute of Technology and Science Indore, (M.P.), India

1ishaanjoshi248@gmail.com, 2harshrajguru345@gmail.com 3mishralalit54@gmail.com, 4preetjain@gmail.com

AbstractTechnology has always contributed for the upliftment of the disabled persons and to help them live a normal life. We propose an automatic gesture recognition approach for Indian Sign Language (ISL), the system automatically recognizes gestures of sign language, convert words and sentences of ISL into voice and text in English and allow the verbally disabled to express themselves in a better way.

The device worn by the verbally challenged senses the way the fingers curls and the way wrist and elbow moves, through which it detects different patterns of the ISL and convert it into speech or text. This device not only recognizes the alphabet but it also interprets common words and phrases used in ISL into text and sounds using Android Technology and Quantum Tunneling Composite(QTC) sensors.

Index Terms Quantum Tunneling Composite (QTC), Indian Sign Language (ISL), Android, Assistive Technology.

1. INTRODUCTION

   Verbally disabled people use sign language for communication but they find difficulty in communicating with others who dont understand sign language. This project aims to lower this barrier in communication. Sign language recognition is a multi-disciplinary research area involving pattern recognition, computer vision and natural language processing; it is major problem because of the complexity of the visual analysis of hand gestures and highly structured nature of sign language. It is very important area not only from the engineering point of view but also for its impact on society. A truly functioning sign language recognition system can provide an opportunity for mute people to communicate with non-signing people without the need of an interpreter.

   Since ISL got standardized only recently and also tutorials on ISL gestures were not available until recently, there were very few research works that has happened in ISL recognition. Our project mainly focuses on ISL recognition.

   This device needs to be worn on the hands of the verbally disabled people and depending on the variation of wrist, fingers and elbow movements the device intelligently converts the gestures/movements into voice and text, the device senses or detects different hand patterns and movement of the mute people and gives the respective output i.e. the related word or letter in the form of text and voice.

   The core idea behind this project is to use computing technology to facilitate communication between two persons

   who cannot converse directly; the goal is to use this project or model at public places like hotels, airports, hospitals, railway stations, etc. This model will help the mute people to contribute or to participate more in the society and will make them open to the world.

2. HISTORY/PREVIOUS WORK

   There has been some prominent research made in the field of sign language recognition. Some of them are:-

   1. Glove Talk and Glove Talk II

      Fels employed data glove for this purpose, with a polhemus tracker attached for position and orientation tracking. Glove talk was based on having a root word determined by hand shape, with an ending depending on the direction of movement. Hand speed and displacement affect speed of speech and stress respectively. For each of these a separate neutral network is employed, with an additional network (strobe network) used for sign separation. The strobe network was by far the most complex since it used five pieces of information derived from ten consecutive frames [1, 2].

   2. Pausch and Davidsons CANDY System

      CANDY(Communication Assistance to Negate Disabilities in Youth) is similar in concept to the glove talk system, although it is extremely simplified , using only two degrees of freedom to model the movement of the mouth and tongue. They used magnetic trackers placed on the bodies of the individuals involved [6]. It is intended for use by people suffering from cerebral palsy and was designed with the intention of being as flexible as possible. From the movement of the subject, they extract two variables: the tongue tip position and the tongue base position. By modeling the tongue in mouth it is possible to determine the sound which a position would generate.

   3. Kramers talking Glove project

      James Kramer and his supervisor, Lary Leifer, have been working on a method for communication between deaf, deaf-blind and non-vocal individuals. It is a complete system, which attempts to integrate a number of technologies together, in such a way that all parties can communicate [2].

3. INDIAN SIGN LANGUAGE (ISL)

Sign languages convey meaning by manual communication and body language instead of acoustically conveyed sound patterns. This can involve simultaneously combining hand shapes, orientation and movement of the hands, arms or body, and facial expressions to fluidly express a speakers thoughts.

There is no single "sign language", wherever communities of deaf people exist, sign languages develop. While they use space for grammar in a way that oral languages do not, sign languages exhibit the same linguistic properties and use the same language faculty as do oral languages. Hundreds of sign languages are in use around the world, some sign languages have obtained some form of legal recognition, while others have no status at all [11].

ISL recognizes English alphabets as shown in

Fig. 1

Fig. 1: ALPHABETS IN INDIAN SIGN LANGUAGE [3]

Analog to digital converter which will be the input for our system.

1. Voltage Divider:

A voltage divider (Fig. 3) is a simple circuit consisting of two resistors that has the useful property of changing a higher voltage (Vin) into a lower one (Vout). It does this by dividing the input voltage by a ratio determined by the values of two resistors R1 and R2 (variable) [5] as shown in Fig. 3

R2 can be flex sensor or QTC sensor, etc.

Fig. 3: Voltage Divider Circuit

Knowing only the finger spelling method is not enough to communicate in sign language, but rather there is a standard way to ask or to tell something in ISL [4] as in Fig. 2.

1. APPLICABLE TECHNOLOGIES

   (1)

   Fig. 2: Sentence formation in ISL [4]

   4. PRINCIPLE

   The principle working behind the gesture recognition is to convert different movements of the wrist and fingers i.e. bending or curling into some sort of electrical signal and for getting these signals we used the resistive property of a material. As we know that the change in length or cross- sectional area in any material changes its resistance, we can use this property of material to have different resistance values for different degree of deformation in materials size and shape.

   We have different variety of materials available which can be used for this purpose. The idea is to wear a sensor made by such material on fingers, wrist and elbow in order to capture their movements in the form of changing resistance. This changing resistance will give us different voltage out of a voltage divider circuit and this variable voltage will be given to

   After the origin of sign language many technologies has emerged out in orderto interpret the sign language into text or sound for the betterment of communication between the challenged and a normal person.

   These technologies used different type of sensors and computational devices. The sensing technique includes flex sensors, fiber optics, image processing, sensors made of potentiometer etc.

   In our research we are suggesting a new way to detect the hand gestures by using a new sensor made of Quantum Tunneling Composite.

   Technologies are:

   1. Flex Sensor:

      Flex sensors are the sensors that change in resistance depending on the amount of bend on the sensor; they convert the change in bend to electrical resistance (more the bend, more the resistance). They are analog resistors and work as variable analog voltage divider. Inside the flex sensors there are carbon resistive elements within a thin flexible substrate when the substrate is bend the sensor produces a resistance output relative to the bending. They are very expensive and less durable.

   2. Image Processing Technique

      Image Processing Technique uses segmentation and detection of hands from each video frame. Segmentation process is the first process for recognizing hand gestures. It is the process of dividing the hand gesture image into regions separated by boundaries. The segmentation process depends on the type of gesture, if it is dynamic gesture then the hand gesture need to be located and tracked, if it is static gesture then the input images have to be segmented only. After this, the segmented image is processed according to the algorithm and gives the respective outputs [14].

      Though image processing is nearly accurate and durable but it is expensive and it is not handy due to the complexity of hardware design.

   3. Quantum Tunneling Composite:

Quantum Tunneling Composite (QTC) is a composite material that varies its electrical resistance according to the force or pressure being applied. It is made from metallic or non-metallic filler particles combined with elastomeric binders, such as silicon rubber.

QTC material was produced in 1996 and is patented technology developed in the UK by Peratech Limited.

The unique method of combining these material results in a composite that exhibits significantly different electrical properties when compared with any other electrically conductive material [7, 8].

QTC material has the unique ability to smoothly change from an electrical insulator to a metal like conductor when placed under pressure. While in an unstressed state the QTC material is a near-perfect insulator; with any form of deformation the material starts to conduct and with sufficient pressure metallic conductivity level can be achieved [7]. Theoretically, the resistance of QTC material decreases exponentially (Fig. 4) with compression-subsequently, allowing increasing current flow through material [12].

\

Fig. 4: The graph between Force (N) and Resistance () shows the effect of varying the current at constant voltage [12]

Conventional Composites having carbon as a filler particle requires a lot more pressure and conduct minute currents through a Percolation process [9].

Percolation process is the process that contain carbon particles within these composites usually has smooth, rounded surface-consequently particles are always in contact with one another creating a constant conduction path. As pressure is applied more particles come into contact and therefore more conduction pathway build up.

QTC material have a different property from percolative composite is that the metal particles are given an irregular structure with a spiked surface which is wetted (Fig. 5) i.e. electrically insulated by the silicon rubber. The wetting allows the metal particles to get close but not touch even when the QTC material is squeezed or densely loaded. The spikes on the surface allow a higher concentration of electron charge to build up at their tips.

The principle effect of the increased charge on the spikes is to decrease the effective width of the potential barrier in quantum tunneling, thus reducing the distance and energy required for the electron charge to tunnel through. By this means the tunneling regime gives a varying conductivity to the QTC material user dynamic condition.

QTC material works as follows:

For tunneling conduction to occur, tunneling probability needs to be high. To increase probability, the width, or apparent width, of the tunneling barrier needs to be lowered. This is achieved in QTC material due to the shape of the conductive filler particles and their high loading in the barrier material which is a non-conductive elastomeric binder.

Spikes on the conductive filler particles produced a localized increase in the electric field (Fig. 6) at the tips which effectively reduce the barriers width and allows conduction to occur. This is known as Field-Assisted tunneling [10].

Fig. 6: Processes of conduction of QTC material when compressed [10]

QTC usually comes in the form of pills (Fig. 7) or sheet, ink/coating, and granule. QTC pills are just tiny little pieces of the material. QTC pills are pressure sensitive variable resistors [8].

Fig. 7: QTC PILLS [13]

The above description about QTC give a very clear sense of how efficient and convenient it will be to use QTC as sensor for detecting hand gestures, using QTC as a sensor will have following advantages :-

1. It is much easier to integrate it in a glove.

2. It will give a better reading of finger bending.

3. The glove will be comfortable to wear.

4. It does not get damaged unless cut into pieces moreover it is water proof.

   All the technologies discussed above can be used for sign language recognition; their comparison in a tabular form is as follows:

   TABLE 1:

   Technology	Wearability	Durability	Accura cy	Cost
   Image Processing	Moderate	Long life	Moderat e	Very Expensive
   Glove with QTC Sensor	High	Long Life	Moderat e	Affordabl e
   Glove with Flex Sensor	Moderate	Short Life	Moderat e	Expensive
   Glove with Potentiomet er	Low	Short Life	Low	Cheap

   From the above table it is clearly seen that the technology using QTC sensor for hand gesture recognition will be the best as compare to rest.

   Fig. 5: Particles in Conventional Composites and QTC [10]

   1. METHODOLOGY

      The hand gestures and patterns play an important role in recognizing the words in Indian sign language. Most of the researches classify gestures recognition in different steps after

      taking the input by different means like images, videos or data glove instrument device.

      In our case, we are using data glove instrument for extracting input from hand gestures with the help of sensors fitted on the glove which must be worn by the verbally challenged person.

      In our research, we are working on recognition of different gestures and hand pattern along with elbow movement in order to get more precise information which will help in better conversion of sign language into text and speech.

      Previous researches in this field focused on finger patterns and wrist movement for sign language recognition which were unable to get the most of the words or information from the action of the verbally challenged person. But in our research, we are also taking elbow movement into account which will help in a better way to translate the action into text or speech.

      Also the most unique feature of our device is that we have incorporated a touch evice with the glove which will give the user a better interface enabling him to quickly express the most commonly used phrases and sentences by just a simple touch.

      The touch device works on Android platform for which an application is developed that will allow the user to express the complex sentences which may not be judged by the machine or by person accurately, and it will also assist them to convey some commonly and repeatedly used sentences and phrases directly from the device which were already fed into the application in the form of templates and thus giving them a far better way to express themselves.

      Thus while communicating to a normal person the user doesnt have to go through the hectic process of gesture formation of complex sentences rather he has to just scroll the list of already fed sentences and choose it from device by a simple touch.

      As we are concentrating on verbally challenged people among which most of them are not capable of spending big bucks on expensive translating machine or devices, we have focused on the cheapest possible way to get the things done and QTC is the best option for this purpose. The whole process of translating the sign language into text and speech starts from getting the input from the mounted on the glove, along with the input from the sensors at the elbow and analyzing them.

      The information gathered from the sensors is fed into microcontrollers. We are using an AVR controller for fast response and better computational ability. The controller process the input according to the algorithm and match the input with the look up table previously stored in it. We have given a wide variety of gestures and pattern combinations to be translated in text and voice into the look up table. After recognizing the gestures and patterns, the microcontroller further proceeds to call for the text or speech to be displayed or articulated.

      8. CONCLUSION

      In this paper we have presented a prototype for a system which converts the sign language into text and sound. This system can play an important role in a challenged persons life and encourage them to come forward and participate in the normal world with a confidence to express themselves more effectively.

      Though the proposed model have some limitations in a way that the complete recognition of a sign language requires the information from face expressions, head movements etc. and also as it has a limited dictionary.

      This model can be modified and extended by incorporating face gesture recognition techniques and developing a better application with enhanced dictionary along with the tools for local language support. Another extension that can be done is to translate the voice of a person into text and display it on the screen for a person with hearing disability. This enhancement will enable a complete two way communication among the challenged and a normal person.

      REFERENCES

      1. Dr. Nasir Sulman, Sadaf Zuberi,Pakistan Sign Language A Synopsis, Pakistan, June 2000

      2. Waleed Kadous, Grasp: Recognition of Australian Sign Language using Instrumental Gloves, http://www.cse.unsw.edu.au/~waleed/thesis/thesis.html, Australia, OCTOBER 1995, pp. 1-2.

      3. Happy Month 2

      4. http://www.inquiry.net/outdoor/native/sign/sentences.htm

      5. http://www.sparkfun.com/tutorials/207

      6. Hersh, M.A. and M.A. Johnson (eds.) (2003), Assistive Technology for the Hearing Impaired, Deaf Blind and Deaf, Springer Verlag.

      7. http://www.peratech.com/qtc-material.html

      8. http://www.materialsforengineering.co.uk/engineering- materials-features/quantum-tunnelling-composites-making-a- switch/43072/

      9. http://openmaterials.org/2009/11/30/materials-101-quantum- tunnelling-composite/

      10. http://www.peratech.com/qtc-science.html

      11. http://en.wikipedia.org/wiki/Sign_language

      12. http://www.peratech.com/qtc-technology.html

      13. http://www.mindsetsonline.co.uk/product_i

      14. nfo.php?products_id=1144

      15. N. Ibraheem, M. Hasan, R. Khan, P. Mishra, (2012). Comparative study of skin color based segmentation techniques, Aligarh Muslim University, A.M.U., Aligarh, India.

   2. EXPERIMENTAL RESULTS

Some of the results from our prototype are shown in Table 2.

TABLE 2: