Performance Optimization of a Flex Sensor Based Glove for Hand Gestures Recognition and Translation

Performance Optimization of a Flex Sensor Based Glove for Hand Gestures Recognition and Translation

Osama Nisar*, Muhammad Anas Imtiaz,Shehwar Hussain, Omer Saleem

Department of Electrical Engineering

National University of Computer and Emerging Sciences Block-B, Faisal Town, Lahore, Pakistan

Abstract – This paper demonstrates essential parameters that can be employed to optimize the performance of a low cost wearable glove interface that will measure a persons hand movements. These wearable interfaces are used in multiple applications; such as robotic tele-operation, computer- interaction, sign-language vocalization and rehabilitation of paralyzed. Since the accuracy and precision of the sensor datais extremely critical in these applications, a hybrid sensing technique is used. The flex bend sensors are used to measure the flexion of fingers, where as an accelerometer is used to sense the orientation of the palm.An 8-bit microcontroller is used to process the sensor data. Various causes of degradation insensor data are described. Solutions are also proposed to enhance the accuracy of the sensor output. The efficacy of the proposed solutions is also manifested in the document.

Keywords—hand gestures; wearable glove; flex sensor; accelerometer, microcontroller.

1. INTRODUCTION

   Gestures play a very important role in daily routine communication. According to many sources more than 50% of the communication consists of non-verbal gestures. The gestures are also becoming an increasingly popular form of human computer interaction. The ways in which the physical work and exertion is reduced are being applied to new technologies for ease of access. The gestures are of many types and each and every human body orientation and position can be treated as a gesture.

   When it comes to interaction, the moving parts of human body play the biggest role. Hand gestures are used in sign language by the people who cannot speak.They are also used in human-computer interaction tools.A lot of work has been done on the recognition, translation and application of hand gestures in the past.In this paper, it is presented how a flex-bend sensor based hand gesture recognition system works, what are the limitations of this glove interface and how can we optimize the performance of such a glove interface.

2. RELATED WORK

   A lot of work has been done in which different forms of cameras, depth imaging, infrared and other imaging devices were used. In all of these works, the recognition demands a device in front of you. There is another very popular form of hand gesture recognition. With the rapid advancement in technology, major breakthroughs are made in the design and characterization of piezo-resistive sensors [1, 2]. These sensors are commonly referred as flex bend sensors since they measure the bending in the flexor tendons, such as fingers. These sensors are installed on a wearable glove, usually one for each finger [3]. The output of the sensor is fed to a microcontroller for further processing [4]. With this interface it becomes very easy to acquire the human fingers movement in neurophysiological settings [5, 6, 7]. These interfaces have several biomedical

   Fig. 1. Schematic Diagram of the proposed Glove System

   applications, such as hand prostheses and gesture vocalization [8]. Moreover, such interfaces are also being extensively used in robotic tele-operation and human- computer interaction tools [9]. Hence, there is a dire need of effective technique(s) to improve the accuracy and integrity of the sensor output that are extremely vital for the above- mentioned applications.

3. EXPERIMENTAL SETUP

   The sensor feedback (flex-bend sensors and accelerometer) is used to provide analog signals regarding the position and orientation of the robot to the microcontroller. The microcontroller processes these signals, transmits the processed data serially at 9600 bauds to the computer. This data is then displayed for further analysis. The schematic diagram of the proposed glove system is shown in Fig. 1.

   1. Flex Sensor Feedback

      The simplest flex sensor based glove is designed using five

      4.5 inch flex sensors, one for each finger. If you see the American Sign Language Chart for static gestures, you will realize that for most of the gestures, this simple glove will work. When the gestures are made through the glove covered hand, the values of the sensors are read and fed to a microcontroller as shown in Fig. 2.The microcontroller ispre-programmed to compute and recognize the gesture formed. The limitation of this glove interface is that the rotation of wrist and orientation of hand in the Cartesian coordinate system cannot be sensed.

   2. Accelerometer based Inertial Feedback

      When it comes to orientation, the addition of an accelerometer to the same glove interface

      Fig. 2. Schematic Diagram of the proposed Glove System [10]

      resolves the problem. We used ADXL335, 3-axis accelerometer.

      It provides analog output across x-, y- and z-axis as shown in Fig. 3. This axial data is fed to the microcontroller and using the Equation (1), the angular tilt (pitch) of the body is calculated.

   3. Hardware Design

      The complete experimental setup is shown in Fig. 4. The designing of the hardware for this glove interface is important because the software can be written with some basic expertise in embedded software development. The hardware has many goals and these goals have to be met with extreme care for better results. The first and the foremost goal is to provide the power to all the sensors attached because powering sensors directly from a microcontroller is not a good practice. On the other hand, the sensors also demand a driving circuit which is necessary to

      Fig. 3. Block Diagram of ADXL335 Accelerometer [11]

      = tan1

      +

      (1)

      Fig. 4. Experimental Setup

      Where, AX, AY, AZ are the normalized voltage output of the accelerometer for each of the three axes

      The number of gestures that can be made and recognized through the glove interface multiplies. Making a gesture using the fingers and then rotating the wrist or tilting it gives rise to innumerable possible gestures. In this way, a glove which was designed to sense the gestures formed from a human hand was upgraded. The static sign language, as discussed above, involves no movement. Therefore certain gestures can only be formed by taking into account the orientation of the palm with respect to the wrist. This angular inclination is acquired by installing an accelerometer at the back of the palm.

      give a meaningful measure to the values of the sensor.

      As an example of the flex sensor, we have to design a transducer circuit to convert the changes in resistance into voltage changes. The flex based glove interface is shown in Fig. 5. Only after conversion to voltage signals a microcontroller will be able to sense these changes.The hardware is also the most important part because it is the first and the most critical part where the values of the sensors have to be stabilized. Otherwise the additive noise and bounded disturbances lead to unstable sensor output. Also, the ripples in the power applied to the sensors bounce their response. Thisleads to erroneous results. For this purpose a signal conditioning circuit has been used, as shown in Fig. 6. The microcontroller serially communicates the processed data to the computer. The network of wires is also a very critical part of the hardware because they too affect the performance.

      Fig. 5. Wearable Glove Interface

   4. PERFORMANCE OPTIMIZATION

      The general understanding from the basic form of the complete workstation created many problems in different dimensions.

      For example, the problem of the unstable values of the flex sensors and the accelerometer, the problem of the unstable voltage from the supply, the problem of speed of recognition of gestures and the percentage of the error in gesture recognition. All of them are addressed here in the techniques which were used to optimize the performance and also the techniques which can be applied to optimize the same interface for more effective and efficient results.

      1. Effect of Temperature on the Flex Sensors

         The effect of temperature on the measurements of the flex sensor was found to be affecting the values all the time. The values showed up different in the day time and different at night, continuously varying with time. As a solution the flex sensors were wrapped in order to insulate them from the atmospheric temperature. Before embedding the flex sensors on the glove interface, they were wrapped in 6 layers of very thin polythene sheets and a scotch tape. When completely covered, it was noticed that the temperature had no effect on the integrity of sensor values.

      2. Effect of Loose Attachment of flex sensors with the Glove

         Even if the flex sensors are properly sewed with a high grade non-breakable thread, they are loose enough to affect the whole systems stability and efficiency. The force applies by the fingers of a human hand is great enough to tear the glove even if the thread does not break. Similarly, the force also varies when different people with different sizes of fingers use the glove. A longer finger will exert more bending force on the flex sensor and consequently, the flex sensor will try to displace from its position.

         Consequently, the required values are not achieved and the response varies. There is a good way to attach the flex sensors to the glove and that is by making use of a strongly adhesive double tape. The glue should not be directly applied to the glove or the sensors because if the sensor needs to be replaced or it is to be disconnected for soldering the wires, the glove and the sensor both are damaged. A good quality double tape will last very long and will not allow the sensor to leave its position when the bending force is applied from the fingers inside the glove.

      3. Effect of the non-Ideal Response of Sensor

         The quality of the flex sensors which are available in the market is good but as they have not been specially designed for very high stability applications, hence they tend to exhibit non-ideal response.When the voltage is applied to the commercial flex sensors, it is observed that even in a fixed posture and a disturbancefree environment,

         Fig. 6. Signal Conditioning Circuit

         their output voltage varies slightly. In normal applications these variations can be neglected but in applications where a good resolution of the data is mandatory, these disturbances can lead to hazardous outcomes. These variations occur due to the ripples present in the voltage source. This effect can be removed by adding a capacitor in parallel with every flex sensor. The capacitor acts as a low pass filter. It helps to filter out the high frequency noise from the sensor output and allows only the DC components, the actual sensor, to be fed to the microcontroller. As the capacitor removes the small ripples in the voltage, the response tends to become stable. A non-polar capacitor of small value (usually picoFarads) is preferred. With a larger capacitor, the time constant increases. It takes large time to charge and discharge and the system response is distorted.

         The same capacitive filtering technique is applied to the accelerometer as it also gives analogvoltage outputs. The response is affected by the same problems which are discussed for flex sensor and the ripples in the response of the accelerometer can also be removed by adding a capacitor in parallel to the output. There is one modification which is needed and that lies in the number of capacitors to be used. The accelerometer generally used for this purpose has three outputs namely: x, y and z-axis. For each of these outputs a capacitor is needed to stabilize the response.

      4. The Effect of Capacitance of the Wires used

         The capacitances of the wires used with these sensors definitely affect the values that are drawn out of the glove. As the whole system is a very low power system and the operating voltages are generally fractions of a volt, the small capacitances get the chance to play their role. To minimize the effect of the capacitances of the wires, it will be a good practice to use a capacitor in series with the positive wire. Adding the capacitor in series will actually get subtracted from the capacitance of the wire and the effect will be minimized. It should be noted that the capacitance value must be very small because the small wires do not have much capacitance.

      5. The Resolution of the sensor values & Mapping

         The resolution of the values from the sensors can vary within a bounded range. The data sheets and results of the tests of sensors altogether provide a reasonable understanding of the maximum and minimum bounds of the range of the sensor values. Since the flex sensors are resistive in nature, their resistance and hence the potential difference across themtends to increase upon bending. The maximum value occurs when theyare fully stretched.This forms the upper limit and vice versa is true for the lower limit the difference between the upper and lower limit is the resolution. In the software, certain thresholds are set for differentiation between the gestures. These thresholdsare decided while keeping in view the vulnerability of any physical change such as a change in size of hand.

         The resolution as observed in the software basically depends on the resolution of the Analogue to Digital Converter. For example if the ADC used has a 10-bit resolution, the range of the values will be from 0 to 1023. Now further increase in the resolution is most of the times necessary for better results and reduction in the error. The simplest method makes use of the mapping function in the software. By the use of a mapping function, the 0 to 1023 range can be further increased. But if the range is too large, it would exceed the computational power of the microcontroller and the calculations will overflow the memory. But it is easy and suitable to increase the resolution to 1500 or 2000, so a one degree bend in the flex sensor will actually exhibit a change of 11.1 units i.e. 2000/180. The same procedure should be applied to the three values coming from the accelerometer.

      6. The Sampling Rate

         The sampling rate plays a vital role in many forms of data acquisition, processing, recognition and conversion. The sampling rate also has a big say in the speed of the whole system. The first decisive factor in sampling rate is the speed of the controller and it governs the maximum number of samples you can take per second. The better the speed, the more efficient is the response. Special consideration must be given in choosing the sampling rate because the sensors have their limitations of providing the samples at a certain maximum sampling rate. It is required that the

         system is tuned to the maximum allowed sampling rate as it will make sure that the speed of the system is good and also when more samples are taken and their average value is used, the result is more accurate.

      7. The Software Filtering

      The values which are depicted by the sensors are changing continuously and also as the sampling rate is very high, in a matter of milliseconds, hundreds of samples are taken. There is a need to use the average of these values because that will be the actual value. There has been observed another difficulty in this system which arises due to the mechanical disturbances when forming a gesture. There is a mechanial vibration when the hand is held still depicting a gesture. This vibration destroys the results which are received from the sensors. The other problem is that before a gesture is completely formed, the hand is in motion or the fingers are in the process of bending to the desired degree. The samples that are taken during these movements actually ruin the average value which is required for a particular gesture. These were the problems which demanded some sort of filtering of the samples taken.

      The research on the responses of different filters was carried out. Since the problems faced for accelerometer and the flex sensors are the same, therefore in this supplication, accelerometer has been used in discussion. The techniques discussed here were also applied to the flex sensors in the similar manner. It was found that the Moving Average Filter and the Low Pass Butterworth filter were of no help. The large jumps which were observed in the samples taken were not filtered by the moving average filter and the average taken had the effects of these jumps. The Butterworth filter was expected to remove all the noise present in the readings and in this case, the removal of noise was not achieved to a satisfactory level. These findings convinced the use of some non-liner filter.

   5. RESULTS

      The median filter was used as it gave the best results. It is a non-linear filter and it acts continuously on the data but not on the actual moving data as the moving average filter does. The median filter acts when it is provided with the samples and that is what was needed. This filter successfully eliminates nearly all the ripples. The noise was considerably reduced.The output value was the average of the remaining samples. The simplest procedure was to take 50 samples, remove the upper and lower 5 samples and then take the average of the remaining 40 samples. The averaged output is the desired result. The authenticity of the proposed technique is verified via two test cases. The pseudo-code is as follows.

      1. START;

      2. TAKE 50 SAMPLES;

      3. SORT VALUES IN ASCENDING ORDER;

      4. REMOVE FIRST 5 VALUES;

      5. REMOVE LAST 5 VALUES;

      6. SUM REMAINIG 40 VALUES;

      7. DIVIDE BY 40;

      8. ANSWER IS THE FILTERED SAMPLE VALUE;

      9. END.

         1. Test Case 1

            In this case, the performance of the flex sensor on the index finger of the human hand is tested, with and without the application of the proposed median filtering technique. The index finger is fully stretched as shown in Fig. 7. The graphical results of the sensor output without the application of median filter and with the application of the filter are shown in Fig. 8.

            Fig. 7. Hand Posture for Test case 1

            The graphs are plotted with the sensors output analog voltage signal in volts (V) against 50 samples of sensor data taken at an interval of 50 msec. The graphical result clearly manifests the enhancement of the systems accuracy due to the application of the filter. The average error in the output without the filter application is 9.8%, while with the filter it reduces to 0.64%.

         2. Test Case 2

      In this case, the performance of the accelerometer, installed at the back of the palm is tested, with and without the application of the proposed median filtering technique. The hand is fully stretched as shown in Fig. 9. The graphical results of the sensor output without the application of median filter and with the application of the filter are shown in Fig. 10.

      Fig. 8. Graphical results for Test Case 1

      Fig. 9. Hand Posture for Test case 2

      The graphs are plotted with the pitch angle in terms of its equivalent analog voltage signal in volts (V) against 50 samples of sensor data taken at an interval of 50 msec. The improvement in the systems accuracy is demonstrated by the graphical results. The average error in the output without the filter application is 6.67%, while with the filter it reduces to 1.1%.

   6. FUTURE RECOMMENDATIONS

      All the problems and procedures which have an impact on the performance and optimization of the whole system of gesture recognition have been discussed above. There were a few limitations to the discussion of the findings of this research. The ADC used was of 10 bit resolution, the Controller Speed of 16 MHz was used and the number of sensors used was six (5 flex sensors, 1 accelerometer).

      solutions can be applied without difficulty and for best results. The authenticity of the proposed solution has been verified by the test results. In the end, it has been discussed that concrete steps should be taken in this regard to design a much useful and practical interface for further research and applications.

      REFERENCES

      Fig. 10. Graphical results for Test Case 2

      The findings above can be applied to any system which is designed for this application or is related to it somehow. It must be mentioned that the ADC of higher resolution will further improve this system both in terms of the resolution of the values and also the speed. This is because a lot of time is spent in analogue to digital conversion of the samples as there are total 8 conversions needed in this system. A microcontroller of higher speed will also improve this system while keeping in view the maximum sampling rate of the sensors. Letting aside the restriction of the sampling rate, the computation speed will also be increased and overall the system will become more efficient. The last and the most important thing for future endeavors in this scenario is the number of sensors used. There is a possibility to develop a system by attaching more than two flex sensors to a single finger to keep an eye on the minute changes. Similarly, other types of sensors can also be added as the touch sensors between the fingers. Increasing the sensors will lead to certain design and implementation difficulties but once these stages are passed, it will be a system having much more to give.

   7. CONCLUSIONS

   The focus of this paper was on the performance optimization of a flex sensor based glove for hand gesture recognition and translation. The need and use of gestures was discussed in the start and various methods for gestures recognition were discussed. The glove designed using flex sensors and an accelerometer can be used effectively for hand gesture recognition. The research was carried on this glove interface to optimize the performance for such an interface. The findings as discussed in the paper throw light on the basic and exceptional problems which have to be realized and solved for the development of a good system. The procedures for performance optimization have been discussed in a very detailed manner. Almost all the problems which affect the performance of this gesture recognition glove interface system have been mentioned along with their effective and simple solutions. These

   1. G. Saggio, G. Latessa, S. Bocchetti, F. Brunetti, A. Reale, F. Giannini,

      Sensor's Design and Characterization for Biomedical Applications, ISSBB, IV National Congress, Santa MargheritaLigure, April 2009

   2. G. Orengo, L. Giovannini, G. Latessa, G. Saggio, F. Giannini,

      Characterization of piezoresistive sensors for goniometric glove in hand prostheses, Wireless Vitae09 – Wireless Communications, Vehicular Technology, Information Theory and Aerospace &Electronic Systems Technology, Aalborg (Denmark), 17-20 May, 2009 pp. 684- 687.

   3. AmbikaGujrati, Kartigya Singh, Khushboo, LovikaSoral, Mrs. Ambikapathy, Hand-Talk Gloves with Flex Sensor: A Review, International Journal of Engineering Science Invention, vol. 2, issue 4,

      April 2013, pp. 43-46

   4. Ata-ur-Rehman, Salman Afghani, Muhammad Akmal, RaheelYousuf,

      Microcontroller and Sensors Based Gesture Vocalizer, Proceedings of the 7th WEAS International Conference on Signal Processing,

      Robotics and Automation, February 2008

   5. Donchul Lee, Youngjin Choi, Development of Compact Data Glove System, the 7th International Conference on Ubiquitous Robots and Ambient Intelligence (URAI), 2010

   6. R. Gentner, J. Classen, Development and evaluation of a low-cost sensor glove for assessment of human finger movements in neurophysiological settings, Journal of Neuroscience Methods 178 (2009) 138147

   7. L. Dipietro, A. M. Sabatini, P. Dario, Evaluation of an instrumented glove for hand movement acquisition, J Rehabil Res Dev 2003;40(2):17990.

   8. L. Dipietro, A. M. Sabatini, P. Dario, A Survey of Glove-Based Systems and Their Applications, IEEE Transactions on Systems, Man, and Cyberneticspart C: Applications and Reviews, vol. 38, no. 4, july 2008, pp. 461482.

   9. Tarek Mohammad, HastiSeifi, Junhao Shi, Design of Gesture based Input Devices for Controlling Presentation, 2008

   10. https://www.sparkfun.com/datasheets/Sensors/Flex/flex22.pdf [11]https://www.sparkfun.com/datasheets/Components/SMD/adxl335.pdf