Performing Basic Tasks on Computer using Hand Gestures & Ultrasonic Sensors

Performing Basic Tasks on Computer using Hand Gestures & Ultrasonic Sensors

Abstract- Innovative approaches like hand gestures play a vital role in this emerging world and evolving time of Industry

1. because they are simply accessible and are efficient in detecting nearby objects. The need for hardware components such as the keyboard and mouse can be drastically reduced with this hand gesture technique. We utilize ultrasonic sensors to identify and characterize hand gestures in real-time in this paper. Ultrasonic sensors use ultrasonic sound waves to measure the distance between a target item and then transform the reflected sound into an electrical signal. The main goal of the paper is to use Arduino and Ultrasonic sensors, as well as various Python packages, to enhance the precision and speed of the computer interface.

   Keywords Hand gesture, IoT, Ultrasonic sensors Arduino-uno, Python, Gesture control

   I INTRODUCTION

   In todays world of automation, hand gesture technology is not just limited to gaming application but also in various areas such as medical application, industry application, IT hubs, Banking sectors, etc. This project is based on a similar concept of hand gesture control-based laptop or computer. Human Machine Interface (HMI) is a system that consists of hardware and software which helps to initiate a communication and information exchange between the user and the machine. We generally use various indicators such as LEDs, Switches, Touch Screens and LCDs as a part of HMI devices. Another unique way of communicating with machines like Robots or computers is with the help of Hand Gestures.

   Instead of using various equipment like keyboards, mouse, joystick, etc, now simply we can use hand moments/hand gesture to control the functions of the computer/laptop. In this project, we have implemented Arduino based hand gesture control where one can control music, video, browser, documents, gaming, navigation and many others tasks. We have used two ultrasonic sensors and attached to Arduino. The ultrasonic sensors are attached at the top of the computer on either side which is used to calculate the distance between the hand and the sensor. Using the following information of the distance between the hand and the sensors, the above operations are executed. Python Pyautogui library is used to perform mentioned tasks on our laptop/computer. From Arduino, the commands are sent to the computer via a serial port. Python reads this incoming data which is running on the computer/laptop and finally based on these incoming read data, the following tasks and operations are executed. [1]

   1. EXISTING METHODS

      There are several techniques to control a computer utilising hand gesture approach. Each method has its advantages and disadvantages. Let's look into different methods in brief.

      1. Controlling computer using color bands

         While using colour bands users use red, blue and green bands and put them to their fingers. So, while using this technique kinect sensor projects an infrared pattern of 307,200 dots in a 640 × 480 mesh and receives reflected pattern CMOS monochrome sensor. This structured light application permits the machine to estimate the intensity of every point using triangulation. Moreover, an RGB camera provides synchronised colour information for each point. [3]

      2. Controlling computer web cameras

         Using web camera technology, we need a projector which projects the display on a clear wall or any other plain surface. User can interact with the projected screen using his fingertips which are tracked in the air by the camera using camshaft tracker. A related study of distinct methods of hand gesture detection has been made. Here an efficient technique is employed to identify the hand gestures which are transformed into relevant actions. [4]

      3. Controlling computer using IR sensors

         When we are working with IR sensors the user is equipped with a glove which was fitted with IR sensors that acts as a bridge between user and computer. The gloves are used to recognize the hand gestures of the disabled people and convert those gestures into meaningful messages in real time. With the amount of bend made by the fingers continuous data stream is obtained as output from Infrared (IR) sensor. IR sensors change the output voltage depending on the strength of the received signal. Change in bend is converted to change in the electrical voltage by IR sensors. The output from IR sensors is processed by a microcontroller and a corresponding message is displayed. [5]

      4. Limitations in existing system

         • Controlling the computer with hand gestures using projectors is only used for gamming applications and this will not sense the objects exactly when the lighting was changed.

         • When controlling the computer with hand gestures using web cameras, it is difficult to interact with system using camshaft in the air and this is time taken thing to project on the plain surface and as it needs extra plain surface along with the requirements of the project to project, we

           demonstrate that the fast and portable hand gesture computers.

         • While controlling the system with hand gestures using IR sensors, it is difficult to always bend the fingers and is expensive to buy IR sensors and also these sensors will not work in lighting.

      So, we demonstrate the easy and cost-effective gesture controlling of computer by using ultrasonic sensors instead of using IR sensors, color bands, web cameras, projectors. As ultrasonic sensors are portable to move, cost effective and easy to operate we implemented the project by using ultrasonic sensors. Hence it is easily affordable by anyone and also easy to operate with the hand just by waving in front of the sensors.

   2. PROPOSED METHOD

      The proposed system adopts a model based on the sensors estimating distance and executing a specific function in response. Some gesture recognition approaches are presented, and then actions are identified using the sensor. We can choose the action to do on the computer-based on the distance recorded by the sensor. The distance of the hand can be calculated using an ultrasonic sensor by measuring the time it takes to emit and receive sound waves using a trigger and echo as an input. A certain function is done based on the distance between the hand and sensor. For rapid operation, the sensor is attached to the computer at the top of the screen on both corners, and we termed them left sensor and right sensor. We're using two sensors to perform a wide range of activities.

      Figure 2: Flow Chart of proposed System

      The time it takes to emit and receive sound waves is recorded here, and the distance is determined using that information. The activities in the system will be done based on the distance. The hand gesture computers will function in this manner.

   3. HARDWARE COMPONENTS SETUP

      1. Basic Hardware Connection

         The gesture control is based on specifying the hand moments from the ultrasonic sensor. This project is implemented using various components. We need to deploy ultrasonic sensors which are connected to Arduino, then place yor hand in-front of the Ultrasonic sensor which will calculate the distance between the hand and the sensor. Using this following information, particular task or operation will be executed.

         ULTRASONIC SENSORS

         ULTRASONIC SENSORS

         Figure 1: Connection Setup of Arduino and Ultrasonic Sensors to the System

         The following figure will explain the steps to perform the operations in the system by using hand gestures. The hand or any item should first be positioned in front of the sensors, after which the sensors will emit sound waves, which will travel through the air and be reflected as echo waves when they strike our hand or any item, that is received by echo hole.

         ARDUINO BOARD

         LAPTOP/PC

         Figure 3: Basic hardware connections

      2. Working of Ultrasonic Sensors

         In the real world, the ultrasonic sensors are used for interacting with robots, computers and other devices. This sensor senses the proximity and detects levels with high

         reliability. The ultrasonic sensor measures the distance to a particular object with the help of ultrasonic sound waves. Ultrasonic sensors make use transducer to send and receive ultrasonic pulses. High-frequency sound waves reflect from boundaries to produce distinct echo patterns.

         Ultrasonic sensors work by emitting sound waves at a particular frequency. The transducer of the sensor acts as a microphone to receive and transmit the ultrasonic sound. The sensors determine the distance to a targeted object. The distance is determined by calculating time lapses between the sending and receiving of the ultrasonic pulse. It sends a 40KHz ultrasonic pulse that travels through the air. If there is an obstacle or an object, it will bounce back the signal towards the sensor. based on the time taken by the waves to travel and the speed of sound, the distance is computed.

         Distance = (Speed of the waves × Time taken to travel)/2 The calculated distance is divided because the waves are first transmitted by the transmitter and then received by the receiver.

         Figure 4: Transmitting and receiving sound waves by ultrasonic sensors

         Figure 5: Pin layout of Ultrasonic sensor

      3. Layout of Arduino Uno Board

         The Arduino is an open- source electronic platform which is used to build the hardware (microcontroller, a small computer) and software projects. Arduino board consists of both hardware part and also software part (IDE). Arduino board is used for controlling certain defined functions. Arduino board is powered by using a USB Cable from computer, etc.

         Figure 6: Arduino Uno Board

         There are specific dedicated pins such as:

         • Ground (GND) connection

         • 3.3V and 5V Power Supply

         • Analog Pins (A0 to A5)

         • Digital Pins (Pin Number: 0 to 13)

         • PWM Pin (Pin Number: 8)

         • AREF (Pin Number: 9)

           Arduino Board has 14 input/output pins. Language used for programming the Arduino board could be C/C++. Arduino board also consists of Reset Button, that temporarily connects the reset pin to the ground and restart the code that is loaded on Arduino Board. Power LED Indicator is also integrated on the Arduino board. This LED will turn on whenever we plug the Arduino into a power source. TX and RX LEDs are used during serial communication. TX and RX Pin are used for transmitting and receiving of data. The main IC on the board belongs to the AT Mega family, 328 AT Mega IC.

      4. Circuit Connection

         The circuit diagram of Arduino and the setup is quite simple. The trigger and echo pins of the first ultrasonic sensor are connected to two pins of Arduino board. For second ultrasonic sensor, the trigger pin and echo pin are also connected to two different pins. The Ultrasonic calculate the distance between the hand moments and the sensor and using the following information, the laptop/computer performs the task/operations. The information of distance from Arduino is collected by Python program. PyAutoGUI, a special library converts the data into keyboard click actions.

         Figure 7: Connection of Ultrasonic Sensors to Arduino Uno Board

      5. Universal Serial Bus (USB) Cable

         USB is an industry-standard that specifies the cables, connections, and protocols for connecting, communicating, and powering personal computers and associated peripheral devices. The USB standard, which was first established in 1996, is currently maintained by USB implementers. USB was developed to regulate the communication and power supply of peripherals such as keyboards, pointing devices, digital still and video cameras, printers, portable media players, disc drives, and network adapters to personal computers. It has largely replaced serial and parallel ports and is now used on a wide range of devices. USB ports are gradually replacing older forms of portable device battery chargers.

         Figure 8: Arduino Uno USB Cable

      6. Jumper Wires

      Jumper wires are simple cables having connector pins on both ends that may be used to connect two points without the use of solder. Jumper wires are commonly used with breadboards and other prototyping tools to allow for quick circuit changes. It's quite straightforward, Jumper wires are the most fundamental of all electrical components.

      Figure 9: Jumper Wires

   4. SETUP OF SOFTWARES & PACKAGES

      1. Installing Arduino IDE

         To program circuit boards effectively, we have an open- source software which is free to use provided by the Arduino named Arduino IDE (Integrated Development Environment). Using this software, we can send instructions to the Arduino board and perform various operations. Programming languages like C, C++ were used to provide instructions and control the board. So using this we work on a wide range of domains like IoT, Wearable devices, 3D printing, circuit boards, etc. We can download the software from Arduino official site and follow installation guidelines to install into the system. https://www.arduino.cc/en/Main/Software

      2. Installing Python IDE

         We need to install python IDE(Integrated Development Environment) to install packages like PyAutoGUI, PySerial, and more. We can download this software and

         install it for free from the python website given below. The version depends on the operating system we are using and also the system requirements, that can be found in the installation instructions. https://www.python.org/downloads/

      3. Installation of PIP Package Installer

         Installation, deletion, and maintenance of packages can be done in python using the Pip package installer. For new versions of the python, the Pip package installer comes pre- installed but for older versions, we need to install it manually. So, we can use the "pip -version" command to check whether pip is installed in the system or not. We can find installation instructions from the pip documentation given below:

         https://pip.pypa.io/en/stable/

      4. Installation of PyAutoGUI Library

         To automate mouse and keyboard interactions with other applications and also their operations we use PyAutoGUI package which helps the python scripts to control these operations. This package can be installed on all the versions and supports all the operating systems. This package can be installed using the following command: "pip install pyautogui"

      5. Installation of PySerialI Library

      This package is used to encapsulate the access for the serial port and also provide backend support for python on various operating systems. The "serial" module helps in automatically finding a suitable backend. To install this package the python version should be 2.7 or newer for python 2 and for python 3 it should be 3.4 or newer. So to install we use following command:

      "pip install pyserial"

   5. OUTPUT & RESULTS

1. Controlling of Video & Audio Player

   Forward, backward, volume up, volum down, next video, previous video, and finally play/pause are fundamental activities for most video players such as Windows media player, VLC, and others. Please note that the same kind of functionalities are used to control any audio players and also this can be used to control YouTube opened on a web browser.

   1. Implementation of Forward Operation:

      Here in the below figure the forward operation for a video will be performed, whenever the object is placed in front of right sensor & in the range specified in the Arduino programming the forward operation for the video will be performed. Here in the below figure the numbers which are appearing on the bottom of the screen represent the time of forwarding the video. The next video operation is also similar to this but the range is little different.

      Figure 10: Forwarding the video using hand gesture.

   2. Implementation of Backward Operation:

      Here in the below figure the backward operation for a video will be performed, whenever the object is placed in front of left sensor & in the range specified in the Arduino programming the backward operation for the video will be performed. Here in the below figure the numbers which are appearing on the bottom of the screen represent the time of the video. The previous video operation is also similar to this but the range is little different.

      Figure 11: Rewinding the video using hand gesture.

   3. Implementation of Volume Up Operation:

      Here in the below figure the volume up operation for a video will be performed, whenever the object is placed in front of right sensor & in the specified range of Arduino programming the volume up operation for the video will be performed. Here in the below figure the numbers which are appearing on the bottom of the screen represent the time of the video.

      Figure 12: Increasing volume of video using hand gesture.

   4. Implementation of Volume Down Operation:

      Here in the below figure the volume down operation for a video will be performed, whenever the object is placed in front of left sensor & in the specified range of Arduino programming the volume down operation for the video will be performed. Here in the below figure the numbers which are appearing on the bottom of the screen represent the time of the video.

      Figure 13: Decreasing volume of video using hand gesture.

   5. Implementation of Play/Pause Operation:

   Here in the below figure the play/pause operation for a video will be performed, whenever the object is placed in front of both sensors & in the specified range of Arduino programming the volume down operation for the video will be performed. Here in the below figure, you can clearly observe that the video is paused. Similarly, to play the video we keep object in front of both the sensors in specified range.

   Figure 14: Pausing of video using hand gesture.

2. Scrolling functions for browser control & reading files Scrolling up, down, sideways are the main functions used to read on browser, files, pdfs, and more. We can read across platforms using one code without the use of any keyboard keys. Apart from these we can perform special functions like zooming of document, switching tabs in browsers, switching from application to application.

   1. Implementation of Scroll Up Operation:

      Here in the below figure the scroll up operation will be performed, whenever the object is placed in front of the left sensor and is within the range mentioned in Arduino programming. This operation is very similar to that of volume up operation in a video player.

      Figure 15: Scrolling up of a document

   2. Implementation of Scroll Down Operation:

      Here in the below figure the scroll-down operation will be performed, whenever the object is placed in front of the

      Figure 16: Scrolling down of a document

      right sensor and is within the range mentioned in Arduino programming. This operation is very similar to that of volume down operation in a video player

   3. Scrolling Sideways:

   Here in the below figure the scrolling sideways of a document will be performed, whenever the object is placed in front of the right sensor and is within the range mentioned in Arduino programming the document is scrolled towards its right and if placed in front of the left sensor it moves towards left. This operation is very similar to that of a forward & rewind operations in a video player.

   Figure 17: Scrolling sideways of a document

3. Controlling Navigation keys for Basic Gaming and Navigations

   Navigation arrows are used to move around the computer and include up, down, left, and right arrows. These arrows are utilized not just for navigating, but also for playing almost any kind of game. The implementation and functionality of each arrow are listed below:

   1. To implement the Up arrow, we need to place an object in front of the right sensor within a small range specified in the Arduino programming. This operation is very similar to forward video in a video player. Using this we can increase the speed of the car in a car race.

   2. To implement the down arrow, we need to place an object in front of the left sensor within a small range specified in the Arduino programming. This operation is very similar to rewind video in a video player. Using this we can slow down the speed of the car in a car race.

   3. To implement the right arrow, we need to place an object in front of the right sensor within a range specified in the Arduino programming. This operation is very similar to the increasing volume of a video in a video player. Using this we can turn the car in the right direction in a car race.

   4. To implement the left arrow, we need to place an object in front of the left sensor within a range specified in the Arduino programming. This operation is very similar to the decreasing volume of a video in a video player. Using this we can turn the car in the left direction in a car race.

CONCLUSION

This paper presents one of the best solutions among others, for operating a computer using hand gestures. It is one of the easiest ways of interaction between humans and computers. It is a cost-effective model which is only based on Arduino UNO and ultrasonic sensor. The python IDE allows seamless integration with Arduino UNO to achieve different processing and controlling methods for creating new gesture control solutions. Using this method, we can perform almost any basic functions and also keyboard shortcuts easily and efficiently. We may use this process to develop our own code for whatever operation we want to execute.

REFERENCES

1. Srirama Dheeraj, R.Chandu, CH.V.N.Anirudh, M.Pavan, Narendran M, Controlling Pc Using Ultrasonic Sensors,

   IJARIIE, Vol. 4, Issue. 2, 2018

2. "Gesture recognition", En.wikipedia.org, 2021. [Online]. Available: https://en.wikipedia.org/wiki/Gesture_recognition. [Accessed: 15- May- 2020]

3. C. J. Cohen; G. Beach; G.Foulk A basic hand gesture control system for PC applications 30th Applied Imagery Pattern Recognition Workshop (AIPR 2001). Analysis and Understanding of Time Varying Imagery.

4. S. Thakur, R. Mehra and B. Prakash, "Vision based computer mouse control using hand gestures," 2015 International Conference on Soft Computing Techniques and Implementations (ICSCTI), Faridabad, 2015, pp. 85-89, doi: 10.1109/ICSCTI.2015.7489570.

5. GergelySziládi; Tibor Ujbányi; JózsefKatona Cost effective hand gesture computer control interface, 2016 7th IEEE International Conference on Cognitive Info communications.

6. Hoshino K, (2017) Hand Gesture Interface for Entertainment Games. In: Nakatsu R., Rauterberg M., Ciancarini P. (eds) Handbook of Digital Games and Entertainment Technologies.

   Springer, Singapore

7. Robust Hand Gesture Recognition for Robotic Hand Control (firstpublished 15th June 2017 by Springer) by Ankit Chaudhary.

8. S. Belgamwar and S. Agrawal, "An Arduino Based Gesture Control System for Human-Computer Interface," 2018 Fourth International Conference on Computing Communication Control and Automation (ICCUBEA), Pune, India, 2018, pp. 1-3, doi: 10.1109/ICCUBEA.2018.8697673.

9. Gopi Manoj Vuyyuru, Dimple Talasila, Hand Gesture Gaming Using Ultrasonic Sensors & Arduino, International Journal of Engineering Research & Technology (IJERT) Volume 09, Issue 06 (June 2020),