IoT based Hand Gesture Control Robot

IoT based Hand Gesture Control Robot

Patel Kathan, Ronak Makwana, Dhruvil Bhavsar, Sandeep Chauhan, Shriji Gandhi

Intrumentation And Control Engineering Department, Government Polytechnic Ambavadi, Ahmedabad, India

ABS TRA CT: The Hand Gesture Control Car is a cutting-edge project that fuses advanced robotics with state-of-the-art computer vision techniques, resulting in a remote-controlled car that can be effortlessly maneuverer through hand gestures. Augmenting its capabilities, a camera module is seamlessly integrated, providing users with real-time video feedback. This project's primary objective is to introduce a more intuitive and captivating means of directing a remote-controlled car, capitalizing on the innate expressiveness of hand movements. The car's hardware setup includes a motorized-wheel-equipped chassis, an ESP32 module, a camera module, and sensors dedicated to recognizing gestures. The ESP32 meticulously processes signals received from the gesture recognition sensors, skilfully translating them into precise commands for the car's movement, encompassing forward, backward, left, and right motions. Simultaneously, the camera module adeptly captures live video from the car's perspective, instantly transmitting it to a user-friendly interface for seamless interaction.

INDEX TER M: Hand Gesture Control, ESP32 controller, motor driver, MPU 605 sensor, Prototype, and Internet of Things,

INTRODU CTION:

A literate survey of IoT-based Hand Gesture Control Robots involves a comprehensive overview of the existing research, technologies, applications, and challenges in this field. This survey will help you understand the current state of the art and trends in IoT-based hand gesture control robots.IoT and Robotics: Introduce the concept of the Internet of Things (IoT) and its integration with robotics, highlighting the significance of IoT in enabling remote control and data exchange for robots. Gesture Control: Explain the concept of hand gesture control and its relevance in human-robot interaction, emphasizing its potential for intuitive and natural communication with robots. Technologies and Components: Sensors: Discuss the various sensors commonly used for hand gesture recognition, such as cameras, IMUs, and ultrasonic sensors. Explain their working principles and advantages/disadvantages. IoT Modules: Describe the IoT communication modules (e.g., Wi-Fi, Bluetooth, cellular) and their role in connecting the robot to the internet and enabling remote control. Microcontrollers/Processors: Explain the role of microcontrollers or single-board computers (e.g., Raspberry Pi, Arduino) in processing sensor data, running gesture recognition algorithms, and controlling robot actuators. Gesture Recognition Algorithms: Discuss the different algorithms and techniques used for interpreting hand gestures, including computer vision, machine learning, and pattern recognition methods.. Gesture Recognition Approaches: Computer Vision-Based: Explore computer vision techniques, such as image processing and deep learning, used for hand gesture recognition in IoT-based robots. Discuss the challenges and recent advancements. Sensor Fusion: Describe how data from multiple sensors (e.g., camera and IMU) can be combined to improve gesture recognition accuracy and robustness. Predefined Gestures vs. Machine Learning: Compare approaches that use predefined gesture patterns with those that employ machine learning to adapt to user-specific gestures. IoT Communication: Communication Protocols: Discuss commonly used communication protocols like MQTT, Web Socket, and RESTful APIs for transmitting gesture data and control commands between the robot and remote interfaces. Low Latency and Real-time Communication: Emphasize the importance of low-latency communication to achieve real-time control of the robot.

DES CRIPTION OF PROJE C T:

This project is focused on creating a "Hand Gesture Control Robot." Its primary goal is to enable the control of a car using human hand gestures. To achieve this, the project relies on an accelerometer to sense these hand gestures accurately. The code is meticulously crafted to ensure that the car responds appropriately to specific human gestures. The project employs two Motor driver shields (L239N) to drive the four gear motors, enabling precise control of the car's movements.

To simplify the complexity of the wiring and circuitry, the project utilizes an ESP32, which streamlines the overall circuit design, making it less intricate. Additionally, an exciting feature has been incorporated into the projecta camera. This camera allows users to view the surroundings from the moving car's perspective and provides the ability to capture images from different angles, including up, down, right, and left. Together, these components and features make the Hand Gesture Control Robot an innovative and interactive project that combines gesture recognition and remote viewing capabilities for an enhanced robotic experience

OBJE CTIV ES :

1. Connect and communicate with physical devices: IoT facilitates the communication between human and machine.

2. Faster and Smart innovation: Speed is very crucial aspect of any tool. Because of use of sensors in IoT devices the required output is given in a good speed with great accuracy.

3. Smart sensing capabilities: Sensors such as accelerometer can sense very minute movement, for instance a little vibration, which humans cannot even recognize. It has tolerance just about 5 -10%. So, the device works very precisely and can be used for such works where errors must be minimized.

IV. Convenience: We can manifest very little movement on very large scale. In this way, we can do maximum work which requires minimum human energy.

COMPO NEN TS :

To create a wireless-controlled robot with motion sensing and potential camera capabilities, a set of essential components is required for both the transmitter and receiver units. The transmitter employs an ESP32 Module, a versatile microcontroller known for its built-in Wi-Fi and Bluetooth capabilities, ideal for IoT projects and wireless communication. It also incorporates an MPU6050 motion-tracking device, featuring a gyroscope and accelerometer, used for measuring orientation and motion. A power supply is necessary to provide electrical power to all transmitter components, while a breadboard and jumper wires facilitate temporary electrical connections and prototyping.

On the receiver side, there are four Gear Motors, electric motors equipped with gears, commonly used for driving wheels or mechanical parts in robotics projects. These motors are complemented by four Rubber Wheels that provide traction and enable the robot's movement. Similar to the transmitter, an ESP32 Module is employed for wireless communication and control. Two Driver Motor Shields, specifically the L293N, are utilized to control the speed and direction of the gear motors. To power the receiver components, a 7-12V DC Battery is essential, particularly for the motors. Single strip wire aids in making electrical connections and soldering in electronic projects, while jumper wires help establish connections on the receiver's breadboard. Lastly, the inclusion of an ESP Cam Module adds a visual component, potentially enabling image or video capture capabilities to the project. Altogether, these components form the foundation for a versatile and capable wireless-controlled robot, thanks to the ESP32 microcontroller's capabilities and the integration of various sensors and motor control mechanisms.

ORKIN G:

Hand Gesture Control Robot is control by human hand. Here there are two Phase of Robot one is transmitter phase and another is receiver phase. We use ESP32 module for controlling the car by gesture.

The "Hand Gesture Control Robot" operates through human hand gestures and consists of two phases: the transmitter and the receiver. In the transmitter phase, an ESP32 module is utilized to control the robot and execute gesture-based algorithms. This ESP32 module, functioning as a Wi-Fi transmitter, sends data to another ESP32 module at the receiver end. To detect motion, the transmitter phase incorporates an MPU6040 module for gyroscope and acceleration data. Commands are transmitted to the robot based on the x, y, and z-axis movements of the hand. To establish a connection between the transmitter and receiver, the MAC address of the receiver's ESP32 is added to the transmitter's code, allowing for automatic pairing when both units receive power.

Moving on to the receiver phase, it forms the core of the robot. Here, another ESP32 module is employed to control the car using specific code, which includes necessary libraries. Two L293N Motor driver shields are directly connected to the ESP32 and control the four motors based on hand gesture movements. These motor drivers are responsible for the front right, front left, back right, and back left motors. In the absence of initial commands, the motors remain stationary. Additionally, an ESP cam module and a two-axis servo assembly are integrated for camera control, enabling movement in various directions, including upward, downward, right, and left. The camera feed can be viewed on a mobile phone. Power is supplied separately to the camera module, and jumper wires are used for convenient connections. The receiver phase is powered by a 7-12V DC power supply, while the transmitter phase is powered by a power bank.

CIRCUI T DI AGRAM ( Gesture ) :

ESP32

(Microcontroller)

MPU6050

(Sensor)

Figure 1: Circuit Diagram of Hand Gesture Control Circuit

This figure showcases a Hand Gesture Circuit Diagram typically includes components such as sensors (like flex sensors or proximity sensors), a microcontroller, power supply elements, LEDs or displays to indicate gestures, and connecting wires. Sensors detect hand movements, which are processed by the microcontroller to trigger specific actions or display corresponding information on the LEDs or displays. The power supply ensures the circuit operates, while wires establish connections between components. Overall, this diagram illustrates how the hardware elements work together to interpret and respond to hand gestures.

CIRCUI T DI AGRAM ( ROBOT) :

Figure 2: Circuit Diagram of Hand Gesture Control Robot

This figure showcases a Hand Gesture Control Robot Circuit Diagram is a schematic representation of the electrical connections and components that enable a robot to be controlled using hand gestures. It typically includes components such as gesture sensors (e.g., accelerometers or cameras), a microcontroller, motor drivers, motors or wheels for movement, a power supply, and connecting wires.

The gesture sensors detect and interpret hand movements, which are then processed by the microcontroller. The microcontroller translates these gestures into commands for the motor drivers, which control the robot's movement. The power supply provides the necessary voltage to power the components, and wires establish the connections between them.

This circuit diagram illustrates how the various hardware elements work together to allow the robot to respond to specific hand gestures, enabling users to control its actions and movements.

PROTOTYPE :

3 (A) 3 (B)

Figure 3 (A) & (B): Actual images of Hand Gesture Control Robot

This figure showcases a Hand Gesture Circuit Prototype is a physical, functional model of a circuit designed to interpret and respond to hand gestures. It typically consists of actual electronic components like sensors, a microcontroller, LEDs or displays for feedback, and connecting wires on a breadboard.

In this prototype, the sensors detect hand movements, which are then processed by the microcontroller. The microcontroller translates these gestures into specific actions, which might include lighting up LEDs or controlling other devices. It serves as a proof of concept or testing platform to demonstrate the feasibility of a hand gesture-based control system before moving to a more permanent and compact implementation.

Overall, a hand gesture circuit prototype helps developers experiment, refine, and showcase the functionality of their gesture control concept.

Merit s a n d d emerits o f th is ro b o t

This robot offers several distinct advantages. Firstly, it doesn't require manual operation or smartphone control; instead, it can be controlled through the movement of a human hand, offering a more intuitive and interactive experience. Secondly, it boasts a good range, allowing it to operate efficiently within various environments. Moreover, it's customizable with the ability to add extra control modes such as Bluetooth control, voice control, and obstacle avoidance.

This versatility allows users to navigate the robot in multiple ways, including forward, backward, leftward, and rightward movements, all while using the same accelerometer sensor to manage the car's throttle. In terms of assembly, the robot's simple circuit connection makes it accessible to a wide range of users while minimizing complexity. Lastly, this robot enhances accessibility, optimizes space utilization, and offers customization options, making it adaptable to different needs and scenarios.

However, there are some downsides to consider. One of the drawbacks is the cost involved in creating this robot, primarily due to the use of two microcontrollers. This cost factor may limit its accessibility to some users. Additionally, the robot consumes a

significant amount of power, which can be a concern for battery life or energy efficiency in certain applications. Another disadvantage is its relatively lower stability compared to other robotic systems, potentially impacting its performance, especially in challenging terrain or dynamic environments. Lastly, the robot has a limited gesture vocabulary, which means that its range of responsive movements may not be as extensive as some other robotic systems, potentially limiting its adaptability in certain scenarios.

CONCLUSION:

To conclude, integrating hand gestures with robotic car control represents a significant advancement in the field of autonomous vehicles. This innovation holds the potential to enhance user experience by providing a more intuitive and interactive way to command and communicate with robot cars. While challenges in accuracy and standardization remain, continued research and development in this area could lead to safer, more efficient, and user-friendly autonomous driving systems in the future.

REFERENCES:

[1] B Deepak Soubagya Nayak and Jalumuru Nalin "Development of Gesture Controlled Robot Using 3-Axis Accelerometer" Journal of Control & Instrumentation. vol. 7 pp. 23-34 2016.

[2] Prajwal Ashwin Jawalekar "Robot Control by Using Human Hand Gestures" International Research Journal of Engineering and Technology (IRJET) 2018.

Chan Wah NG, Surendra Ranganath, Real-time Gesture Recognition System And Application Image And Vision Computing(20): 993-1007, 2002.

[3] Argha Samanta Sangharsh Kumar Deepak Saraswat Bharnab Das Vibhuti Kumar and Harpreet Kaur Channi Modeling and Designing of Gesture Control Robot 2017.

[4] Sergios Theodoridis, Konstantinos Koutroumbas, Pattern Recognition, Elsevier Publication, Second Edition, 2003.

[5] [Seyed Eghbal Ghobadi, Omar Edmond Loepprich, Farid Ahmadov Jens Bernshausen, Real Time Hand Based Robot Control Using Multimodal Images, Iaengm International Journal Of Computer cience, 35:4, IJCS_35_4_08, Nov 2008.

[6] Premangshu Chanda Kanti Mukherjee Pallab Modak Subrata Nath and Asoke "Gesture Controlled Robot using Arduino and Android" International Journal of Advanced Research in Computer Science and Software Engineering (IJIRCSSE) vol. 6 pp. 227-234 2016.

Archika Setia Surbhi Mittal Padmini Nigam Shalini Singh and Surendra Gangwar "Hand Gesture Recognition Based Robot Using Accelerometer Sensor" International Journal of Advanced Research in Electrical Electronics and Instrumentation Engineering 2015.