Human Computer Interaction: Game Developed in Python using Gesture Recognition

Human Computer Interaction: Game Developed in Python using Gesture Recognition

Shikhar Singh

SCOPE School, Vellore Institute of Technology, Tamil Nadu , India.

Dr. Swarnalatha P

SCOPE School, Vellore Institute Of Technology Tamil Nadu, India

Abstract This paper has been taken up to demonstrate the an emerging field of Human Computer Interaction i.e. Gesture Recognition. Gesture recognition has found its way into many applications ranging all across from basic home automation to navigation and gaming. In this paper, I have explored the capabilities of gesture recognition by demonstrating the classic Snake game as a gesture controlled snake . We use Computer vision techniques and the OpenCV library to achieve our results, which are similar if not more fluid then the original mechanical version that makes use of the arrow keys. The key aspects of the game and how it fairs to the original are also listed in this document. The future scope of such an application is also discussed.

KeywordsGesture recognition; computer vision; OpenCV.

1. INTRODUCTION

   In vision-based interfaces for video games, gestures are used as commands for the games instead of pressing buttons on a keyboard or moving a mouse. In these interfaces, unintentional movements and continuous gestures must be supported to provide the user with a more natural interface.

   The paper I decided to work on is the classic snake game. This is a game in which, the user uses a red coloured object and moves it around in front of a webcam so that a trail follows the object on its image and represents the shape of a snake. The size of this snake keeps on increasing in size and the game becomes progressively more difficult to play.

   This project has been coded in Python . The libraries used are NumPy and OpenCV . This project is considered to be under the field of gesture recognition which is a field which is gaining widespread popularity as it makes it very easy for the user get some particular tasks done. Gesture recognition has slowly but surely created a stronghold in the gaming industry and now moving beyond it in every aspect. Together with computer vision it is enabling for a much better user experience and dynamic and user-friendly interfaces.

   In our paper, we used techniques like colour detection contour detection drawing, placing png over image and checking for intersection of line segments to achieve results.

2. BACKGROUND

   The background of the implementation lies in the idea to develop an interesting gaming application for the children in which they can get immersed and have a fabulous experience.

   The aim was to deliver the following features to the user :

   • Error free calculation of score

   • Prompts to follow next instructions.

   • Accurate object recognition to prevent false results.

   • Negligible response time to react to object movement.

   • Prompt readjustment of target location.

   • Easy to exit application.

   The game should be addictive and provide a strong will to succeed and stay in the game. Every object class has its own special features that help in classifying the object.

   object recognition is that sub-domain of computer vision which helps in identifying objects in an image or video sequence. With more efficient algorithms, objects can even be recognized even when they are partially obstructed from the direct view. Various approaches to this task have been implemented in the past years.

3. HUMAN COMPUTER INTERACTION

HCI is the domain of computer science which deals with how users interact with their systems and how the user experience and the user interfaces can be optimized to improve user satisfaction.

The following points discuss this project with respect to the HCI paradigm.

The input channel used is the snake head which is an object that is recognized by the interface through Computer Vision and its relative position is mapped at every instance of the way. Movement of the said object causes a change in the relative motion of the snake drawn on top of the image captured through the webcam. Thus the interface is quick and responsive.

object recognition and color detection in computerized pictures has gotten one of the most significant applications for enterprises to ease client, spare time and to accomplish parallelism. This is certainly not

another method, yet improvement in object recognition is still required so as to accomplish the focus on target to be all the more productively and precise.

The principle point of considering and investigating computer vision is to recreate the conduct and way of human eyes legitimately by utilizing a computer and later on build up a framework that diminishes human effort.

Computer vision is such sort of research field which attempts to see and speaks to the 3D data for world items. Its fundamental reason for existing is remaking the visual parts of 3D protests in the wake of breaking down the 2D data extricated. Genuine 3D objects are spoken to by 2D pictures. The procedure of item discovery examination is to decide the number, area, size, position of

Design

Figure 1 : Design of gesture recognition process

the articles in the input image.

object detection is the essential idea for following and acknowledgment of items, which influences the proficiency and precision of object acknowledgment.

The normal object recognition technique is the color based methodology, distinguishing objects dependent on their color intensities. The technique is utilized on account of its solid flexibility and power, be that as it may, the recognition speed should be improved, in light of the fact that it requires testing every single imaginable window by thorough pursuit and has high computational unpredictability.

List of Nielsen heuristics applicable to the game

1. Visibility of system status : The score is updated in real time and user is always aware of his/her status.

2. Match between system and real world : The game communicates with the user in a language that the user will understand.

3. User Control and freedom : Supports undo and redo as it is has three states I.e start, playing and end.

4. Consistency and standards : The rules of the game are clearly defined and standardized for all plays.

5. Recognition rather than recall : Game is played based on recognition of target and reaction of user.

6. Flexibility and Efficiency of use : The interaction takes place at real time and is there is no concept of lag

7. Aesthetic and minimalist design.

Not applicable :

1. Help and Documentation.

2. Help users, recognize ,diagnose and recover from errors.

3. Error prevention.

1. Conversion of each frame of the video into hsv format from rgb format.

2. Setting of lower and upper limits of hsv values being tracked.

3. These lower values are then converted to Numpy array values which are processed by the computer.

4. Colour is located on the screen and their position coordinates are tracked.

5. The centre of the object is identified after using morphological transformation.

6. The object is tracked and a trail mask is created which follows the course of the red object and takes the form of the body of the snake.

7. The snake increases in size at the rate of 40 px per second and the game becomes progressively more difficult to play.

8. An apple is generated at random positions inside the frame. Whenever the snake passes through the apple, the score increases by 1.

9. Whenever the snake intersects with its own body, the game ends and GAME OVER and the game ends.

10. The user can then press any key to exit.

Figure 2 : Flowchart for Snake Game

TECHNOLOGIES USED

Python

Python is an interpreted high level programming language used for a wide host of functionalities.

Libraries used :

1. OpenCv

   OpenCV (open source computer vision) is a library of programming functions mainly aimed at real-time computer vision.

2. NumPy

   NumPy is a library for the Python programming language, adding support for large, multi- dimensional arrays and matrices, along with a large collection of high-level mathematical functions to operate on these arrays.

   ADVANTAGES

   The conceivable outcomes of utilizing computer vision to take care of genuine issues are many. The essentials of item recognition alongside different methods for accomplishing it and its extension has been talked about. Python has been favored over MATLAB for coordinating with OpenCV on the grounds that when a Matlab program is run on a PC, it gets caught up with attempting to translate all that Matlab code as Matlab code is based on Java.

   RESULTS

   Player 1 ( <10 yrs )	3
   Player 2 (10 – 20 yrs)	6.2
   Player 3 ( >20 yrs )	15.5
   Player 4 ( >20 yrs )	22
   Player 5 ( >20 yrs )	32.3

   Player 1 ( <10 yrs )	3
   Player 2 (10 – 20 yrs)	6.2
   Player 3 ( >20 yrs )	15.5
   Player 4 ( >20 yrs )	22
   Player 5 ( >20 yrs )	32.3

   User Performance in the game . Average score in five plays. Player Avg Score

   Table 1 : User Performance

   FUTURE SCOPE

   Computer vision is as yet a creating discipline, it has not been developed to that level where it very well may be applied legitimately to genuine issues.

   After few yearsComputer vision and especially the item location would not be any progressively advanced and will be universal. For the present, we can consider object location as a sub-part of AI.

   REFERENCES

   1. HOWSE, JOSEPH. OPENCV: COMPUTER VISION PROJECTS WITH PYTHON. PACKT PUBLISHING LIMITED, 2016