Camera Based Color Identification Robot for Typecasting

Camera Based Color Identification Robot for Typecasting

Manjunatha V G

Project Assistant, Department of Aerospace Engineering, Indian Institute of Science.

Abstract – The goal of this project is to implement a programmable industrial robot for color sorting. This investigates the development of an intelligent and low-cost monitoring system for color identification and segregation. The main purpose is to optimize the productivity and avoid human mistakes. A serial image acquisition device (camera) is used to capture the image and are sent to SD card through a microcontroller. The microcontroller performs color detection algorithm to recognize the dominant color of the object, and it sends commands to the robotic arm to pick and place the objects to their respective locations. A robotic arm is a manipulator, which has about same number of degrees of freedom as in human arm. We use DC motors for joint rotations involved in the robotic arm, these motors are interfaced with microcontroller through motor driver circuits. These motor drivers are able to efficiently control the speed and direction of motors.

Keywords: color identification and segregation, PIC controller, serial camera, SD card, robotic arm and motor drivers.

1. INTRODUCTION

   Traditionally, most of the industrial work is done by the operator manually. One of the major and the most commonly performed work in automation industries are picking and placing of objects from one place to another. Pick and place robots are used over a wide variety of material transfer applications. A good example is a robotic arm picking up objects off a conveyer belt and placing them into packaging boxes as shown in figure 1.

   The existing method is the color sorting system with robot arm that will be operated using a computer and 8 bit controller [1]. In this method entire implementation is done on a Peripheral Interface Controller without active participation of a computer. The main objective of the project is to sort the object according to their color using a serial camera and also to sort the object to the station accordingly. The benefits are that it will increase the speed and accuracy of the color sorting process. It will cut down the cost of color sorting process. This project will also optimize the overall productivity of that particular industry.

   Figure 1. A Typical Pick and Place Robot.

2. RELATED WORK

   Vindhya Devalla and Dr. Rajesh Singh [1], proposed Design and Development of Object Recognition and Sorting Robot for Material Handling in Packaging and Logistic Industries. In this proposal a Matlab code is written for the Objrec algorithm for performing the operation. The Objrec algorithm is executed to identify the object and send the appropriate commands to the microcontroller using serial communication for the robot to perform the sorting operation.

   Weilin Cai Tao Xiong [2] proposed, pick-and-place robot is usually 4 degrees-of freedom (DOF) apparatus and indispensable in semiconductor equipment, such as SMT (surface mount technology) equipment, die chip bonder. Its operation object is chip which is light in weight and small in size. It is difficult to manipulate this small object and pick- and-place robot must be accurate enough to accomplish this operation. Placement accuracy is the most important index to evaluate it. To improve placement accuracy, machine vision is generally utilized to provide the exact pose parameters of chip and pad, which two must be aligned with each other before placement. This paper proposes

   an effective vision-based kinematic calibration method which calibrates 3 translational DOFs and 1 rotational DOF of pick-and-place robot, respectively. For the 3 translational DOFs, a down-look vision system takes an image and obtains position of a fixed reference cross mark when it moves with pick-and-place robot and the relationship between displacement of pick-and place robot and position parameters of the cross mark in down look vision system is the key factor to calibrate it. For the rotational DOF, error parameters of it can be derived after the projective curve of its trajectory in the image plane of up-look vision system is fitted with an ellipse. Placement accuracy is improved to better than ±30_m in the experiment after kinematic calibration. Result shows this vision-based kinematic calibration method is valid and very suitable for 4-DOF pick-and-place robot in flip chip bonder.

   Antonio J. Sanchez, et.al [3] implemented, this paper presents a programming robot-arm system for carrying out flexible pick and place behaviour using visual perception. Object manipulation from visual data involves determining the pose of the object with respect to the manipulator. Taking into account visual positioning is an ill-posed problem due to the perspective projection, this system uses a camera and a sensor distance, and both of them mounted on a robot-arm tool adapter, for locating (positioning and orienting) objects. This programming system is modular, composed by different dynamic link libraries to be independent with the hardware and offers a friendly graphic interface, where the user can define pick and place object locations on the image space.

   Ilhan Konukseven [4] implemented, robotics research continuously seeks to improve productivity in manufacturing automation. With the recent advances in the areas of vision and sensing, robots have become a major element of today's industrial world. The objective is to develop a multisensory controlled robotic tracking and automatic pick and place system. The system is designed for recognizing and tracking an object which is selected from multiple objects that are unknown and randomly placed on a moving conveyor belt, using a vision, infrared and encoder sensors in the feedback loop. The vision processing and object recognition portions of the program are given top priority because of the variable computation time. The goal is to make robots more adaptive and flexible in unstructured or frequently changing environments, and to enable robots to execute intelligent tasks. Thus the robot productivity as well as applicability can be improved.

   L.A. Reeves [5] presented, with a demand for increased productivity from robotic manipulators, multiple robot arms are being used for applications which are more time- consuming, difficult or impossible for a single arm. Because of increased computational complexity, a single processor is opening inadequate to meet real-time demands. We present a pick-arid-place operation for two robot arms, controlled and synchronized by a multiprocessor real-time operating system.

3. PROPOSED SYSTEM

   The functional block diagram is as shown in the above figure2. Initially microcontroller has to send commands to

   the camera for synchronization and initialization. Once these are set the camera is ready to capture the image and stores in its module. The stored raw data of the image will be sent to the microcontroller through interfacing circuit using UART protocol. Microcontroller will receive data for each pixel and stores it to SD card. Later these data are fetch back to calculate the dominant color. A counter is assigned for red, green and blue color components, it increment only if the pixel value exceeds threshold value of the color intensity. This process is done for the entire picture. Based on the counter value, color is been decided and appropriate signals are given to the robotic arm to pick and place the object to the respective location and return back to its initial position. Robotic arm is connected through driver circuits which are interfacd to the microcontroller.

   Figure 2. Functional Block Diagram.

4. HARDWARE IMPLEMENTATIONS

   The whole system design can be divided into following sections

   • Camera module

   • Pic controller

   • Motor and Robotic arm.

     1. Camera Module

        We use uCAM (MicroCAM), it is a highly integrated and compact camera module which can be interfaced to any host system that requires an image acquisition device. uCAM module uses Omni Vision CMOS VGA colour sensor accompanied by an inbuilt JPEG compression chip as shown in fig 3. This module is a low powered and low cost camera system that can be operated at either 3.3V or 5V. It has an on-board UART serial (TTL) interface that is suitable for a direct communication between camera and any host micro-controller or a PC COM port.

        Figure 3. Serial JPEG Camera.

        Camera is configured for 160×120 resolution without JPEG compression and baud rate is set at 9600bps. The basic commands for working of the camera are mentioned in

        the table 1. Initially we need to send commands through microcontroller for synchronization of the camera. The controller will receive acknowledgement from the camera. Once it receives, controller has to send commands for initializing, taking snapshot and getting picture. Once the picture is been taken, the raw data of the image will be transferred to controller by pixel by pixel form. Each pixel is 16 bit wide, where the first 5 bits is for RED colour, and the next 6 bits for Green colour and last 5 bits for BLUE.

        Table 1. Commands for Camera.

        Functionality	Camera Commands
        SYNC	0xAAh, 0x0Dh, 0x00h, 0x00h, 0x00h, 0x00h
        ACK	0xAAh, 0x0Eh, 0xFFh, 0x00h, 0x00h, 0x00h
        INITIAL	0xAAh, 0x01h, 0x00h, 0x06h, 0x01h, 0x03h
        SNAP SHOT	0xAAh, 0x05h, 0x01h, 0x00h, 0x00h, 0x00h
        GET PICTURE	0xAAh, 0x04h, 0x02h, 0x00h, 0x00h, 0x00h

     2. Pic Microcontroller

        In 1989, Microchip Technology Corporation introduced an 8 bit microcontroller called Peripheral Interface Controller (PIC). PIC18x series has the highest performance of all the families of 8-bit PIC controllers. The PIC18x has a RISC architecture that comes with some standard features like, on-chip program ROM, data RAM, data EEPROM, timers, ADC and USART. Normally it has 10 bit ADC and the number of channels can varies from 5 to 16. It can have up to four timers besides watchdog timer and 5 I/O ports viz, PORTA, PORTB, PORTC, PORTD and PORTE.

        Detailed circuit connections and dedicated functionality of each port is explained below, however we are not using PORTA and PORTE since they are analog I/O ports.

        1. PORTB Register and its connections. PORTB occupies a total 8 pins (RB0 RB7). To use the pins of PORTB as both input and output ports, each bit must be connected externally to the pin by enabling the bits of register TRISB. It is also called data direction register, by configuring it we can control the direction of data transfer. The three pins of PORTB register are multiplexed with the Low-Voltage Programming function and In-Circuit Debugger: PGM/ RB5, PGC/ RB6/ and PGD/ RB7. Each of the pins in PORTB register has a weak internal pull-up, so we used external pull up resistor to provide sufficient voltage drop across the terminals. A single control bit can also be used to turn on all the pull-ups. This is accomplished by making RBPU bit zero. But pull-ups are disabled when the port pins are configured as an output.

           Figure 4. PORTB with motor connections.

           We used PORTB for interfacing DC motor drivers. The connections are as shown in the figure 4.

   • PB7- Base motor pin1

   • PB6- Base motor pin2

   • PB5- Arm motor pin1

   • PB4- Arm motor pin2

   • PB3- Gripper motor pin1

   • PB2- Gripper motor pin2

     1. PORTC Register and its connections. PORTC occupies a total 8 pins (RC0 RC7). To use the pins of PORTC as both input and output ports, each bit must be connected externally to the pin by enabling the bits of register TRISC. We are using this port for interfacing Serial Camera and SD card, which supports UART protocol and SPI as shown in figure 5. RC6 and RC7 are responsible for USART communication, which serves as RX and TX functions. Pin RC6 is connected to Rx pin of the camera and RC7 to TX pin of camera. RC5 and RC4 are used for data transfer in SPI and RC3 for clock generation.

     2. PORTD Registers and its connections. PORTD is an 8-bit bidirectional port with Schmitt Trigger input buffers. We are using this port to interface LCD (figure 6) The Liquid Crystal Display module is a parallel interface 16 pin module. The first 3 pins of LCD module are used for contrast adjusting and next three pins are control lines.

     Figure 5. PORTC with uCam interfacing.

     The pins, 7-14 are data lines (D0 to D7). In this particular circuit the data lines D4 to D7 are used because the LCD is operated in 4-bit mode.

     Figure 6. PORTD with LCD interfacing.

     1. Motor And Robotic Arm.

        1. Robotic Arm. A robotic arm is a programmable mechanical arm, analogous with functionality of a human arm. The links of a robotic arm are interconnected by joints allowing either translational or rotational motion. Robotic arm has three segments namely base, elbow and gripper. Each segment has one degree of freedom which means it will have six movements. Motor will be used to move the robot arm and will be located at every joint of the robot arm. Table 2 gives the details about functionality of each segments.

           Table 2. Robotic arm functions

           Segmen ts	Function ality	Connection to controller	Specific ation	Angle of movement
           Gripper	Pick and release	PB4, PB5	100 RPM, 12V	Limited by gear assembly
           Elbow	Up and down	PB6, PB7	100 RPM, 12V	180°
           Base	Rotation al	PB2, PB3	60 RPM, 12V	360°

        2. Motors. Permanent-magnet direct-current (PMDC) motor is used. DC Motor is inexpensive, small and powerful motors. These motors are flexible because both their direction and speed can be controlled; direction by polarity and speed by voltage. DC Motors are extensively used in robotics because of their lesser size and more energy output, they are excellent for driving the mechanical assemblies.

        3. Motor Driver. A push-pull high current, high voltage four channel driver, L293D is used to control the robotic arm. It is designed to source bidirectional drive currents of up to 1Amperes (A) at voltages from 4.5 – 36 Volts (V). When the enable input is set, the associated drivers are activated, when the enable input is clear, those drivers are deactivated and their outputs are low and in the high- impedance state.

5. SOFTWARE IMPLEMENTATION

   1. MikroC IDE

      MikroC is a rich featured and user friendly integrated development tool for Peripheral Interface Controllers (PIC). It is designed to deliver the user with the easiest possible solution for creating an embedded applications, without compromising in the performance or control. PIC and C language fit together well: PIC is the most rated microcontroller in the world, used in a wide range of applications, and C, prized for its flexibility and efficiency. MikroC affords an effective match featuring highly advanced IDE, broad set of hardware libraries, ANSI compliant compiler, coprehensive documentation, and plenty of ready to run examples.

      Figure 7. Workspace of MikroC Compiler.

   2. Circuit Diagram

   The circuit diagram of this project is shown in figure 8.

   Figure 8. Circuit diagram.

6. APPLICATIONS

   Waste Management Applications: – It is used in garbage classifier to separate bio-degradable, non-degradable, metal pieces, and plastics based on color. It can also be used to segregate and separate dry waste and wet waste based on color.

   Automotive Agro Applications: – The Pick and Place robot is used in agriculture for collecting ripened fruits based on the color. The ripe fruits have a different color compared to the unripe fruits. For example, ripened tomato is bright red in color and the unripen one is green. So the pick and place robot can pick the fruits according to their necessity by identifying the color.

   Industrial Applications: – As the assembly of the machine parts involving color is a tedious task for execution, the robots are conveniently used to pick and place the parts. It can be used in packing fruits and vegetables of only specific quality in industries like the good quality apples which will be bright red in color.

   Auto industry: – The auto industries are the largest user of robots, which industrialize the production of various components and assemble the body of the finished vehicle based on colours. Car production is one primary example of the employment of complex and large robots. Pick and place robots based on color identification are used in that process for the picking and placing the components for assembly of the body of the automobiles.

7. RESULTS AND ANALYSIS

   These are few snapshots taken from the serial JPEG camera. One is in color as shown in the figure 9 and the other one is in black and white mode as shown in figure 10.

   Figure 9. 8 bit Color mode.

   Figure 10. Black & White mode.

   The snapshot of entire project hardware as a product is also been shown in figure 11. It consists of the robotic arm

   and the serial JPEG camera interfaced with the PIC 18f452 microcontroller.

   Figure 11. Snapshot of our project kit.

   The table 3 shows the analytical report of the project. The numbers of red colored objects of average load pick and placed to the desired location by the robot in one minute is three, three blue colored objects and two green objects.

   Table 3. Analysis of work

   Color	Red	Blue	Green
   No of objects to pick and place in 1 minute	3	3	2
   No of objects to pick and place in 10 minute	37	36	23

   • Maximum distance at which the camera detects the color in the object is 3.6mm.

   • The range of load the gripper can hold is 10grams to 250 grams.

   The comparison of the conventional method, the existing system and our proposed project is been shown in the table 4.

   Parameters	Conventio nal Method	Existing system	Our Proposed project
   The color detecting device	Human eye	USB Camera	Serial JPEG camera
   Programmer	Human	Host PC	8 PIC microcontro ller
   The device used to pick and place the objects	Human hand	Robot using DC motors	Robot using DC motors
   Ease of work	Difficult	Average	Very easy
   Speed at which the work is completed	Slow	Average (depends on clock speed of the computer)	Fast
   Time consumed to complete the work	Takes too much time	Average	Takes less time to complete the work
   Accuracy	Less accurate	High	More accurate
   Efficiency	Less efficient	High	More efficient
   Cost of the system	Huge labour cost	6,000 + Cost of PC	6000-7000
   Maintenance cost	——-	High	Low
   Drawback	Humans tend to make mistakes	Computer is required	—–

   Table 4. Comparison of the various systems.

   responding the situation. Future pick and place robots will be automated and able to move under certain navigation systems.

8. CONCLUSION

   Robotics is a technology with a future, and it is a technology for the future. In this project we are introducing a robot which can pick and place objects based on the color. Earlier this was done using color sensors but we have improved the entire project by using a serial jpeg camera which can capture and identify or recognize the color of the image and eventually the robot picks and places that object accordingly. By doing so the speed and the accuracy of the color sorting process is increased. The cost for the color sorting process is considerably reduced. And most importantly there is overall optimization in the productivity if an industry. This robot arm can perform an action which is much similar to human. Although there is significant progress in robotics, still its usage is limited due to less availability of resources and high cost of production. If we able to overcome these restrictions, more benefits can be achieved from robotics.

   We can put forward a likely profile of the future pick and place robot based on the several research activities that are presently being performed. This project can be enhanced to recognize shape and size recognition as well. The pick and place robot will have a wide range of sensing capabilities including vision, tactile sensing, shape and size, and others. This is vital to enable robots to accomplish complex tasks that require some active controlling in

9. REFERENCES

1. Vindhya Devalla, Dr. Rajesh Singh, Amit Kumar Mondal and Vivek Kaundal, Design and Development of Object Recognition and Sorting Robot for Material Handling in Packaging and Logistic Industries International Journal of Science and Advanced Technology , Volume 2 No 9 September 2012.

2. Weilin Cai, Tao Xiong and Zhouping Yin, Vision-based Kinematic Calibration of a 4-DOF Pick-and-place Robot, International Conference on Mechatronics and Automation, 978-1-4673-1278-3, Proceedings of 2012 IEEE.

3. Antonio J. Sanchez and Josk M. Martinez Robot-arm Pick and Place Behaviour programming System Using Visual Perception, Universidad Politinica de Valencia, Spain0- 7695-0750-6, Proceedings of 2000 IEEE.

4. Ilhan Konukseven, Bilgin Kaftanoglu Moving Part Recognition and Automatic Pick and Place Using an Industrial Robot Department of Mechanical Engineering Middle East Technical University 06531 Ankara, Turkey, 0- 7803-41 19, Proceedings of 1997 IEEE.

5. L.A. Reeves and M. Farooq A Multiprocessor Pick-and- Place Operation for TWO Robot Arms Dept. of Electrical and Computer Engineering, Royal Military College of Canada, Kingston, Ontario,0-7803-1443-3 ,Proceedings of 1993 IEEE.