Classification of Affected Fruits using Machine Learning

Classification of Affected Fruits using Machine Learning

Heena Shaikh, Yashashree Wagh, Siddhesh Shinde, Shankar M. Patil

Department of Information Technology Bharati Vidyapeeth College of Engineering

Navi Mumbai, India

Abstract Consuming good, healthy and quality fruit is the utmost requirement of the consumer. Hence, automation in food industries is growing in recent times. Because it is impractical for humans to manually inspect the fruits as it requires a large number of labors as well as requires a lot of time and effort. This paper proposes a system which is capable of detecting and classifying the fruits as affected or not based on surface using Faster R-CNN model. As this system is automatic, it cuts down the traditional/manual method of fruit inspection, which takes a lot of time, labor and effort. This system will not only save the cost of labors but will also give a high accuracy.

Keywordsquality, automation, Faster R-CNN, Object detection, TensorFlow.

1. INTRODUCTION

   Fruits are essential for balanced diet and good health. Fruits are good sources of vitamins and minerals without which human body cannot maintain proper health and develop resistance to disease they also contain pectin, cellulose, fats, proteins etc. Importance of fruits in human diet is well recognized. Man cannot live on cereals alone. Fruit growing is one of the important and age old practices, practiced in India since ancient times. Cultivation of fruit crops plays an important role in overall status of the mankind and the nation. The standard of living of the people of a country is depending upon the production and per capita consumption of fruits. Fruit growing have more economic advantages. [1]

   The fruit industry plays a vital role in a countrys economic growth. They account for a fraction of the agricultural output produced by a country. It forms a part

   of the food processing industry. Fruits are a major source of energy, vitamins, minerals, fibre and other nutrients. They contribute to an essential part of our diet. Fruits come in varying shapes, colour and sizes. Some of them are exported, thereby yielding profit to the industry. Fruit sorting and grading are performed before export. This determines the quality of the fruits which is an important factor in the food processing industry. Nutritionist advocates 60-85g of fruits and 360 gm. Classification of fruits by processing and preparation methods could be especially important in cultures where there is reliance on a limited number of local crops and the processing techniques alter the composition so as to limit the intake of critical food components. For populations that have access to a wide variety of fruit and a range of processing and preparation methods, these methods are not likely to be useful as classification terms.

   Quest for increased performance and reliability has made it imperative to develop techniques for utilization of resources to make informed decisions. [2]. Hence, automatic inspection of raw ingredients as well as end products in food industry is getting more consideration in recent times. The ultimate goal is to produce a high quality product by means of an efficient, rapid and non- destructive production process. In previous years, attempts have made for non-destructive investigation of products from agriculture and food industry. Presence of worm damage and other defects has been determined in a non-destructive way. Current standard inspection methods are still unable to detect many defects. For the food industry, prior work has established the fact that images are very useful for assessment of ingredients as good or bad. In fruit inspection, the key objective is, not to pass the defected/affected fruit to the consumer. A method is required which may identify and separate the

   infected or damaged fruits from the healthy ones in an automatic way. For quality assessment, several kinds of features have been extracted from images. Hence, our work proposes to develop a system that is capable for automatic detection of fruits and quality inspection.[3][4][5][6][7]

2. LITERATURE SURVEY

   Xiaochun Mai et al. [8] have used classier fusion strategy in region proposal stage to improve localization of proposal candidates for small fruits. Classier fusion combines classication of classiers learned from low level and high-level features, which further increases reliability of object classication for small fruits. Experimental results demonstrate that their proposed model is feasible for detecting small fruits. They have also explored the effectiveness of this strategy in fast R- CNN to improve classication of objects, and can use this strategy in both region proposal network and fast RCNN. Secondly, deep learning methods have been shown to be effective on limited dataset for fruit detection. In order to guarantee detection performance under various types of fruits and variant environments, a bigger dataset would have worked. Moreover, there are some mis-labelling and annotation errors in ground truth, which results in uncertainty. Hence, based on existing detection methods, they have designed an annotation correction method to nd errors in annotation and reduce errors.

   Basri et al. [9] have presented a system that classify fruit with an accuracy level of 88% (Mango), 83% (Limes), and 99% (Pitaya) and with an average computation cost of 0.0131 m/s. They have tracked and calculated fruit sequentially without using additional sensors. Additionally, they have also checked the defect rate on fruit accurately using the video streaming camera. The datasets are taken in the field areas then trained using the FRCNN Framework using the Tensor flow platform. Faster R-CNN is one model that has proved possible in solving complex computer vision problems with the same principle and have shown incredible results at the beginning deep learning revolution. This algorithm has provided a solution for solving cases of fruit quality detection in real time using multi-classes. They would have used the tracking video system to calculate the quantity of fruits on the conveyor runway. This would have also track and calculate fruit sequentially without using additional sensors and check the defect rate on fruit using the video streaming camera more accurately and with greater ease. Also, they would have

   experimented and made an Android Apps to integrate with the analytic system.

   Hasan Basri et al. [10] have experimented by classifying fruit using Google's model architecture, the convolutional neural network has distinguishable depth wise using Mobile Nets. They have proved that the Faster R-CNN can build a training model rapidly and classify of fruits using width multiplier with checking the rightness level shrink size and latency. They have used the Mobile Net model on TensorFlow platform and has achieved the accuracy score of about 99%. This method is very appropriate for developed the process of sorting multi-fruits in real-time to maintain the quality of the fruit. The dataset of fruits, in this method is very appropriate for developing a sorting machine for classification multi-fruit in real-time with the camera and detecting moving fruits above the sorting machine for ensuring the quality of the fruit fresh. Only Mangoes and Pitaya fruits were used for classification. More fruits would have been used for classification, this would have also improved the accuracy, and would have filled up 1% gap.

   Linjuan Ma et al. [11] have introduced a fruit detector based on fruits-360, fruit-360 is dataset to perform the experiments, which contains 38409 images of fruits spreadacross 60 labels and 75 classes of the fruits dataset and has applied Faster R-CNN, which is the state-of-the art detection framework. Fruit detection is of great significance in the agriculture, which can improve labour efficiency and market price competition. More tools and framework would have been used. They would have improved the performance of our deep network to detect more fruits and meet the needs in our actual life.

3. METHODOLOGY

   In this research, we proposed a system that can detect and classify apple, pear and banana as defected or not based on surface of the fruit. We have implemented three things in this project, first is that it classifies the fruit in one of the 6 categories as Healthy Apple, Bad Apple, Healthy Pear, Bad Pear, Healthy Banana and Bad Banana. Secondly, it shows the percentage of the Healthy or Bad fruit. Finally, it uses web cam for live detection and classification of the fruit. In web cam, we hold the fruit in front of the camera and it detects the fruit as one of the 6 category and also shows the percentage of classification.

   1. System Design

      Fig 1. system design

   2. Faster R-CNN

      The architecture of Faster R-CNN is complex because it has several moving parts. Starting with a high level overview, and then getting over the details for each of the components. It all starts with an image, from which we want to obtain:

      • a list of bounding boxes.

      • a label assigned to each bounding box.

      • a probability for each label and bounding box.

        Fig 2. Complete faster R-CNN architecture

        The input images are represented as Height×Width×Depth tensors (multidimensional arrays), which are passed through a pre-trained CNN up until an intermediate layer, ending up with a convolutional feature map. We use this as a feature extractor for the next part.

        This technique is very commonly used in the context of Transfer Learning, especially for training a classifier on a small dataset using the weights of a network trained on a bigger dataset. Well take a deeper look at this in the following sections.

        Next, we have what is called a Region Proposal Network (RPN, for short). Using the features that the CNN computed, it is used to find up to a predefined number of regions (bounding boxes), which may contain objects.

        Probably the hardest issue with using Deep Learning (DL) for object detection is generating a variable-length list of bounding boxes. When modeling deep neural

        networks, the last block is usually a fixed sized tensor output

        The variable-length problem is solved in the RPN by using anchors: fixed sized reference bounding boxes which are placed uniformly throughout the original image.

        After having a list of possible relevant objects and their locations in the original image, it becomes a more straightforward problem to solve. Using the features extracted by the CNN and the bounding boxes with relevant objects, we apply Region of Interest (RoI) Pooling and extract those features which would correspond to the relevant objects into a new tensor.

        Finally, comes the R-CNN module, which uses that information to:

        • Classify the content in the bounding box (or discard it, using background as a label).

        • Adjust the bounding box coordinates (so it better fits the object).

   Obviously, some major bits of information are missing, but thats basically the general idea of how Faster R- CNN works. [12]

4. IMPLEMENTATION

   1. Data gathering

      Data gathering and analysis is the first and most crucial step because it plays an important role for decision support irrespective of type of industry. [13]. In this study, we made our own dataset. We took the pictures of real fruits and performed training on that data. This dataset includes both the clicked pictures of fruits as well as it also consists of some images of fruits downloaded from internet. The dataset consists of 488 images of apple, pear and banana. Out of which 344 belongs to training data and 144 belongs to test data. These images are further categorized in 6 categories such as Healthy apple, Bad Apple, Healthy Pear, Bad Pear, Healthy Banana and Bad Banana. Following is the table showing all categories of fruits:

      Fruit	Category/Label	Training data	Testing data
      Apple	1.Healthy Apple 2.Bad Apple	52 50	22 20
      Pear	3.Healthy Pear	66	27

      	4.Bad Pear	59	25
      Banana	5.Healthy Banana 6.Bad Banana	58 59	24 22

      Table 1. Fruits dataset

      Fig 3. Fruit dataset

   2. Image Labelling

      After collecting the dataset, the next step is to label the images. We crop the images and apply the label from one of 6 categories i.e. Healthy apple, Bad Apple, Healthy Pear, Bad Pear, Healthy Banana and Bad Banana. To do this we can use various labelling tools available. We have used LabelImg tool to label the images. This labelling is done for each and every image in the dataset. Following figure 3 shows how labelling is done in LabelImg.

      Fig 4. Labelling the image

      Once you label the image, the image is saved in XML file. This XML file consists of various information about the image such as height, width, coordinates. Following Fig 4. Shows how image is saved in .xml

      extension and fig 5. Shows information present inside xml file.

      Fig 5. Save as XML file

      Fig 6. Info. Inside XML

      Once you have the XML file, the contents of that can be converted to CSV file.

   3. Training

   After completing the the labelling process, the next step is training the images. For this purpose, first of all split the dataset into two new folders (train and test). Now copy 10% images of the total dataset along with their respective XML files into the test folder and copy remaining images along with their repective XML files into train folder. In our case, we contain the total dataset of 488 images without XML and 976 with XML. So we copied 144 images along with their XML in test folder and 344 images along with XML in train folder. We have used Faster R-CNN model for training

   purpose. We have used paperspace services for training the model. We run faster R-CNN with 6 classes along with number of steps=20k. You can view your loss graph as well. The loss should be as low as possible (0.2). This indicates that the accuracy will be high. Once, the training is completed now we proceed for testing and results.

   Fig 7. Total loss vs steps

5. TESTING AND RESULTS

   After the training is completed, now comes our testing and results phase. The below table shows the results of the live testing done with web cam. The fruits are placed in front of the webcam to test the different categories of fruits along with the percentage mentioned.

   N o	Image	Expecte d class	Predicte d class
   1		Bad Banana	Bad Banana
   2		Healthy banana	Healthy banana

   3		Healthy Pear	Healthy Pear
   4		Bad Pear	Bad Pear
   4		Healthy apple	Healthy apple
   5		Bad apple	Bad apple

   Table 2. Testing

   Sr no	<>Categories of fruits	Accuracy (in %)
   1	Bad Banana	77
   2	Healthy Banana	96
   3	Healthy Pear	99
   4	Bad Pear	98
   5	Healthy Apple	70
   6	Bad Apple	60

   Table 3. Result

   The above table 3. Shows the results of the testing.

6. CONCLUSION

   By implementing this system, we can say that Faster R- CNN is the quite fastest model and gives most accurate results. Accuracies for different categories of fruits lies between: Healthy apple = (60-75)%, Bad Apple = (60- 70)%, Healthy Pear = (85-99)%, Bad Pear = (80- 98)%, Healthy Banana = (80-97)%, Bad Banana = (70-80)%. Hence, this system can be very useful in automatic sorting machines where it can detect as well as classify the fruit and their defects. Therefore, it will help in ensuring the quality and richness of the fruit.

7. REFERENCES

1. http://ecoursesonline.iasri.res.in/mod/page/view.php?i d=97351

2. Shankar Shankar M. Patil and A. K. Malik, Corelation based real-time data analysis of graduate students behaviour Springer Nature Singapore Pte Ltd. 2019

3. Defect Detection in Food Ingredients Using Multilayer Perceptron Neural Network by Ikramullah Khosa and Eros Pasero; Department of Electronics and Telecommunication; Torion, Italy.

4. P.M Keagy and T.F.Schatzki, Machine recognition of weevil damage in wheat radiograph Cereal chemistry 70 (1993): 696-696.

5. P.M Keagy, B.Parvin and T.F.Schatzki, Machine recognition of navel orange worm damage in X-ray images of pistachio nuts (1996);140-145.

6. T.F.Schatzki,et al. Defect detection in apples by means of X-ray imaging Transactions of the ASAE 40.5(1997):1407-1415.

7. Y.Yin and G.Y.Tian, Feature extraction and optimisation for x-ray weld image classification Proc. 17th World Conf. on Nondestructive Testing, 2008.

8. Faster R-CNN with Classier Fusion for Small Fruit Detection by Xiaochun Mai, Hong Zhang and Max Q.-H. MengJ., 2018 IEEE International Conference on Robotics and Automation (ICRA) May 21-25, 2018,

   Brisbane, Australia

9. INTELLIGENT SYSTEM FOR AUTOMATIC CLASSIFICATION OF FRUIT DEFECT USING FASTER REGION-BASED CONVOLUTIONAL NEURAL NETWORK (FASTER R-CNN) by aHasan Basri, bIwan Syarif, cSritrusta Sukaridhoto, dMuhammad Fajrul Falah,aDepartment of Informatics Management, Politeknik Negeri Fakfak, West Papua, Indonesia b,c,dDepartment of Informatics and

   Computer Engineering, Politeknik Elektronika Negeri Surabaya, Indonesia E-mail: hasanbasri@polinef.id, iwanarif@pens.ac.id, dhoto@pens.ac.id

10. Faster R-CNN Implementation Method for Multi- Fruit Detection Using Tensorflow Platform by Hasan Basri, Iwan Syarif, Sritrustra Sukaridhoto Department of Information and Computer Engineering Graduate Program Of Engineering Technology Politeknik Elektronika Negeri Surabaya hasanbasri@pasca.student.pens.ac.id,

    {iwanarif,sritrustra}@pens.ac.id

11. Fruit Detection Using Faster R-CNN Based On Deep Network by Linjuan Ma1, Fuquan Zhang2, Lin Xu3; School of Computer Science and Technology, Beijing Institute of Technology, 100081 Beijing, P.R. China 2 Fujian Provincial Key Laboratory of Information Processing and Intelligent Control(Minjiang Univeristy), Fuzhou, 350121, P.R. China * Corresponding author: Fuquan Zhang, e-mail: 8528750@qq.com. 3 Key Laboratory of Nondestructive Testing, Fuqing Branch of Fujian Normal University, Fuzhou, 350300, P.R.China

12. https://tryolabs.com/blog/2018/01/18/faster-r-cnn- down-the-rabbit-hole-of-modern-object-detection/

13. Shankar M. Patil and Dr. Praveen Kumar, Data Mining Model For Effective Data Analysis Of Higher Education Students Using MapReduce, IJERMT, 2017