A Review on Deep Learning for Plant Species Classification using Leaf Vein

A Review on Deep Learning for Plant Species Classification using Leaf Vein

Neethu Mariam Sunny

PG Student, Computer Science & Engineering College of Engineering, Chengannur

Abstract:- Plants have a wide range of application in agriculture and medical purpose, and is especially significant to the bio diversity research. Plants are essential for the environmental protection. For the easy identification of plants it is important to identify and classify them accurately. By classifying plants we can learn about different kinds of plants and their features, similarities and differences. This work will help to classify plant species using leaf vein. The proposed approach consists of four main steps to extract the leaf vein – image sampling, image preprocessing (RGB to gray scale conversion,sobel edge detection, skeletionization), feature extraction and feature classification by CapsNet. This work is based on plant classification using leaf vein. It is specially designed for plants having compound leaves.

Keywords: : Plant classification, bio diversity, sobel edge detection, skeletionization, Compound leaves.

INTRODUCTION

Plants provide the essential components of life on earth such as oxygen, food, medicine and environmental protection. Plant identification branch include the modern practices of medicine, culinary arts, agriculture, and botany

. This science starts in the sheer practicality of identifying plants. It is a good practice to know which plants are edible and useful, and which ones produce itchy rashes or even death. About 3 million species of plants have been named and classified. It is impossible for a botanist to know the total number of named species and to identify all the plant species on earth. One way to identify and classify the plant is by their leaves. Each plant leaf serves as a tool to plant biologists and botanists for distinguishing plant species. Due to the unstoppable growth in human population and varying climate, there is an alarming threat to many plants. Many of the plants are on the verge of extinction. They are easily identified based on flowers and fruits. However these are three dimensional objects and increases complexity but plant leaves are two-dimensional in nature. Thus, they are most suitable for machine processing. Plant classification based on leaves is the easiest way to identify plant since they are the easily visible and they can be easily found and collected at all seasons, while flowers can only be obtained at blooming season.

Plants are an integral part of our ecosystems. Plant identification and classification has always been a matter of interest for the botanists as well as for the laymen. In the 4 million recognized plant species, only a tiny part of the plants is known. So as the leaves of the plant carry a lot of information about the plant species. These extracted features are used as a basis for an automated identification

and classification. The advancement in image processing has made this a quick and easy process and the advancement of Information Technology, systems with more functionality such as automatic labeling and flexible searching are required and it is achieved through image processing and machine learning techniques. Sampling leaves as well as photogening them is inexpensive and expedient. It is easy to transfer the leaf image to a computer and then the computer can extract necessary features automatically using image processing techniques and subsequently can recognize the plant / leaf using machine learning techniques. The first step in this process is to train the computer leaf features of plants . The leaf are characterized by its color, texture, vein structure and shape which reveals a vast amount of information about the plant. It is taken as the basis for the development of the system which will classify the given plant leaf into its belonging class using image processing techniques. The aim of plant classification is to place the plants into a class of ranks or categories such as species, genera, families and so on.

The characters of a leaf are their features or attributes (leaf width, stamen range, corolla length, locule range, placentation, and so forth) possessed by using the organism that can be composed, measured, counted, described or otherwise assessed.This means that indifferences, similarities and discontinuities between plants and taxa are reflected in their character. The characters of a leaf are determined by observing or analyzing samples of individuals leaves and recording the observations or by conducting controlled experiments. In compound leaves the arrangement of leaflets depends on the rachis and on the order of complexity. There are two main types of compound leaves they are pinnate and palmate leaves. Pinnate compound leaves have leaflets that occur in succession along a rachis where as a palmate compound leaves have leaflets borne at the tip of the rachis, and can be further categorized as being either peltate or non-peltate compound leaaves. Peltate leaves have leaflets that are present around the entire circumference of a radial, unifacial petiole. Non-peltate leaves have leaflets present around a portion of a bifacial petiole. This work aims in classifying plants by leaf vein. Leaves are commonly used in plant species recognition due to their availability throughout the year, especially, in the tropical areas. Many useful features can be acquired from a single leaf; such as, shape, texture, venation pattern, and colour.

LITERATURE REVIEW

This segment includes survey of various related works and the techniques used for feature extraction.

Convolutional Neural Network

Convolutional Neural Networks are usually used as deep learmning method. It is used to extract deep, semantic capabilities of an object. It include four layers they are Convolutional Layer, Rectified Linear Unit (ReLu), Max pooling Layer and absolutely related Layer. In artificial neural networks, ConvNets is used forimages recognition and images classifications. Objects detections, recognition faces , are some of the other areas where CNNs are widely used. CNN models is used to train and test, each input image thatb will pass through a series of convolution layers with filters (Kernals), Pooling, fully connected layers (FC) and finally apply Softmax function to classify an object ( with probabilistic values between 0 and 1).

Deep learnig CNN technique become used for plant type the usage of leaf vein [1]. on this, a pre-trained CNN AlexNet version changed into used on this research. AlexNet is made from 9 layers which include 5 convolution layers, 3 fully connected layers, and a softmax class layer.Softmax category layer is mentioned the softmax feature which yield the anticipated probability of every organization and is completely linked to the final full connected layer. In [1], a CNN version was advanced to extract the capabilities from the photos as opposed to pleasant-tuning the AlexNet model. It includes a complete

6 layers of CNN model with 3 convolution layers (Convolution stage, ReLU, Pooling stage), three completely linked layers, and a softmax classification layer. The dataset used for [1] can be acquired from DLeaf database.

Gray Level Co-occurrence Matrix

A statistical method of inspecting texture that considers the spatial dating of pixels is the grey-level co-occurrence matrix (GLCM), additionally referred to as the grey-degree spatial dependence matrix. The GLCM capabilities symbolize the feel of an photo with the aid of calculating how often pairs of pixel with precise values and in a specific spatial courting occur in an image, developing a GLCM and then extracting statistical measures from this matrix. It has a few advantages, one of them is that histograms of the photo and its rotation picture are the equal, and any other is that the dimensions of storage location for histogram is decrease than the storage length of the photo f. in the second-order, the relationship among pixels is taken into consideration. In [8] and [9], GLCM is used as a characteristic extraction method is a way of extracting 2nd order statistical texture features. The approach has been utilized in a number of applications, 1/3 and higher order textures take into account the relationships amongst three or greater pixels. The dataset used are pl@ntleaves[8] and Flavia[9].

Probabilistic Neural Network

A probabilistic neural network (PNN) is a feed-forward neural network, which is broadly utilized in category and sample reputation problems. In the PNN algorithm, the parent possibility distribution feature (PDF) of each elegance is approximated by a Parzen window and a non- parametric characteristic. Then, the use of PDF of each magnificence, the class opportunity of a new enter facts is envisioned and Bayes rule is then hired to allocate the magnificence with highest posterior opportunity to new enter records. through this method, the possibility of misclassification is minimized. This type of ANN become derived from the Bayesian network and a statistical set of rules referred to as Kernel fisher discriminant evaluation.

PNN is frequently used in type troubles.when an enter is gift, the first layer computes the space from the enter vector to the education input vectors. This produces a vector in which its factors indicate how close the input is to the education input. the second one layer sums the contribution for every class of inputs and produces its net output as a vector of chances. eventually, a whole switch characteristic at the output of the second one layer selections the maximum of these possibilities, and produces a 1 (positive identification) for that class and a zero (negative identification) for non-targeted lessons. In [8] and [9], PNN is used as a characteristic category approach for matching vein sample.

input Layer:-every neuron in the enter layer represents a predictor variable. In specific variables, N-1 neurons are used whilst there are N wide variety of classes. It standardizes the variety of the values by way of subtracting the median and dividing by the inter quartile range. Then the input neurons feed the values to every of the neurons in the hidden layer.

pattern layer:-

this deposit contains one neuron for each case in the education information set. It stores the values of the predictor variables for the case along side the target price. A hidden neuron computes the euclidean distance of the test case from the neurons center point after which applies the radial basis function kernel characteristic the usage of the sigma values

Summation Layer:-

For PNN there's one pattern neuron for every category of the target variable. The real goal category of every training case is saved with each hidden neuron; the weighted fee coming out of a hidden neuron is fed best to the pattern neuron that corresponds to the hidden neurons class. The sample neurons add the values for the magnificence they constitute

Output Layer:-

The output layer compares the weighted votes for each goal class gathered in the pattern layer and makes use of the largest vote to are expecting the goal category.

Multilayer Perceptron

A multilayer perceptron (MLP) is a class of feed-forward artificial neural network(ANN). The term MLP is used

ambiguously, from time to time loosely to refer to any feed-ahead ANN, sometimes strictly to refer to networks composed of multiple layers of perceptrons (with threshold activation). Multilayer perceptrons are from time to time colloquially referred to as "vanilla" neural networks, specially when they have a single hidden layer. An MLP consists of at the least 3 layers of nodes as in [15]:- an input layer, a hidden layer and an output layer. except for the enter nodes, every node is a neuron that makes use of a nonlinear activation feature. MLP utilizes a supervised mastering technique referred to as back propagation for education. it is a a couple of layers and non-linear activation feature that distinguish MLP from a linear perceptron. it is able to distinguish records that isn't always linearly separable. The MLP includes three or more layers (an input and an output layer with one or greater hidden layers) of non-linearly activating nodes. gaining knowledge of takes place within the perceptron by converting connection weights after every piece of data is processed, primarily based on the amount of errors inside the output in comparison to the anticipated end result. that is an example of supervised gaining knowledge of, and is carried out via backpropagation, a generalization of the least mean squares algorithm within the linear perceptron.

Zernike Moments

Zernike moments are used to extracting the features of revealed digits in grayscale pix[4]. The Zernike moments uniquely describe functions on the unit disk, and may be prolonged to pictures. There in-variance homes make them attractive as descriptors for optical character popularity. simple translational and scale invariant residences. by means of preprocessing an picture the usage of the everyday moments we will get an photo to be translational and scale invariant earlier than walking Zernike moments. Zernike moments are accurate descriptors despite exceptionally few information points. Reconstruction of Zernike moments may be used to decide the quantity of moments important to make an accurate descriptor.

Scope of the work

When gone through several research works associated with plant class based totally on leaf. it's miles clean that maximum of the works had been done the use of neural networks. [1] includes mainly 4 most important steps as shown in parent three.1, which are, sampling, photo pre- processing, feature extraction and feature category.The leaf samples of this research had been amassed from three locations inside the university of Malaya, Kuala Lumpur, Malaysia. these locations were areas round Varsity Lake (VL), principal library (ML) and Dewan Tunku Canselor corridor (DTC). The leaf is the part of the tree that is constantly selected because the samples to be studied instead of culmination, flowers or a few other elements because of its availability throughout the year. forty three

species of tropical bushes with 30 samples consistent with species were accrued. consequently, a complete of 1290 leaf pix had been collected for this studies.

The leaf pictures have been acquired by using the usage of a Nikon D750 version of DSLR digicam. The samples have been placed on a field with white background and fluorescent lighting were located beneath the container. as the lighting of this setup is from the bottom, it is able to help to reduce glaze and shadow on the leaf.uncooked photographs are no longer suitable for analysis purposes and want to be transformed into the processed format, which include, jpeg, jpg and tiff for in addition evaluation. In this studies, the obtained pictures have been stored in a layout of Nikon camera, named Nikon electronic file (NEF) with 6016 x 4016 decision. Adobe photo-save became used to transform these raw images into Tagged picture file format (TIFF). similarly, the historical past noises of the images were reduced with the aid of the usage of Adobe picture-shop and MATLAB R2016a changed into used for resizing and photo conversion. two extraordinary pre- processing methods have been carried out, namely, picture reconstruction for CNN and vein morphometric measurements (segmented by means of Sobel). characteristic extraction is the key level on this studies, which is used to extract the essential capabilities from the leaf images. The capabilities which might be commonly used for plant identiy systems are shape, texture , shade and vein. severa techniques can be employed for characteristic extraction including Histogram of oriented Gradient (HOG), Zernike Moments, Hus second and others. A deep mastering set of rules, particularly, the Convolutional Neural community, become hired to extract the features in this research. A traditional approach of the usage of morphological feature extraction to phase the leaf venation with the aid of using sobel area detection approach changed into investigated and compared.

PROPOSED SYSTEM

A new method is proposed to eliminate the constraints of present device up to an enlarge. It includes particularly four predominant steps as shown in Fig.4.1, which are sampling, image preprocessing, function extraction and class. The pics required are received the use of a camera and from DLeaf information set. within the preprocessing level the leaf images are transformed to a wellknown length (a hundred and twenty*120cm) after which the noise is eliminated the usage of sobel area detection. within the feature extraction degree a CapsNet approach is used to extract the vein sample of leaf. ultimately, the flora are recognized the use of classifier. A CapsNet is a machine mastering machine, used for a higher model hierarchical courting.,

Fig. 1. CapsNet architecture

Figure 1 suggests a CapsNet structure. In CapsNet you can upload extra layers inner a unmarried layer. Or in other words nest a neural layer inner some other. The nation of the neurons interior a pill capture the above residences of one entity inside an photo. A capsule outputs a vector to represent the life of the entity.CapsNets have following benefits over convolutional neural networks.

Point of view in-variance:- the usage of pose matrices lets in pill networks to apprehend objects irrespective of the angle from which they may be considered.

Fewer parameters:- considering that drugs organization neurons, the connections among layers require fewer parameters. Capsnets reject the pooling layer approach of

CNN that reduces the amount of detail to be processed at the following higher layer.

Higher generalization to new viewpoints:- CNNs, whilst educated to understand rotations, frequently learn that an object can be viewed similarly from numerous different rotations. but, capsule networks generalize higher to new viewpoints because pose matrices can seize those traits as linear transformations.

Defense against white-box adversarial attacks:- The Fast Gradient Sign Method (FGSM) is a typical is an average approach for attacking CNNs. It evaluates the gradient of every pixel towards the lack of the network and adjustments every pixel by using at maximum epsilon (the mistake term) to maximize the loss.

Fig. 2. Proposed system block diagram

The block diagram shows overall steps of the proposed system. Figure 2 describes the plant classification system. This project consist of four main steps as shown in Fig. 2, which are, sampling, image preprocessing, feature extraction and classification. First, the leaf samples were collected and images were acquired. The leaf images were then preprocessed and fed into the feature extraction step to retrieve the important information from the leaves using CapsNet and Sobel edge detection approach. Lastly, the extracted features were trained and classified by using machine learning method.

Image Sampling

The leaf is the part of the tree that is always chosen as the samples to be studied, instead of fruits, flowers or some other parts due to its availability throughout the year.The leaf images of this dataset were collected from the common

tropical trees, which can be found easily. . 43 species of tropical trees with 30 samples per species were collected. Thus, a total of 1290 leaf images were collected for this research. The leaf samples in this dataset possessed a high similarity in shape and colour. Some of the leaf images were acquired by using a Nikon camera. The samples were put on a box with white background and fluorescent lights were placed under the box in order to capture good quality images with standard background. As the lighting of this setup is from the bottom, it can help to reduce glaze and shadow on the leaf.

Image Preprocessing

Raw images are not appropriate for analysis purposes and need to be converted into the processed format, such as, jpeg, jpg and tiff for further analysis. In this project, the acquired images were stored in 6016 x 4016 resolution.

Sobel is an edge detection method, which is simple and effective in detecting the edge of an object. Sobel was used as a segmented method to extract the vein architectures of all the leaf samples. First, all images were converted from

Decoder

dynamic routing). This layer outputs 16D vectors that contain all the instantiation parameters required for rebuilding the object.

RGB images into grey-scale images . Then, Sobel was used to segment out the region of interest (ROI) from the images. After segmentation, the images were then post- processed and skeletonized to ensure a clean vein architecture could be obtained.

Feature Extraction

Feature extraction is the key stage in this project, which is used to extract the important features from the leaf images. The features that are commonly used for plant identification systems are shape , texture , colour and vein. numerous techniques can be employed for characteristic extraction which include Histogram of orientated Gradient (HOG), Zernike Moments, Hus moment and others.The vein features were extracted from the segmented images (with Sobel) by measuring the vein morphological features.

62 vein features were extracted, such as, number of branching points, number of ending points, number of branches, and number of areoles. The leaf area was computed for the density calculation of veins, branching points, ending points and areoles.

Encoder

The Encoder takes the photograph enter and learns how to constitute it as a 16-dimensional vector which incorporates all the records had to essentially render the picture.

1. The Conv Layer detects features that are later analyzed by the capsules. As proposed inside the paper, consists of 256 kernels of length 9x9x1.

2. The Capsule Layer is the lower level capsule layer which I described previously.

3. The Git Layer is the higher level capsule layer which the Primary Capsules would route to(using

The decoder takes the 16D vector from the Digit capsule and learns a way to decode the instantiation parameters given into an photograph of the item it's miles detecting. The decoder is used with a Euclidean distance loss function to determine how similar the reconstructed function is as compared to the real function that it's miles being skilled from. This makes sure that the tablets best maintain statistics so as to advantage in spotting digits inner its vectors.

Feature Classification

Classification, as the remaining step for an automated plant popularity device, is an sensible set of rules in schooling facts to recognize the precise features of each character plant species and categorizing a new sample as the best species. A supervised system mastering method, is conceded as one of the effective type methods because of its high capability in dealing with excessive dimensional area and records points which are not linearly separated. Statistical classifier makes tries to predict the elegance with an unknown sample based totally on opportunity .

RESULTS AND DISCUSSIONS

When gone through several research works associated with plant category primarily based on leaf. it is clear that most of the works have been executed the use of neural networks. [1] consists of particularly 4 most important steps as shown in discern three.1, which are, sampling, photo pre-processing, function extraction and feature category. raw snap shots are not suitable for evaluation functions and want to be converted into the processed ormat, along with, jpeg, jpg and tiff for further analysis. Sobel become used as a segmented technique to extract the vein architectures of all the leaf samples.

Fig. 3. Output Images

First, all pics had been transformed from RGB pictures into grey-scale pix. Then, Sobel become used to segment out the place of hobby (ROI) from the pix. After segmentation, the pictures were then post-processed and skeletonized to make certain that a smooth vein structure should be obtained. feature extraction is the key stage on this research, which is used to extract the crucial features from the leaf pics. The capabilities which might be usually used for plant identity systems are form, texture , shade and vein. severa techniques may be employed for function extraction which includes Histogram of oriented Gradient (HOG), Zernike Moments, Hus second and others. A deep learning set of rules, specifically, the Convolutional Neural community, turned into hired to extract the features in this studies. A CapsNet is a device getting to know machine, used for a higher model hierarchical relationship. discern four.2 shows a CapsNet architecture. In CapsNet you could upload extra layers inner a single layer. Or in other phrases nest a neural layer inside every other. The country of the neurons interior a tablet capture the above properties of one entity inside an image. A capsule outputs a vector to represent the life of the entity.

CONCLUSIONS AND FUTURE WORK

In this work, when surveyed through numerous techniques of deep learning to know for plant species category the use of leaves. it is clear that most of the research works were executed the use of CNN, GLCM, naive bayesian classifier, RBF neural network, PNN, MLP and zernike moments and located out their limitations. Then to lessen those barriers to some extend new method called CapsNet has been proposed with extra quantity of location primarily based info and smoothing clear out. This assignment will help to classify plant species the usage of leaf vein. The proposed method includes four primary steps to extract the leaf vein – photograph sampling, photo preprocessing (RGB to grey scale conversion, sobel area detection, skeletionization), characteristic extraction by CapsNet technique and characteristic type by using PNN approach. The proposed project carried out a plant species identification gadget the use of leaf vein pattern by way of CapsNet technique. As a future work the same paintings may be prolonged for greater compound leaf vegetation.

REFERENCES

1. Jing Wei Tan , Siow-Wee Chang, Sameem Binti Abdul Kareem, Hwa Jen Yap, Kien-ThaiYongDeep Learning for Plant Species Classification using Leaf Vein Morphometric,inIEEE/ACM Transactions on Computational Biology and Bioinformatics,

   Page(s): 1 – 1 ,19June 2018

2. Hoshang Kolivand, Bong Mei Fern,Tanzila Saba, Mohd Shafry Mohd Rahim AmjadRehmanA new leaf venation technique for plant species classification,, Volume 44, Issue4, pp 33153327 ,April 2017.

3. A. Kadir,L. E. NugrohoNeural network application on foliage plant identifi-cation, International Journal of Computer Applications (0975 8887)Volume 29 No.9,September 2011

4. Elhariri, E., El-Bendary, N., Hassanien, A. E.Plant classification system based on leaffeatures, 9th International Conference on Computer Engineering Systems (ICCES).2014

5. P.Jona Innisai Rani,C.Suresh Kumar,S.VenkatakrishnanA leaf recognition technique forplant classification using RBPNN and zernike moments, International Journal of Innova-tive Science and Research Technology,February 2018

6. Hong fu, zheu chi, Dagan FengMachine learning for ontology based leaf classification, 8th International Conference onControl,

   Automation, Robotics and VisionKunming, 9thDecember 2004

7. Olfa Mzoughi, Nozha Boujemaa, Ezzeddine ZagroubaMultiple leaflets based identifica-tion approach for compound leaf species, ACM Conference on Environmental Multime-dia Retrieval 2014

8. AbdolvahabEhsanir AD and Sharath Kumar Y.Leaf recognition for plant classificationusing GLCM and PCA method,Oriental journal of computer science and technology,Vol.3(1),31-36 (2010)

9. Hermawan Syahputra, Agus Harjoko, Retantyo Wardoyo and Reza PulunganPlant recog-nition using stereoleaf image using GLCM ,International journal of computer science andtelecommunication, 2012

10. C. S. Sumathi and A. V. Senthil KumarEdge and texture fusion for plant leaf classifi-cation ,International Journal of Computer Science and Telecommunications [Volume 3,Issue 6, June 2012]

11. M. Z. Rashad, B.S.el-Desouky , and Manal S

    .KhawasikPlant image classificationbased on textural features using combined classifier,International journal of computer sci- ence and information technology,2011

12. Khadije Mahdikhanlou, Hossein Ebrahimnezhad Plant leaf classification using cen-troid distance and axis of least inertia method ,22nd Iranian Conference on ElectricalEngineering (ICEE),2014

13. Ji-Xiang Dua,b, Xiao-Feng Wangb, Guo-Jun ZhangaLeaf shape based plant speciesrecognition , International Journal of Computer Science and Telecommunications [Vol-ume 3, Issue 6, June 2012]

14. M. M. Amlekar, A. T. Gaikwad ; R. R. Manza ; P.L. YannawarLeaf shape extraction forplant classification,

    International Conference on Pervasive Computing (ICPC),2015

15. Parul Mittal, Manie Kansal, Hardeep kaur JhajjCombined Classifier for Plant Clas-sification and Identification from Leaf Image Based on Visual Attributes, InternationalConference on Intelligent Circuits and Systems (ICICS),2018

16. Azeil Louisse Codizar, Geoffrey SolanoPlant leaf recognition by venation and shape us-ing artificial neural networks, 7th International Conference on Information, Intelligence,Systems , 19 December 2016