Automatic License Plate Recognition using MSER and CNN

Automatic License Plate Recognition using MSER and CNN

Abhipsa S Kalyanshetti1*

Department of Computer Science and Engineering, SJB Institute of technology

Bengaluru, India

K Hemanth Ratna Kumar2

Department of Computer Science and Engineering SJB Institute of Technology

Bengaluru, India

Chethan R3 Karthick R4

Department of Computer Science and Engineering Department of Computer Science and Engineering SJB Institute of Technology SJB Institute of Technology

Bengaluru, India Bengaluru, India

Abstract Automatic license plate recognition has become very important in daily life because of increase in the vehicles which makes it impossible to track and monitor in some areas like parking toll, traffic rule violation etc. There are many challenges in recognition of the license plate which includes different orientations, size, broken plates, fonts, images taken in different environments etc. The system includes different types of techniques, like object detection, image processing. The vehicles license plate may be in different shapes, in different languages and characters can be of different fonts. This paper introduces Automatic License Plate Recognition system which uses Maximally Stable Extremal Regions (MSERs) technique to locate the license plates in an image, image processing techniques to segment characters and convolutional neural network to recognise the characters segmented from the license plate.

KeywordsMSER (Maximally stable extremal region), SWT (Stroke width transform), CNN (Convolutional neural network)

1. INTRODUCTION

   The Automatic license plate recognition (ALPR) is a large scale method used to scrutinize plates on vehicles. They can use available closed-circuit television or road-rule enforcement cameras, or ones designed exclusively for getting the vehicle images. It can be used in fields like traffic rule violation, parking lots and so on. ALPR uses the image given as input and outputs the license plate number. The features of vehicle plates which include presence of characters, aspect ratio, color, size and rectangular shape are used to localize the plate. The first ALPR was invented in 1976 at the Police Scientific, Development Branch in the UK. The system is generally made up of three modules or steps: license plate detection, license plate segmentation and its character recognition. Fig. 1 shows the three modules of the system. License plate recognition is the process of locating the region of interest (ROI) that is the license plate in the image. License plate segmentation is cropping out the located license plate from the image. Last step is the recognition of the segmented characters in the license plate. For locating the license plate MSER method is used. MSER is a technique used to detect the extremal regions in the image. Image processing techniques such as Sobel edge filtering and thresholding are applied to segment the region. After skew

   correction of the detected plate individual alphanumeric characters are segmented and recognized using convolutional neural network (CNN). The extracted numbers are filtered at last by using pattern matching technique. The system is implemented in Python.

   Fig. 1. Three modules of ALPR

   Following are the challenges encountered in the process:

   1. Plate variations:

      1. Position: license plates may be present in different locations in an image.

      2. Number of license plates: the input image might have zero or more number of plates.

      3. Shape and Size: shape and size of the plate varies with respect to the angle and distance of the camera from the vehicle.

      4. Color: license plates may contain characters which are not part of the license number and different colors with respect to region.

      5. Fonts: license plate numbers in different countries can be printed in different fonts and language.

      6. Obstruction: license plates could be covered with dirt or blocked by other objects.

      7. Frames and screws: along with above challenges license plates consist of frames and screws which

         get detected and recognized along with the license number.

   2. Surrounding variations:

      1. Illumination: input images may have different types of illumination, mainly due to environmental lighting and vehicle headlights.

      2. Backdrop: the vehicle image may contain other non- license plate objects such as company names, stickers or stamps or other plate-like objects.

2. RELATED WORKS

   In the past many license plate recognition systems have been proposed. It is still a difficult task to locate the license plate from a varying angle and with varying surrounding conditions. In the entire process of license plate recognition, the most crucial and challenging section is detection and segmentation of the license plate from the composite images.

   1. Stroke width transform

      Stroke width Transform (SWT) [3][4]is used to detect textual regions in the image. It is a local operator that computes per pixel the width of the most likely stroke containing the pixel. A stroke can be defined as a continuous part of an image that forms a band which has the same or constant width. The original width of the stroke is not known but rather it could be recovered. Initially, the stroke width of all the pixels is assigned to . Then in order to recover the strokes, the edge of the image is calculated using different edge detection operators. The edges are considered as stroke boundaries, and the width of such a stroke is going to be found. Characters in nature have large strokes which makes SWT suitable for detecting it in the process. Input to the operator is grey or color image and the corresponding output is an image of same size but with non-text regions removed.

   2. Maximally stable extremal regions

      Maximally stable extremal regions (MSER for short) [1][4] is used to find the extreme regions in the images. Maximal Stable Extremal Regions are used as a method of blob detection – blob detection refers to detecting regions in an image that differs in properties, such as brightness or color. These blobs are typically found by searching for regions that maintain a consistent intensity when a wide range of thresholds is applied to an image. As the contrast of text within an image with its background is typically significant or the color of every letter in the text stays consistent, MSER is a great choice for text detection. However, MSER is sensitive to image blur. And, many non- text regions are also detected in the image. Filtering techniques are required to remove many false positives from the result.

   3. Template matching

      Template Matching [2][9]is a method of comparing portions of images with one another. Template matching is

      used to recognize the characters from the license plate. This method is basically used for classifying objects. Templates are created which are used to recognize characters, numbers and other objects. But this method does not need the "segmentation" process of license plate characters. After the license plate segmentation phase, INFORMATION RECOGNITION PHASE (IPR) is

      applied. In the template matching method, templates are matched with the source image. The process involves moving the template to all the regions in the input image and it gives a numerical index as output. The index indicates to what extent the current template matches the image in a particular position. Matching is done on a pixel by pixel basis. Matching or correlation is ameasure of degree to which two variables agree. The variables are the pixel values in two images, template and source image. The correlation value ranges from -1 to +1. If the value is high that means, there is a strong relation between the template and the source image. Templates are used for individual characters. Since the size of the template is fixed, it may give wrong predictions.

   4. Connected component analysis

      Connected-component Analysis (CCA) [5][6]is an application of graph theory, where subsets of connected components are uniquely labelled based on a given heuristic. It uses binary images for detecting the connected regions. CCA is used for counting, filtering and tracking of connected components which is also called as Blobs. A pixel is said to be connected to another pixel if both have the same value and are adjacent to each other. Pixels with the same value, belong to the same object. Each pixel is labelled with a value depending on the component to which it was assigned. The algorithm can be used to detect the license plate in the images. The detected regions are compared with the given dimensions to qualify as a license plate.

3. PROPOSED SYSTEM

   The proposed system consists of following steps after taking the digital image as input:

   1. Pre-processing input image.

   2. Detecting license plate region using MSER.

   3. Extracting license plate region.

   4. Skew correction of license plate.

   5. Segmenting characters from license plate.

   6. Recognition of characters using CNN model. Fig. 2. shows the flow chart of the proposed system.

      1. Pre-processing

         Pre-processing is the initial step in the process where the digital image is processed to increase its quality and making it suitable for further computational processing. The colored image is converted to gray scale. Unwanted noises are removed. Image is resized to our requirement. Image is deblurred and its brightness is normalized. Fig. 3 and Fig. 4 shows the original and the pre- processed image. The steps involved are:

         1. Image resized (E.g. To width of 500).

         2. Conversion from RGB to gray scale.

         3. Applying filter (E.g. Gaussian Filter).

         4. Brightness normalization.

         Fig. 2. Stages of the proposed ALPR

         Fig. 3. Original image.

         Fig. 4. After preprocessing image.

      2. Detecting License plate region using MSER

         After the image is pre-processed MSER is used to detect the license plate. Since characters exhibit extreme regions property, the license plate which contains only characters is detected in the process. Along with the license plate many false positives are also detected. The non-extremal or non-textual regions are filtered out by applying a mask on the original image. The mask is produced from the coordinates and the bounding boxes of each region. Only

         MSER detected regions are seen in the output of this step. Fig. 5 shows the MSER detected regions.

      3. Extracting license plate region

         This step is to eliminate many false positives in the MSER detected regions which are to be considered as regions of interest. Only the number plate is extracted from the input image at the end. The image is passed through Canny edge detection. Canny edge operator is applied on the image for getting accurate boundaries of objects. Then, the edge detected image is thresholded using optimum threshold value obtained. Morphological Erosion and Dilation operations are performed on the image to get the rectangular shaped license plate. Fig.6 and Fig. 7 shows the output of extracting detected license plates.

      4. Skew correction of license plate

         License plates may be at any angle with respect to the camera. Before segmenting the characters from the image, it needs to be aligned properly. Contours are applied on the image which creates a rectangular box around the license plate. minAreaRect() a library function in OpenCV gives the angle made by the rectangular object with the positive or negative x-axis respectively. It gives a Box2D object which has the following details – ((top-left corner (x, y), (objects width, objects height), (its angle of rotation with respect to x-axis)). To draw the rectangle around the license plate cv2.drawContours () function is used. The license plate is skewed by the angle given from minAreaRect(). To rotate the vertically projected license plate (shown in Fig. 8), the slope of the diagonal from the first corner point from below to the opposite corner point that is the third corner point of the rectangle is considered (red colored points). The slope of the diagonal in the non-skewed rectangle is calculated. If the slope is positive then, the vertically projected license plate is rotated 90 degrees or else if it is negative it is rotated -90 degrees with respect to the y-axis.

         Fig. 5. MSER detected regions.

      5. Segmenting characters from license plate

         After the license plate is extracted and skewed to correct position, characters are segmented one-by-one. The license plate is processed through the following steps in this process:

         1. Image sharpening.

         2. Applying threshold to the image.

         3. Clearing edges of the plate up to specified radius.

         4. Eliminating smaller objects in the plate.

         5. Applying contours to segment individual characters. License plate is sharpened and thresholded to form a binary image. The edges are cleared out to the required radius from the plate. Then, objects with lower area are also eliminated. Contours are applied later to form bounding boxes around characters. Finally, the individual characters are segmented. Fig.9 and Fig. 10 shows the segmentation of characters.

      6. Recognition of characters using CNN model

   Last step is passing the individual characters to a CNN model. The model consists of 3 hidden layers each with 64 neurons. Output layer is a dense layer with 37 (36+1) neurons for classes of characters. The model is trained with character images. The trained model has the accuracy of 98%. The dataset consists of binary image of each class or character. Each image is of size 18×18 size. The individual characters segmented are fed into the model to predict the corresponding or class. The resulting string matched to a pattern which qualifies for a license plate number. Fig. 11 shows the recognized plate number.

   Fig. 6. Bounding box around license plate.

   Fig. 7. Bounding box around license plate.

   Fig.8. Rotating vertically projected license plate.

   Fig. 9. Segmenting license plate

   Fig. 10. Segmenting each character in the license plate

   Fig. 11. Detected license plate number

4. CONCLUSION

In this paper we presented a model for detecting license plate number from an image. The license plate consists of at least 4 characters to recognize. The image is first converted into gray- scale and filtered using a bilateral filter. Then, MSER is used to detect the license plate in the image and the plate region is extracted. Plate region is skew corrected if needed and each character is segmented. Individual characters are given as input to a CNN model to predict the character or class. The resulting license plate number is extracted.

REFERENCES

1. Chao Gou, Kunfeng Wang, Zhongdong Yu and Haitao Xie, License plate recognition using MSER and HOG based on ELM, IEEE International Conference on Service Operations and Logistics, and Informatics, 20 November. 2014.

2. Y.P. Huang, S. Y. Lai, and W. P. Chuang, A template-based model for license plate recognition, in Proc. IEEE International Conference on Networking, Sensing and Control, Vol. 2, pp. 737- 742, March. 20

3. Boris Epshtein, Eyal Ofek, Yonatan Wexler, Detecting text in natural scenes with stroke width transform, IEEE Computer Society Conference o Computer Vision and Pattern Recognition, 05 august 2010.

4. S.Shiyamala, S Suganya, Detection, Localization of Text in Images by Mser and Enhanced Swt, International Journal of Innovative Technology and Exploring Engineering (IJITEE),

   ISSN: 2278- 3075, Volume-8 Issue-10, August 2019

5. N. Vishwanath ; S. Somasundaram ; M. R. Rupesh Ravi ; N. Krishnan Nallaperumal, Connected component analysis for Indian license plate infra-red and color image character segmentation, IEEE International Conference on Computational Intelligence and Computing Research, 02 May 2012.

6. S. Stephy Golda Mercy1, Dr. I. Muthulakshmi2 1 PG Scholar, 2 Professor, Automatic number plate recognition using connected components analysis algorithm, International Journal for Technological Research in Engineering Volume 5, Issue 7, March- 2018.

7. Mrs. S. Mahalakshmi1, Ms. P. Shreeroja2, MSER Region Based Number Plate Recognition System, International Journal for Research in Applied Science & Engineering Technology (IJRASET), Volume 5 Issue V, May 2017.

8. Kethineni venkateswarlu, sreerama murthy velaga, text detection on scene images using mser, international journal of research in computer and communication technology, vol 4, issue 7, july-2015.

9. Savita Goel ; Savita Dabas, Vehicle registration plate recognition system using template matching, INTERNATIONAL CONFERENCE ON SIGNAL PROCESSING AND COMMUNICATION (ICSC), 23 January 2014.

10. Zheng, D., Zhao, Y., and Wang, 1., "An efficient method of license plate location," Pattern Recognition Letters 26, 243 1- 243 8 (Nov. 2005).

11. Cao, G., Chen, and Jiang, "An adaptive approach to vehicle license plate localization," The 29th Annual Conference of the IEEE Industrial Electronics Society, 2003. IECON '03. 1 786- 1791 Vo1.2.

12. S. Du, M. Ibrahim, M. Shehata and W. Badawy, "Automatic License Plate Recognition (ALPR): A State-of-the-Art Review," IEEE Trans. Circuits and Systems for Video Technology, vol. 23, no. 2, pp. 311325, Feb. 2013.

13. Y. P. Huang, S. Y. Lai, and W. P.Chuang, A template-based model for license plate recognition, in Proc. IEEE International Conference on Networking,Sensing and Control, Vol. 2, pp. 737-742, March. 2004.

14. N. N. Kyaw, G. R. Sinha and K. L. Mon, "License Plate Recognition of Myanmar Vehicle Number Plates A Critical Review," IEEE 7th Global Conference on Consumer Electronics (GCCE), Nara, pp. 771774, 2018.

15. S. Prabu, V. Balamurugan, and K. Vengatesan, Design of cognitive image filters for suppression of noise level in medical images, Measurement, Vol. 141, pp. 296-301, 2019.

16. B.D. Parameshachari, H.T. Panduranga, and S.K. Naveenkumar, Partial encryption of medical images by dual DNA addition using DNA encoding, in Proc. International conference on recent innovations in signal processing and embedded systems (RISE), pp. 310-314, 2017.