Detection and Recognition of Road Traffic Signs in Various Illumination and Weather Conditions

Detection and Recognition of Road Traffic Signs in Various Illumination and Weather Conditions

Ms. Rani V. Lunge

ME, Student, Dept. of Electronics & Telecommunication PVPIT, University of Pune,

Pune, India

Prof. B. V. Pawar

Dept. of Electronics & Telecommunication PVPIT, University of Pune,

Pune, India

AbstractThis paper presents a novel system for the detection and recognition of road traffic signs. Road Traffic signs are used to regulate the traffic and to indicate the state of the road to guide and warn drivers for their safety. The proposed system consist of two main stages i.e. detection and recognition. The proposed system is able to detect traffic sign regions as maximally stable extremal regions, which is insensitive to variations in lighting conditions. Recognition is based on a support vector machine (SVM) classifiers using Radial basis Function kernel, that are trained using histogram of oriented gradient (HOG) features. The training data are generated from synthetic template images that are freely available from an online database. SVM classifier is trained using histogram of oriented gradient (HOG) features. The proposed system can operate in various illumination and weather conditions.

KeywordsCandidate region, histogram of oriented gradient features (HOG), maximally stable extremal regions (MSERs), support vector machines (SVMs), traffic sign detection and recognition.

1. INTRODUCTION

   The detection and recognition of traffic sign is an important part of an advanced driver assistance system and a great issue for research recently and plays a great role in intelligent transport system. Traffic signs are used to regulate the traffic and to indicate the state of the road to guide and warn drivers and pedestrians. They have several distinguishing features like specific colors and shapes, with the text or symbol in high contrast to the background that may be used for their detection and identification. The visibility of traffic sign is very important for the drivers safety. There are many problems involved in traffic-sign detection like variations in perspective, variations in illumination, occlusion of signs, motion blur, and weatherworn deterioration of signs. These different conditions are brought about by changes in the weather such as sunny, cloudy etc, the time of the day or night, and the state of the road sign itself subject to deterioration.

   The proposed method consists of the two stages: detection and recognition. The first is detection of traffic sign in image using image processing. The detection is performed using a novel application of maximally stable extremal regions (MSERs) [1]. The second one is recognition of detected sign.

   The recognition is performed with histogram of oriented gradient (HOG) features, which are classified using support vector machine (SVM). The result of the TSDR research effort can be used as a support system for the driver. Traffic Sign detection and recognition can also be useful for highway maintenance so that a human operator does not have to check the presence and condition of the signs. The goal of this work is to find and classify road signs in various illuminations and different weather conditions such as rainy, sunny, cloudy, etc.

   In this paper, Section II focuses on previous work and Section III consist of the overview of the system. The experimental work and results are illustrated in section IV and V respectively. Finally, Section VI presents the conclusions.

2. RELATED WORK

   In most cases, traffic sign recognition algorithms are divided into two stages: 1) detection and 2) recognition. Traffic signs have varying illumination in various weather conditions. Many of system use the color information as method of segmenting the image. But the color-based road sign detection methods gives often reduced performance in scenes with poor lighting or adverse weather conditions such as fog. To overcome this issues the RGB image is converted into different Color spaces, such as huesaturation-value HSV [6], YUV (luma Y and two chrominance UV) [12], and CIECAM97 (color appearance model) [7]. The method presenting a system for detection and recognition of road signs with red boundaries and black symbols inside is represented in [6]. The detection is invariant to varying lighting conditions and shadows. The detection technique [12] includes histogram equalization, color segmentation and light control. In this system, both the HSV and the YUV color spaces are used to get better segmentation results than that of one color space technique. Gao et al. [7] used a quad-tree histogram method to segment the image based on the hue and chroma values of the CIECAM97 color model.

   In contrast, there are several approaches that use only shape information instead of color information from grayscale images. Gareth Loy [11] introduced a shape based technique that extends the concept of the fast radial symmetry transform to detect triangular, square and octagonal road signs. The system proposed by S. Maldonado Bascon et al. consists of color based segmentation, traffic sign detection by

   shape classification with linear SVMs and recognition based on Gaussian-kernel SVMs [5]. E. Cardarelli et al. presented the recognition system based on neural network [4]. These both systems consist of classifiers that were trained using hand labelled real images, which is repetitive, time

   consuming and error-prone process. But the system can overcome these issues using synthetic graphical representations of signs from an online road sign database. There is another novel system for the automatic detection and recognition of traffic signs [1].

   Fig. 1. Block diagram of the system.

   (a) Original Image (b) Threshold = 70 (c) Threshold = 90 (d) Threshold = 110

   (e) Threshold = 130 (f) Threshold = 150 (g) Threshold = 170 (h) Threshold = 190 Fig. 2. Fig(a)-(i) Connected components at several thresholds for original image.

   Here traffic symbols are detected as MSERs. MSER method for detection is robust to variations in contrast and lighting conditions. Here detection is based on background color of the sign. Recognition is based on a cascade of support vector machine (SVM) classifiers that were trained using histogram of oriented gradient (HOG) features.

3. OVERVIEW OF THE PROPOSED SYSTEM

   The proposed system consists of the two main stages: detection and recognition. The input image of the road traffic sign is detected and recognized by the system as shown in Fig.1. Here candidates for the traffic symbol are detected as MSERs. In detection stage, image is thresholded at several levels and the regions that maintain their shape through several levels are selected as MSERs. This method of detection is selected due to its robustness to variations in contrast and lighting conditions. This algorithm detects candidates based on the background color of the sign, because these backgrounds persist within the MSER process. Recognition is based on support vector machine classifiers that are trained using histogram of oriented gradient (HOG) features.

   1. Detection of Traffic Signs as MSERs

      MSER is a method for blob detection in images. The MSER algorithm extracts a number of co-variant regions from an image, called MSERs. The concept more simply can be explained by thresholding. All the pixels below a given threshold value get 'black' and all those above or equal get 'white'. If we are shown a sequence of thresholded images with frame t corresponding to threshold t, we get first a white image, then 'black spots corresponding to local intensity minima will appear and then grow larger. These 'black' spots will merge eventually, until the whole image is black. The set of all these connected components in the sequence is the set of all extremal regions. Fig.2 shows connected components at several thresholds for an image.

      Generally traffic signs are with red or blue color. To find the MSERs, the image is first transformed from RGB image into normalized image. For this, the ratio of red channel to sum of all channels and ratio of blue channel to ratio of blue channel to sum of all channels are found for each pixel of the original image. Then, greater ratio value of between two is selected as pixel value of the normalized image i.e.

4. EXPERIMENTAL WORK

   The experimental work that has been carried out for the proposed system is explained in the following sections. The software used for this system is MATLAB (8.2.0.701).

   (1)

   = 1 =

   ++

   , 2 = .

   ++

   This normalized image ( ) is binarized at number of different threshold levels to find the connected components. The connected components that maintain their shape through several thresholds are selected as MSERs. For this, a threshold range is selected. The threshold is evenly spaced

   between ranges 70 to 190 for traffic symbols, because the MSERs that represent road signs generally appear within this range. The number of thresholds selected is 24. The Fig. 3 (a) shows the

   1. Traffic symbol with blue background b) Normalized image c) Binarized image to get MSERs

      1. Full scale image of MSERs e) MSER at mth= 12 f) Binary image after area opening operation Fig. 3. Fig (a)-(f) detection of traffic sign image as MSERs.

         Cell

         1. Extracted image divided into 4 cell b) Gradient directions as 1, 2, 3, or 4 c) Histogram of oriented gradient features Fig. 4. Fig (a)-(c) HOG feature extraction.

      TABLE I. PROPERTIES TO SORT CONNECTED COMPONENTS

      Properties	Maximum	Minimum
      Width	100	20
      Height	100	20
      Height/Width	1.2	0.8
      Width/ Height	1.2	0.8

      traffic sign image with blue background and Fig. 3 (b) shows the normalized image for this image.

      After getting the MSERs for the traffic sign image, image is converted into full scale image using function imagesc (C). The imagesc function scales image data to the full range of the current colormap and displays the image. In detection stage, threshold for MSERs as mth is obtained through the trial and error method and kept it constant equal to 12. The

      MSER value below 12, get converted into black and above 12 get converted into white. After this operation, a binary image is obtained. Then from binary image, all connected component (objects) are removed that have fewer than 50 pixels by using area opening operation in MATLAB. Fig. 3 shows the results for detection of traffic sign image as MSERs. For the remaining connected components, the basic properties such as area, centroid and bounding box are measured. Using the properties of connected components, the sign from original image is extracted. After extraction of image, HOG features are calculated for this image. And to classify the features of extracted traffic sign image only, we decided the properties for bounding box of extracted traffic sign. TABLE I. shows the properties of bonding box of extracted image. After satisfying the conditions for bounding box, the SVM classifier is applied to the HOG features of extracted image to classify the traffic symbol.

   1. Classification of Road Signs

   In recognition stage, to confirm the shape of candidate region as traffic sign and exact type of sign, HOG features are extracted from the image, which represent the occurrence of gradient orientations in the image. Gradient histograms measure the orientations and strengths of image gradients within an image region. Here we divide the extracted image

   into 4 × 4 cell i.e total 16 cells as shown in Fig. 4(a). Here we use the Canny edge detector to detect a wide range of edges in image. Canny algorithm uses four filters to detect horizontal, vertical and two diagonal edges in the blurred image. The angle of edge direction is rounded to one of four angles

   (for example 0, 45, 90 and 135 degrees) representing vertical, horizontal and the two diagonals. Here, for each cell we are calculating 4 directions. Therefore we get total 64 features.

   (2)

   = exp

   | xx |2 + b

   2

   2

   Here each cell is divided into 3 × 3 pixel as shown in Fig. 4(b). For central pixel, we are calculating position of neighborhood pixel i.e. b0 to b7 and calculating the 4 feature vectors. Fig. 4 (c) shows the histogram of oriented gradient features, where x-axis shows orientations and y-axis shows the frequency of gradient features.

   After extraction of HOG features, the SVM classifier is applied to the HOG features to classify the traffic symbol. Here, SVM classifier with Radial Basis Function Kernel is used. A SVM is a binary classifier i.e. it classifies data between two classes. But traffic sign data cannot be classified using two classes. Hence to train the SVM classifier, the pair wise classification method of training images is used. Here, the classifier is trained for each possible pair of classes. For M classes, this results in (M1)*M / 2 binary classifiers. An unknown point x is classified by applying each of the binary classifiers and count how many times point x was assigned to that class label. Class label with highest count is then considered the label for unknown point x. The SVM with RBF is illustrated as

   Where N is the number of support vector, is either 1 or 1, indicating the class to which the point x belongs, x is support vector, b is the bias i.e. intercepts of the hyperplane that separates the two groups in normalized data space, is the vector of weights for the support vectors.

5. RESULTS AND DISCUSSIONS

   The proposed system is tested for the traffic sign images in various illumination and weather conditions. The Fig. 5 shows the traffic symbol in night illumination and results are shown for detection and recognition of it. The Fig. 6 shows the results for the traffic symbol in rainy weather. .Fig. 7 shows the results for detection of traffic sign in sunny weather. The software is also tested for detection of traffic sign in cloudy weather and gives good results.

   TABLE II. shows the accuracy for detection and recognition of traffic sign with color background and white background. These results show that, accuracy of the detection of sign is high (> 89.47%) and developed software satisfies the aim of proposed system.

   (a)Traffic sign in night illumination (b) Normalized image (c) MSER at mth= 12 (d) Binary image after area opening operation

   (e) Extracted traffic sign image f) Recognized Traffic sign.

   Fig. 5. Fig.(a)-(f) detection and recognition of traffic sign in night illumination.

   (a)Traffic sign in rainy weather (b) Normalized image (c) MSER at mth= 12 (d) Binary image after area opening operation

   (e) Extracted image (f) Recognized traffic sign

   Fig. 6 Fig.(a)-(f) detection and recognition of traffic sign in rainy weather.

   1. Traffic sign in sunny weather (b) Normalized image (c) MSER at mth= 12 (d) Binary image after area opening operation

      (e) Extracted image (f) Recognized Traffic sign

      Fig. 7. Fig.(a)-(f) detection and recognition of traffic sign in sunny weather.

      TABLE II. RESULTS FOR TRAFFIC SIGN WITH WHITE AND COLOR BACKGROUND

      Traffic Sign /td>	Total Traffic Signs	Traffic Signs Correctly Classified	Signs Misclassified	Signs not Classified	Accuracy in %
      Traffic Signs with White Background	38	34	2	2	89.47
      Traffic Signs With Color Background	31	28	2	1	90.32

6. CONCLUSION

The software developed and implemented for Detection and Recognition of Road Traffic Signs in various illumination and weather conditions works satisfactorily and the following conclusions are drawn.

• The developed software gives the satisfactory results under various illumination and weather conditions (Fig. 5, Fig. 6, and Fig.7).

• Accuracy of detection of sign is high ( 89.47%) which mostly satisfies the requirement of sign detection.

REFERENCES

1. Greenhalgh and Majid Mirmehdi, Real-time detection and recognition of road traffic signs, in Proc. IEEE Trans. Intell. Transp. Syst. vol. 13, Dec. 2012.

2. M. Hossain, M. Hasan,M. Ali, M. Kabir, and A. Ali, Automatic detection and recognition of traffic signs, in Proc. RAM, pp. 286291, 2010.

3. I. Creusen, R. Wijnhoven, and E. Herbschleb, Color exploitation in HOG-based traffic sign detection, in Proc. ICIP, pp. 26692672, 2010.

4. E. Cardarelli, P. Medici, P. P. Porta, and G. Ghisio, Road sign shapes detection based on Sobel phase analysis, in Proc. IEEE IVS, 2009, pp. 376381, 2009.

5. S. Maldonado-Bascón, S. Lafuente-Arroyo, P. Gil-Jimenez, H. Gomez- Moreno, and F. Lopez-Ferreras, Road-sign detection and recognition based on support vector machines, IEEE Trans. Intell. Transp. Syst., vol.8, pp. 264-278, Jun. 2007.

6. R. Malik, J. Khurshid, and S. Ahmad, Road sign detection and recognition using color segmentation, shape analysis and template matching, in Proc. ICMLC, vol. 6, pp. 35563560, Aug. 2007.

7. X. Gao, L. Podladchikova, D. Shaposhnikov, K. Hong, and N. Shevtsova, Recognition of traffic signs based on their color and shape features extracted using human vision models, J. Vis. Commun. Imag. Represent.,vol. 17, no. 4, pp. 675685, Aug. 2006.

8. C. Bahlmann, Y. Zhu, V. Ramesh,M. Pellkofer, and T. Koehler, A system for traffic sign detection, tracking, and recognition using color, shape, and motion information, in Proc. IVS, pp. 255260, 2005.

9. N. Dalal and B. Triggs, Histograms of oriented gradients for human detection, in Proc. CVPR, pp. 886893, 2005.

10. J. Matas, Robust wide-baseline stereo from maximally stable extremal regions, Image Vis. Comput., vol. 22, no. 10, pp. 761767, Sep. 2004.

11. G. Loy, Fast shape-based road sign detection for a driver assistance system, in Proc. IROS, pp. 7075, 2004.

12. W. Shadeed, D. Abu-Al-Nadi, and M. Mismar, Road traffic sign detection in color images, in Proc. ICECS, vol. 2, pp. 890893, 2003.