A Real-Time System for Detection and Recognition of Traffic Signs

A Real-Time System for Detection and Recognition of Traffic Signs

Sudha S. K.

Associate Professor Department of Computer Science

Sarabhai College of Engineering & Technology

Mahathi L

M. Tech. Student Computer and Information Science

Sarabhai College of Engineering & Technology

AbstractA traffic sign detection and recognition system developed and implemented as a part of Intelligent Transportation System (ITS), aims to minimize the number of accidents due to drivers negligence or wrong decisions. It helps the road users by guiding and providing all traffic related information about the law, to warn the driver about the dangerous condition ahead and current status of the road in advance, so that driver can find his/her way safely. This real time computerized system produces the output simultaneously as the driver go ahead, and keep informing about the latest situation before they take place on the road automatically. The objective of this paper is to identify a traffic sign from a real-world scene. The sign may be viewed from various angles and in many diverse background situations. The identified sign will then be conveyed to the driver after identification. Images captured through a camera are pre-processed and traffic signs are detected from the images. The detected signs are then recognized based on a classifier after extracting its features. The recognised sign is then informed to the driver through a speech output. Thus this system will enhance road safety by informing the driver, about speed limits, road safety regulation and also information on the oncoming dangers such as hump ahead, railway crossing, school ahead, etc. As an addition to road safety engineering, this system can also be used to detect damaged road signs.

Keywords Traffic signs, Detection, Classification, Neural Network, Thresholding

1. INTRODUCTION

   According to the World Health Organisations (WHO) global status report on road safety 2013 estimates that more than 231 000 people are killed in road traffic crashes in India every year [15]. Technology can be very helpful in decreasing the number of road accidents and in turn a decrease in causalities. Quite often, the traffic signs on the road sides becomes difficult to watch for the drivers and the driver may sometimes miss the warning signs. These signs include bump ahead or narrow bridge or even accident zone etc. This becomes even more difficult during many times and at nights. Sometimes because of the traffic or the road condition driver may not be able to read anything. This paper aims at developing a solution for this problem using image processing technique.

   A Traffic Sign Detection and Recognition (TSDR) system helps the road users by guiding and providing all traffic related information about the law, to warn the driver about the dangerous condition ahead and current status of the road in advance, so that driver can find his/her way safely. This real time computerized system produces the output simultaneously as the driver go ahead and keep informing about the latest situation before they take place on the road automatically. Thus a typical TSDR system uses computer vision and pattern recognition for the automatic detection and recognition of traffic signs and symbols. TSDR systems can be used in various applications such as advanced driver assistance systems, road surveying, and inventory purposes, autonomous and assisted driving, etc. These systems provide an additional level of driver assistance, leading to improved safety for passengers, road users and vehicles.

2. PHYSICAL CHARACTERISTICS OF A ROAD SIGN

   Road signs, in India, were incorporated in the Motor Vehicles Act, 1988. Road signs are designed to regulate the flow of the vehicles, give specific information to the traffic participants, or warn against unexpected road circumstances. Road signs are always mounted in places that are easy to spot by the drivers without distracting them from manoeuvring the vehicle, e.g. on posts by the roadside or over the motorway lanes. Besides, their pictograms are designed in a way that admits easy discrimination between multiple signs, even from the considerable distance and under poor lightning and weather conditions.

   There are a number of traffic signs in India categorized as:

   1. Cautionary/ Warning (36)

   2. Mandatory/ Regulatory (38)

   3. Informatory (14);

      making a total of 88 traffic signs all together. These signs are mainly characterized by their different colour and shape. Fig 1. shows the different shapes and colours of the traffic signs that can be seen in India.

      Fig 1: Different shapes and colour

      1. Cautionary/Warning Sign: A triangle with red coloured border and white background represents a cautionary sign. Different pictograms in black are used to represent the various warnings. These signs warns the driver of the existence of certain hazardous condition either on or adjacent to the roadway.

         Fig 2: Some examples of cautionary signs

      2. Mandatory/RegulatorySign: Mandatory signs uses circle with red border and white background or a blue circle with white border with a white pictogram of arrow. These signs driver of certain laws and regulations that has to be followed to provide safety and free flow of traffic. Mandatory signsare placed to control the action of the drivers on road. They are used to regulate the traffic flow and vehicles moving on road.Signs with a cross prohibit the driver from certain decision such as no left turn or no U turn etc. Speed limit signs are also included in this category of traffic signs, with speed limit as the pictogram. An exception to the shape is octagon with red background with STOP in white.

         Fig 3: Some examples of mandatory signs

      3. Information Sign:Informatory signsguides road users along routes, inform them about destination and distance, identify points of geographical and historical interest and provide other information.Important information like

   nearby hospitals, telephone booth, first aid, petrol pumps etc. come under this category. This information helps the driver in emergency in need. It is represented in a white rectangle with thick blue border and the necessary pictogram represents the required pictogram. Parking information is also included in this category. Informatory signs also include the signs which provide information on location and distance. They are represented in a green background with white text.

   Fig 4: Some examples of informatory signs

   Even though the physical characteristics of traffic signs are simple enough to be detected, identification of traffic sign is still a challenging task due to different geographic and weather conditions like cloudy day, raining or foggy day. The lighting conditions are uncontrollable since it is time dependent and seasonal. Moreover, the signs are of different types. Distance between the sign and video capturing device is a factor. The blurring of the image is dependent on the speed of the moving vehicle. Other problems include: sign may be disoriented, damaged, faded or occluded. There may be similar objects as in colour or shape. Hence most sign detection system use both colour and shape as distinguishing feature and pre- processing techniques for image enhancement for coping with the varying lighting conditions.

3. RELATED WORKS

   As early as 1987, Japan researchers H. Akatsuka et al.

   [1] began the work on automatic traffic sign detection and recognition. Over the past twenty years, French researcher De Saint Blancard M. develped red sign recognition systems [2], in 1992, these developed systems used a red filter, edge detection method, a closed curve detection method, and a neural network recognition method to complete the extraction process and target classification. In 1994, German scientists S. Estable et al. launched the world's most advanced TSDR system [3] at that time. The system used color segmentation, shape analysis and statistical pattern recognition techniques. The research on TSDR system has achieved a few practical results with its evolution over the last few decades.

   In 2006, inventors W. Wu, X. Chen, and J. Yang [4] developed a TSDR system for images on video. It uses a combination of corner detection algorithms, Guassian mixture models and geometric analysis and an assumption was made that traffic sign text appeared on a vertical plane with respect to motion and optical axis of the camera.

   In 2006, scientists A. Reina et al. [5] developed methods for road traffic sign panel text extraction. They segmented the regions of interest (ROI) based on color information, by applying a threshold to chrominance and luminance channels. Rectangular regions were thus found by comparing the FFT signature of each blob to FFT signature of rectangular reference. Points found thus were used to vertically align text characters.

   In 2012, F. Zaklouta and B. Stanciulescu [6] used tree- classifiers for real-time traffic-sign recognition. They evaluated the performance of k-d trees, random forests, and support vector machines (SVMs) for traffic-sign classification using different-sized histogram-of-oriented- gradient (HOG) descriptors and distance transforms (DTs). Fishers criterion and random forests were then used for the feature selection to reduce the memory requirements and enhance the performance.

   In 2012, A. Møgelmose et al., [7] in their survey of the vision-based traffic sign detection analysed the contributions of recent works to the various stages in TSDR such as: segmentation, feature extraction, and final sign detection. They also researched on the integration of context and localization

   In 2014, inventors A. González et al., [8] used visual appearance for text detection and recognition on traffic panels from street-level imagery. They developed an approach to detect traffic panels in street-level images and to recognize the information contained on them, as an application to intelligent transportation systems (ITS). It extracts local descriptors at some interest keypoints after applying blue and white color segmentation. Images are represented as a bag of visual words and classified using Naïve Bayes or support vector machines.

   In 2015, Jack Greenhalgh and Majid Mirmehdi [9] developed a novel system for the automatic detection and recognition of text in traffic signs. Scene structure is used to define search regions within the image, in which traffic sign candidates are then found. Maximally stable extremal regions (MSERs) and hue, saturation, and value color thresholding are used to locate a large number of candidates, which are then reduced by applying constraints based on temporal and structural information. A recognition stage interprets the text contained within detected candidate regions. Individual text characters are detected as MSERs and are grouped into lines, before being interpreted using optical character recognition (OCR). Recognition accuracy is vastly improved through the temporal fusion of text results across consecutive frames.

   TSDR system is a still ongoing research work. More and more approaches are being invented and suggested that

   improve the TSDR in all aspects. The works included in the literature survey can be summarized as follows. Problems such as illumination and shadows, blurring, occlusion, sign deterioration affect the TSDR. The TSDR must be robust enough to handle all kind of situations. No system has been yet found that can detect and recognize texts written in regional languages. In India, many textual sign boards are written in regional languages. So a TSDR system cannot be used which can recognize only texts written in English. TSDR system recognizes all traffic signs and symbols and there is no option of selecting just a type of sign/symbol recognition. User will be provided information on all the traffic signs and symbols on road and does not have the option to choose the detection and recognition of a single text or symbol. Detection and recognition in diverse weather conditions not optimally addressed. Adverse weather conditions such as snow, fog, hail, etc. will affect the viewing of sign boards. This is where a TSDRS is mainly needed. But TSDRS in very adverse weather conditions has not been addressed optimally as such. TSDR systems in India may be only applicable in urban roads and might not be feasible in a rural environment where sign boards may not be available.

4. TRAFFIC SIGN DETECTION AND RECOGNITION SYSTEM

   A Traffic Sign Detection and Recognition (TSDR) system works in two stages.

   1. Detection- In this stage, the presence or absence of a traffic sign is detected.

   2. Recognition- In this stage, the detected traffic sign is recognised and classified.

   A prototype of the proposed system for the detection of Indian traffic sign is shown in fig 5

   Fig 5: A prototype of the traffic sign detection & recognition system

   It mainly consists of four stages: Image acquisition, pre-processing, detection and classification.

   In the detection phase, the acquired image is pre- processed, enhanced, and segmented according to the sign properties of color and shape. Images are captured using HD car surveillance camera placed on the roof of the car, to get images according to a particular frame rate. These images are processed for image enhancement according to environmental conditions. An analysis of colour and shape is performed on the frame to detect the possibility of a traffic sign in that frame. In colour analysis,objects with similar colour (red or blue) as that of traffic sign is segmented out of the image as sign candidate. To confirm whether these candidate are in fact traffic signs a shape segmentation is used. If a traffic sign is not discovered in the current frame, it is discarded and the next frame is taken for analysis.

   The detection stage must be fast and robust to speed up the computation of the real time detection system. The output of the detection stage is input to the recognition stage where neural networks are used for classification. Since the three different types of traffic signs has some distinguishing physical properties, three separate, different database can be used for training. Thus, classification is easier, robust and faster. The main steps included can be summarized as:

   1. Image capturing from camera.

   2. Image enhancement for different lighting and whether conditions.

   3. Selecting candidate objects based on colour.

   4. Filtering the candidate objects based on shape.

   5. Recognising the filtered object using neural network

   6. Give visual image of the recognised sign and audio for the type of category of the traffic sign.

      1. Image Capturing from Camera

         Images are captured by a HD car cam that can take images of high definition (720p) mounted on the roof of a moving vehicle. The camera is configured to capture images at a particular frequency rate. i.e., the camera will capture images at a pre-set rate of frames per second. The HD cam can also capture good images at night using infrared night vision technology.

         Since images are captured using cameras installed in a moving vehicle the images might appear blurred. Also the brightness, contrast, clarity are affected by time of day, location and weather conditions. Due to climatic conditions or calibration difference, noise may also be encrypted along with the image. This would increase the difficulty of detection and therefore some pre-processing is required to reduce their influence accoding to varying conditions. The Gaussian noise filter and Weiner filter is used to remove the noise.Each of the image frames are then pre-processed to increase their clarity and then sent to the detection stage one by one.

      2. Sign Detection Stage

         Traffic signs are going to be detected colour andshape analysis. First the captured image which is in RGB colour space, is converted to an YCbCr colour space. The YCbCr color space is widely used for digital video. In this format, luminance information is stored as a single component (Y), and chrominance information is stored as two color- difference components (Cb and Cr). Cb represents the difference between the blue component and a reference value. Cr represents the difference between the red component and a reference value.

         The YCbCr image can be converted to/from RGB image. To convert from RGB to YCbCr:

         Y = 0.2126 *red+0.7152*green+0.0722*blue Cb = 0.5389*(blue-Y)

         Cr = 0.6350*(red-Y)

         Fig 6: A YCbCr colour space model

         YCbCr holds a better option during detection due to the following reasons:

         1. Though RGB colour space represent colour image directly human eye has different sensitivity to colour and brightness.

         2. YCbCr represents colour as brightness and two colour components as the difference of two signals, whereas RGB represents colour as Red, Green and Blue.

         3. The illumination variation can be solved by converting RGB to YCbCr colour space.

         4. YCbCr is a colour space used in JPEG.

            Detected sign lies in the region of interest. After identifying this region, we extract important features of the region and classify this region of interest.

            1. Colour Based Analysis:

               Traffic sign can be classes into two groups mainly as red traffic sign and blue traffic signs. Since colours are distinguishing feature of traffic sign, colour extraction is an effective solution for candidate selection from an image. Hence colour segmentation is performed in YCbCr color space.

               Thresholding technique is then used to separate the red and blue colour objects in the image. Output is a binary image. Morphological operation is done to close open area thus removing all the stray pixels.

            2. Shape Based Analysis:

            Though colour segmentation is used for initially selecting the candidate objects, shape detection is used for filtering off the candidates. First step is labelling the connected regions. All connected candidate pixels are grouped using 8-neighbors. There may be more than one candidate object. Hence an effective measure for filtering is required without losing the traffic signs. The shape characteristic is hence used.

            There are mainly four shapes. Triangle representing alert sign, circle representing compulsory, rectangle for information sign and octagon meant only for STOP. Shape detection is based on a similarity measure between the binary image of the colour segmented road sign and the objects of the template database. For this, both the sample and segmented images must have the same dimensions or else resized. The distance measure (Euclidean distance) is used for cross correlation for template matching. The colour segmented image after thresholding is matched with the different templates in the database. If the similarity measure for the match is greater than a threshold value then it is detected as a traffic sign and classified into any of the three classes.

            Fig 7: Detection process of a stop sign

      3. Sign Recognition/ Classification Stage

   Once the presence of a traffic sign has been established during the detection stage, the resized frame is sent to the next stage for recognition and classification process.

   1. Pictogram thresholding:

      Once the traffic sign has been filtered and the presence of traffic sign in the frame is confirmed, we can now recognize the traffic sign. The pictogram inside the traffic sign is the key to recognize the traffic sign correctly. Thus, the pictogram in the frame is to be segmented out. For this, a thresholding operation is performed. The frame from detection stage is analysed pixel-by-pixel. All pixels that are above a set threshold value are made into white pixels and all pixels that are below the threshold value are converted to a black pixel. Thus the pictogram inside the traffic signs gets extracted. There may some stray pixels

      after thresholding, which can be removed using morphological operation. Thus, after this process we are left with a binary image of the pictogram inside the traffic sign. This pictogram will be used for classification.

      Fig 8: Extracted pictogram showing the stop

   2. Classification using neural network:

   The classification process uses neural network which is very efficient and reliable.Neural networks are a suitable for recognition and classification of road signs. Main two advantages of using neural networks is that, the input image does not have to be transformed into another representation space. Second, the result of the classification depends on the correlation between the network weights and the network itself which are chosen from the beginning. The main disadvantage using multi-layer neural network is the requirement of considerably large training data images. Since a standard database does not exist for Indian traffic signs this proves to be a drawback. Thus a database has to be created first, either of manual data or synthetic data or a combination of both.

   A multilayer feed forward neural network with supervised learning method is more reliable and efficient for this purpose. The network applies back propagation- learning algorithm that is a systematic method for training multiple layer ANNs. The objective of the training is to adjust the weight so that the input produce desired output.

   The main task of the classification module is to classify the extracted regions of interest presented to its input into the road-sign category they belongs. Here the MLP networks have been employed to implement the classification module because they have proven to be good classifiers and have been able to successfully solve several object recognition problems.

   In a network, the input information moves in onlyone direction which is from the input nodes, through thehidden layers node and to the output nodes as shown inmodel in Figure 9.A three-layer feedforward neural network as in Figure 9 is sufficient to classify the various classes of traffic sign.

   Fig 9: Basic model of neural network

   Inputs for the neural network are corresponding to thefeature vector length extracted from the trained images. The output layer is set to have 3 neuronnodes which are the number of classes of traffic sign. The number of neurons in the hidden layer is found by multiple testing of the system. (It may be set to half the number of input neurons).

   Since, the type of traffic sign is acquired during the detection stage itself, the neural network classification will become easier and faster. The network will process the image in any one of the three sample database and find a closest match from the database. Thus, the sign gets classified.

5. APPLICATIONS

A traffic sign detection and recognition system can be used to inform a driver about the approaching traffic signs and informs the driver via a visual or auditory aid. A TSDR system can also be used in various other application to reduce manual work and increase efficiency. Some application of such a system include:

1. Highway maintenance:Currently, a human operator has to watch a videotape to check the presence and condition of the signs. It is a tedious & wearisome task because of the amount of manual work to be done and depended on.

2. Sign inventory: Similar to highway maintenance, sign inventory keeps track of positions and placement of traffic signs in cities and towns. In this case, it is more difficult than highways. The signs are not alays placed perpendicular to the movement of the vehicles, producing a deformed image of the signs; besides, there are occlusions, and other objects with the same colour.

3. Driver Support Systems. Traffic sign detection and classification is one of the main field of Driver Support Systems. The general idea is to support the driver in some tasks, allowing him or her to concentrate in driving.

1. CONCLUSION

   This paper describes the method of classifying and recognizing the traffic signs in real-time. The paper describes the system that is used to recognize Indian Traffic Signs that is subdivided in three classes according to the shapes. The system works in two basic steps, it detects the traffic sign, classifies it and finally recognizes the traffic sign. A pattern matching technique is used for comparing the traffic signs. The system is fast and robust as the system discards the images having signs of shapes other than the basic shape at the initial phase. Future modification can make the system more robust to weather conditions and tilted signs.

2. REFERNCES

1. H. Akatsuka and S. Imai, Road signposts recognition system,"

   Training Journal, pp. 12-11, 2013

2. J. Greenhalgh and M. Mirmehdi, Real-time detection and recognition of road traffic signs, IEEE Trans. Intell. Transp. Syst., vol. 13, no. 4, pp. 14981506, Dec. 2012.

3. A. González, L. Bergasa, and J. Yebes, Text detection and recognition on traffic panels from street-level imagery using visual appearance, IEEE Trans. Intell. Transp. Syst., vol. 15, no. 1, pp. 228238, Feb. 2014

4. FatinZaklouta and Bogdan Stanciulescu, Real-Time Traffic-Sign Recognition Using Tree Classifiers,IEEE TRANSACTIONS ON INTELLIGENT TRANSPORTATION SYSTEMS, VOL. 13,

   NO. 4, DECEMBER 2012 1507

5. F. Zaklouta and B. Stanciulescu, Real-time traffic-sign recognition using tree classifiers, IEEE Trans. Intell. Transp. Syst., vol. 13, no. 4, pp. 1507 1514, Dec. 2012.

6. TuhanÇalayan, HabibullahAhmadzay and GökhanKofraz, Real-Time Traffic Sign Recognition Based on Shape and Color Classification, IEEE 2015

7. Mrs. P. Shopa, Mrs. N. Sumitha, Dr. P.S.K Patra, Traffic Sign Detection and Recognition Using OpenCV, IEEE 2014

8. NursabillilahMohd Ali, NurMaisarahMohdSobran, M. M. Ghazaly, S. A. Shukor and A. F. Tuani Ibrahim, Traffic Sign Detection and Classification for Driver Assistant System The 8th International Conference on Robotic, Vision, Signal Processing & Power Applications, 2014

9. G. Piccioli, E. De Michelib, P. Parodi, M. Campani, Robust method for road sign detection and recognition, 1996 Elsevier

10. Nilesh Uke1 and Dr. Ravindra C Thool Road Sign Detection A Vision based Driver Support, Elsevier, 2013

11. Chunsheng Liu, Faliang Chang, Zhenxue Chen, and Dongmei Liu Fast Traffic Sign Recognition via High-Contras Region Extraction and Extended Sparse representation, IEEE TRANSACTIONS ON INTELLIGENT TRANSPORTATION SYSTEMS, 2015

12. Aly A. Farag and Alaa E. Abdel-Hakim, DETECTION, CATEGORIZATION AND RECOGNITION OF ROAD SIGNS FOR AUTONOMOUS NAVIGATION, Proceedings of Acivs 2004

13. U. Zakir, A. N. J. Leonce and E. A. Edirisinghe, Road sign segmentation based on colour spaces: A Comparative Study, In Proceedings of the 11th Iasted International Conference on Computer Graphics and Imgaing, Innsbruck, Austria, 2010.

14. tefan Toth, Usage Smartphones for Traffic Sign Recognition and Data Acquisition

15. http://www.who.int/violence_injury_prevention/surveillance/en/