Regression Test Framework using Golden Dataset for Adaptive Cruise Control

Regression Test Framework using Golden Dataset for Adaptive Cruise Control

Vishal Gholap1,

1School of Electrical Engineering, Vellore Institute of Technology, Vellore, Tamil Nadu-632014, India,

Ramani Kannan3

1. Bagyaveereswaran2, 2School of Electrical Engineering, Vellore Institute of Technology Vellore, Tamil Nadu-632014, India,

   3Department of Electrical and Electronics Engineering, Universiti Teknologi, Petronas – 32610

   Bandar Seri Iskandar, Perak Darul Ridzuan, Malaysia,

   Abstract – The Advanced Driver Assistance Systems (ADAS) used in automobiles involve the integration of many sophisticated technologies into the vehicle. With the increasing complexity of ADAS systems, ensuring the safety of car occupants, pedestrians and vehicles on the road are of paramount importance. In this regard, performing regression test to exercise the ADAS software, ensuring coverage of different traffic scenarios, is a very critical step in the product development. This primary scope of the project assignment is to create a golden data set of vehicle logs containing different road and traffic scenarios, create the ground truth information for the golden data set, create a batch re- simulate script to automatically re-simulate all the logs in the data set with any version of tracker software.

   Keywords: Advanced Driver Assistance System, Adaptive Cruise Control, Data Association, Automotive Radar, Data Fusion, Ground-truthing, Kalman Filter.

   1. INTRODUCTION

      The Active safety systems are one of the important features in any vehicle these days. The actions from these systems can be used for building advanced and efficient control system in automobiles [1]. The overall safety in automobiles can be categorized under active and passive features. The active safety features are implemented for actions to be taken before the accidents or to avoid the accidents and passive features are implemented for actions to be taken immediately after accidents occur. These are nothing but preventive and curative measures. With the use of cameras, radars and various sensors hazardous events are detected. An attempt is made to avoid these events. If the hazard avoidance is inevitable then an attempt is made to reduce the consequences. Forward collision warning systems, lane departure warning systems, anti-lock braking systems, and emergency braking system are some of the active safety features. These features work in real time, i.e., continuous monitoring is done for various aspects, on the host as well as surrounding vehicle status and environment. This is done to avoid the potential hazards by taking necessary actions. These features cannot replace a human driver completely but an attempt is made to avoid crashes.

      The Adaptive Cruise Control (ACC) is used for activation of mild acceleration or braking to maintain the speed and to maintain the constant distance from the vehicle immediately in front. ACC ensures car and passenger safety [2][3]. For every car, ACC parameters must be tuned for various traffic scenarios using golden data- set. Radar and Cameras are mainly used for perceiving the surrounding scenario and measure the properties of the objects such as range (longitudinal and latitudinal distance), range- rate, azimuth angle, etc. ACC must be able to perform well in environmental conditions such as rainy and snowy conditions, night time, day time, etc. It must perform well in a tunnel, village roads where lane markings are not present, on highways, in traffic scenarios, etc.

      The Isometric radar in automotive systems is used to detect or track relevant objects surrounding the vehicle. It has transmitter and receiver which will transmit the radio waves and receive after reflection from the target or object [4]. Todays challenge is to come up with compact high performance radars with a reasonable cost. Apart from target detection, radars are also used for blind spot detection, lane change assistance and lane departure warning, cross traffic alert, collision detection warning and mitigation.

      Automotive radars can be classified into three categories: Long range, medium range and short range radars. Frequency modulation continuous wave radars are used for improved range and reduced antenna height and overall cost. To understand the concept of distance measurement using radar, let us consider a stationary host and a stationary target object. The transmitted and received wave chirps along frequency and time axis are shown in Fig. 1. It is also showing the chirps with different slopes. The bandwidth of sinusoidal waveform and pulse duration are inversely proportional to each other. Range resolution of radar is dependent on the bandwidth of the signal transmitted. So, signals of short duration are required for good range resolution. Usually, radars use Frequency modulated continuous wave (FMCW). In this frequency changing pattern, frequency transmitted is changed linearly within each time period. This change in frequency increases the bandwidth and hence, leads to narrow the signal in time- domain. Resolution and range are dependent on the slope of

      the chirps. For smaller slope, the resolution is less and range is high. For higher slope, the resolution is more and range coverage is less. The formula to calculate range of the radar is given in (1). Hence, with the increase in slope keeping the chirp time constant, bandwidth increases, hence the range coverage of the radar decreases. Similarly, with a decrease in slope keeping the chirp time constant, bandwidth

      decreases and range of radar increases. The formula to calculate the resolution of the radar is given in (2). Hence, with the increase in slope, bandwidth is more, hence the resolution of radar increases. Similarly, with a decrease in slope, bandwidth decreases and resolution of radar decreases.

      Range C *

      2

      dF * T Bandwidth

      (1)

      Where, dF is beat frequency or sweep frequency, dT is time interval between transmitted and received waveform, C is speed of the electromagnetic wave. The range is calculated by

      taking FFT across samples in chirps. The FFT of range data across chirps gives range rate. The FFT of range/ rate data across antenna elements will give us azimuth angle

      resolution Bandwidth

      T

      Figure 1. Distance measurement using radar

      Figure 2. Chirps with different slopes

      (2)

      The ACC is dependent mainly upon vision-based detection where image recognition algorithms are run to track the selection targets and driver state. The camera resolution is increasing with a decrease in cost. The connectivity options (FPD- link, LVDS) are expanding. Radars, radars can be connected as supplementary technologies [3].This is done to increase the robustness of ACC. Following are some of the vision-based detection systems [5].

      1. Lane Departure and Lane Keep Assist Warning: Camera is used to detect the lane markings and give the warning signal to the driver and if necessary take required actions.

      2. Pedestrian Detection: Detect the pedestrians and warn driver for potential detection and take braking actions if required.

      3. Forward collision warning system: Detect the relevant targets around the host vehicle and warn the driver in critical situations to avoid the crash and take required actions.

   2. RELATED STUDY

      2.1 Radar and Camera Data Fusion

      The camera and radar are complimentary sensors. They both have their strength and weaknesses. The cameras are good at object classification and measuring the azimuth angle of the objects, they cannot measure the distance and speed of he objects accurately. The radars, on the other hand, have very

      good accuracy and resolution in measuring the distance to the objects and detecting their speed. But they are not good at object classification. Therefore, using sensor fusion, we can achieve better ACC performance. For testing and tuning the active safety algorithms, it is essential to collect sensor data on roads. This raw data collected from cameras and radars can later be used in the simulation environment to test various algorithms. This raw data is recorded in the form of vectors and stored in the form of logs and used for product development. The object fusion generation process is shown in Fig.3. The logs are set

      Figure 3. Fusion Objects Generation Process

      Figure 4. Data Fusion Flow-diagram

      of files which contain data collected from CAN, Ethernet, flex-ray, LVDS, CVBS, etc.[6].When data has to be retrieved, it is passed through various software modules as shown in Fig.4. The resimulation environment can be used to develop the algorithms offline in a PC environment. There are various stages in resimulation.

      • Radar low level data can be used to develop/test DSP algorithms which form several radar detections per object. Detections represent scatter points on the surface of an object.

      • The radar detections can then be tracked, matched to the objects detected by the camera and then form fused objects.

      • A number of targets can then be selected to be processed by the ACC algorithm.

      Radar captured data is stored in Ethernet data file in the form of streams. Ethernet data file will store radar and camera data and fusion data. Following properties are measured by the radar:

      1. Range: The distance from the Radar sensor to the detected object.

      2. Range Rate: The rate-of-change of the range between the radar and the object (i.e., the derivative of the range over time).

      3. Azimuth Angle: The angle between a line perpendicular to the center of the radar and the detected object.

      4. Radar Cross Section: A measure of how detectable an object is, compared to a smooth sphere with a cross-sectional area of one-meter square.

      5. Radar Objects: Refers to the virtual objects that are being tracked with each object having a set of properties.

      6. Vision Objects: A virtual object based on camera sensor measurements.

      7. Fusion Objects: A virtual object that groups radar object array and Vision objects. Fuses the measurements of all associated objects.

      The camera collects the data in the form of image frames. These frames are fed to vision algorithms along with vehicle

      data. Various machine learning features such as SIFT and support vector machine are used for object classification and target selection. The output of vision algorithm module is called vision objects [7]. The Radar Objects and vision objects are fed to fusion algorithm to get fusion data.

   3. KALMAN FILTER AND DATA ASSOCIATION

      1. Kalman Filter

         The Kalman filter (KF) is time-tested technique for tracking algorithms, estimation of parameter and data fusion algorithms. Rudolf E. Kalman invented the filter. Its small computational procedure, elegant recursive properties are the key to the great success. It is an optimal estimator and used for linear systems with Gaussian error statistics. KF is typically used for estimating parameters of interest and removing the noise or smoothing noisy data. From a theoretical standpoint, the KF uses linear dynamical state space model for exact inference. The KF, Extended KF and Unscented KF are popular data fusion algorithms and most celebrated in information processing. KFs are at work in every smartphone, many computer games and in satellite navigation device [8].The KF is a minimum mean squared estimator derived using vector algebra. It is dependent on a property that the product of two Gaussian distribution is a Gaussian distribution. It finds the state of the system parameter by considering the prior state [9]. There are two steps in the KF algorithm. It is shown in Fig.5.

      2. Data Association

      Figure 5. Kalman Filter steps which are called iteratively

      Now the data association algorithm groups all the radar

      From the fusion data, we get many fusion objects for various objects in the surrounding. Nearby fusion objects which have the same position, velocity, yaw rate are grouped together and considered as the same object. These n number of radar objects for the same vehicle are due to reflection from the various points on the same vehicle [10]. Estimating exact location of an object include the use of Kalman filter. The various radar objects for the same vehicle with the cross symbol is shown in Fig.6. Using Kalman filter the position and velocity of the vehicle is estimated. It is denoted by square symbol.

      and vision objects that correspond to a physical object. The KF predicts the exact location of the vehicle based on individual measurements with the help of the state vector of the model and the covariance matrix [11].The exact location of the vehicle is estimated based on the fusion data object and estimated vehicle position using Kalman gain. Covariance matrices are derived from sensor characteristics and the motion model of the system.

      Once the vehicle is located, using position, velocity and yaw rate data, we can estimate the next position of the vehicle. When measurements are available, estimated values are updated.

   4. SELECTION TARGET EVENTS

      Figure 6. Data Association

      adjacent lane are labelled ST3 and ST5. The two nearest

      In any road scenario, ACC software will characterize vehicles in front of the host vehicle as selection target ST1, ST2, ST3, ST4 till STn and STS. The front radar will be located behind the front bumper and vehicle will be detected when it is in the field of view. A camera will be placed behind the windshield. The selected targets are labelled from ST1 to STn based on their longitudinal and lateral positions with respect to the host. Two nearest targets in the same lane as the host are labelled ST1 and ST2. The two nearest targets in the left

      targets in the right adjacent lane are labelled ST4 and ST6. Similarly, selection targets are classified till STn as per requirement. STS are stationary targets. Stationary targets are out of the scope of this paper. Whenever a selected target is recorded, it is considered as a particular event. If the host or target vehicle changes the lane, then the vehicles position changes. So the current events must be ended and new events must begin. Fig.7 shows the pictorial representation of selection target.

   5. GOLDEN DATA SET

      Figure 7. Selection Targets

      straight road, curvy road, highway, village road, uphill,

      So far we have seen how the data is recorded in logs and how it is retrieved using various algorithms and machine learning features. We have also seen how vehicles are classified as selection targets in road scenario. Now, collecting a number of logs and training the product over those logs is necessary for any product development. Logs must be collected for various environmental, road scenarios. Created logs must add on to the safety issues. Logs must be collected for daylight, evening, night time as well as for rainy, cloudy, snowy, sunny weather. Combinations like day rain, night rain, day snow, night snow, etc. The various kind road must be recorded like

      downhill, multiple-lane road, one-way and two- way road, high traffic and low traffic road, S-curve, roundabout, etc. Various events such as host lane change, target lane change, sharp turn, etc. must be recorded.

      In a tunnel, various false targets are detected due to the number of reflections from tunnel walls. Same is the case on roads with barricades and guardrails. In this case, the data from radar is less reliable, so camera data is used which is very useful for object/vehicle identification.So, ACC software must be well trained for such situations. In case of

      merging lanes, the ACC feature may not be able to classify the selection targets. This is because merging lanes act as a transient period and hence the ACC feature is not decided whether the host is moving to other lane or target is moving to its lane or simultaneous lane change is happening. Hence, the product must be designed to tackle all these scenarios and such logs must be included.

      This whole set of logs is termed as Golden Data-set. The shape of vehicle differ from country to country, the marking on roads differ, the kind of diversion symbols on barricades and hoardings differ. All these things must be taken into consideration. These logs are run on the product and output is recorded. The records are used to generate some statistics in order to validate the output. Once the golden data-set is created ground truthing is done followed by automated resimulation.

   6. GROUND TRUTHING AND STATISTICS GENERATION

      Once golden data-set is collected, the logs are run through the tracker software and the results (target detections and classification) are recorded. These results are then used for generating some statistics. Ground truthing is the procedure where actual events in the log are recorded manually using the software tool (GT tool) and some statistics are generated

      automatically with the help of tool records. Now, these records from tracker software and GT tool are compared for checking the authenticity of tracker software.

      Logs are run on the ground truthing tool. The avi file is run along with the plot of fusion objects/data simultaneously. Fig.8 and Fig.9 show how records can be made for various events. Fusion objects can be used to classify the vehicle as stationary or moving by using different colours. The various environmental conditions, road type and vehicle manoeuvre can be selected accordingly and records can be made in a tabular format.

      Some of the statistics which are mainly considered while comparing the records are missed, dropped, false and delayed events. If the tracker software completely fails to detect an event, then it is a missed event. If tracker software detects an event for some time and drops it in the middle, then it is a dropped event. If the tracker software detects an event even though it is not there in the ground-truth table, then such event is a false event. If tracker software delays in detection, then it is a delayed event. Calibration parameters limit is given for different statistics. If those tolerance limits are not satisfied then required changes are made in next version of tracker software, so that tolerance limits are satisfied.

   7. RE-SIMULATION

      Figure 8. Ground Truthing Tool part 1

      continues or the process is terminated. Now, tags for data are

      Re-Simulation means, execution of logs over the new version of tracker software. It gives us results about how the new version of the software will perform in real time environment. To check whether the tolerance limits are satisfied or more changes are required. The flowchart of resim process is shown in Fig. 10. Initially, it is checked whether the log file to be resimed is available in the folder. If so, the process

      set which needs modification. In our case, we consider the radar object arrays from radar, vision objects from camera and fusion data. Then a check is made to confirm that the executable file over which resim is carried out is present or not. Then the app is run and required outputs are generated. These results are then used to generate statistics and again compared with ground truth data.

      Figure 9. Ground Truthing Tool part 2

      Multiple logs can be run and comparison can be made for statistics. This can be done for logs from various folders by automating the process using the batch script. It is shown in

      Fig. 11. The file name can be renamed which includes the tracker software version and required data.

      Fig. 10. Log Re-sim Flowchart Fig. 11. Multiple Log Re-sim Flowchart

   8. CONCLUSION

The golden data set (in the form of logs) was collected for different environmental and road scenarios. Ground truthing was done for every set of a log. The required changes were made in ground truthing tool for better usability. Statistics were generated and compared using ground truth data and original results from tracker software. After required changes are made in the tracker software, the logs are resimed again and compared with ground truth statistics. Resim is done regressively for the number of logs by process automation using the batch script.

REFERENCES

1. Chakraborty, Shouvik, Sachidananda Sen, Ashoke Sutradhar, and Anindita Sengupta, Estimation of tire-road friction coefficient and frictional force for Active Vehicle safety system, Proc. IEEE International Conf. on Industrial Instrumentation and Control (ICIC), 2015, 674-679.

2. Ioannou, Petros A., and Margareta Stefanovic, Evaluation of ACC vehicles in mixed traffic: Lane change effects and sensitivity analysis, IEEE Transactions on Intelligent Transportation Systems, 6(1), 2005, 79-89.

3. Hammond, Maen, Guangzhi Qu, and Osamah A. Rawashdeh, Deploying and Scheduling Vision Based Advanced Driver Assistance Systems (ADAS) on Heterogeneous Multicore

   Embedded Platform, Proc. Ninth IEEE International Conference on Frontier of Computer Science and Technology (FCST), 2015, 172-177.

4. Arab, Homa, S. O. Tatu, and Cevdet Akyel, Design and Characterization of a 77 GHz Six-Port Modulator for an Automobile Radar, Proc. 84th IEEE conference on Vehicular Technology Conference (VTC-Fall), 2016, 1-5.

5. Nikolic, Zoran, Gaurav Agarwal, B. Williams, and S. Pearson, Ti gives sight to vision-enabled automotive technologies, Texas Instruments, Tech. Rep. ,2013.

6. Srinivasa, Narayan, Yang Chen, and Cindy Daniell. A fusion system for real-time forward collision warning in automobiles. Proc. IEEE conference on Intelligent Transportation Systems, 2003, vol. 1, 457-462.

7. Ma, Wenhua, Zhenjiang Miao, and Qiang Zhang, Vehicle classification based on multi-feature fusion, 2013, 4-6.

8. Hinkkanen, Marko, Lennart Harnefors, and Jorma Luomi, Reduced-order flux observers with stator-resistance adaptation for speed-sensorless induction motor drives, IEEE Transactions on Power Electronics, 25(5), 2010, 1173-1183.

9. Kalman, Rudolph Emil, A new approach to linear filtering and prediction problem, Journal of basic Engineering, 82(1), 1960, 35-45.

10. Folster, Florian, and Hermann Rohling, Data association and tracking for automotive radar networks, IEEE Transactions on Intelligent Transportation Systems, 6(4), 2005, 370-377.

11. Lee, M-S., and Y-H. Kim, New data association method for automotive radar tracking, IEEE Proceedings-Radar, Sonar and Navigation, 148(5), 2001, 297-301.