Analysis of Market-Ready Traffic Sign Recognition Systems in Cars

energies

Article Analysis of Market-Ready Trafﬁc Sign Recognition Systems in Cars: A Test Field Study

Darko Babić , Dario Babić* , Mario Fiolićand Željko Šarić

Faculty of Transport and Trafﬁc Sciences, University of Zagreb, 10000 Zagreb, Croatia; [email protected] (D.B.); mario.ﬁ[email protected] (M.F.); [email protected] (Ž.Š.) * Correspondence: [email protected]

Abstract: Advanced Driver Assistance System (ADAS) represents a collection of vehicle-based intelligent safety systems. One in particular, Trafﬁc Sign Recognition System (TSRS), is designed to detect and interpret roadside information in the form of signage. Even though TSRS has been on the market for more than a decade now, the available ones differ in hardware and software solutions they use, as well as in quantity and typology of signs they recognize. The aim of this study is to determine whether differences between detection and readability accuracy of market-ready TSRS exist and to what extent, as well as how different levels of “graphical changes” on the signs affect their accuracy. For this purpose, signs (“speed limit” and “prohibition of overtaking”) were placed on a test ﬁeld and 17 vehicles from 14 different car brands underwent testing. Overall, the results showed that sign detection and readability by TSRS differ between car brands and that even small changes in the design of signs can drastically affect TSRS accuracy. Even in a controlled environment where no sign has been altered, there has been a 5% margin of misread signs.

Citation: Babić,D.; Babić,D.; Fiolić, Keywords: traffic signs; ADAS; traffic sign recognition system; automated driving M.; Šarić,Ž. Analysis of Market-Ready Traffic Sign Recognition Systems in Cars: A Test Field Study. Energies 2021, 14, 3697. 1. Introduction https://doi.org/10.3390/en14123697 Road traffic accidents are a significant social problem and it is estimated that, de- pending on the country, their costs amount to 1% up to 3% of gross domestic product [1]. Academic Editors: Arno Eichberger, Although road safety is improving in most European countries, the progress remains slow Zsolt Szalay, Martin Fellendorf and and misaligned with the established targets. This slow progress is partially due to the Henry Liu dynamic and complex nature of road traffic, and safety performance depends on a number of interconnected factors related to roadway environment, vehicles and humans. Received: 12 May 2021 In the past decade, with new technological breakthroughs, a significant effort has been Accepted: 17 June 2021 Published: 21 June 2021 devoted to improving vehicle safety systems. These safety systems can be divided into two categories: passive safety systems and active safety systems. A passive safety system

Publisher’s Note: MDPI stays neutral reduces injuries sustained by passengers when an accident occurs, while active ones try to with regard to jurisdictional claims in keep a vehicle under control and avoid accidents [2,3]. published maps and institutional affil- In general, Advanced Driver Assistance System (ADAS) is a collection of numerous iations. intelligent units integrated into the vehicle itself that perform different tasks and assist the human driver in driving. Common ADAS functions include adaptive speed control, lane departure warning, forward collision warning, automatic high beam assist, traffic sign recognition, pedestrian and object detection, automatic emergency braking, etc. All these functions base their operations on different cameras, RADARs, LIDARs and other Copyright: © 2021 by the authors. Licensee MDPI, Basel, Switzerland. sensors which “scan” the environment around the vehicle in order to gather the needed This article is an open access article information. Since the efficiency of such systems majorly depends on the data collected distributed under the terms and from the surrounding environment, it is clear that different road infrastructure elements, conditions of the Creative Commons such as traffic signs or road markings, provide necessary cues not only to human drivers Attribution (CC BY) license (https:// but also to built-in vehicle technologies. creativecommons.org/licenses/by/ Traffic Sign Recognition System (TSRS) is designed to detect and interpret roadside 4.0/). information in the form of signage. Its basic infrastructure can be generalized into three

Energies 2021, 14, 3697. https://doi.org/10.3390/en14123697 https://www.mdpi.com/journal/energies Energies 2021, 14, 3697 2 of 11

specific components: visual sensor, image processor and vehicle display [4]. Acquiring information from traffic signs involves traffic sign detection (TSD), which consists of finding the location, orientation and size of traffic signs in natural scene images, and traffic sign recognition (TSR), or classifying the detected traffic signs into types and categories in order to extract the information they are providing to drivers [5]. In order to detect and recognize traffic signs, as mentioned before, vehicles are equipped with different technologies. Cameras are the most common sensors and can be used for TSR, TSD or both at the same time. LIDAR sensors have been used for TSD. Their 3D perceptive capabilities are useful to determine the position of a sign and its shape, and can also use the intensity of reflected light to improve detection accuracy based on the high reflectivity of traffic signs [5]. Potentially, the best solution presents the combination of LIDAR and cameras. Such fusion enables the collection of information from different sources, their comparison and analysis, and thus better detection and recognition of signs. Besides detecting and recognizing road signs, TSRS also use digital maps with already implemented signs (mainly related to the speed limit signs). After sensors and front-facing cameras collect data, algorithms are used to segment and analyze the stimuli. This process includes shape, color and symbol detection as well as classification of signs based on the aforementioned characteristics [4]. A vast body of literature has analyzed the efficiency of different algorithms for segmentation and classification of signs [6–9]. Several review papers on this subject have also been published in the past few years [10–13]. Besides the overall review of the working procedures, studies identified main issues and challenges regarding the accuracy of TSRS. They are generally related to fading and blurring of traffic signs, visibility levels of signs in comparison to the environment, differences between existing traffic sign systems, multiple appearances of signs, damaged or partially obscured signs, correct location of signs, unavailability of public databases, electronic signs, etc. An on-road study conducted in Australia and New Zealand confirmed some of the aforementioned issues [4]. For the purpose of this study, authors used five cars with TSRS and drove a number of trials designed and conducted in order to identify key issues existing in the current TSRS and to investigate potential causes of the found issues. The study highlighted several applicable changes that could improve traffic sign readability, including electronic signs, installation and maintenance, sign positioning and location, sign face design, vehicle-mounted signs and other advisory and information signs. From the literature review, one can conclude that general problems regarding TSRS are well known. However, different car manufacturers use different hardware and software solutions that may differ in traffic sign detection and readability accuracy. In addition, there are some differences between the signs being detected and presented to the driver. Namely, some cars can only read “speed limit” signs, while others in addition to “speed limit” signs can read other signs such as “prohibition of overtaking”, “end of all restrictions”, “start and end of highway”, etc. For this reason, the main aim of this study is to determine whether differences between detection and readability accuracy of market-ready TSRS exist, and to what extent. Moreover, since the problem of partially obscured signs was identified in literature as one of the main challenges of TSRS, the objective of the study was to test how simulated “graphical changes” affect their detection accuracy.

2. Materials and Methods In this section, the research methodology is presented. The section consists of four subsections each describing a part of the methodology, from the testing track, vehicles, scenarios and test procedures to data analysis.

2.1. Testing Track The experiment was conducted on a road inside the campus of the University of Zagreb. The road is a typical two-way road with almost no trafﬁc since it is only used Energies 2021, 14, x FOR PEER REVIEW 3 of 11

Energies 2021, 14, 3697 2.1. Testing Track 3 of 11 The experiment was conducted on a road inside the campus of the University of Za- greb. The road is a typical two-way road with almost no traffic since it is only used for forconnecting connecting buildings buildings on the on thecampus. campus. The total The totallength length of the of road the is road 1.2 km is 1.2 and km it consists and it consistsof straight of straightsections, sections, four curves four and curves five inte andrsections five intersections with the right with of the way right in the of direction way in thein which direction tests in were which conducted. tests were Nine conducted. traffic signs Nine were traffic placed signs on were straight placed sections on straight of the sectionstesting track, of the as testing shown track, in Figure as shown 1. Seven in Figure of them1. Sevenwere used, of them i.e., were their used, graphical i.e., theirimage graphicalwas changed image according was changed to scenario according designs, to wh scenarioile the remaining designs, while two (8 the and remaining 9) were placed two (8for and control 9) were purposes placed foronly control and their purposes “reading onlys” and were their not “readings” recorded and were analyzed. not recorded Only andspeed analyzed. limit and Only prohibition speed limit of andovertaking prohibition signs of were overtaking used since signs the were majority used sinceof current the majorityTSRS read of currentonly these TSRS signs. read In only total, these four signs. “speed Intotal, limit” four signs “speed (50 km/h, limit” 60 signs km/h, (50 90 km/h, km/h 60and km/h, 100 km/h) 90 km/h and and three 100 “prohibition km/h) and three of ov “prohibitionertaking” signs of overtaking” were used. signsThe speed were used.limits The(50 speedkm/h, limits60 km/h, (50 90 km/h, km/h) 60 were km/h, chosen 90 km/h) based on were the chosen fact that based they onare thethe factmost that common they areand the that most their common meaning and could that theireasily meaning be altered. could For easily example, be altered. with a Forvery example, small modifica- with a verytion, small 60 km/h modification, could easily 60 km/h be altered could to easily 80 km/h. be altered The to100 80 km/h km/h. speed The 100limit km/h was speedchosen limitsince was it contains chosen sincethree it digits. contains All threesigns digits. were newly All signs made were and newly fulfilled made all and technical fulfilled prop- all technicalerties (visibility, properties chromaticity, (visibility, chromaticity,etc.) defined by etc.) the defined Croatian by standard the Croatian [14]. standard The design [14 ].of Thethe designsigns was of the also signs according was also to accordingthe Croatian to the standard, Croatian which standard, is based which on isthe based Vienna on thecon- Viennavention. convention. All of them All are of them 60 cm are in 60 diameter cm in diameter and were and made were madeusing using class classII sheeting II sheeting with withprismatic prismatic retroreflection. retroreflection. Furthermore, Furthermore, all all signs signs were were placed placed according according toto thethe CroatianCroatian standardstandard [ 14[14],], which which implies implies a a 30–75 30–75 cm cm distance distance from from the the edge edge of of the the road road and and a a height height betweenbetween 1.2 1.2 to to 1.5 1.5 m. m. The The distance distance between between each each sign sign was was set set to to a a minimum minimum of of 100 100 m m and and thethe locations locations of of the the signs signs did did not havenot have any environmentalany environmental “disturbances” “disturbances” which which could affectcould theaffect TSRS. the AlthoughTSRS. Although vegetation vegetation next to thenext road to the is present,road is present, all signs all were signs clearly were visible, clearly i.e.,visible, they i.e., were they not were covered not withcovered vegetation. with vegeta Thetion. layout The of layout the signs of the at eachsigns location at each location was as follows:was as Locationfollows: Location 1—50 km/h 1—50 speed km/h limit; speed Location limit; 2—prohibitionLocation 2—prohibition of overtaking; of overtaking; Location 3—60Location km/h 3—60 speed km/h limit; speed Location limit; Location 4—prohibition 4—prohibition of overtaking; of overta Locationking; Location 5—90 km/h 5—90 speedkm/h limit;speed Location limit; Location 6—prohibition 6—prohibition of overtaking; of overtaking; Location Location 7—100 km/h7—100 speedkm/h limit;speed Locationlimit; Location 8—prohibition 8—prohibition of overtaking of overtaking (control (control sign);Location sign); Locati 9—60on km/h9—60 speedkm/h limitspeed (control sign). limit (control sign).

Figure 1. Layout of the test route and placement of the signs. Figure 1. Layout of the test route and placement of the signs.

2.2.Vehicles Seventeen (17) cars in total were included in the study. Fourteen of them were from different manufacturers while the remaining three were from the same manufacturer. Most of those cars are new vehicles (12 of them were produced in 2020), while the other ﬁve ranged from 2014 to 2019. Each vehicle has TSRS which is based on a camera located at the

Energies 2021, 14, x FOR PEER REVIEW 4 of 11

2.2. Vehicles Seventeen (17) cars in total were included in the study. Fourteen of them were from Energies 2021, 14, 3697 4 of 11 different manufacturers while the remaining three were from the same manufacturer. Most of those cars are new vehicles (12 of them were produced in 2020), while the other five ranged from 2014 to 2019. Each vehicle has TSRS which is based on a camera located at thefront front windshield windshield inside inside the vehicle.the vehicle. The vehiclesThe vehicles were, were, for the for sole the purpose sole purpose of the of research, the research,lent to lent us by to their us by ofﬁcial their official distributors distri forbutors their for respective their respecti manufacturingve manufacturing brand in brand Croatia. in Croatia.It is worth It is noting worth that noting the that technical the techni aspectscal aspects of the system of the system were not were provided not provided to us. Sinceto us.the Since aim the of theaim study of the is study to detect is to the detect differences the differences between existingbetween TSRS existing and TSRS not to and determine not to determinewhich car brandwhich hascar thebrand “best” has TSRS,the “best” brand TSRS, names brand are innames code. are in code.

2.3.2.3. Scenarios Scenarios and and Test Test Procedure Procedure In orderIn order to totest test the the accuracy accuracy of ofTSRS, TSRS, nine nine scenarios scenarios were were created. created. The The first first one one was was a a controlcontrol scenario scenario in in which which signs did notnot havehave anyany graphical graphical changes. changes. In In the the second second scenario, scenario,the redthe outlinered outline of the of signsthe signs was coveredwas covere withd blackwith black paper. paper. In the thirdIn the and third fourth and scenarios,fourth scenarios,minor changesminor changes on symbols on symbols were made. were made. For example, For example, a piece a piece of black of black paper paper was placedwas placedso that so that 60 km/h 60 km/h speed speed limit limit looks looks similar simila tor 80to 80 km/h km/h or or black black lines lines were were placed placed on on the theprohibition prohibition ofof overtakingovertaking signs.signs. In In the the fift fifth,h, sixth sixth and and seventh seventh scenarios, scenarios, a half a half of ofthe the signs’signs’ face face was was covered covered with with black black paper. paper. The The difference difference between between the the scenarios scenarios was was in the in the orientationorientation of the of the coverage. coverage. In the In the eighth eighth scenario, scenario, the the last last digit digit (0) (0)on onthe the speed speed limit limit sign sign was covered with white paper. On the two prohibition of overtaking signs, the symbols of was covered with white paper. On the two prohibition of overtaking signs, the symbols cars were covered with white paper while on the third prohibition of overtaking sign, the of cars were covered with white paper while on the third prohibition of overtaking sign, symbols were mirrored. In the last scenario, symbols were covered with graffiti. The visual the symbols were mirrored. In the last scenario, symbols were covered with graffiti. The presentation of signs in each scenario is shown in Figure2. visual presentation of signs in each scenario is shown in Figure 2.

Scenario 1

Scenario 2

Scenario 3

Scenario 4

Scenario 5

Scenario 6

Scenario 7

Figure 2. Cont.

Energies 2021, 14, 3697 5 of 11 Energies 2021, 14, x FOR PEER REVIEW 5 of 11

Scenario 8

Scenario 9

FigureFigure 2. Visual 2. Visual presentation presentation of signs of signs used used in each in each scenario. scenario.

TestsTests were were conducted conducted during during daytime daytime between between 9:00 9:00 a.m. a.m. and and 3:00 3:00 p.m. p.m. in the in theperiod period of threeof three days. days. During During all three all three days, days, weather weather conditions conditions were were almost almost the thesame, same, i.e., i.e., normal normal (sunny)(sunny) weather. weather. Before Before the thestart, start, we weheld held a “practice” a “practice” run run with with each each vehicle vehicle in order in order to to checkcheck if the if theTSRS TSRS were were functioning functioning properly. properly. During During the thetest, test, each each vehicle vehicle passed passed through through the thetest test track track once once per per each each scenario. scenario. Alongside Alongside the the driver, driver, the the researcher researcher was presentpresent inin the thevehicle vehicle duringduring tests,tests, controllingcontrolling and recording if the signs were or were were not not displayed displayed to to thethe driver driver and, and, if if yes, yes, what what was was displaye displayed.d. Only Only one one vehicle vehicle was was permitted permitted on on the the test- testing ingtrack trackduring during each each run run and and other other traffic traffic was was not not present. present. Because Because of of the the road road configuration configuration(campus (campus site), site), the the driving driving speed speed was was between between 40 and 40 and 60 km/h. 60 km/h. 2.4. Data Analysis 2.4. Data Analysis For the first part of the analysis, we grouped the data into three categories: (1) correctly For the first part of the analysis, we grouped the data into three categories: (1) cor- recognized signs, (2) unrecognized signs and (3) wrongly recognized signs. For each rectly recognized signs, (2) unrecognized signs and (3) wrongly recognized signs. For each category, a percentage distribution of the results was calculated and compared. For the category, a percentage distribution of the results was calculated and compared. For the second part, the results from groups (2) and (3) were joined into one group, unrecognized second part, the results from groups (2) and (3) were joined into one group, unrecognized or wrongly recognized signs, since they all in all presented the error of TSRS. On top of or wrongly recognized signs, since they all in all presented the error of TSRS. On top of that, scenarios were grouped based on the level of modification they were submitted to: that, scenarios were grouped based on the level of modification they were submitted to: control (scenario 1), minor changes (scenarios 2 + 3 + 4), medium changes (scenarios 8 + 9) control (scenario 1), minor changes (scenarios 2 + 3 + 4), medium changes (scenarios 8 + 9) and major changes (scenarios 5 + 6 + 7) in order to analyze the differences between the and major changes (scenarios 5 + 6 + 7) in order to analyze the differences between the control scenario (1) and each level of graphical change. Mean values for each vehicle and control scenario (1) and each level of graphical change. Mean values for each vehicle and group were computed and repeated measures ANOVA with Bonferroni adjustment (alpha grouplevel were 0.05) computed were used and to testrepeated the differences measures between ANOVA the with control Bonferroni condition adjustment and each (al- group. pha levelIn 0.05) the secondwere used part to of test the the analysis, differences we tested between the the differences control condition between vehiclesand each per group.scenario. Since the values in the data set for this analysis were dichotomous, Cochran’s Q testIn wasthe second used (alpha part of level the analysis, 0.05). As we one test ofed the the assumptions differences between of Cochran’s vehicles Q test per is sce- that a nario.sample Since has the two values mutually in the exclusivedata set for categories, this analysis the analysiswere dichotomous, was conducted Cochran’s on the Q correctly test wasrecognized used (alpha signs level categories 0.05). As one and of not the on assumptions unrecognized of orCochran’s wrongly Q recognized test is that signs.a sample has two mutually exclusive categories, the analysis was conducted on the correctly recog- nized3. Resultssigns categories and not on unrecognized or wrongly recognized signs. In total, 1071 recognitions were possible (7 signs per scenario × 9 scenarios × 17 ve- 3. Resultshicles). In 21.20% of the cases, signs were recognized correctly. In 72.46%, signs were notIn recognizedtotal, 1071 recognitions at all, while were in 6.35%, possible signs (7 signs were per wrongly scenario recognized × 9 scenarios by vehicles.× 17 vehi- Of cles).course, In 21.20% these of percentages the cases, varysigns between were re eachcognized scenario. correctly. As expected, In 72.46%, in the signs control were scenario not recognized(scenario at 1), all, the while highest in 6.35%, level of signs correctly were recognized wrongly recognized signs was recordedby vehicles. (73.11%). Of course, On the theseother percentages hand, in thevary same between scenario, each 21.58%scenario. of As the expected, signs were in not therecognized control scenario at all (mainly(sce- narioprohibition 1), the highest of overtaking, level of butcorrectly also in reco somegnized cases signs speed was limit recorded signs). This (73.11%). is due On to the the fact otherthat hand, the type in the of signssame whichscenario, TSRS 21.58% recognize of the and signs display were to not the recognized driver varies at toall some (mainly extent prohibitionbetween of brands. overtaking, Moreover, but also some in some of the case signss speed (5.04%) limit were signs). wrongly This is recognized due to the even fact in thatthe the control type of scenariosigns which (for TSRS example, recognize 90 km/h and display was recognized to the driver as 30 varies km/h). to some In the extent second betweenscenario brands. (black Moreover, outline of thesome signs), of the the signs number (5.04%) of unrecognized were wrongly signs recognized increased even to 36.13%, in thewhile control the scenario percentage (for ofexample, wrongly 90 recognized km/h was signsrecognized stayed as the 30 same km/h). as In in the scenario second 1 (5.04%).scenario (blackA major outline decrease of the insigns), sign the recognition number of occurred unrecognized in scenarios signs 3increased and 4 in to which 36.13%, minor whilechanges the percentage were made of wrongly to the signs. recognized Namely, signs the stayed percentage the same of correctlyas in scenario recognized 1 (5.04%). signs in scenarios 3 and 4 fell to 20.17% and 4.20%, respectively. In addition to the increase in

Energies 2021, 14, x FOR PEER REVIEW 6 of 11

Energies 2021, 14, 3697 6 of 11 A major decrease in sign recognition occurred in scenarios 3 and 4 in which minor changes were made to the signs. Namely, the percentage of correctly recognized signs in scenarios 3 and 4 fell to 20.17% and 4.20%, respectively. In addition to the increase in un- unrecognizedrecognized signs, signs, there there was was also also an anincrease increase in wrongly in wrongly recognized recognized signs—26.89% signs—26.89% in sce- in scenarionario 3 and 3 and 14.29% 14.29% in scenario in scenario 4. 4. WhenWhen signssigns werewere halfhalf coveredcovered byby blackblack paperpaper (scenarios(scenarios 4,4, 55 andand 6)6) asas wellwell asas byby grafﬁtigraffiti (scenario(scenario 9),9), thethe percentagepercentage ofof unrecognizedunrecognized signssigns waswas betweenbetween 98%98% andand 100%.100%. InIn scenarioscenario 8,8, 33.61%33.61% ofof thethe signssigns werewere correctlycorrectly recognizedrecognized (63.03%(63.03% ofof signssigns unrecognized),unrecognized), whilewhile 3.36%3.36% werewere wronglywrongly recognizedrecognized (mainly(mainly speedspeed limitlimit signs).signs). AA graphicalgraphical presentationpresentation of of the the aforementioned aforementioned results results is shownis shown in Figurein Figure3 while 3 while the overallthe overall results results for eachfor each vehicle vehicle per eachper each scenario scenario are presentedare presented in Appendix in AppendixA. A.

100% 90% 80% 70% 60% 50% 40% 30%

20% 10% 0% Scenario 1 Scenario 2 Scenario 3 Scenario 4 Scenario 5 Scenario 6 Scenario 7 Scenario 8 Scenario 9

% of correctly recognized signs % of unrecognized signs % of wrongly recognized signs

Figure 3. Percentages of “correctly recognized”, “unrecognized” and “wrongly recognized” signs per scenario. Figure 3. Percentages of “correctly recognized”, “unrecognized” and “wrongly recognized” signs per scenario. As mentioned in the Data Analysis section, the scenarios were grouped into four As mentioned in the Data Analysis section, the scenarios were grouped into four cat- categories: control (scenario 1), minor changes (scenarios 2, 3 and 4), medium changes egories: control (scenario 1), minor changes (scenarios 2, 3 and 4), medium changes (sce- (scenarios 8 and 9) and major changes (scenarios 5, 6 and 7). This was done in order tonarios analyze 8 and how 9) and different major levelschanges of changes(scenarios on 5, signs 6 and affect 7). This recognition was done accuracy.in order to For analyze each vehiclehow different and group, levels mean of changes value was on computed.signs affect Overall, recognition the decrease accuracy. in For correctly each vehicle recognized and Energies 2021, 14, x FOR PEER REVIEW group, mean value was computed. Overall, the decrease in correctly recognized signs7 of 11 be- signs between controlled conditions and minor, medium and major changes is 62%, 77% andtween 99%, controlled respectively conditions (Figure and4). minor, medium and major changes is 62%, 77% and 99%, respectively (Figure 4).

100%

90%

80% 73.12% 70%

60%

50%

40% 27.82% 30% 16.88% 20% Percentages Percentages signs of recognized 10% 0.29% 0% Control Minor changes Medium changes Major changes

FigureFigure 4. Percentages 4. Percentages of ofrecognized recognized signs signs per per each each category. category.

Although differences between each category are evident, a repeated measure ANOVA with Bonferroni adjustment was used to test the differences between the control condition and each group. The results of ANOVA analysis shown in Table 1 confirms that a statistical difference (p < 0.05) between the control condition and each group representing different levels of graphical changes of signs exists.

Table 1. Results of ANOVA.

95% Confidence Interval Mean Dif. Std. Er- (I) (J) p for Difference (I–J) ror Lower Bound Upper Bound Minor changes 0.454 0.051 0.000 0.301 0.607 Control Medium changes 0.563 0.065 0.000 0.368 0.758 Major changes 0.728 0.059 0.000 0.550 0.906

Furthermore, we tested the difference between traffic sign recognition of each vehicle in each scenario with Cochran’s Q test. Since scenarios 6, 7 and 9 after grouping did not have variations in values (no values for group “correctly recognized signs”), Cochran’s Q test was not performed on them. Overall, a statistical difference (p < 0.05) in traffic sign recognition between tested vehicles was recorded for scenarios 1, 2, 3 and 8, while for scenarios 4 and 5, no statistical difference was found (p > 0.05), as shown in Table 2.

Table 2. Results of Cochran’s Q test—significant differences in traffic sign recognition between tested vehicles per analyzed scenarios.

Scenario Asymptotic Sig. 1 0.002 2 0.000 3 0.041 4 0.699 5 0.453 8 0.000

Energies 2021, 14, 3697 7 of 11

Although differences between each category are evident, a repeated measure ANOVA with Bonferroni adjustment was used to test the differences between the control condition and each group. The results of ANOVA analysis shown in Table1 conﬁrms that a statistical difference (p < 0.05) between the control condition and each group representing different levels of graphical changes of signs exists.

Table 1. Results of ANOVA.

95% Conﬁdence Interval for Difference (I) (J) Mean Dif. (I–J) Std. Error p Lower Bound Upper Bound Minor changes 0.454 0.051 0.000 0.301 0.607 Control Medium 0.563 0.065 0.000 0.368 0.758 changes Major changes 0.728 0.059 0.000 0.550 0.906

Furthermore, we tested the difference between trafﬁc sign recognition of each vehicle in each scenario with Cochran’s Q test. Since scenarios 6, 7 and 9 after grouping did not have variations in values (no values for group “correctly recognized signs”), Cochran’s Q test was not performed on them. Overall, a statistical difference (p < 0.05) in trafﬁc sign recognition between tested vehicles was recorded for scenarios 1, 2, 3 and 8, while for scenarios 4 and 5, no statistical difference was found (p > 0.05), as shown in Table2.

Table 2. Results of Cochran’s Q test—signiﬁcant differences in trafﬁc sign recognition between tested vehicles per analyzed scenarios.

Scenario Asymptotic Sig. 1 0.002 2 0.000 3 0.041 4 0.699 5 0.453 8 0.000

4. Discussion Due to their great potential in reducing road accidents, ADAS technologies have become one of the fastest-growing safety application areas. In the European Union, ADAS systems are mandatory for all new and certified vehicles starting from 2022 and all new registered vehicles by 2024 [15]. One of those systems is traffic sign recognition, which provides drivers with information about traffic signs. However, the analysis of market-ready TSRS shows that the functionalities of systems differ to some extent between car brands, mainly in the type and the number of signs that can be recognized and displayed to the driver. Some of the cars display only speed limit signs, some besides speed limit display prohibition of overtaking, while some are, in addition to the above, displaying main warning signs such as “dangerous curve”, “pedestrian crossing”, etc. Furthermore, while the majority of TSRS signs are displayed in color, there are a few brands that display signs in black and white. The results of this study show that TSRS accuracy differs between car brands and graphical clarity of the sign. In other words, each scenario with graphical changes on signs had significantly lower recognition level compared to the control condition (ranging from 62% to 99%). It is important to note that even the control condition did not have a 100% recognition and, contrary to expectations, had 5% of wrongly recognized signs. This is precisely due to the differences in TSRS between car brands. Moreover, between each category of graphical change and the control condition, there has been a statistically significant difference in the number of correctly recognized signs. The results suggest Energies 2021, 14, 3697 8 of 11

that even small changes in the design of a sign, such as the change of the outline color or a minor change in the symbol, can drastically affect TSRS accuracy. This result is in accordance with previous studies [4,12]. Although graphical changes on the signs were created artificially in this study, similar situations exist in real road conditions [16,17] and may result in TSRS errors. Such errors may affect the driver in different ways. Studies have shown that drivers in general perceive a relatively low amount of traffic signs [18–20], and thus an additional display of signs in the vehicle (especially speed limits) should increase overall compliance with traffic regulations. On the other hand, if the sign is not recognized by the TSRS, the driver will not have additional information, which may result in inappropriate and/or improper driving behavior. Furthermore, if TSRS recognizes the signs but wrongly (for example 60 km/h as 80 km/h), such error may confuse drivers and thus distract them, which again may lead to inappropriate and/or improper driving behavior. This result confirms the importance of proper maintenance of signs and their surroundings as highlighted in several recommendations and publications [4,21,22]. Since the general functioning and accuracy of TSRS significantly differ between car brands, a “standardization” of such systems is needed. Standardization, in this sense, implies defining the minimal number and types of signs which every TSRS should recognize, the way signs are displayed to the drivers and minimal levels of recognition accuracy at least for properly placed and maintained signs during daytime and low visibility conditions (night-time, rain, fog, etc.). Although the last requirement is difficult to define since recognition accuracy depends on the number of factors [10–13], the standardization of TSRS could potentially accelerate their development and possibly eliminate some of the current problems. In addition, recent studies emphasize that, although ADAS systems provide a significant progress in safety, they also may distract drivers [23,24]. Thus, the education of drivers regarding the functionality and limitations of TSRS is needed as well in order to avoid or at least decrease potential confusion of drivers. The education altogether with the standardization could decrease the driver’s distraction caused by TSRS. Finally, in order to increase the accuracy of TSRS, a database of traffic signs for each country should be established. The database in this sense should have all the variations of signs and their meaning for each country. In the primal stage, it could be developed at least for speed limits so the algorithms used for sign classification and recognition can be “taught” and thus the accuracy of the whole system may be increased. Although this study provides valuable results, there are some limitations. The study was conducted in controlled conditions (practically a closed road section, dry daytime weather, lack of other traffic, all new traffic signs, relatively low driving speed, etc.) that are not generally present in the real world. However, such controlled conditions represent the best-case scenario, meaning that TSRS accuracy would decrease even more in real world conditions. Moreover, the study included only 17 cars from 14 brands. Even though the used sample is relatively small, most of the major car brands were included and all cars had the best equipment provided by the manufacturer. Finally, due to unavailability of the technical data, proper comparison of the TSRS was not possible.

5. Conclusions The main aim of this field study was to test whether differences between detection and readability accuracy of market-ready TSRS exist and to what extent, as well as how different “graphical changes” affect their accuracy. A total of 17 cars from 14 brands were tested. Fourteen cars were from different manufacturers while three were from the same manufacturer. The results confirm that TSRS accuracy differs between car brands and even between the same brand, although, due to a limited sample, this needs to be further tested. Furthermore, graphical changes significantly affected the accuracy of the TSRS in all vehicles compared to the control condition (ranging from 62% to 99%). Based on the results and limitations of this study, future research should investigate TSRS accuracy on real roads with other traffic present and in different conditions as well Energies 2021, 14, 3697 9 of 11

as different types and quality of trafﬁc signs. In order to develop a deeper insight, the technical data about the installed TSRS should be analyzed in order to evaluate each system and its limitations. Furthermore, a detailed elaboration of standardization methodology and principles should also be developed.

Author Contributions: Conceptualization, D.B. (Darko Babić)and D.B. (Dario Babić);Methodology, D.B. (Darko Babić),D.B. (Dario Babić), M.F. and Ž.Š.; Formal analysis, M.F., D.B. (Dario Babić)and Ž.Š.; Investigation, D.B. (Dario Babić)and M.F.; Resources, D.B. (Dario Babić),M.F. and Ž.Š.; Writing— D.B. (Darko Babić)and D.B. (Dario Babić);Supervision, D.B. (Darko Babić);Project administration, D.B. (Darko Babić);Funding acquisition, D.B. (Darko Babić).All authors have read and agreed to the published version of the manuscript. Funding: This research was a part of the project entitled “Establishing a Methodology for Testing and Evaluating the ADAS Systems” funded by the University of Zagreb (Potpore za temeljno financiranje znanstvene i umjetniˇckedjelatnosti Sveuˇcilištau Zagrebu u ak. god. 2019./2020.). Data Availability Statement: The data used to support the findings of this study are available from the corresponding author upon request. Conflicts of Interest: The authors declare no conflict of interest.

Appendix A Results for Each Vehicle per Each Scenario

Vehicles S TS V1 V2 V3 V4 V5 V6 V7 V8 V9 V10 V11 V12 V13 V14 V15 V16 V17 111111111111111111 211010000111001111 311100111111111111 1 411010000111001111 5 1 0 −1 1 1 1 1 1 1 1 1 1 −1 −1 −1 1 1 611010000111001011 7111111111111 −1 0 −1 1 1 111000111101111111 211000000101001111 311000111101111111 2 411000000101001111 5110001111011 −1 −1 −1 1 1 611000000101001111 7110001111011 −1 −1 −1 1 1 111111111011011011 210000000000001000 3 −1 −1 −1 0 0 −1 −1 −1 0 0 0 −1 0 0 0 −1 −1 3 410000000001001010 5 −1 0 −1 −1 −1 −1 −1 −1 0 −1 −1 −1 −1 −1 0 1 −1 600000000000000000 7 −1 0 1 −1 −1 −1 −1 −1 0 −1 −1 1 0 −1 0 1 −1 1 −1 0 −1 −1 1 1 −1 −1 −1 −1 −1 −1 0 −1 −1 −1 −1 21000000000 −1 0 0 −1 1 0 0 300000000000000000 4 400000000000000000 50000000000000001 −1 600000000000000000 70000000000 −1 0 0 0 0 0 0 Energies 2021, 14, 3697 10 of 11

100000000000000010 200000000000000000 300000000000000000 5 400000000000000000 500000000000000000 600000000000000000 700000000000000000 S—Scenario; TS—Trafﬁc signs; V—Vehicle

References 1. World Health Organization. Road Traffic Injuries; World Health Organization: Geneva, Switzerland, 2017. 2. Lu, M.; Wevers, K.; van der Heijden, R. Technical feasibility of advanced driver assistance systems (ADAS) for road traffic safety. Transp. Plan. Technol. 2005, 28, 167–187. [CrossRef] 3. Dörterler, M.; Bay Ömer, F. Neural network based vehicular location prediction model for cooperative active safety systems. Promet Traffic Transp. 2018, 30, 205–215. [CrossRef] 4. Austroads. Implications of Traffic Sign Recognition (TSR) Systems for Road Operators; Technical report AP-R580-18; Austroads: Sydney, Australia, 2018. 5. Martí, E.; de Miguel, M.Á.; García, F.; Pérez, J. A review of sensor technologies for perception in automated driving. IEEE Intell. Transp. Syst. Mag. 2019, 11, 94–108. [CrossRef] 6. Cao, J.; Song, C.; Peng, S.; Xiao, F.; Song, S. Improved traffic sign detection and recognition algorithm for intelligent vehicles. Sensors 2019, 19, 4021. [CrossRef][PubMed] 7. Dewi, C.; Chen, R.C.; Yu, H. Weight analysis for various prohibitory sign detection and recognition using deep learning. Multimed. Tools Appl. 2020, 79, 32897–32915. [CrossRef] 8. Gil Jiménez, P.; Bascón, S.M.; Moreno, H.G.; Arroyo, S.L.; Ferreras, F.L. Traffic sign shape classification and localization based on the normalized FFT of the signature of blobs and 2D homographies. Signal. Process. 2008, 88, 2943–2955. [CrossRef] 9. Tian, Y.; Gelernter, J.; Wang, X.; Li, J.; Yu, Y. Traffic sign detection using a multi-scale recurrent attention network. IEEE Trans. Intell. Transp. Syst. 2019, 20, 4466–4475. [CrossRef] 10. Gudigar, A.; Chokkadi, S.; Raghavendra, U. A review on automatic detection and recognition of traffic sign. Multimed. Tools Appl. 2016, 75, 333–364. [CrossRef] 11. Liu, C.; Li, S.; Chang, F.; Wang, Y. Machine vision based traffic sign detection methods: Review, analyses and perspectives. IEEE Access 2019, 7, 86578–86596. [CrossRef] 12. Wali, S.B.; Abdullah, M.A.; Hannan, M.A.; Hussain, A.; Samad, S.A.; Ker, P.J.; Bin Mansor, M. Vision-based traffic sign detection and recognition systems: Current trends and challenges. Sensors 2019, 19, 2093. [CrossRef][PubMed] 13. Zourlidou, S.; Sester, M. Traffic regulator detection and identification from crowdsourced data—A systematic literature review. ISPRS Int. J. Geo-Inf. 2019, 8, 491. [CrossRef] 14. Pravilnik o Prometnim Znakovima Signalizaciji i Opremi na Cestama (NN 92/19) (2019). [In Croatian; National Regulations on Traffic Signs, Equipment and Signalization (NN 92/19) (2019)]. Available online: https://narodne-novine.nn.hr/clanci/sluzbeni/ 2019_09_92_1823.html (accessed on 17 June 2021). 15. European Parliament and the Council of the European Union. General Safety Regulation; Official Journal of the European Union: Brussels, Belgium, 2019. 16. Khalilikhah, M.; Heaslip, K. Prediction of traffic sign vandalism that obstructs critical messages to drivers. Transport 2016, 33, 399–407. [CrossRef] 17. Babić,D.; Fiolić,M.; Babić,D.; Šˇcukanec,A. Testing the quality of traffic signs and road markings: Republic of Croatia case study. In Proceedings of the Transport Research Arena 2018, Vienna, Austria, 16–19 April 2018. 18. Babić,D.; Babić,D.; Šˇcukanec,A. The impact of road familiarity on the perception of traffic signs—Eye tracking case study. In Proceedings of the 10th International Conference: “Environmental Engineering”, Vilnius, Lithuania, 27–28 April 2017. 19. Macdonald, W.A.; Hoffmann, E.R. Drivers’ awareness of traffic sign information. Ergonomics 1991, 34, 585–612. [CrossRef] 20. Milosevic, S.; Gajic, R. Presentation factors and driver characteristics affecting road-sign registration. Ergonomics 1986, 29, 807–815. [CrossRef] 21. European Union Road Federation. Improved Signage for Better Roads; European Union Road Federation: Brussels, Belgium, 2015. 22. EuroRap. Roads That Cars Can Read; EuroRap: Basingstoke, UK, 2013. 23. Fundation for Road Safety. Understanding the Impact of Technology: Do Advanced Driver Assistance and Semi-automated Vehicle Systems Lead to Improper Driving Behavior? Technical report; Fundation for Road Safety: Basingstoke, UK, 2019. 24. Cades, D.; Crump, C.; Lester, B.; Douglas, Y. Driver distraction and advanced vehicle assistive systems (ADAS): Investigating effects on driver behavior. In Advances in Human Aspects of Transportation; Springer: Berlin/Heidelberg, Germany, 2017.