I EYE TRACKER ANALYSIS of DRIVER VISUAL FOCUS AREAS at SIMULATED INTERSECTIONS by Jacob Mauk Submitted in Partial Fulfillment Of

Home , Eye tracking

EYE TRACKER ANALYSIS OF DRIVER VISUAL FOCUS AREAS AT SIMULATED INTERSECTIONS

Jacob Mauk

Submitted in Partial Fulfillment of the Requirements

for the Degree of

Master of Computing and Information Systems

YOUNGSTOWN STATE UNIVERSITY

December, 2020

EYE TRACKER ANALYSIS OF DRIVER VISUAL FOCUS AREAS AT SIMULATED INTERSECTIONS

Jake Mauk

I hereby release this thesis to the public. I understand that this thesis will be made available from the OhioLINK ETD Center and the Maag Library Circulation Desk for public access. I also authorize the University or other individuals to make copies of this thesis as needed for scholarly research.

Signature: ______Jacob Mauk, Student Date

Approvals: ______John Sullins, Thesis Advisor Date

______Alina Lazar, Committee Member Date

______Abdu Arslanyilmaz, Committee Member Date

______Dr. Salvatore A. Sanders, Dean of Graduate Studies Date

Abstract

Automobiles are rapidly transitioning from a human controlled machine, to a machine that controls itself. With every passing year, new works are published detailing how self-driving vehicles are becoming closer to a reality that we will encounter on our streets. While the technology to get these vehicles has been pushed more, one of the biggest obstacles they face is what actions to take at intersections, where they are faced with traffic signals, cross traffic, cars making turns, and pedestrians.

In this thesis, an analysis to determine what areas a human driver focuses on aims to provide insight for a more accurate review of where driver attention is focused.

Participants used a driving simulation to drive through several intersections with planned distractions and events at each to record their reactions. The goal of this is to provide more data for a more direct application that involves intersection dangers and driving awareness, such as in self-driving cars. By identifying the areas where a human will focus, a model can use this data for its own observations to make improvements. The results indicated that a driver would tend to focus most of their attention while driving on the road directly in front of them, which may not always be the most efficient way to detect potential problems.

iii

Acknowledgements

First, I would like to thank my family, for helping to support me at the beginning and throughout my time in college so that I could adequately focus on my studies and be where I am now.

I would like to thank my advisor Dr. John Sullins, for helping me with scheduling questions throughout both my undergraduate and graduate program, as well as being the main source of help for advisement on this thesis.

I have to thank all of my friends, from those who directly supported me in times of need, to those who were always by my side for every step along the way, without them

I would not be here today.

Lastly, I would like to thank the girl from “Lo-fi Hip Hop Radio – Beats to

Relax/Study to” for always providing me with a working companion and music to keep me focused during long hours of work.

TABLE OF CONTENTS

LIST OF FIGURES...... vii

LIST OF TABLES ...... ix

CHAPTER 1 INTRODUCTION ...... 1

1.1 Motivation ...... 2

1.2 Research Questions ...... 3

1.3 Contributions ...... 4

1.4 Organization ...... 5

CHAPTER 2 BACKGROUND AND RELATED WORK ...... 6

2.1 Eye Tracking Technology ...... 6

2.2 Autonomous Vehicles ...... 7

CHAPTER 3 SOFTWARE AND HARDWARE DESCRIPTION...... 10

3.1 Eye Tracking Hardware ...... 10

3.2 Simulation Hardware ...... 12

3.3 Simulation Software ...... 13

CHAPTER 4 EXPERIMENT METHODOLOGY ...... 15

4.1 COVID-19 Safety Precautions ...... 15

4.2 Simulation Steps ...... 16

4.3 Trial Recording ...... 16

4.4 Scenarios ...... 17

CHAPTER 5 RESULTS ...... 28

5.1 Interactions ...... 28

5.1.1 Lights ...... 29

5.1.1 Pedestrians ...... 30

5.1.1 Active Vs. Passive ...... 31

5.2 Expectations ...... 32

5.3 Discussion ...... 34

CHAPTER 6 CONCLUSIONS AND FUTURE WORK ...... 36

6.1 Conclusions ...... 36

6.2 Future Work ...... 36

APPENDIX A: STUDY MATERIAL ...... 38

REFERENCES ...... 47

LIST OF FIGURES

Figure 1. An example of a heatmap ...... 10

Figure 2. An example of a fixation map...... 11

Figure 3. The simulator seating and setup...... 12

Figure 4. Scenario 1 ...... 17

Figure 5. Scenario 2 ...... 18

Figure 6. Scenario 3 ...... 19

Figure 7. Scenario 4 ...... 19

Figure 8. Scenario 5 ...... 20

Figure 9. Scenario 6 ...... 20

Figure 10. Scenario 7...... 21

Figure 11. Scenario 8...... 22

Figure 12. Scenario 9...... 22

Figure 13. Scenario 10...... 23

Figure 14. Scenario 11...... 24

Figure 15. Scenario 12...... 24

Figure 16. Scenario 13...... 25

Figure 17. Scenario 14...... 26

Figure 18. Scenario 15...... 26

Figure 19. The “Jaywalker” Scenario ...... 31

vii

Figure 20. A composite of all users heatmaps ...... 32

Figure 21. User 4’s heatmap ...... 33

viii

LIST OF TABLES

Table 1 A table of overall driver performance at each scenario...... 28

CHAPTER 1

INTRODUCTION

Autonomous driving is an emerging field of technology that has made rapid advancements in recent years, and demand for perfect technology continues to drive it forward. One of the most complex situations for an autonomous vehicle to encounter is an intersection. To “see” the world around them, an autonomous car is fitted with a variety of sensors, which can interpret most things a car might interact with. However, at intersections the vehicle sensors will be detecting traffic lights, signs, pedestrians, and of course other vehicles. Processing all this data is very costly and can inhibit a vehicles timely reaction to an emergency event.

Negating the issues that come with this processing can significantly improve reaction time and overall safety of an autonomous vehicle. The most efficient way to do this is by filtering what must be processed. Filtering is a process that removes certain data or assigns it a lower priority to reserve resources for a more important purpose. Humans have the most experience driving vehicles and should have an appropriate grasp of the best driving practices. It is proposed that by analyzing where a human driver will tend to focus their attention when driving, we can adjust an autonomous vehicles’ sensors to prioritize regions that humans focus on while driving. To gather this information, a vehicle drivers eye gaze and focus can be harvested using eye tracking technology.

Utilizing eye tracking technology to collect data on participants has been used to refine thoughts and studies for several years, however implementing them into a real- world scenario is usually not possible. To perform a task such as this would require devices added to a vehicle dashboard, along with other measurement devices, which

could become distracting and dangerous to test. So, to test a question regarding driving behavior, simulations have often been used as a safer alternative instead. In this work a driving simulation has been utilized in conjunction with eye tracking technology to reveal where drivers may potentially draw their attention and focus towards while driving a vehicle. The eye tracking is performed by a specialized device and through a software that records and monitors the eye movement of participants in the trials.

The goal of this research is to provide data that can be used to make a more educated choice regarding focus from drivers while on the road, so that it may be applied in other projects. The most important of these projects would be in the application of self- driving vehicles, or autonomous vehicles. These vehicles have many sensors used on them, which all relay back important feedback to support decision making for vehicles.

By obtaining data on where humans tend to focus, we can specify that these details should be weighted more heavily by the model that will decide what the vehicle should do next. To calculate this data, participants will have their eye movements recorded while using the simulator to drive through an urban environment. While in this simulation, several scenarios have been planned out that will occur to the drivers, and their attention and focus for these events will be recorded to later be analyzed for more details. From that data, it will be possible to make conclusions on what types of events humans regularly notice and do not notice, so that a more accurate model may be applied.

1.1 Motivation

Prior work done by Kawashima (2001) has attempted to implement a self-driving vehicle using human driving as a basis, however this study was conducted long before technology such as eye tracking and gaze monitoring was as common as it is now. In

Kawashima’s (2001) study it is specifically mentioned that “image processing for recognizing environmental situations is the most time-consuming part of processing for autonomous navigation”, and more recently Urmson et. al (2008) also addressed how thousands of sensor results must be processed very quickly. One of the primary ways of improving results when working with a large amount of data is to trim unnecessary data.

This work will be done in intersections, as these are the most common place of accidents and are also often the most detailed locations a car will have to drive in terms of distractions (Stevens et al. 2017).

Trimming of data from sensors could be done by disregarding some of the data, or by focusing on sensor data from specific areas, which is part of the motivation for this paper. Identifying certain regions that have more importance in scenarios can allow for data not pertaining to these regions to be filtered out, improving overall speed and reactivity of the system.

1.2 Research Questions

There are two main questions that this study hopes to discover.

RQ1: Where does a driver tend to focus most of their attention towards while driving?

RQ2: Are a human driver’s habits an ideal model for a self-driving car to follow, regarding the observations they make while driving?

The first question seeks to figure out where most of the information that a driver takes in is derived from. While humans are imperfect drivers, it is natural to absorb more information to keep oneself safe while driving, and where drivers choose to gather this information from would point towards areas that are more relevant.

The second question serves as a basis for potential future projects. If the human based model is an accurate system, can a system be developed that tries to closely mimic human driving habits? If a human-like system is implemented, it could potentially integrate with other human drivers on the road with less hassle. Systems like this have been attempted before with Kawashima’s (2001) work, and if data from the eye tracking supports this, it could simplify potential systems in automated vehicles. Wei Shi et al.

(2013) has attempted a study before that utilizes eye tracking data as a model for future projects as well.

1.3 Contributions

This study hopes to contribute additional data on where the driver of a vehicle will focus their attention at various scenarios at intersections. By collecting and reporting on this data using eye tracking technology, this data can be used to improve future applications in fields such as public safety, automotive engineering, and artificial intelligence.

Public safety analysts can benefit from this by having new data on intersection distractions and focus areas of drivers, which might allow certain traffic patterns to be better built around and planned for. The automotive engineering and artificial intelligence fields can apply this data to the same problem, vision, and analysis of the road for an autonomous vehicle. Proper utilization of data this study finds could give higher data quality and a stronger, more confident analysis of the environment in certain scenarios an autonomous vehicle might find itself in.

1.4 Organization

This thesis is organized as follows. Chapter two will give an overview of eye tracking technology, and autonomous vehicles from prior research. Chapter three will discuss the software and hardware used in this study. Chapter four discusses the methodology of the trials that were conducted to collect data. Chapter five details the study’s results and observations, with chapter six concluding the thesis and explaining how future work can improve upon this study.

CHAPTER 2

BACKGROUND AND RELATED WORK

This chapter contains an overview on the history and use of eye tracking technology, as well as background regarding autonomous vehicles. As this project has a main goal of using driver focus areas to improve autonomous vehicles, their current usages and limits should be known.

2.1 Eye Tracking Technology

Eye tracking technology is a modern and revolutionary way to examine and analyze data that was not previously possible before. Eye tracking involves a sensor that after calibration can follow the precise location a user is looking at a screen. This allows researchers to analyze scenarios by seeing directly what the participant focused on rather than relying solely on the participants thoughts or recollections in a post-test survey.

Cooke (2005) discussed in her study how eye tracking can be used in fields such as advertising and agreed with the sentiment that a user’s vision will closely and accurately correlate to their thinking process.

This technology has had many implementations so far, with many researchers using it to make models that allow for future projects to base core decisions on what data has shown. An example of this is Knoepfle et al.’s (2009) study, which attempted to anticipate how players would learn and act while playing games.

A study was also proposed that involved making an even more realistic driving simulation using virtual reality headsets. This was decided against pursuing, as it would potentially lead to a higher rate of motion sickness in the simulator (Nelson et al. 2010).

Additionally, while a virtual reality headset would give information such as where a

driver chooses to look, it would not give precise measurements of the driver’s eye gaze which is essential data for this type of work that is only obtainable from eye tracking technology.

2.2 Autonomous Vehicles

Autonomous vehicles, or self-driving cars, have been a technology that sounds futuristic, but is within grasp. These cars already exist in some respects, with the Tesla brand car models having significant self-driving capabilities, but the work in this field is not done yet. Self-driving vehicles have many complex components, including lane control, object recognition, and everything else that must be used to ensure a safe driving experience. Many variables come into play when designing even simple path planning for an autonomous vehicle, which makes it a much more challenging task (Frazzoli et al.

2002).

Adding features to self-driving cars has been gradual over the years, with various systems put in place, often as one part of the system as a luxury option in common cars.

Cars with lane assist, which lets the car automatically correct its position if it drifts slightly out of lane, are common now. There have even been plans to include driver monitoring, such as alerting when cell phones are used, or if fatigue is detected and potentially causing unsafe driving (Solomon and Wang 2007). Another small feature that has been shown to work well is traffic light recognition at intersections by Park and Kee

(2019). Competitions that exist solely to test the limits of modern self-driving cars are seen in events like the Urban Grand Challenge, which Urmson et al. (2008) and

Petrovskaya and Thrun (2008) both compete in, utilizing their own self-driving cars fitted with sensors known as “Boss” and “Junior” respectively.

In addition to these features, many are proponents that this technology is part of the future of safety. As Dingus et al. (2016) notes, over 68% of crashes observed in their study found that driver distraction was present as the main culprit in an accident. This is not a problem unique to cars, as all vehicles are impacted by distractions and would benefit from automation. Systems like ANGEL (Austin et al. 2000) are already implemented or in development in certain aircraft, which help detect potential hazards for pilots. Another system for ships has been proposed by Fu (2019) to help prevent nautical disasters as traffic increases from worldwide shipping. Additionally, as these systems do not have any regulating bodies, they are not necessarily constrained to any specific guidelines. Wagner and Koopman (2016) propose that autonomous vehicles should have a standard and propose their own in their work.

Challenges to this technology has critics across many fields and professions. One of these critiques comes from Stilgoe (2018), who says that while humans are error prone and cause 90% of crashes, self-driving cars will just create new unexplored issues, like social dilemmas when accidents happen. These dilemmas present new challenges to the public and legal systems, as discussed by Nyholm and Smids (2016) who compare the implementation of self-driving vehicles to the proverbial “Trolley Problem”. They present a specific problem where many passengers may die in a crash unless the car automatically swerves out of the way, but this would result in the death of an innocent bystander. Problems like these have been theorized to occur in self driving cars, and even found that based on surveys, people would prefer the passengers should accept the crash to prevent bystanders from unnecessary harm, but those same participants would prefer they did not ride in such a car (Bonnefon et al. 2016).

One of the best ways to prevent these concerns and challenges that arise from self-driving cars is better detection systems, which this study will aim to help alleviate by improving detection algorithms by giving additional insight to what types of visual information are most important to effective decision making at intersections.

CHAPTER 3

SOFTWARE AND HARDWARE DESCRIPTION

This chapter explains the software and hardware that was used for conducting the experiments, as well as why they were chosen. Within are why these specific tools were used compared to competitors and the advantages that these gave, as well as shortcomings that they cannot address.

3.1 Eye Tracking Hardware

For this experiment, a Gazepoint GP2 was used in conjunction with Gazepoint’s software system. Utilizing both of their systems ensured that no data would be lost by using non-proprietary software. The eye tracking hardware is calibrated on a per-user basis and can accurately track a user’s eye movements across the screen. This ran concurrently with the eye tracking software and simulation software to record live data.

The Gazepoint software features several mapping outputs for analyzing data, including heatmaps and specific point-by-point tracking called a fixation map.

Figure 1. An example of a heatmap.

Heatmaps allow one to get a measurement of where a user’s eye gaze was recorded over the course of several seconds, which is extremely useful for generalizing about where focus is commonly directed. This is represented in blue for areas that were looked at briefly or sparingly and scaling to shades of red for areas that were heavily focused on for large durations of time. Figure 1 shows an example of a heatmap, where the middle was heavily focused on, while the sides were focused on gradually less.

Figure 2. An example of a fixation map.

Point-by-point data is useful for a slow analysis, as it records a user’s precise eye gaze location that the user is currently staring at several times per second. This is detailed in Figure 2, which labels various points and has a corresponding time of focus on each point. Using this, you can see exactly where a user is looking at while approaching any distractions that they may occur upon while in the simulation. Using these two methods together allows for a detailed analysis of where user’s focus is directed at on both precise and generalized levels.

3.2 Simulation Hardware

The hardware that this simulation was conducted on utilized a customized desktop provided by Youngstown State University and featured a Logitech G920 Driving Kit.

This driving kit was fitted with a steering wheel to closely mimic real driving, a car-like seat, and a foot pedal system for acceleration and deceleration.

Figure 3. The simulator seating and setup.

The desktop computer was designed with having a driving simulation planned to operate on it in mind, so it also featured a monitor cage which held 3 monitors, angled slightly around the seat to give a larger field of view to the participant to make the simulation feel more realistic. Special care was given to ensure a smooth environment so that participants in the study would be comfortable and give accurate data while not suffering from any of the ailments that can be caused by using a simulator. (Nelson et al. 2010)

3.3 Simulation Software

This simulation was developed by me and other students, which was additionally utilized in studies to test driver take over in autonomous vehicles (Arslanyilmaz 2020) in

Unity over the course of the Fall 2019 and Spring 2020 semesters. The software consists of a built Unity package which launches on all windows in full screen mode. The software implements features such as occlusion culling and checkpoint-based level loading to maintain sixty frames per second. Sixty was chosen as according to Claypool and Claypool (2007) users perform better at interactive tasks when a higher framerate is achieved, shown to thirty on their study. Additionally, Bryson (2001) found that this is also supported in tasks where users would be tracking objects. Raising this to sixty for smoother performance was chosen to ensure that data quality is not affected by poor video performance, and even if there was a dip in quality it would still be above the recommended thirty frames per second.

Occlusion culling is the act of not rendering objects that are not currently within a user’s field of view to prevent the processing units from using unnecessary power to create the user experience. By implementing this, it allowed for several thousands of objects to be in a scene, but without hindering performance. The checkpoint-based level loading also entirely removed scenes that were not visible to the user yet or would not be seen again to allow for optimal performance.

The software features a controllable car that a user will use to navigate through a series of fifteen intersections, each of which have been specifically curated to feature various distractions to gauge how a user will react to them. At or slightly before each intersection, triggers are implemented that detect when the user has reached a certain

area, which then activated a set of distractions depending on what intersection they are at.

This allows for a precise timing to ensure that a distraction event will be noticed by the user. A snippet of the code used to manage events at intersections in this project can be found in the Appendix. These distractions include cars, either parked, waiting at a light, or actively driving and possibly turning/switching lanes, and pedestrians which are either stationary or walking along the sidewalk, or potentially along crosswalks.

CHAPTER 4

METHODOLOGY

In this chapter, the methods that were used to conduct the experiment are explained in detail, along with additional precautions that had to be taken due to the ongoing COVID-19 pandemic.

4.1 COVID-19 Safety Precautions

At the time of this thesis’s conception and throughout its lifecycle, there has been an ongoing global pandemic which has brought about several challenges to making projects such as this one possible. Initially it was thought that due to the global COVID-

19 pandemic, performing an actual test of the project would not be possible, but working with the Institutional Review Board, a few conditions were proposed and accepted that allowed the project to continue.

Some of the conditions were very basic, such as requiring that a student is not actively infected with the disease, displayed no symptoms, and was otherwise considered healthy. Along with this, facemasks were always to be worn during the trials, and the examiner would maintain a 6-foot distance from the participant as per the CDC’s guidelines as a further preventative measure (2020).

Another of the conditions was proper sanitization. This involved using the

University provided sanitization equipment, which was used thoroughly on all equipment the student might interact with, including but not limited to the mouse, keyboard, chair, and steering wheel. This was to ensure that if someone was not displaying symptoms but was infected, any potential pathogens they may have left behind would be cleaned up afterwards.

One of the last steps that was taken was that only a select few number of participants would be involved. Only four participants were tested, all of whom were already in close contact with each other to further minimize any potential spread that would be a result from this study.

4.2 Simulation Steps

This section details the specific steps that were taken with each trial in the simulation.

At the beginning of the trial the user is put on a road that allows for them to get used to the feel of the simulator for approximately 20 seconds, before beginning the actual course that has been designed for them. This extra time allows the user to become more comfortable with the steering and acceleration of the car before they are posed with any actual scenarios. Once they were on the course, they followed the road and were expected to abide by any standard traffic laws that would apply. Along the road as the users drove, they came across a total of fifteen intersections, each with specific conditions requiring a driver’s attention. The traffic signals at these intersections would either be green, begin to turn yellow as the driver passes by close, or will turn red before the user gets to the intersection. After the user passed through all fifteen intersections, they reached an area designated as the end of the trial and recording ceased.

4.3 Trial Recording

To begin each trial, participants are first given a run-down of what to expect in the simulation, without giving away anything specific about the scenarios themselves. They will be told that they are going to be driving a car, and that their gazes will be recorded as

they drive through the course. After they have understood what they will be doing, the eye tracker calibration can begin.

For the Gazepoint GP2 to track the participants eye gaze, it must be properly calibrated to each participant. It is calibrated by first adjusting the base so that it is level with a participant’s eyes, and then running software that requires the participant to follow a series of dots across the screen. Following calibration, a short practice test is done to verify the gaze tracking is accurate. Following this, recording will begin, the simulation is launched, and the participants will then work through and finish the trial.

4.4 Scenarios

In this section each scenario is detailed in terms of what it entails and what distractions the driver needs to track, along with a picture of each scenario. Active distractions are circled in red markers, while passive distractions will be marked in green.

Additionally, in each scenario the traffic light is considered a potential distraction but will not be marked in these images.

Figure 4. Scenario 1.

In Scenario 1, there are no major or notable distractions. This is intended as a control scenario, in which there are no distractions near the user. As Figure 4 shows,

there are no major distractions. Prior to reaching the first intersection they will pass both a car and a pedestrian, so they are familiar with them.

Figure 5. Scenario 2.

In Scenario 2, there are again no major distractions. Figure 5 shows that while there is a car pulled out far ahead of the user and driving away, it is never near the participant’s car and poses no obvious threat. Additionally, at this scenario the user will observe a red light for the first time, and this scenario is meant to be a control scenario as well, however it is a control scenario where the user will have to stop. This was done so that it can be used when comparing an intersection with no dangers to an intersection with potential dangers.

Figure 6. Scenario 3.

In Scenario 3, the first potential hazard is introduced, by adding a small crowd of pedestrians near the street. Figure 6 shows the pedestrians right at the corner of the intersections, which is where people would gather if they were about to cross the street.

These pedestrians are stationary and do not walk, however their existence is intended to draw a potential reaction from the driver.

Figure 7. Scenario 4.

In Scenario 4, the same group of pedestrians is at the intersection, however this intersection also has the light change, which could indicate to the driver that pedestrians could potentially cross. Figure 7 shows they are also positioned very similarly to the

previous scenario and should get a similar response to the previous scenario. Even though the light changes and the pedestrians could cross, they do not move in this scenario, but are intended to gauge the user’s possible reaction to pedestrians at a stop light.

Figure 8. Scenario 5.

In Scenario 5, walking pedestrians are added to several locations before and at the intersection. As Figure 8 shows, they are not as grouped together, and in the left side of the image there are even pedestrians still standing, which act as a potential distraction opposite the walking pedestrians.

Figure 9. Scenario 6.

In Scenario 6, the first active distraction takes place. Circled in green in Figure 9, a group of pedestrians is again at the corners of the intersection, however in this scenario a jaywalker will cross just as the driver gets near. This jaywalker is circled in red in

Figure 9. Depending on the speed and caution of the driver, they will have a different reaction to this distraction. If the user is driving fast, and is not focused on the jaywalker, they may not notice them until they are already crossing and the driver will have to make a sharp reaction, but if the driver is cautious and aware of the pedestrians at the intersection they will notice the jaywalker earlier and have more time to react reasonably.

Figure 10. Scenario 7.

As shown in Figure 10, the 7th scenario will have the user pass through another greenlight with no distractions except for some vehicles on the left of the road that have their turning signals on. The flashing lights of the cars should distract the user, or at least draw their attention from anything else.

Figure 11. Scenario 8.

Scenario 8 is almost a repeat of Scenario 7, with the main difference being that the user will come to a stoplight which will turn red before the user gets to the intersection. Figure 11 shows that the light turns long before the driver will arrive at the intersection, so this will also check where the user focuses as they stop with moving distractions. Once the light turns red in this scenario, the cars will then drive onto the road in front of the user and turn at the next intersection.

Figure 12. Scenario 9.

In Scenario 9, the only distraction is an oncoming car with its turning signal on.

As shown in Figure 12, this car will then turn out ahead of the user and cross traffic. This

is like Scenario 6 with the jaywalker, in that depending on the speed and focus of the driver they will have to react either sharply or reasonably to avoid the car opposite of them.

Figure 13. Scenario 10.

Scenario 10 is like that of Scenario 9, where a car will pull out ahead of the driver. In Figure 13 the car appears to be very far away when it does this, but this is due to the driver from this image being very cautious at this stoplight after seeing it turn yellow. The car in the image does not turn until right before the light turns red. In the next several scenarios some distractions may appear to be far away or inconsequential, but the driver in this trial was generally more cautious than others, which led to them often being a bit further away from any active distractions or hazardous situations.

Figure 14. Scenario 11.

In Scenario 11, the user is forced into the right-hand lane as there is a car with hazard lights on stuck ahead, which is meant to force them into the right lane where the next hazard will appear. After being forced over, just as they approach the intersection a car will pull ahead of the driver and continue along the road. This determines how the user will react either by slamming the brakes at the vehicles sudden entrance, or if they are at a safe distance slowing down just enough to avoid a collision.

Figure 15. Scenario 12.

In Scenario 12 the pedestrians make a return while they are walking down the streets alongside some pedestrians that are waiting at the intersection corners.

Additionally, as Figure 15 shows, similarly to Scenario 11 a car will turn out from the opposing street and merge into the right-hand lane. This is intended to see how the user reacts to the same scenario as before, but with different distractions.

Figure 16. Scenario 13.

In Scenario 13, several vehicles are distractions for the user. To begin, there are several cars on the opposite side of the road driving appropriately. There is also a car with a turn signal on in the left-hand side of the road that the user is driving on. Lastly, in

Figure 16 you can also see that there is a car that will make a turn before the user arrives at the intersection, which is specifically placed to be hidden behind the turning signal car until it turns. This complex scenario tests the drivers focus on multiple moving objects, and what their reaction will be to another scenario like that in Scenario 6 and Scenario 9, where the road ahead of them is cut off by something else.

Figure 17. Scenario 14.

In Scenario 14, pedestrians and a car with their turning signal are waiting at the intersection. This scenario is less complex than the previous two intersections. In this scenario neither the pedestrians nor the car circled in red in Figure 17 make any movements, but this should test how the driver reacts after seeing several moving objects in previous scenarios.

Figure 18. Scenario 15.

The final intersection is Scenario 15, and Figure 18 identifies several of the distractions, which this scenario features the most of. There are oncoming cars before the intersection, and at the intersection a mix of all prior events await. In the left lane is a car

with a turning signal on, a car that will cut across traffic to make a turn, a car that merges in front of the user, and additionally pedestrians on the side of the road.

CHAPTER 5

RESULTS

The results are discussed in two forms, the first of which is with raw observation of where the drivers would focus attention towards at various scenarios, and the second is what was expected, with educated reasoning for how they differed or were the same.

5.1 Interactions

While driving through the course, participants would interact with several different events of varying importance. Table 1 describes the performance of drivers in the simulation.

Scenario User 1 User 2 User 3 User 4 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 Table 1. A table of overall participant performance at scenarios.

Each of the colored cells in Table 1 is a marking of each of the participants performances at the scenarios. A dark green cell indicates that the user focused on the main hazards of the scenario, a red cell indicates that the user did not notice the main hazard of the scenario, and a yellow cell indicates they noticed some of the hazards that are present at a scenario if there was more than one.

This chart shows a vast difference in what participants noticed throughout the trial. Each of the users tended to focus on different things more often than others, such as some focusing more on a potential light change, while others were more concerned with what cars would do around them. The first two scenarios were designed to be noticed by all, as these were the control scenarios with nothing in them except a light. However, as the chart shows other than the control scenarios, only scenario 4 saw all the participants noticing the main hazard. In every other scenario there was at least one hazard that at least one participant did not notice. Additionally, scenario 7, 8 and 15 saw none of the participants noticing all the hazards.

5.1.1 Lights

Of the fifteen scenarios, four of them featured a light that would either change to red or yellow as the driver approached. Following an analysis of the user performance, none of the drivers ran through a light that turned red before their approach. The lights tended to be focused on more than any other event, as the participants would almost always glance at the lights at least once in each intersection, and would often repeatedly glance back to the light in case the light changed color.

The frequent attention on the traffic light led to issues for other events however, as three of the four participants were focused on a light and missed at least one hazardous

event throughout their trials. This was most noticeable in scenario 6, where users 1, 2 and

4 were all looking at the light and did not notice or anticipate a potential jaywalker until they had already started walking in the road, and had to slam on their brakes to prevent an accident. This information shows that in some situations the traffic light can be more distracting than beneficial.

5.1.2 Pedestrians

Pedestrians were present in seven of the fifteen scenarios and were often ignored.

Of the 28 times pedestrians would have been seen across all participants, 15 of those times the pedestrians were not even glanced at. The CDC’s WISQARS (Web-Based

Injury Statistics Query and Reporting System) shows that an estimated 137,000 pedestrians were treated in hospitals for crash-related incidents in 2017 (2020).

Furthermore, these numbers are even more exaggerated and common in dense, urban roadways like those in large cities. The National Highway Traffic Safety Administration records that in cities with populations over 500,000, the rate of pedestrian accidents and fatalities is often doubled or even tripled compared to state averages (2019).

There were a few occasions where pedestrians were more focused on than a more active and hazardous event, such as a car turning or a light changing color. Oftentimes however, pedestrians were ignored, and this may be more dangerous, as a driver could notice a pedestrian hazard too late and potentially swerve to avoid the pedestrian. Doing this could cause even more potential damage to both the driver, other drivers, and any other pedestrians nearby.

Figure 19. The “Jaywalker” Scenario.

An example of this is the scenario in which a jaywalker crosses traffic without warning. In this scenario, most of the participants had to slam on their brakes due to having their attention focused on something else. The scene shown in Figure 19 is from

User 3, and User 3 is the only participant who actively anticipated a jaywalker throughout the trials and continued to monitor pedestrians closely throughout their entire trial.

5.1.3 Active Vs. Passive

Most of the distractions can be broken into two categories, active or passive.

Some of the passive distractions that can be considered are pedestrians standing on sidewalks, cars stopped, or even lights which should be checked frequently. These passive events generally did not pose any actual hazard but are constant reminders of what could be dangerous. Depending on the driver, passive distractions were focused on in varying capacity. For example, User 3 tended to focus more on passive distractions than the others, and in a few events was more preoccupied focusing on a crowd of pedestrians or a potential light change than noticing a more pressing hazard.

Active distractions are generally more noticeable and include hazards such as a light actively changing color, a moving vehicle, or walking pedestrians. These present a 31

more obvious distraction and hazard, and as a result when compared to the passive distractions were often noticed more and would draw in more attention from the users.

5.2 Expectations

Initial expectations were that drivers would notice most if not all distractions in each event, and that they would be able to react to them all appropriately. The results of this do not entirely line up with this hypothesis. For example, the first of my research questions I wanted to answer is that I wanted to know where users would spend most of their time looking. Since this was a small environment that was event dense, I expected them to be constantly scanning the screen for potential hazards. However, based on the heatmaps collected from the users, in general they would most often focus directly ahead of them. The heatmaps that are generated in the following figures were taken at the same moment from the trials, which was averaged over 30 seconds, and features data from 2 intersections, one of which had 2 passive distractions, and the other had 2 active distractions.

Figure 20. A composite of all heatmaps from users.

Figure 20 shows that drivers still spent most of their time focused on the road directly in front of them despite the presence of possible hazards to the side. While the drivers would occasionally glance to the side, most of a driver’s attention seemed to remain focused ahead on the lane they were in. All the users have their heatmaps listed individually in Appendix A. The only heatmap with a much larger spread was from User

Figure 21. User 4’s heatmap.

User 4’s heatmap is singled out in Figure 21, as they noticeably had a much wider range than anyone else. The conclusion I drew from this when paired with the other observations I made during their trial is that they were a more reckless driver on average.

User 4 had the most missed distractions, and additionally when possible, they chose to spend most of their time driving in the left lane, rather than the right lane of the road. As a result, their entire graph was shifted to the left a little bit (as they were looking ahead of them in the left lane opposed to the right). During certain scenarios, a car would drive on the right side of the road and it was presumed that most would follow it, however User 4 instead tried driving around the car.

5.3 Discussion

There was another result I drew from these results that will answer the second research question, which was if a human driver’s habits were an ideal model for a self - driving car to follow. This was found to be a bad model, as many of the distractions and hazards a driver would come across was either not properly acknowledged or missed entirely. The lack of safety was evident in examples where a participant’s perception to a hazard or distraction would affect their response to it. This was shown subtly sometimes, where if a participant had not noticed a distraction they would do a few sudden eye jerking movements, swiftly darting back and forth to that hazard and the road a few times to see if anything was going to happen. Other times these reactions were more pronounced, such as in the jaywalker scenario where all participants except for User 3 slammed the brakes, trying to slow down quickly. Afterwards participants tried paying more attention to the pedestrians, possibly cautious of another jaywalker. After seeing these results, the opposite of a human’s driving habits might be ideal in some cases.

A self-driving car would have many different systems onboard and compartmentalizing these systems would seem to be the most efficient model based on these results. If the main model for a car’s sensors were to use the human’s results as their basis, the sensors would focus mostly in front of them. Weighing the area directly ahead of the car would lead to missing out on any potential distractions that occur just off the road, such as cars turning onto the road out of another location, or most pedestrians.

This is what happened in many of the user trials, as it was shown that they would often miss at least some of the potential hazards. A reversal of this would result in a system

scanning mostly off the road that is directly ahead of them. This system could actually work exceptionally well, as it can help navigate most of the dangers of the road, with the only supplement requiring there being a separate system that monitors the vehicles position in a lane, and directly ahead. These technologies already have proof of concepts shown in prior experiments by Park and Kee (2019) and Urmson et al. (2008).

Additionally, several cars produced currently have some form of implementation of these systems in the form of early collision detection or lane assist.

CHAPTER 6

CONCLUSIONS AND FUTURE WORK

In this study, data was gathered and interpreted to help the future understanding of vehicle driver attention. From these results gathered here, some preliminary conclusions can be drawn.

6.1 Conclusions

The first of these conclusions is that while drivers do spend part of their time driving observing potential distractions and the environment around them, a vast majority of the time is still spent focused directly in front of them on the road ahead. This then results in the human driver missing many potential hazards that they could come across.

This implies that in a scenario where you are using human data as a model for a self - driving car, you would be weighing the lane and cars ahead of you above all else. It could instead be said that a better way to interpret this data is to say that a self-driving car should focus on everywhere except for directly in front of it assuming other systems are in place.

6.2 Future Work

No work is established or executed perfectly, and there are always ways to improve the quality. Future work on this project could include some more scenarios that have different events, or even a different ordering of events to see if the apparent order of events could have adjusted the results. Future tests with a much larger sample size of participants when the COVID-19 pandemic is over would also give better data quality.

Also, as was noted earlier in this study, some drivers became conditioned to focus on certain distractions such as pedestrians more when a noteworthy event like the jaywalker

event occurred, which may condition them for future intersections and adjust their behavior. Elimination of this bias somehow could produce different results. Furthermore, all these trials were conducted while driving straight through an intersection. This took out variables such as what a driver might have to do when making turns at intersections, which Kusano and Gabler (2015) notes make up over 70% of intersection crashes.

Additional future work from this project could hopefully put the hypothesis about this data for a model for a self-driving car to the test as well.

APPENDIX A: STUDY MATERIALS

Figure A.1. The Informed Consent forms that volunteers signed.

Figure A.2. User 1’s heatmap.

Figure A.3. User 2’s heatmap.

Figure A.4. User 3’s heatmap.

Figure A.5. User 4’s heatmap.

Figure A.6. A snippet of code that handled events at intersections.

Figure A.7. IRB Application Form 1/5.

Figure A.8. IRB Application Form 2/5.

Figure A.9. IRB Application Form 3/5.

Figure A.10. IRB Application Form 4/5.

Figure A.11. IRB Application Form 5/5.

Figure A.12 IRB Approval Form

REFERENCES

Abdurrahman Arslanyilmaz, Salman Al Matouq, and Durmus Doner. Driver Readiness in

Autonomous Vehicle Take-Overs. World Academy of Science, Engineering, and

Technology.

B.J. Austin, K.T. Edwards, E.J. Hennings, J.L. Vian, and N.W. Warner. 2000. Active

Network Guidance and Emergency Logic (ANGEL) Program. SAE Technical

Paper Series 109 (2000), 312–315. DOI: http://dx.doi.org/10.4271/2000-01-2095

Jean-Francois Bonnefon, Azim Shariff, and Iyad Rahwan. 2016. The social dilemma of

autonomous vehicles. Science 35 (June 2016). DOI:

http://dx.doi.org/10.1126/science.aaf2654

Steve Bryson. 2001. Effects of Lag and Frame Rate on Various Tracking Tasks.

Proceedings of SPIE: The International Society for Optical Engineering (January

2001). DOI: http://dx.doi.org/10.1117/12.157034

CDC. 2020. Social Distancing, Quarantine, and Isolation. (July 2020). Retrieved October

1, 2020 from https://www.cdc.gov/coronavirus/2019-ncov/prevent-getting-

sick/social-distancing.html

CDC. 2020. WISQARS (Web-based Injury Statistics Query and Reporting

System)|Injury Center CDC. (July 2020). Retrieved November 2, 2020 from

https://www.cdc.gov/injury/wisqars/index.html

Kajal T. Claypool and Mark Claypool. 2007. On frame rate and player performance in

first person shooter games. Multimedia Systems 13, 1 (2007), 3–17. DOI:

http://dx.doi.org/10.1007/s00530-007-0081-1

Lynne Cooke. 2005. Eye Tracking: How It Works and How It Relates to Usability.

Technical Communication 52, 4 (November 2005), 456–463.

https://www.jstor.org/stable/43093613

Thomas Dingus et al. 2016. Driver Crash Risk Factors and Prevalence Evaluation using

Naturalistic Driving Data. Proceedings of the National Academy of Sciences of

the United States of America 113, 10 (March 2016), 2623–2641. DOI:

http://dx.doi.org/10.1073/pnas.1513271113

Emilio Frazzoli, Munther Dahleh, and Eric Feron. 2002. Real-Time Motion Planning for

Agile Autonomous Vehicles. Journal of Guidance, Control, and Dynamics 25, 1

(2002), 116–129. DOI: http://dx.doi.org/10.2514/2.4856

Shan Fu. 2019. The Ship Driving Safety Control System Based on Visual Effect

Analysis. Journal of Coastal Research 98, Special (2019), 88–91. DOI:

http://dx.doi.org/10.2112/si98-022.1

Aoi Kawashima, Kazuyuki Kobayashi, and Kajiro Watanabe. 2001. Implementation of

Human-Like Driving for Autonomous Vehicle. SAE Technical Paper Series 110

(2001), 450–455. DOI: http://dx.doi.org/10.4271/2001-01-0805

Daniel Knoepfle, Colin Camerer, and Joseph Wang. 2009. Studying Learning in Games

Using Eye-Tracking. Journal of the European Economic Association 7, 2-3 (May

2009), 388–398. DOI: http://dx.doi.org/10.1162/jeea.2009.7.2-3.388

Philip Koopman and Michael Wagner. 2016. Challenges in Autonomous Vehicle Testing

and Validation. SAE International Journal of Transportation Safety 4, 1 (April

2016), 15–24.

Kristofer D. Kusano and Hampton C. Gabler. 2015. Target Population for Intersection

Advanced Driver Assistance Systems in the U.S. SAE International Journal of

Transportation Safety 3, 1 (April 2015), 1–16. DOI:

http://dx.doi.org/10.4271/2015-01-1408

Erik T. Nelson, David G. Kidd, and David M. Cades. 2010. Examining Patterns of

Simulator Sickness during Increased Exposure to a Motion-base Driving

Simulator over Time. Journal of the Washington Academy of Sciences 96, 3

(2010), 1–14.

NHTSA. 2019. Traffic Safety Facts 2017 Data: Pedestrians (2019).

Sven Nyholm and Jilles Smids. 2016. The Ethics of Accident-Algorithms for Self-

Driving Cars: An Applied Trolley Problem? Ethical Theory and Moral Practice

19, 5 (November 2016), 1275–1289. DOI: http://dx.doi.org/10.1007/s10677-016-

9745-2

Changhyeon Park and Seok-Cheol Kee. 2019. Implementation of Autonomous Driving

Vehicle at an Intersection with Traffic Light Recognition and Vehicle Controls.

Proceedings of the 5th International Conference on Vehicle Technology and

Intelligent Transport Systems 1 (May 2019), 542–549. DOI:

http://dx.doi.org/doi:10.5220/0007760905420549

Anna Petrovskaya and Sebastian Thrun. 2008. Model Based Vehicle Tracking for

Autonomous Driving in Urban Environments. Robotics: Science and Systems IV

(June 2008). DOI: http://dx.doi.org/10.15607/rss.2008.iv.023

Cleshain Solomon and Zenghui Wang. 2015. Driver Attention and Behavior Detection

with Kinect. Journal of Image and Graphics 3, 2 (December 2015), 84–89. DOI:

http://dx.doi.org/doi:10.18178/joig.3.2.84-89

Scott Stevens, Jeffrey Bellone, Philip Azeredo, and Marisol Medri. 2017. Baseline

Analysis of Driver Performance at Intersections for the Left-Turn Assist

Application. SAE International Journal of Transportation Safety 5, 1 (April

2017), 13–29. DOI: http://dx.doi.org/10.4271/2017-01-0033

Jack Stilgoe. 2018. We Need New Rules for Self-Driving Cars. Issues in Science and

Technology 34, 3 (2018), 52–57.

Chris Urmson et al. 2008. Autonomous driving in urban environments: Boss and the

Urban Challenge. Journal of Field Robotics, 425-466. DOI:

http://dx.doi.org/doi:10.1002/rob.20255

Savannah Wei Shi, Michel Wedel, and F.G.M. (Rik) Pieters. 2013. Information

Acquisition During Online Decision Making: A Model-Based Exploration Using

Eye-Tracking Data. Management Science 59, 5 (May 2013), 1009–1026. DOI:

http://dx.doi.org/10.1287/mnsc.1120.1625