A Large-Scale Field Study of Browser Security Warning Effectiveness

Alice in Warningland: A Large-Scale Field Study of Browser Security Warning Effectiveness

Devdatta Akhawe Adrienne Porter Felt University of California, Berkeley\ast Google, Inc. [email protected] [email protected]

Abstract The security community’s perception of the “oblivious” user evolved from the results of a number of laboratory We empirically assess whether browser security warn- studies on browser security indicators [5, 11, 13, 15, 26, ings are as ineffective as suggested by popular opinion 30, 34]. However, these studies are not necessarily rep- and previous literature. We used Mozilla Firefox and resentative of the current state of browser warnings in Google Chrome’s in-browser telemetry to observe over 2013. Most of the studies evaluated warnings that have 25 million warning impressions in situ. During our field since been deprecated or significantly modified, often in study, users continued through a tenth of Mozilla Fire- response to criticisms in the aforementioned studies. Our fox’s malware and phishing warnings, a quarter of Google goal is to investigate whether modern browser security Chrome’s malware and phishing warnings, and a third of warnings protect users in practice. Mozilla Firefox’s SSL warnings. This demonstrates that We performed a large-scale field study of user deci- security warnings can be effective in practice; security sions after seeing browser security warnings. Our study experts and system architects should not dismiss the goal encompassed 25,405,944 warning impressions in Google of communicating security information to end users. We Chrome and Mozilla Firefox in May and June 2013. We also find that user behavior varies across warnings. In con- collected the data using the browsers’ telemetry frame- trast to the other warnings, users continued through 70.2% works, which are a mechanism for browser vendors to of Google Chrome’s SSL warnings. This indicates that collect pseudonymous data from end users. Telemetry the user experience of a warning can have a significant allowed us to unobtrusively measure user behavior during impact on user behavior. Based on our findings, we make normal browsing activities. This design provides realism: recommendations for warning designers and researchers. our data reflects users’ actual behavior when presented with security warnings. 1 Introduction In this paper, we present the rates at which users click through (i.e., bypass) malware, phishing, and SSL warn- An oft-repeated maxim in the security community is the ings. Low clickthrough rates are desirable because they futility of relying on end users to make security decisions. indicate that users notice and heed the warnings. Click- Felten and McGraw famously wrote, “Given a choice through rates for the two browsers’ malware and phish- between dancing pigs and security, the user will pick ing warnings ranged from 9% to 23%, and users clicked dancing pigs every time [21].” Herley elaborates [17], through 33.0% of Mozilla Firefox’s SSL warnings. This demonstrates that browser security warnings can effec- Not only do users take no precautions against tively protect most users in practice. elaborate attacks, they appear to neglect even Unfortunately, users clicked through Google Chrome’s basic ones. For example, a growing body of SSL warning 70.2% of the time. This implies that the measurement studies make clear that ...[users] user experience of a warning can have a significant impact are oblivious to security cues [26], ignore cer- on user behavior. We discuss several factors that might tificate error warnings [30] and cannot tell legit- contribute to this warning’s higher clickthrough rates. Our imate web-sites from phishing imitations [11].1 positive findings for the other five warnings suggest that the clickthrough rate for Google Chrome’s SSL warning \ast The Mozilla Firefox experiments were implemented while the au- can be improved. thor was an intern at Mozilla Corporation. 1Citations updated to match our bibliography. We also consider user behaviors that are indicative of attention to warnings. We find that Google Chrome’s SSL clickthrough rates vary by the specific type of error. In Mozilla Firefox, a fifth of users who choose to click through an SSL warning remove a default option, showing they are making cognitive choices while bypassing the warning. Together, these results contradict the stereotype of the wholly oblivious user with no interest in security. We conclude that users can demonstrate agency when confronted with browser security warnings. Users do not always ignore security warnings in favor of their desired content. Consequently, security experts and platform designers should not dismiss the role of the user. We find that the user experience of warnings can have an enormous impact on user behavior, justifying efforts to build usable warnings. Figure 1: Malware warning for Google Chrome Contributions. We make the following contributions:

• To our knowledge, we present the ﬁrst in-depth, large-scale ﬁeld study of browser security warnings.

• We survey prior laboratory studies of browser security warnings and discuss why our ﬁeld study data differs from prior research.

• We analyze how demographics (operating system and browser channel), warning frequency, and warning complexity affect users’ decisions. Notably, Figure 2: Malware warning for Mozilla Firefox we ﬁnd evidence suggesting that technically skilled users ignore warnings more often, and warning frequency is inversely correlated with user attention.

• We provide suggestions for browser warning designers and make recommendations for future studies.

2 Background

Web browsers show warnings to users when an attack Figure 3: SSL warning for Google Chrome. The first paragraph might be occurring. If the browser is certain that an attack changes depending on the specific SSL error. is occurring, it will show an error page that the user cannot bypass. If there is a chance that the perceived attack is a false positive, the browser will show a bypassable warning that discourages the user from continuing. We study only bypassable warnings because we focus on user decisions. A user clicks through a warning to dismiss it and proceed with her original task. A user leaves the warning when she navigates away and does not continue with her original task. A clickthrough rate describes the proportion of users who clicked through a warning type. When a user clicks through a warning, the user has (1) ignored the warning because she did not read or understand it or (2) made an informed decision to proceed because she be- lieves that the warning is a false positive or her computer Figure 4: SSL warning for Mozilla Firefox is safe against these attacks (e.g., due to an antivirus). Warning Design. Figures1 and2 show the Google Chrome and Mozilla Firefox warnings. Their phishing warnings are similar to their respective malware warnings. When a browser presents the user with a malware or phishing warning, she has three options: leave the page via the warning’s escape button, leave the page by closing the window or typing a new URL, or click through the warning and proceed to the page. The warnings also allow the user to seek more information about the error. Click Count. Mozilla Firefox users who want to bypass the warning need to click one button: the “Ignore this warning” link at the bottom right. On the other hand, Chrome users who want to bypass the warning need to click twice: first on the “Advanced” link, and then on “Proceed at your own risk.” Figure 5: SSL Add Exception Dialog for Mozilla Firefox We focus on three types of browser security warnings: 2.2 SSL Warnings malware, phishing, and SSL warnings. At present, all The Secure Sockets Layer (SSL/TLS) protocol provides three types of warnings are full-page, interstitial warnings secure channels between browsers and web servers, mak- that discourage the user from proceeding. ing it fundamental to user security and privacy on the web. As a critical step, the browser verifies a server’s identity by validating its public-key certificate against a 2.1 Malware and Phishing Warnings set of trusted root authorities. This validation will fail in Malware and phishing warnings aim to prevent users from the presence of a man-in-the-middle (MITM) attack. visiting websites that serve malicious executables or try Authentication failures can also occur in a wide vari- to trick users. Google Chrome and Mozilla Firefox rely ety of benign scenarios, such as server misconfigurations. on the Google Safe Browsing list [25] to identify mal- Browsers usually cannot distinguish these benign scenar- ware and phishing websites. The browsers warn users ios from real MITM attacks. Instead, browsers present away from the sites instead of blocking them because the users with a warning; users have the option to bypass the Safe Browsing service occasionally has false positives, warning, in case the warning is a false positive. although the false positive rate is very low [25]. Clickthrough Rate. We hope for a 0% clickthrough rate for SSL warnings shown during MITM attacks. However, Clickthrough Rate. If a malware or phishing warning many SSL warnings may be false positives (e.g., server is a true positive, clicking through exposes the user to a misconfigurations). There are two competing views re- dangerous situation. Nearly all Safe Browsing warnings garding SSL false positives. In the first, warning text are true positives; the false positive rate is low enough should discourage users from clicking through both true to be negligible. The ideal clickthrough rate for malware and false positives, in order to incentivize developers and phishing warnings is therefore close to 0%. to get valid SSL certificates. In the other, warning text Warning Mechanisms. The browsers routinely fetch a should provide users with enough information to correctly list of suspicious (i.e., malware or phishing) sites from identify and dismiss false positives. The desired click- Safe Browsing servers. If a user tries to visit a site that through rates for false-positive warnings would be 0% is on the locally cached list, the browser checks with the and 100%, respectively. In either case, false positives are Safe Browsing service that the URL is still on the malware undesirable for the user experience because we do not or phishing list. If the site is still on one of the lists, the want to annoy users with invalid warnings. Our goal is browser presents a warning. therefore a 0% clickthrough rate for all SSL warnings: The two browsers behave differently if a page loads users should heed all valid warnings, and the browser a third-party resource (e.g., a script) from a URL on the should minimize the number of false positives. Safe Browsing list. Google Chrome stops the page load Warning Design. Figures3 and4 present Google Chrome and replaces the page with a warning. Mozilla Firefox and Mozilla Firefox’s SSL warnings. The user can leave blocks the third-party resource with no warning. As a via the warning’s escape button, manually navigate away, result, Mozilla Firefox users can see fewer warnings than or click through the warning. In Mozilla Firefox, the Google Chrome users, despite both browsers using the user must also click through a second dialog (Figure 5) to same Safe Browsing list. bypass the warning. The browsers differ in their presentation of the techni- 3 Prior Laboratory Studies cal details of the error. Google Chrome places information about the specific error in the main warning (Figure 3, first We survey prior laboratory studies of SSL and phishing paragraph), whereas Firefox puts the error information warnings. The body of literature paints a grim picture in the hidden “Technical Details” section and the second of browser security warnings, but most of the warnings “Add Exception” dialog (Figure 5). have since been deprecated or modified. In some cases, Click Count. Mozilla Firefox’s SSL warning requires warnings were changed in response to these studies. more clicks to bypass. Google Chrome users click Only two studies evaluated warnings that are similar through a single warning button to proceed. On the other to the modern (June 2013) browser warnings that we hand, Mozilla Firefox’s warning requires three clicks: study in this paper. Sunshine et al. and Sotirakopoulos (1) click on “I Understand The Risks,” (2) click on the et al. reported clickthrough rates of 55% to 80% for the “Add Exception” button, which raises a second dialog, Firefox 3 and 3.5 SSL warnings, which are similar but (3) click on “Confirm Security Exception” in the second not identical to the current Firefox SSL warning [29,30]. dialog. By default, Firefox permanently remembers the However, Sotirakopoulos et al. concluded that laboratory exception and will not show the warning again if the user biases had inflated both studies’ clickthrough rates [29]. reencounters the same certificate for that website. In contrast, Chrome presents the warning every time and 3.1 SSL Warnings does not remember the user’s past choices. SSL warnings are the most studied type of browser warn- 2.3 Browser Release Channels ing. Usability researchers have evaluated SSL warnings in both SSL-specific studies and phishing studies because Mozilla and Google both follow rapid release cycles. SSL warnings and passive indicators were once viewed They release official versions of their browsers every six as a way to identify phishing attacks.2 or seven weeks, and both browsers update automatically. Dhamija et al. performed the first laboratory study of The official, default version of a browser is referred to as SSL warnings in 2006. They challenged 22 study partic- “stable” (Google Chrome) or “release” (Mozilla Firefox). ipants to differentiate between phishing and legitimate If users are interested in testing pre-release browser websites in Mozilla Firefox 1.0.1 [11]. In this version, versions, they can switch to a different channel. The sta- the warning was a modal dialog that allowed the user to ble/release channel is the recommended channel for end permanently accept, temporarily accept, or reject the cer- users, but a minority of users choose to use earlier chan- tificate. When viewing the last test website, participants nels to test cutting-edge features. The “Beta” channel is encountered an SSL warning. The researchers reported several weeks ahead of the stable/release channel. The that 15 of their 22 subjects (68%) quickly clicked through “developer” (Google Chrome) or “Aurora” (Mozilla Fire- the warning without reading it. Only one user was later fox) channel is delivered even earlier. Both browsers also able to tell the researchers what the warning had said. offer a “nightly” (Mozilla Firefox) or “Canary” (Google The authors considered the clickthrough rate of 68% a Chrome) release channel, which updates every day and conservative lower bound because participants knew that closely follows the development repository. they should be looking for security indicators. The pre-release channels are intended for advanced In 2007, Schechter et al. studied user reactions to Inter- users who want to experience the latest-and-greatest fea- net Explorer 7’s SSL warning, which is the same one-click tures and improvements. They give website, extension, interstitial that is present in all subsequent versions of In- and add-on developers time to test their code on upcom- ternet Explorer [26]. Participants encountered the warning ing versions before they are deployed to end users. The while logging into a bank website to look up information. early channels are not recommended for typical end users The researchers were aware of ecological validity con- because they can have stability issues, due to being un- cerns with laboratory studies and split their participants der active development. The rest of this paper assumes into three groups: participants who entered their own cre- a positive correlation between pre-release channels use dentials, a role-playing group that entered fake passwords, and technical ability. While this matches the intention and a security-primed role-playing group that entered of browser developers, we did not carry out any study to fake passwords. Overall, 53% of the total 57 participants validate this assumption. clicked through. However, only 36% of the non-role- playing group clicked through. The difference between the role-playing participants and non-role-playing partic-

2There is evidence that modern phishing sites can have valid SSL certificates [24]. ipants was statistically significant, illustrating one chal- the lock icon or color of the URL bar to tell the user when lenge of experiments in artificial environments. a website has an Extended Validation (EV) SSL certifi- Sunshine et al. performed multiple studies of SSL warn- cate. Jackson et al. asked 27 study subjects to classify ings in 2009 [30]. First, they conducted an online survey. 12 websites as either phishing or legitimate sites, but the They asked 409 people about Firefox 2, Firefox 3, and EV certificates did not help subjects identify the phish- Internet Explorer 7 warnings. Firefox 2 had a modal dia- ing sites [19]. In a follow-up study, Sobey et al. found log like Firefox 1.0.1, and Firefox 3’s warning is similar that none of their 28 subjects clicked on the EV indica- but not identical to the current Firefox warning. Less tors, and the presence of EV indicators did not affect than half of respondents said they would continue to the decision-making [28]. Similarly, Biddle et al. found that website after seeing the warning. As a follow-up, Sun- study participants did not understand Internet Explorer’s shine et al. also conducted a laboratory study that exposed certificate summaries [3]. 100 participants to SSL warnings while completing infor- In 2012, a Google Chrome engineer mentioned high mation lookup tasks. The clickthrough rates were 90%, clickthrough rates for SSL warnings on his blog [20]. We 55%, and 90% when participants tried to access their bank expand on this with a more accurate and detailed view of websites in Firefox 2, Firefox 3, and Internet Explorer SSL clickthrough rates in Google Chrome. 7, respectively. The clickthrough rates increased to 95%, 60%, and 100% when participants saw an SSL warning 3.2 Phishing Warnings while trying to visit the university library website. Sotirakopoulos et al. replicated Sunshine’s laboratory Phishing warnings in contemporary browsers are active, SSL study with a more representative population sam- interstitial warnings; in the past, they have been passive ple [29]. In their study, 80% of participants using Firefox indicators in toolbars. Researchers have studied whether 3.5 and 72% of participants using Internet Explorer 7 they are effective at preventing people from entering their clicked through an SSL warning on their bank website. credentials into phishing websites. More than 40% of their participants said that the labora- Wu et al. studied both interstitial and passive phish- tory environment had influenced them to click through ing warnings [34]. Neither of the warnings that they the warnings, either because they felt safe in the study evaluated are currently in use in browsers. First, they environment or were trying to complete the experimental launched phishing attacks on 30 participants. The par- task. Sotirakopoulos et al. concluded that the laboratory ticipants role-played during the experiment while using environment biased their results, and they suspect that security toolbars that display passive phishing warnings. these biases are also present in similar laboratory studies. Despite the toolbars, at least one attack fooled 20 out Bravo-Lillo et al. interviewed people about an SSL of 30 participants. In their next experiment, they asked warning from an unspecified browser [5]. They asked 20 10 study participants to perform tasks on PayPal and a participants about the purpose of the warning, what would shopping wish list website; they injected modal phishing happen if a friend were to click through, and whether a warnings into the websites. None of the subjects entered friend should click through the warning. Participants the credentials into the PayPal site, but the attack on the were separated into “advanced” and “novice” browser wish list site fooled 4 subjects. The authors do not report users. “Advanced” participants said they would not click the warning clickthrough rates. through an SSL warning on a bank website, but “novice” Egelman et al. subjected 60 people to simulated phish- participants said they would. ing attacks in Internet Explorer 7 or Mozilla Firefox 2.0 [13]. Firefox 2.0 had a modal phishing dialog that is Passive Indicators. Some studies focused on passive not comparable to the current Mozilla Firefox phishing SSL indicators, which non-interruptively show the status dialog, and Internet Explorer had both passive and active of the HTTP(S) connection in the browser UI. Although warnings. Participants believed that they were taking part browsers still have passive SSL indicators, interruptive in a laboratory study about shopping. The researchers SSL and phishing warnings are now the primary tool for asked participants to check their e-mail, which contained communicating security information to users. both legitimate shopping confirmation e-mails and similar Friedman et al. asked participants whether screenshots spear phishing e-mails sent by the researchers. Users who of websites depicted secure connections; many partici- clicked on the links in the phishing e-mails saw a phishing pants could not reliably determine whether a connection warning. Participants who saw Mozilla Firefox’s active was secure [15]. Whalen and Inkpen used eye-tracking warning, Internet Explorer’s active warning, or Internet software to determine that none of their 16 participants Explorer’s passive warning were phished 0%, 45%, and looked at the lock or key icon in the URL bar, HTTP(S) 90% of the time, respectively. The clickthrough rates status in the URL bar, or the SSL certificate when asked to were an unspecified superset of the rates at which people browse websites “normally” [33]. Some browsers modify fell for the phishing attacks. 3.3 Malware Download Warnings 4.2 Measuring Time Spent on Warnings Google Chrome and Microsoft Internet Explorer also dis- We also used the Google Chrome telemetry framework to play non-blocking warning dialogs when users attempt observe how much time Google Chrome users spent on to download malicious executables. In a blog post, a Mi- SSL warnings. Timing began as soon as an SSL warning crosoft employee stated that the clickthrough rate for Inter- came to the foreground in a tab. In particular, net Explorer’s SmartScreen warning was under 5% [16]. We did not study this warning for Google Chrome, and • We recorded the time spent on a warning and associ- Mozilla Firefox does not have this warning. ated it with the outcome (click through or leave). • We recorded the time spent on a warning and associ- 4 Methodology ated it with the error type, if it was one of the three most common error types (untrusted authority, name We rely on the telemetry features implemented in Mozilla mismatch, and expired certificate). Firefox and Google Chrome to measure clickthrough rates in situ. Telemetry is a mechanism for browser vendors to Together, these correspond to five timing measurements collect pseudonymous data from end users who opt in to (two for outcome and three for error type). For scalability, statistics reporting. Google Chrome and Mozilla Firefox the telemetry mechanism in Google Chrome only allows use similar telemetry platforms. timing measurements in discrete buckets. As a result, our analysis also treats time as a discrete, ordinal variable. We used log-scaled bucket sizes (e.g., the first bucket 4.1 Measuring Clickthrough Rates size is 45ms but the last is 90,279ms) with 50 buckets, We implemented metrics in both browsers to count the ranging from 0ms to 1,200,000ms, for the two outcome number of times that a user sees, clicks through, or leaves histograms. The three error type histograms had 75 buck- a malware, phishing, or SSL warning. Based on this data, ets each, ranging from 0ms to 900,000ms. We used more we can calculate clickthrough rates for each warning type. buckets for the error histograms because we anticipated As discussed in Section2, we report only the clickthrough that they would be more similar to each other. rates for warnings that the user can bypass. We measured the prevalence of non-bypassable warnings separately. To 4.3 Ethics supplement the clickthrough rates, we recorded whether users clicked on links like “Help me understand,” “View,” We collected data from users who participate in their or “Technical Details.” browsers’ broad, unpaid user metrics programs. At first Bypassing some warnings takes multiple clicks, and run of a browser, the browser asks the user to share usage our clickthrough rates for these warnings represent the data. If the user consents, the browser collects data on number of users who completed all of the steps to proceed performance, features, and stability. In some pre-release to the page. For Mozilla Firefox’s SSL warning (which developer channels, data collection is enabled by default. takes three clicks to proceed), we recorded how often The browser periodically sends this pseudonymous data users perform two intermediate clicks (on “Add Excep- over SSL to the central Mozilla or Google servers for tion” or “Confirm Security Exception”) as well as the analysis. The servers see the IP addresses of clients by overall clickthrough rate. necessity, but they are not attached to telemetry data. All We also measured how often users encounter and click telemetry data is subject to strict privacy policies and through specific SSL errors. In addition to the overall participants can opt out by changing their settings [7, 23]. clickthrough rates for the warnings, we collected click- Multiple Google Chrome committers and Mozilla Firefox through data for each type of Mozilla Firefox SSL error contributors reviewed the data collection code to ensure and the three most common Google Chrome SSL errors. that the metrics did not collect any private data. Our Mozilla Firefox data set does not allow us to track This work is not considered human subjects research by specific telemetry participants. In Google Chrome, we UC Berkeley because the student did not have access to can correlate warning impressions with psuedonymous database identifiers or personally identifying information. browser client IDs; however, the sample size for most individual users is too small to draw conclusions. We 4.4 Data Collection therefore report the results of measurements aggregated across all users unless otherwise specified. The telemetry Collection Period. Google Chrome’s malware and phish- frameworks do not provide us with any personal or demo- ing measurement code was in place in Chrome 24 prior graphic information except for the operating system and to our work, and our SSL measurement code was added browser version for each warning impression. to Google Chrome 25. The Google Chrome data in this paper was collected April 28 - May 31, 2013. Our Mozilla and found that the clickthrough rates do not differ if Firefox measurement code was added to Firefox 17, and we remove non-independent clients. Our large sample a bug in the SSL measurement code was fixed in Firefox sizes and small critical value (\alpha = 0.001) should further 23. The data on the Firefox malware warning, phishing ameliorate these concerns. warning, and SSL “Add Exception” dialog was collected Frames. Our original measurement for Mozilla Firefox May 1-31, 2013. The data on Firefox SSL warnings was did not differentiate between warnings shown in top-level collected June 1 - July 5, 2013, as the Firefox 23 fix frames (i.e., warnings that fill the whole tab) and warnings progressed through the various release channels. shown in iframes. In contrast, Google Chrome always Sample Sizes. In Google Chrome, we recorded 6,040,082 shows malware and phishing warnings in the top-level malware warning impressions, 386,350 phishing warning frame and does not render any warning type in iframes. impressions, and 16,704,666 SSL warning impressions. Since users might not notice warnings in iframes, the two In Mozilla Firefox, we recorded 2,163,866 malware warn- metrics are not necessarily directly comparable. ing impressions, 100,004 phishing warning impressions, Upon discovering this issue, we modified our Firefox and 10,976 SSL warning impressions. AppendixA fur- measurement implementation to take frame level into ther breaks downs these sample sizes by OS and channel. account. Our new implementation is not available to all Number of Users. For Mozilla Firefox, we recorded Firefox users yet, but we have data for recent pre-release warning impressions from the approximately 1% of Fire- channels. For malware and phishing warning impressions fox users who opt in to share data with Mozilla via teleme- collected from the beta channel, the clickthrough rate for try. In Google Chrome, we observed malware, phishing, the top-level frame is within two percentage points of and SSL warning impressions on 2,148,026; 204,462; and the overall clickthrough rate. This is due to the relative 4,491,767 clients (i.e., browser installs), respectively. infrequency of malware and phishing warnings in iframes and the low overall clickthrough rate. Since the frame level does not make a notable difference for malware and 4.5 Method Limitations phishing warnings, we present the overall rates (including both top-level frames and iframes) for the full sample Private Data. Due to privacy constraints, we could not sizes in Section 5.1. The difference is more important collect information about users’ personal demographics for SSL warnings: the clickthrough rate for top-level or browsing habits. Consequently, we cannot measure frames is 28.7 percentage points higher than the overall whether user behavior differs based on personal character- clickthrough rate of 4.3%. Consequently, Section 5.2 istics, the target site, or the source of the link to the site. presents only the top-level frame rate for SSL warnings, We also cannot identify SSL false positives due to captive although it limits our sample to pre-release users. portals, network proxies, or server misconfigurations. Sampling Bias. The participants in our field study are not 5 Clickthrough Rates a random population sample. Our study only represents users who opt in to browser telemetry programs. This We present the clickthrough data from our measurement might present a bias. The users who volunteered might be study. Section 5.1 discusses malware and phishing warn- more likely to click through dialogs and less concerned ings together because they share a visual appearance. We about privacy. Thus, the clickthrough rates we measure then present rates for SSL warnings in Section 5.2. could be higher than population-wide rates. Given that most of our observed rates are low, this bias augments our claim that clickthrough rates are lower than anticipated. 5.1 Malware and Phishing Warnings Overrepresentation. We present clickthrough rates The clickthrough rates for malware warnings were 7.2% across all warnings shown to all users. A subset of users and 23.2% in stable versions of Mozilla Firefox and could potentially be overrepresented in our analysis. Google Chrome, respectively. For phishing warnings, Within the Google Chrome data set, we identified and we found clickthrough rates of 9.1% and 18.0%. In this removed a small number of overrepresented clients who section, we discuss the effects of warning type, demo- we believe are either crawlers or malware researchers. graphics, and browser on the clickthrough rates. We were unable to remove individual clients from the Mozilla Firefox set, but we do not believe this represents 5.1.1 Malware Rates by Date a bias because we know that the overrepresented clients in Chrome still contributed fewer than 1% of warning The malware warning clickthrough rates for Google impressions. Some clients experienced multiple types Chrome vary widely by date. We have observed click- of warning impressions; we investigated this in Chrome through rates ranging from 11.2% to 24.9%, depending Operating Malware Phishing is listed as both malware and phishing.3 However, the System Firefox Chrome Firefox Chrome practical difference is small: 7.2% vs. 9.1%. Windows 7.1% 23.5% 8.9% 17.9% In Google Chrome, the average malware clickthrough MacOS 11.2% 16.6% 12.5% 17.0% rate is higher than the phishing clickthrough rate. How- Linux 18.2% 13.9% 34.8% 31.0% ever, the malware clickthrough rate fluctuates widely (Sec- tion 5.1.1); the malware clickthrough rate is sometimes Table 1: User operating system vs. clickthrough rates for mal- lower than the phishing clickthrough rate. ware and phishing warnings. The data comes from stable (i.e., release) versions. 5.1.3 Malware/Phishing Rates by Demographics Malware Phishing We consider whether users of different operating systems Channel Firefox Chrome Firefox Chrome and browser release channels react differently to warn- Stable 7.2% 23.2% 9.1% 18.0% ings. As Table 1 depicts, Linux users have significantly Beta 8.7% 22.0% 11.2% 28.1% higher clickthrough rates than Mac and Windows users Dev 9.4% 28.1% 11.6% 22.0% combined for the Firefox malware warning, Firefox phish- Nightly 7.1% 54.8% 25.9% 20.4% ing warning, and Chrome phishing warning (\chi 2 tests: p(1) < 0.0001). While the low prevalence of malware Table 2: Release channel vs. clickthrough rates for malware and for Linux could explain the higher clickthrough rates for phishing warnings, for all operating systems. the Firefox malware warning, use of Linux does not provide any additional protection against phishing attacks. on the week, since the current version of the warning The Chrome malware warning does not follow the same was released in August 2012. In contrast, the Mozilla pattern: Windows users have a significantly higher click- Firefox malware warning clickthrough rate across weeks through rate (\chi 2 tests: p(1) < 0.0001). stays within one percentage point of the month-long We also see differences between software release average. We did not observe similar temporal variations channels (Table 2). Nightly users click through Google for phishing or SSL warnings. Chrome malware and Firefox phishing warnings at much Recall from Section 2.1 that Google Chrome and higher rates than stable users, although they click through Mozilla Firefox’s malware warnings differ with respect Firefox malware and Google Chrome phishing warnings to secondary resources: Google Chrome shows an at approximately the same rates. interstitial malware warning if a website includes In several cases, Linux users and early adopters click secondary resources from a domain on the Safe Browsing through malware and phishing warnings at higher rates. list, whereas Mozilla Firefox silently blocks the resource. One possible explanation is that a greater degree of tech- We believe that this makes Google Chrome’s malware nical skill – as indicated by use of Linux or early-adopter clickthrough rates more sensitive to the contents of the versions of browsers – corresponds to reduced risk aver- Safe Browsing list. For example, consider the case where sion and an increased willingness to click through warn- a well-known website accidentally loads an advertisement ings. This does not hold true for all categories and warn- from a malicious domain. Google Chrome would show ings (e.g., nightly and stable users click through the Fire- a warning, which users might not believe because they fox malware warning at the same rate), suggesting the trust the website. Mozilla Firefox users would not see need for further study. any warning. Furthermore, Chrome phishing warnings are less likely to be due to secondary resources, and that 5.1.4 Malware/Phishing Rates by Browser warning’s clickthrough rates do not vary much by time. Google Chrome stable users click through phishing warnings more often than Mozilla Firefox stable users. This 5.1.2 Malware/Phishing Rates by Warning Type holds true even when we account for differences in how the browsers treat iframes (Section 4.5). Mozilla Fire- In Mozilla Firefox, we find a significantly higher click- fox’s beta channel users still click through warnings at a through rate for phishing warnings than malware warn- lower rate when we exclude iframes: 9.6% for malware ings (\chi 2 test: p(1) < 0.0001). This behavior is rational: warnings, and 10.8% for phishing warnings. a malware website can infect the user’s computer without One possibility is that Mozilla Firefox’s warnings are any action on the user’s part, but a phishing website can more frightening or more convincing. Another possi- only cause harm by tricking the user at a later point in 3Google Chrome will display both warnings. To preserve inde- time. Mozilla Firefox makes this priority ordering explicit pendence, our measurement does not include any warnings with both by choosing to display the malware warning if a website phishing and malware error messages. Dual messages are infrequent. bility is that the browsers have different demographics Operating SSL Warnings with different levels of risk tolerance, which is reflected System Firefox Chrome in their clickthrough rates. There might be differences Windows 32.5% 71.1% in technical education, gender, socioeconomic status, or MacOS 39.3% 68.8% other factors that we cannot account for in this study. In Linux 58.7% 64.2% support of this theory, we find that differences between Android NC 64.6% the browsers do not hold steady across operating systems or channels. The gap between the browsers narrows or Table 3: User operating system vs. clickthrough rates for SSL warnings. The Google Chrome data is from the stable channel, reverses for some categories of users, such as Linux users and the Mozilla Firefox data is from the beta channel. and nightly release users. SSL Warnings 5.2 SSL Warnings Channel Firefox Chrome The clickthrough rates for SSL warnings were 33.0% and Release NC 70.2% 70.2% for Mozilla Firefox (beta channel) and Google Beta 32.2% 73.3% Chrome (stable channel), respectively. Dev 35.0% 75.9% 5.2.1 SSL Rates by Demographic Nightly 43.0% 74.0% In Section 5.1, we observed that malware and phishing Table 4: Channel vs. clickthrough rates for SSL warnings. clickthrough rates differed across operating systems and channels. For SSL, the differences are less pronounced. clicks in Mozilla Firefox also perform the third. This As with the malware and phishing warnings, nightly indicates that the extra click is not a determining deci- users click through SSL warnings at a higher rate for both sion point. Unfortunately, we do not have data on the 2 Firefox and Chrome (\chi tests: p < 0.0001). difference between the first and second clicks. The effect of users’ operating systems on SSL clickthrough rates differs for the two browsers. In Firefox, Warning Appearance. The two warnings differ in sev- Linux users are much more likely to click through SSL eral ways. Mozilla Firefox’s warning includes an image warnings than Windows and Mac users combined (\chi2 test: of a policeman and uses the word “untrusted” in the title. p < 0.0001), although it is worth noting that the Firefox These differences likely contribute to the rate gap. How- Linux sample size is quite small (58). In Chrome, Win- ever, we do not think warning appearance is the sole or dows users are very slightly more likely to click through primary factor; the browsers’ malware and phishing warn- SSL warnings than Linux and Mac users combined (\chi2 ings also differ, but there is only about a 10% difference test: p < 0.0001). between browsers for these warnings. 5.2.2 SSL Rates by Browser Certificate Pinning. Google Chrome ships with a list of “pinned” certificates and preloaded HTTP Strict Trans- We find a large difference between the Mozilla Firefox port Security (HSTS) sites. Users cannot click through and Google Chrome clickthrough rates: Google Chrome SSL warnings on sites protected by these features. Certifi- users are 2.1 times more likely to click through an SSL cate pinning and HSTS cover some websites with impor- warning than Mozilla Firefox users. We explore five tant private data such as Google, PayPal, and Twitter [8]. possible causes. In contrast, Mozilla Firefox does not come with many Number of Clicks. Google Chrome users click one but- preloaded “pinned” certificates or any pre-specified HSTS ton to dismiss an SSL warning, but Mozilla Firefox users sites. As a result, Chrome shows more non-bypassable need to click three buttons. It is possible that the addi- warnings: our field study found that 20% of all Google tional clicks deter people from clicking through. However, Chrome SSL warning impressions are non-bypassable, as we do not believe this is the cause of the rate gap. compared to 1% for Mozilla Firefox. First, the number of clicks does not appear to affect Based on this, we know that Mozilla Firefox users see the clickthrough rates for malware and phishing warn- more warnings for several critical websites. If we assume ings. Mozilla Firefox’s malware and phishing warnings that users are less likely to click through SSL warnings require one click to proceed, whereas Google Chrome’s on these critical websites, then it follows that Mozilla malware and phishing warnings require two. The Google Firefox’s clickthrough rate will be lower. This potential Chrome malware and phishing warnings with two clicks bias could account for up to 15 points of the 37-point gap do not have lower clickthrough rates than the Mozilla between the two clickthrough rates, if we were to assume Firefox warnings with one click. Second, as we discuss that Google Chrome users would never click through SSL in Section 5.2.3, 84% of users who perform the first two errors on critical websites if given the chance. Remembering Exceptions. Due to the “permanently Certificate Error Percentage Clickthrough store this exception” feature in Mozilla Firefox, Mozilla of Total Rate Firefox users see SSL warnings only for websites without Untrusted Issuer 56.0% 81.8% saved exceptions. This means that Mozilla Firefox users Name Mismatch 25.0% 62.8% might ultimately interact with websites with SSL errors Expired 17.6% 57.4% at the same rate as Google Chrome users despite having Other Error 1.4% – lower clickthrough rates. For example, imagine a user All Error Types 100.0% 70.2% that encounters two websites with erroneous SSL config- uration: she leaves the first after seeing a warning, but Table 5: Prevalence and clickthrough rates of error types for the visits the second website nine times despite the warning. Google Chrome SSL warning. Google Chrome only displays This user would have a 50% clickthrough rate in Mozilla the most critical warning; we list the error types in order, with Firefox but a 90% clickthrough rate in Google Chrome, untrusted issuer errors as the most critical. Data is for the stable despite visiting the second website at the same rate. channel across all operating systems. We did not measure how often people revisit websites with SSL errors. However, we suspect that people do errors appear on unimportant sites, leading to higher click- repeatedly visit sites with warnings (e.g., a favorite site through rates without user attention or comprehension; with a self-signed certificate). If future work were to however, the Mozilla Firefox data suggests otherwise. confirm this, there could be two implications. First, if An alternative explanation could be that expired certifi- users are repeatedly visiting the same websites with errors, cates, which often occur for websites with previously the errors are likely false positives; this would mean that valid certificates [1], surprise the user. In contrast, un- the lack of an exception-storing mechanism noticably trusted certificate errors always occur for a website and raises the false positive rate in Google Chrome. Second, conform with expectations. warning fatigue could be a factor. If Google Chrome users Mozilla Firefox. Mozilla Firefox’s SSL warning does not are exposed to more SSL warnings because they cannot inform the user about the particular SSL error by default.4 save exceptions, they might pay less attention to each Instead, the secondary “Add Exception” dialog presents warning that they encounter. all errors in the SSL certificate. The user must confirm Demographics. It’s possible that the browsers have dif- this dialog to proceed. ferent demographics with different levels of risk toler- Table6 presents the rates at which users confirm the ance. However, this factor likely only accounts for a “Add Exception” dialog in Mozilla Firefox. The error few percentage points because the same demographic ef- types do not greatly influence the exception confirmation fect applies to malware and phishing warnings, and the rate. This indicates that the “Add Exception” dialog does difference between browsers for malware and phishing not do an adequate job of explaining particular error cate- warnings is much smaller. gories and their meaning to the users. Thus, users ignore the categories and click through errors at the same rate. This finding also suggests that the differences in click- 5.2.3 SSL Rates by Certificate Error Type through rates across error types in Google Chrome cannot To gain insight into the factors that drive clickthrough be attributed to untrusted issuer errors corresponding to rates, we study whether the particular certificate error unimportant websites; if that were the case, we would affects user behavior. expect to see the same phenomenon in Firefox. Google Chrome. Google Chrome’s SSL warning in- Error Prevalence. The frequency of error types cludes a short explanation of the particular error, and encountered by users in our field study also indicates the clicking on “Help me understand” will open a more- base rate of SSL errors on the web. Our Google Chrome detailed explanation. In case a certificate has multiple data contradicts a previous network telemetry study, errors, Google Chrome only shows the first error out of which suggested that untrusted issuer errors correspond untrusted issuer error, name mismatch error, and certifi- to 80% of certificate errors seen on the wire [18]. Also, cate expiration error, respectively. Google Chrome users see fewer untrusted issuer errors Table5 presents the clickthrough rates by error types than Mozilla Firefox users; this may be because Mozilla for Google Chrome. If Google Chrome users are paying Firefox users are more likely to click on the “Add attention to and understanding the warnings, one would Exception” dialog for untrusted issuer errors. Recall that expect different clickthrough rates based on the warning we collect the Mozilla Firefox error type statistics only types. We find a 24.4-point difference between the click- after a user clicks on the “Add Exception” button. through rates for untrusted issuer errors and expired certifi- 4This information is available under the “Technical details” link, but cate errors. One explanation could be that untrusted issuer our measurements indicate that it is rarely opened (Section 5.2.4). Milliseconds Proceed Don’t Proceed 0 199774 70261 0.0186445131 0.021408557 400 5994 14352 0.0005594082 0.0043730606 473 10075 20206 0.0009402799 0.0061567769 559 23140 27289 0.0021596105 0.0083149701 660 80848 37879 0.0075453843 0.0115417476 780 282062 51828 0.0263242897 0.0157920139 922 680478 70520 0.0635076685 0.0214874744 1089 1128250 100089 0.1052973454 0.0304971614 1287 1363312 139722 0.1272352179 0.0425733536 1521 1318852 178562 0.1230858539 0.0544079183 1797 1119848 203459 0.1045132033 0.0619940449 2123 900719 212131 0.0840623263 0.0646364071 2508 711590 209503 0.0664112901 0.0638356543 2963 556981 200574 0.0519819373 0.0611149842 3501 434936 189304 0.0405917184 0.0576810103 4136 341933 176668 0.0319119319 0.0538308157 4887 270189 164303 0.025216206 0.0500631949 5774 213619 151076 0.0199366396 0.0460329223 6822 169926 138836 0.0158588582 0.042303389 8060 136820 126655 0.0127691405 0.0385918331 9523 111645 114267 0.0104196075 0.0348172041 11251 92590 103704 0.0086412419 0.0315986535 13293 77384 95096 0.007222096 0.0289757922 15705 65378 90998 0.0061015997 0.0277271298 18555 56332 86081 0.0052573544 0.0262289178 21922 47889 75766 0.0044693858 0.0230859328 25901 41403 66391 0.0038640603 0.0202293663 30602 35884 60753 0.0033489829 0.0185114653 36156 30733 55351 0.0028682502 0.0168654735 42718 26709 47934 0.0024926982 0.0146055104 50471 22757 43018 0.0021238659 0.0131076031 59631 19634 38047 0.0018324025 0.0115929373 70453 16508 32494 0.0015406591 0.0099009358 83239 13966 27680 0.0013034192 0.0084341079 98346 11882 24022 0.0011089236 0.0073195137 116194 10305 20621 0.0009617453 0.0062832276 137282 8998 17888 0.0008397656 0.0054504813 162197 7557 14961 0.0007052799 0.0045586231 191633 6680 13286 0.0006234312 0.0040482499 226412 5935 11925 0.0005539018 0.0036335526 267502 5211 10694 0.0004863323 0.0032584664 316050 4621 9884 0.0004312688 0.0030116591 373408 4094 8785 0.0003820849 0.0026767933 441176 3637 8363 0.000339434 0.0025482097 521243 3525 7824 0.0003289813 0.0023839762 615841 3125 7306 0.0002916501 0.0022261413 727607 2751 6811 0.0002567454 0.0020753146 859657 2625 6599 0.0002449861 0.0020107181 1015671 2358 6212 0.0002200675 0.0018927991 1200000 23433 78358 0.0021869556 0.0238757164

Chart 1 15% Certiﬁcate Error Percentage Conﬁrmation Proceed of Total Rate Don’t Proceed Untrusted Issuer 38% 87.1% 12.5% Untrusted and Name 26.4% 87.9% 10% Mismatch Name Mismatch 15.7% 80.3% 7.5% Expired 10.2% 80.7% Expired, Untrusted and 4.7% 87.6% 5% Name Mismatch 2.5% Expired and Untrusted 4.1% 83.6% Expired and Name Mis- 0.7% 85.2% 0% match 0 559 922 1521 2508 4136 6822

None of the above <0.1% 77.9% 11251 18555 30602 50471 All error types 100.0% 85.4% Figure 6: Google Chrome SSL clickthrough times (ms), by outcome. The graph shows the percent of warning impressions that Table 6: Prevalence and confirmation rates of error types for fall in each timing bucket. The x-axis increases logarithmically, the Mozilla Firefox “Add Exception” dialog. The confirmation and we cut off the distribution at 90% due to the long tail. rate measures the percentage of users who click on “Confirm Security Exception” (Figure 5). The Mozilla Firefox dialog lists all the errors that occur for a certificate. Data is for the release channel across all operating systems; we did not need to limit users proceed to the page after opening the “Add Excep- it to the beta channel because frame level issues do not affect tion” dialog: 14.6% of the time that a dialog is opened, clickthrough rates inside the “Add Exception” dialog. the user cancels the exception. These occurrences indicate that at least a minority of users consider the text in The high frequency of untrusted issuer errors high- the dialog before confirming the exception. lights the usability benefits of “network view” SSL cer- Remember Exception. By default, the “Remember Ex- tificate verification systems like Perspectives and Conver- ception” checkbox is checked in the Mozilla Firefox “Add gence [10,32], which do not need certificates from trusted Exception” dialog. Our measurements found that 21.3% authorities. All of the untrusted certificate warnings— of the time that the dialog is opened, the user un-ticks the between 38% and 56% of the total—would disappear. checkbox. We hypothesize that these users are still wary Warnings with other errors in addition to an untrusted cer- of the website even if they choose to proceed. tificate error would remain. Nonetheless, our study also shows that these mechanisms are not a panacea: name mismatch errors constitute a large fraction of errors, and 6 Time Spent On SSL Warnings new systems like Perspectives and Convergence still perform this check.5 In addition to MITM attacks, SSL warnings can occur due to server misconfigurations. Previous work found that 5.2.4 Additional SSL Metrics 20% of the thousand most popular SSL sites triggered a false warning due to such misconfigurations [30]. Con- We collected several additional metrics to complement sequently, it may be safe and rational to click through the overall clickthrough rates. such false warnings. The prevalence of a large number of More Information. Google Chrome and Mozilla Firefox such false warnings can potentially train users to consider both place additional information about the warning be- all SSL warnings false alarms and click through them hind links. However, very few users took the opportunity without considering the context. to view this extra information. The “Help me understand” In order to determine whether users examine SSL warn- button was clicked during 1.6% of Google Chrome SSL ings before making a decision, we measured how much warning impressions. For Mozilla Firefox warnings, 0 time people spent on SSL warning pages. In this section, users clicked on “Technical Details,” and 3% of viewers we compare the click times by outcome (clickthrough or of the “Add Exception” dialog clicked on “View Certifi- leave) and error type to gain insight into user attention. cate.” This additional content therefore has no meaningful Our timing data is for all operating systems and channels. impact on the overall clickthrough rates. Add Exception Cancellation. Not all Mozilla Firefox 6.1 Time by Outcome

5Convergence does not check the certiﬁcate issuer, relying on net- Figure 6 presents the click times for different outcomes. work views instead. However, it performs name checks [10]. Users who leave spend more time on the warning than Date Invalid Name Invalid Authority Untrusted Name Expired Chart 2 Invalid Issuer Mismatch 8% 0 29726 57841 181330 0.0195129206 0.0172819263 0.0186663887 Untrusted Issuer security warnings can be highly effective at preventing 400 1162 2575 8338 0.0008972521 0.0007693671 0.0007296758 7% Name Mismatch users from visiting websites: as few as a tenth of users Expired 445 1488 3179 10348 0.0011135482 0.0009498322 0.0009343869 6% click through Firefox’s malware and phishing warnings. 0.0014341184 0.0012937318 0.0012860379 We consider these warnings very successful. 495 2048 4330 13327 5% 550 2586 6129 17672 0.0019016838 0.001831243 0.0016238741 We found clickthrough rates of 18.0% and 23.2% for 4% 611 3679 9237 26943 0.0028993361 0.0027598616 0.0023102215 Google Chrome’s phishing and malware warnings, respectively, and 31.6% for Firefox’s SSL warning. These 679 5722 16407 49754 0.0053540277 0.004902138 0.0035931197 3% warnings prevent 70% (or more) of attempted visits to 755 9773 31289 98207 0.0105680549 0.0093486314 0.0061369379 2% potentially dangerous websites. Although these warnings 839 17250 56722 189550 0.0203974747 0.0169475877 0.0108321067 1% could be improved, we likewise consider these warnings 933 28694 91402 318968 0.0343241452 0.0273093934 0.0180183461 0% successful at persuading and protecting users. 1037 41621 131258 468412 0.0504058134 0.0392177016 0.0261358327 0 Google Chrome’s SSL warning had a clickthrough rate 550 839

1153 56998 167760 594135 0.063934865 0.0501238905 0.0357917925 1282 1957 2988 4560 6959

10621 16210 24741 37761 of 70.2%. Such a high clickthrough rate is undesirable: 1282 69097 190512 671696 0.0722812039 0.0569218087 0.0433893379 either users are not heeding valid warnings, or the browser 1425 78215 199358 691696 0.0744334038 0.059564846 0.0491149698 Figure 7: Google Chrome SSL clickthrough times (ms), by error is annoying users with invalid warnings and possibly caus- 1584 83027 196592 667097 0.0717863055 0.0587384113 0.0521366566 type. The graph shows the percent of warning impressions that ing warning fatigue. Our positive findings for the other 1761 82860 186615 604480 0.0650480904 0.0557574501 0.0520317893 fall in each timing bucket. The x-axis increases logarithmically, warnings demonstrate that this warning has the poten- and we cut off the distribution at 90% due to the long tail. 1957 80363 175240 536431 0.0577253378 0.0523587898 0.0504638026 tial for improvement. We hope that this study motivates 2175 76433 161351 471226 0.050708628 0.0482089882 0.0479959661 further studies to determine and address the cause of its 2418 72087 146299 412341 0.0443720134 0.0437117016 0.0452669031 users who click through and proceed to the page. 47% of higher clickthrough rate. We plan to test an exception- 2688 67063 132085 356626 0.0383765224 0.0394647954 0.0421120913 users who clicked through the warning made the decision remembering feature to investigate the influence of repeat 2988 61199 118225 310620 0.033425817 0.0353236585 0.0384298029 within 1.5s, whereas 47% of users who left the page did so exposures to warnings. At Google, we have also begun a 3321 55696 106502 271295 0.0291940539 0.0318210216 0.0349742039 within 3.5s. We interpret this to mean that users who click series of A/B tests in the field to measure the impact of a 3691 50183 97773 237722 0.0255812635 0.029212942 0.0315123254 through the warning often do so after less consideration. number of improvements. 4103 45687 87280 207645 0.0223446776 0.0260778086 0.0286890702 7.1.2 User Attention 4560 40904 81185 182721 0.0196626061 0.0242567242 0.0256855939 6.2 Time by Error Type 5068 36318 75563 160877 0.0173119733 0.0225769643 0.0228058233 Although we did not directly study user attention, two 5633 33073 70029 141367 0.0152125023 0.0209234974 0.0207681314 Figure 7 depicts the click times for three error types (un- results of our study suggest that at least a minority of 6261 29324 64795 124313 0.0133773214 0.0193596655 0.0184139535 trusted authority, name mismatch, and expired certificate users pay attention to browser security warnings. 6959 26578 60270 110218 0.0118605585 0.018007671 0.0166896077 errors). Users clicked through 49% of untrusted issuer • There is a 24.4-point difference between the click- 0.0105219978 0.0166326701 0.0151605538 warning impressions within 1.7s, but clicked through 50% 7735 24143 55668 97779 through rates for untrusted issuer errors (81.8%) and 8597 21750 52553 87571 0.009423515 0.01570196 0.0136578737 of name and date errors within 2.2s and 2.7s, respectively. We believe that this data is indicative of warning fatigue: expired certificate errors (57.4%) in Google Chrome. 9556 19838 48189 78589 0.008456962 0.0143980696 0.0124572367 users click through more-frequent errors more quickly. 10621 17971 44742 70654 0.0076030767 0.0133681635 0.0112848574 • 21.3% of the time that Mozilla Firefox users viewed The frequency and clickthrough rate of each error type 11805 16648 40896 64167 0.0069050106 0.0122190428 0.0104540819 the “Add Exception” dialog, they un-checked the (as reported in Section 5.2) are inversely correlated with 0.0063736325 0.0112748875 0.0098286455 default “Permanently store this exception” option. 13121 15652 37736 59229 that error type’s timing variance and mode (Figure 7). 14584 14865 34761 55095 0.0059287727 0.0103860071 0.0093344502 These results contradict the stereotype of wholly obliv- 0.0056016383 0.009642933 0.009058153 16210 14425 32274 52055 ious users with no interest in security. 18017 13620 29410 50314 0.0054142893 0.0087872176 0.0085526547 7 Implications 20026 13533 26744 47125 0.0050711211 0.0079906612 0.0084980232 7.2 Comparison with Prior Research Our primary finding is that browser security warnings can 22259 13308 23974 41911 0.0045100425 0.0071630314 0.0083567349 be effective security mechanisms in practice, but their As Bravo-Lillo et al. wrote [5]: 24741 13388 21045 38018 0.0040911168 0.0062878951 0.0084069707 effectiveness varies widely. This should motivate more 27499 13141 18499 34747 0.0037391245 0.0055271927 0.0082518675 Evidence from experimental studies indicates attention to improving security warnings. In this section, 0.0035273481 0.0048498509 0.0083975515 that most people don’t read computer warnings, 30565 13373 16232 32779 we summarize our findings and their implications, present 0.0032727428 0.0042749918 0.0084540668 don’t understand them, or simply don’t heed 33973 13463 14308 30413 suggestions for warning designers, and make recommen- them, even when the situation is clearly haz- 37761 13537 12399 28010 0.003014156 0.0037046144 0.008500535 dations for future warning studies. 41971 12989 11176 25100 0.0027010109 0.0033392024 0.0081564194 ardous. 46650 13014 9665 22125 0.0023808712 0.0028877408 0.0081721181 7.1 Warning Effectiveness In contrast, a majority of users heeded five of the six 51851 12640 8486 20483 0.0022041755 0.0025354753 0.0079372655 types of browser warnings that we studied. This section 57632 11783 7531 19297 0.0020765501 0.0022501372 0.0073991138 7.1.1 Clickthrough Rates explores why our results differ from prior research. 64058 10720 6725 17605 0.001894474 0.0020093179 0.0067316049 Popular opinion holds that browser security warnings are Browser Changes. Most prior browser research was con- 71200 9677 5985 15860 0.0017066945 0.0017882182 0.0060766549 ineffective. However, our study demonstrates that browser ducted between 2002 and 2009. Browsers were rapidly 79138 8442 5512 14535 0.0015641113 0.0016468937 0.0053011388 87961 7709 4859 13183 0.0014186226 0.0014517882 0.0048408528 97768 7036 4433 12065 0.0012983146 0.0013245065 0.0044182437 108669 6123 4100 10953 0.0011786523 0.0012250116 0.0038449269 120785 5295 3751 10454 0.0011249549 0.0011207362 0.0033249858 134252 4576 3417 9439 0.0010157308 0.0010209426 0.002873491 149220 4073 3169 8936 0.0009616029 0.0009468444 0.0025576331 165857 3315 2866 8059 0.000867229 0.000856313 0.0020816483 184349 2981 2658 7711 0.0008297807 0.0007941661 0.0018719136 204903 2619 2626 7004 0.0007537004 0.000784605 0.0016445964 227748 2275 2370 6777 0.0007292729 0.0007081165 0.0014285822 253140 2101 2174 6307 0.0006786962 0.0006495549 0.0013193192 281363 1817 2129 6049 0.0006509329 0.0006361097 0.0011409819 312733 1567 2080 5759 0.000619726 0.0006214693 0.0009839949 347601 1409 1840 5464 0.000587981 0.0005497613 0.000884779 386356 1258 1765 5189 0.0005583883 0.0005273526 0.0007899589 429432 1131 1654 4816 0.0005182497 0.0004941876 0.0007102094 477311 1085 1582 4793 0.0005157747 0.0004726752 0.0006813238 530528 1089 1558 4535 0.0004880113 0.0004655044 0.0006838356 589678 885 1532 4442 0.0004780036 0.0004577361 0.0005557342 655423 851 1416 4200 0.000451962 0.0004230772 0.0005343839 728498 766 1322 4037 0.0004344216 0.0003949916 0.0004810083 809721 741 1293 3829 0.0004120387 0.0003863268 0.0004653096 900000 11012 24265 81127 0.0087300761 0.0072499774 0.0069149658 changing during this time period; some changes were with specific demographics (e.g., students), it can help directly motivated by published user studies. Notably, understand the impact of these factors on user behavior. passive indicators are no longer considered primary secu- Similar studies could help understand SSL clickthrough rity tools, and phishing toolbars have been replaced with rates; recent work addressed how to reproduce certificate browser-provided, full-page interstitial warnings. As a re- validation at the network monitor [1]. sult, studies of passive indicators and phishing toolbars no longer represent the state of modern browser technology. 7.3 Demographics Two studies tested an older version of the Mozilla Fire- fox SSL warning, in which the warning was a modal We found that clickthrough rates differ by operating sys- (instead of full-page) dialog. Dhamija et al. observed tem and browser channel. Our findings suggest that higher a 68% clickthrough rate, and Sunshine et al. recorded technical skill (as indicated by use of Linux and pre- clickthrough rates of 90%-95% depending on the type of release channels) may predispose users to click through page [11, 30]. The change in warning design could be some types of warnings. We recommend further inves- responsible for our lower observed clickthrough rates. tigation of user demographics and their impact on user behavior. Large-scale demographic studies might uncover Ecological Invalidity. Sunshine et al. and Sotirakopoulos additional demographic factors that we were unable to et al. recorded 55%-60% and 80% clickthrough rates, re- study with our methodology. If so, can warning design spectively, for a slightly outdated version of the Mozilla address and overcome those demographic differences? Firefox SSL warning [29, 30]. They evaluated the Fire- Technically advanced users might feel more confident fox 3 and 3.5 warnings, which had the same layout and in the security of their computers, be more curious about appearance as the current (Firefox 4+) warning but with blocked websites, or feel patronized by warnings. Studies different wording. It’s possible that changes in word- of these users could help improve their warning responses. ing caused clickthrough rates to drop from 55%-80% to 33.0%. However, during an exit survey, 46% of Soti- rakopoulos’s subjects said they clicked through the warn- 7.4 Number of Clicks ing because they either felt safe in the laboratory envi- Our data suggests that the amount of effort (i.e., number ronment or wanted to complete the task [29]. Since their of clicks) required to bypass a warning does not always study methodology was intentionally similar to the Sun- have a large impact on user behavior. To bypass Google shine study, Sotirakopoulos et al. concluded that both Chrome’s malware and phishing warnings, the user must studies suffered from biases that raised their clickthrough click twice: once on a small “Advanced” link, and then rates [29]. We therefore attribute some of the discrepancy again to “proceed.” Despite the hidden button, users click between our field study data and these two laboratory through Google Chrome’s malware/phishing warning at a studies to the difficulty of establishing ecological validity higher rate than Mozilla Firefox’s simpler warning. Fur- in a laboratory environment. thermore, 84% of users who open Mozilla Firefox’s “Add In light of this, we recommend a renewed emphasis Exception” dialog proceed through it. on field techniques for running and confirming user stud- We find this result surprising. Common wisdom in ies of warnings. Although we used in-browser telemetry, e-commerce holds that extra clicks decrease clickthrough there are other ways of obtaining field data. For exam- rates (hence, one-click shopping) [12, 31]. Google ple, experience sampling is a field study methodology Chrome’s warning designers introduced the extra step in that asks participants to periodically answer questions the malware/phishing warning because they expected it about a topic [2,6,9, 27]. Researchers could install a to serve as a strong deterrent. One possible explanation is browser extension on participants’ computers to observe that users make a single cognitive decision when faced their responses to normally occurring warnings and dis- with a warning. The decision might be based on the URL, play a survey after each warning. This technique allows warning appearance, or warning message. Once the user researchers to collect data about participants’ emotions, has decided to proceed, additional clicks or information comprehension, and demographics. Participants may be- is unlikely to change his or her decision. come more cautious or attentive to warnings if the pur- Our data suggests that browser-warning designers pose of the study is apparent, so researchers could obscure should not rely on extra clicks to deter users. However, the purpose by surveying subjects about other browser we did not explicitly design our study to examine the topics. Network-based field measurements also provide effects of multiple clicks. Future studies on multi-click an alternative methodology with high ecological validity. warnings could shed light on user decision models and A network monitor could maintain its own copy of the impact security warning design. It is possible that extra Safe Browsing list and identify users who click through clicks do not serve as a deterrent until they reach some warnings. If the monitor can associate network flows threshold of difficulty. 7.5 Warning Fatigue Acknowledgements

We observed behavior that is consistent with the theory of We thank the participants in Google and Mozilla’s teleme- warning fatigue. In Google Chrome, users click through try programs for providing us with valuable insight into the most common SSL error faster and more frequently our warnings. At Google, we would like to thank Matt than other errors. Our findings support recent literature Mueller for setting up the malware and phishing measure- that has modeled user attention to security warnings as ments, Adam Langley for making suggestions about how a finite resource [4] and proposed warning mechanisms to implement SSL measurements, and many others for based on this constraint [14]. providing insightful feedback. At Mozilla, we would like Based on this finding, we echo the recommendation to thank Sid Stamm for his mentorship and help collecting that security practitioners should limit the number of warn- telemetry data, Dan Veditz for gathering data from Fire- ings that users encounter. Designers of new warning fox 23, Brian Smith for providing information about the mechanisms should always perform an analysis of the telemetry mechanisms, and the Mozilla contributors who number of times the system is projected to raise a warn- reviewed our code and helped land this telemetry [22]. ing, and security practitioners should consider the effects We also thank David Wagner, Vern Paxson, Serge Egel- that warning architectures have on warning fatigue. man, Stuart Schechter, and the anonymous reviewers for providing feedback on drafts of the paper.

References 7.6 “More Information” [1] AKHAWE,D.,AMANN,B.,VALLENTIN,M., AND SOMMER, R. Here’s My Cert, So Trust Me, Maybe? Understanding TLS Users rarely click on the explanatory links such as “More Errors on the Web. In Proceedings of the 2013 World Wide Web Information” or “Learn More” (Section 5.2.4). Designers Conference (2013). who utilize such links should ensure that they do not hide [2] BEN ABDESSLEM, F., PARRIS,I., AND HENDERSON, T. Mobile a detail that is important to the decision-making process. Experience Sampling: Reaching the Parts of Facebook Other Methods Cannot Reach. In Privacy and Usability Methods Pow- Mozilla Firefox places information about SSL errors wow (2010). under “Technical Details” and in the “Add Exception” [3] BIDDLE,R., VAN OORSCHOT, P. C., PATRICK,A.S.,SOBEY, dialog instead of the primary warning. Thus, the error J., AND WHALEN, T. Browser interfaces and extended validation type has little impact on clickthrough rates. In contrast, SSL certificates: an empirical study. In Proceedings of the ACM Workshop on Cloud Computing Security Google Chrome places error details in the main text of (2009). its SSL warning, and the error has a large effect on user [4] BOHME¨ ,R., AND GROSSKLAGS,J. The Security Cost of Cheap User Interaction. In Proceedings of the New Security Paradigms behavior. It is possible that moving this information into Workshop (NSPW) (2011). Mozilla Firefox’s primary warning could reduce their [5] BRAVO-LILLO,C.,CRANOR, L. F., DOWNS,J.S., AND KO- clickthrough rates even further for some errors. MANDURI,S. Bridging the Gap in Computer Security Warnings: A Mental Model Approach. In IEEE Security and Privacy (March 2011), vol. 9. [6] CHRISTENSEN, T., BARRETT,L.,BLISS-MOREAU,E.,LEBO, 8 Conclusion K., AND KASCHUB,C. A Practical Guide to Experience- Sampling Procedures. In Journal of Happiness Studies (2003), vol. 4. We performed a field study with Google Chrome and [7] Google Chrome Privacy Notice. http://www.google.com/ Mozilla Firefox’s telemetry platforms, allowing us to col- chrome/intl/en/privacy.html. lect data on 25,405,944 warning impressions. We find [8] CHROMIUM AUTHORS. HSTS Preload and Certificate Pinning that browser security warnings can be successful: users List. https://src.chromium.org/viewvc/chrome/ trunk/src/net/base/transport_security_ clicked through fewer than a quarter of both browser’s state_static.json. malware and phishing warnings and a third of Mozilla [9] CONSOLVO,S., AND WALKER,M. Using the Experience Sam- Firefox’s SSL warnings. We also find clickthrough rates pling Method to Evaluate Ubicomp Applications. In Pervasive as high as 70.2% for Google Chrome SSL warnings, in- Computing (2003). dicating that the user experience of a warning can have a [10] Convergence. http://www.convergence.io. tremendous impact on user behavior. However, warning [11] DHAMIJA,R.,TYGAR,J.D., AND HEARST,M. Why phishing effectiveness varies between demographic groups. Our works. In Proceedings of the SIGCHI Conference on Human findings motivate more work on browser security warn- Factors in Computing Systems (2006). ings, with particular attention paid to demographics. At [12] DUTTA,R.,JARVENPAA,S., AND TOMAK,K. Impact of Feed- back and Usability of Online Payment Processes on Consumer Google, we have begun experimenting with new warning Decision Making. In Proceedings of the International Conference designs to further improve our warnings. on Information Systems (2003). [13] EGELMAN,S.,CRANOR, L. F., AND HONG,J. You’ve Been [30] SUNSHINE,J.,EGELMAN,S.,ALMUHIMEDI,H.,ATRI,N., Warned: An Empirical Study of the Effectiveness of Web Browser AND CRANOR, L. F. Crying Wolf: An Empirical Study of SSL Phishing Warnings. In Proceedings of the ACM CHI Conference Warning Effectiveness. In Proceedings of the USENIX Security on Human Factors in Computing Systems (2008). Symposium (2009).

[14] FELT, A. P., EGELMAN,S.,FINIFTER,M.,AKHAWE,D., AND [31] TILSON,R.,DONG,J.,MARTIN,S., AND KIEKE,E. Factors WAGNER,D. How to ask for permission. In Proceedings of the and Principles Affecting the Usability of Four E-commerce Sites. USENIX Conference on Hot Topics in Security (HotSec) (2012). In Our Global Community Conference Proceedings (1998). [32] WENDLANDT,D.,ANDERSEN,D.G., AND PERRIG,A. Perspec- [15]F RIEDMAN,B.,HURLEY,D.,HOWE,D.C.,FELTEN,E., AND tives: Improving SSH-style Host Authentication with Multi-Path NISSENBAUM,H. Users’ Conceptions of Web Security: A Com- parative Study. In CHI Extended Abstracts on Human Factors in Probing. In USENIX Annual Technical Conference (2008). Computing Systems (2002). [33] WHALEN, T., AND INKPEN,K.M. Gathering evidence: Use of visual security cues in web browsers. In Proceedings of the [16] HABER,J. Smartscreen application reputa- Graphics Interface Conference (2005). tion in ie9, May 2011. http://blogs. msdn.com/b/ie/archive/2011/05/17/ [34] WU,M.,MILLER,R.C., AND GARFINKEL,S.L. Do Security smartscreen-174-application-reputation-in-ie9. Toolbars Actually Prevent Phishing Attacks? In Proceedings of aspx. the SIGCHI Conference on Human Factors in Computing Systems (2006). [17] HERLEY,C. The plight of the targeted attacker in a world of scale. In Proceedings of the Workshop on the Economics of Information Security (WEIS) (2010). A Sample Sizes [18] HOLZ,R.,BRAUN,L.,KAMMENHUBER,N., AND CARLE,G. The ssl landscape: a thorough analysis of the x.509 pki using active and passive measurements. In Proceedings of the ACM Malware Phish- SSL Add SIGCOMM Internet Measurement Conference (IMC) (2011). ing Exception [19]J ACKSON,C.,SIMON,D.R.,TAN,D.S., AND BARTH, A. An Release 1,968,707 89,948 NC 1,805,928 evaluation of extended validation and picture-in-picture phishing attacks. In Proceedings of the Workshop on Usable Security Beta 74,782 3,058 10,976 66,694 (USEC) (2007). Dev 61,588 2,759 15,560 53,001

[20] LANGLEY,A. SSL Interstitial Bypass Rates, February Nightly 58,789 4,239 18,617 64,725 2012. http://www.imperialviolet.org/2012/07/ 20/sslbypassrates.html. Table 7: Warning impression sample sizes for Mozilla Firefox [21] MCGRAW,G.,FELTEN,E., AND MACMICHAEL,R. Securing warnings, by channel, for all operating systems. Java: getting down to business with mobile code. Wiley Computer Pub., 1999.

[22] MOZILLA BUGZILLA. Bug 767676: Implement Security UI Malware Phish- SSL Add Telemetry. https://bugzil.la/767676. ing Exception [23] Mozilla ﬁrefox privacy policy. http://www.mozilla.org/ Mac 71,371 3,951 534 154,129 en-US/legal/privacy/firefox.html#telemetry. Win 1,892,285 85,598 10384 1,634,193 [24] NETCRAFT. Phishing on sites using ssl cer- Linux 1,750 112 58 17,606 tiﬁcates, August 2012. http://news. netcraft.com/archives/2012/08/22/ Table 8: Warning impression sample sizes for Mozilla Firefox phishing-on-sites-using-ssl-certificates. warnings, by operating system. The malware, phishing, and the html. “Add Exception” samples are from the release channel, whereas [25] PROVOS,N. Safe Browsing - Protecting Web Users for 5 the SSL samples are from the beta channel. The frame issue Years and Counting. Google Online Security Blog. http: //googleonlinesecurity.blogspot.com/2012/ does not affect statistics that pertain only to the “Add Exception” 06/safe-browsing-protecting-web-users-for. dialog. html, June 2012.

[26] SCHECHTER,S.E.,DHAMIJA,R.,OZMENT,A., AND FISCHER, I. The Emperor’s New Security Indicators. In Proceedings of the Malware Phishing SSL IEEE Symposium on Security and Privacy (2007). Stable 5,946,057 381,027 16,363,048 [27] SCOLLON,C.N.,KIM-PRIETO,C., AND DIENER,E. Experi- Beta 44,742 3,525 232,676 ence Sampling: Promises and Pitfalls, Strengths and Weaknesses. Dev 14,022 1,186 66,922 In Journal of Happiness Studies (2003), vol. 4. Canary 35,261 612 42,020 [28] SOBEY,J.,BIDDLE,R., VAN OORSCHOT, P., AND PATRICK, A.S. Exploring user reactions to new browser cues for extended Table 9: Warning impression sample sizes for Google Chrome validation certiﬁcates. In Proceedings of the European Symposium warnings, by channel, for all operating systems. on Research in Computer Security (2008). [29] SOTIRAKOPOULOS,A.,HAWKEY,K., AND BEZNOSOV,K. On In Google Chrome, we recorded 6,040,082 malware the Challenges in Usable Security Lab Studies: Lessons Learned from Replicating a Study on SSL Warnings. In Proceedings of the warning impressions, 386,350 phishing warning impres- Symposium on Usable Privacy and Security (2011). sions, and 16,704,666 SSL warning impressions. In Malware Phishing SSL Mac 598,680 20,623 947,971 Windows 9,775,104 333,522 13,399,820 Linux 15,456 577 515,319 Android NC NC 1,499,938

Table 10: Warning impression sample sizes for Google Chrome warnings, by operating system, for the stable channel.

Mozilla Firefox, we recorded 2,163,866 malware warning impressions, 100,004 phishing warning impressions, and 45,153 SSL warning impressions. Tables7,8,9, and 10 further separate the sample sizes based on OS and release channel.