How People Habituate to Mobile Security Warnings in Daily Life: A Longitudinal Field Study Jeff Jenkins, Brock Kirwan, Daniel Bjornn, Bonnie Brinton Anderson, Anthony Vance Brigham Young University {jeffrey_jenkins, kirwan, dbjornn, bonnie_anderson, anthony.vance}@byu.edu Abstract Second, past studies did not examine how habitua- tion influences actual warning adherence behavior Research in the fields of information security and in the field but instead used laboratory experiments human–computer interaction has shown that habit- that presented unrealistically high numbers of uation—decreased response to repeated stimula- warnings to participants in a short session. Because tion—is a serious threat to the effectiveness of se- users typically receive security warnings infre- curity warnings. Although habituation is a phe- quently, presenting an artificially high number of nomenon that develops over time, past studies have warnings in a short time is too far removed from only examined this problem cross-sectionally. Fur- real life to be ecologically valid [24]. Consequent- ther, past studies have not examined how habitua- ly, for these reasons, the full extent of the problem tion influences actual security warning behavior in of habituation is unknown. the field. For these reasons, the full extent of the Third, previous research [3; 4] proposed that re- problem is unknown. peatedly updating the appearance of a warning (i.e., We addressed these gaps by conducting a three- a polymorphic warning design) can be effective in week field experiment in which users were natural- reducing habituation. However, their findings were ly exposed to privacy permission warnings as they subject to the same limitations above. Therefore, it installed apps on their mobile devices. We found is not clear (1) whether polymorphic warnings are that (1) users’ warning adherence substantially de- effective over time or if users will quickly learn to creased over the three weeks, validating previous ignore them and (2) whether the polymorphic de- cross-sectional studies, (2) the general decline in sign can actually lead to better security warning warning adherence was partially offset by a recov- behavior. ery effect—a key characteristic of habituation— We address these gaps in this paper by presenting when permission warnings were not displayed be- the results of a longitudinal three-week field exper- tween days, and (3) for users who received poly- iment in which users were naturally exposed to morphic permission warnings—warnings that up- privacy permission warnings as they installed apps date their appearance with each repeated expo- on their mobile devices. Consistent with previous sure—adherence dropped at a substantially lower cross-sectional experimental results, users’ warning rate and remained high after three weeks compared adherence behavior substantially decreased over to users who received standard warnings. the three weeks. However, for users who received These findings provide the most complete view yet polymorphic permission warnings, adherence of the problem of habituation to security warnings dropped at a substantially lower rate and remained and demonstrate that polymorphic warnings can high after three weeks compared to users who re- substantially improve warning adherence behavior. ceived standard warnings. Together, these findings Keywords: habituation, security warning, longitu- provide the most complete view yet of the problem dinal field experiment, mobile devices. of habituation to security warnings and demon- strate that polymorphic warnings can substantially 1. Introduction improve warning adherence behavior. Research in the fields of information systems and 2. Literature Review and Theory human–computer interaction has shown that habit- uation—“decreased response to repeated stimula- Habituation has been identified as a key contributor tion” [26, p.419]—is a serious threat to the effec- to the failure of warnings [14; 19; 20]. Several re- tiveness of security warnings. However, past stud- searchers have inferred warning habituation in ies share three critical limitations. First, they only cross-sectional laboratory experiments [2-4; 8; 11; examined habituation cross-sectionally (see Table 12; 15; 19; 23; 25]. In addition, two studies sup- 1). This is a substantial limitation, because habitua- ported cross-sectional habituation in warning ad- tion is a phenomenon that develops over time [17]. herence behavior using Amazon Mechanical Turk Furthermore, a key characteristic of habituation is [6; 7]. While these studies provide important in- recovery—the increase of a response after a rest sights into the problem of habituation to security period in which the stimulus is absent [17]. With- warnings, they share a fundamental limitation: they out a longitudinal design, it is not possible to exam- only examine a single point in time. However, in ine whether recovery can sufficiently counteract the fields of neuroscience and neurobiology, it is the effect of habituation to warnings. well recognized that the effects of habituation

change over time [17]. For this reason, cross- previously lost because of habituation” [28, p. 55]. sectional studies can only provide a partial view of Changing the appearance of a warning creates nov- the effects of habituation. For example, the two elty, and the warning will therefore be less similar most prevalent characteristics of habituation are (1) to existing mental models. As a result of this dis- response decrement—an attenuation of a response similarity, the response strength will recover [22]. after multiple exposures—and (2) response recov- DPT describes this as sensitization, an energizing ery—the increase of a response after a rest period process that strengthens attention [13]. Sensitiza- in which the stimulus is absent [17]. Without a lon- tion counterbalances or decreases habituation [17]. gitudinal design, it is not possible to observe how As a result of sensitization, users will pay closer (or whether) users recover from habituation to attention to warnings and reject risky permissions warnings between exposures. For this reason, it is more accurately. not clear from previous cross-sectional research H2: Users’ accuracy in rejecting risky permission whether response recovery can offset the negative warnings over time will decrease more slowly when impact of response decrements observed in previ- viewing polymorphic warnings as compared to ous habituation research. static warnings. Hypotheses 1 and 2 explore how users become less accurate in rejecting risky permissions over time 2.2 Response Recovery with repeated viewings and how polymorphic Although users will habituate to warnings, we pre- warnings can mitigate this effect. Hypotheses 3 and dict that they will partially recover from the habitu- 4 explore how users’ responses to warnings recover ation after a rest period. Decay theory [5] explains after the warning is withheld and how polymorphic that memory becomes weaker due to the passage of warnings enhance this recovery. time. When a warning is withheld for some time, the mental model of the warning weakens. There- 2.1 Response Decrement fore, when users see a warning in the future, it will We first hypothesize that users’ accuracy in reject- be less likely to match the mental model and will ing risky app permissions will decrease when view- appear novel, increasing sensitization and users’ ing multiple warnings across days. Dual-process attention to the warning [9]. This time between theory (DPT) [13] states that when users see a re- warnings should thus result in an increase in accu- peated stimulus, they compare it to a mental model racy in rejecting risky permissions. of that stimulus. If the two match, users evaluate H3: Time between warnings will improve users’ the actual stimulus less carefully and rely on the accuracy in rejecting risky permission warnings. mental model instead. This is referred to as a “re- sponse decrement,” and may result in paying less We predict that the amount of recovery after a time attention and responding less thoughtfully to the period will be greater for polymorphic warnings stimulus. In the context of mobile app permission than for static warnings. As previously discussed, warnings, users will unconsciously compare warn- the mental models of polymorphic warnings are ings to their mental model of warnings they have weaker and less stable than the models of static seen when previously downloading other apps. If warnings. Less stable mental models (i.e., mental users determine that a warning is similar to the models that have not received as much reinforce- mental model (even if, in fact, it lists different per- ment) fade more quickly than stable models [17]. missions), they will give it less attention. In future Thus, after users have not seen a warning for a time exposures, users will rely even more on the model period, they are more likely to perceive the poly- and respond even less thoughtfully. As a result, morphic warning as novel. As a result, the respons- users who view similar permission warnings over es of users who view polymorphic warnings will time will give less attention to them, and habitua- recover to a greater degree than the responses of tion will inhibit the ability to identify and reject users who view static warnings. risky permission warnings. H4: Users’ accuracy in rejecting risky permission H1: Multiple exposures to permission warnings warnings over time will increase more after a over time will decrease users’ accuracy in rejecting withholding time period when viewing polymorphic risky permission warnings. warnings as compared to static warnings. We hypothesize that users will habituate more slowly to polymorphic permission warnings— 3. Experimental Design permission warnings that change their appearance with repeated exposures [2]—than to static permis- 3.1 Motivation sion warnings. Wogalter states that “habituation To test our hypotheses in an ecologically valid con- can occur even with well-designed warnings… text, we examined longitudinal habituation to mo- Where feasible, changing the warning’s appearance bile app permission warnings. Users see many noti- may be useful in reinvigorating attention switch fications on mobile devices daily. An analysis of 40,191 Android users suggests that users encounter 134 subjects, 26 failed to participate past the first an average of nearly 100 notifications per day on week, so we had 108 valid responses. These sub- their mobile phones (app notifications, email noti- jects were 63% male and had average age of 21.9 fications, system notifications, etc.) [21]. years (SD 2 years). In addition to the extra credit, and to encourage them to continue participating in A subset of these notifications reflects app permis- the study, participants were given $10 for complet- sion warnings (i.e., warnings that are shown before ing the first week, $10 for completing the second an app is granted access to information or re- week, and an additional $20 if they completed all sources). These warnings can be shown when the days in the third week, for a total of $40. app is downloaded or when the app attempts to access a resource (i.e., just-in-time warnings). Per- 3.3 Ethics mission warnings are frequent. In 2015, 25 billion iOS apps and 50 billion Android apps were down- The university’s Institutional Review Board ap- loaded by smartphone users. The average Android proved the deception protocol described below. user has 95 apps installed on his or her mobile de- After the experiment, participants were debriefed vice, most of which displayed a permission warn- on its true purpose. ing during installation or use [18]. Furthermore, users often see multiple permission warnings in a 3.4 Study Design short period of time during an interaction. For ex- Participants were asked to rank apps on an app ample, when configuring a new phone, people may store created specifically for this study (see Figure download many apps (and see many warnings) in a 1). This operated as a legitimate app store, and par- short period of time. When using apps with just-in- ticipants were unaware that it was created and time warnings, it is typical for the user to see a se- managed by the research team for the purpose of ries of separate permission requests when first the experiment. The study used a deception proto- opening the app. Furthermore, when evaluating col to increase realism. The store was presented as apps, it is common for people to download multiple a third-party app store not affiliated with the re- apps in a short period of time. Mobile apps there- search team. We told participants that the purpose fore represent a realistic scenario where people of our study was to observe how people rank An- frequently see a given warning (permission warn- droid apps in various categories. ing), and thus this is an appropriate context for studying longitudinal habituation to security mes- The app store presented apps from a different cate- sages. gory (e.g., utilities, education, entertainment, trav- el, finance) each day. Participants were instructed In this experiment, we asked participants to evalu- to download, install, and evaluate three apps within ate apps at a third-party Android app store. Third- the daily category on their personal Android device party app stores are common on the Android plat- and rank each app from 1 (best) to 3 (worst). Par- form (e.g., Amazon Underground, Getjar, Mob- ticipants then completed an apparently unaffiliated ogenie, Slideme, Appbrain, Aptoide Cloud Store, daily survey from the research team that allowed BAM, Top Apps, AppGratis, Myapp, MIUI, Baidu, them to report their results. These steps were re- and F-Droid). Some of these compete with the peated each day for three weeks (excluding week- Google Play app store by offering app specials ends). (e.g., free or reduced-price apps) and serve markets that have restricted access to Google Play (e.g., GetJar in China). Others complement Google Play by providing customized experiences (e.g., app of the day, in-depth app reviews, categorized apps, or recommended apps) with apps that link directly to Google Play (e.g., AppGratis). Some of these stores are standalone apps that can be downloaded (e.g., Amazon Underground), while others must be ac- cessed via a web browser on the Android phone (e.g., Mobogenie). In our experiment, we created a browser-based third-party app store and monitored participants’ responses to permission warnings across time.

3.2 Participants Participants were students from a variety of majors recruited at a university in the western United Figure 1. A screenshot of the web-based app store States. They received course credit for their partic- created for the field experiment. ipation in the experiment. Of an initial group of When participants clicked to download an app, phone. Not only is your own device at risk if they were shown a warning listing the app’s per- you install these apps, but if you positively re- missions. These permissions were randomly drawn view these apps, it will also put future users at from two categories: safe and risky (see Table 1). risk. Therefore, if you review too many apps Safe permissions were taken from the Android De- with dangerous permissions, you may not re- veloper Guide [1] and were selected because we ceive the course credit for this experiment.” determined participants would consider these to be Before starting the experiment, participants were low-risk across app categories. We also created required to pass a short quiz verifying they knew four risky permissions to (1) heighten respondents’ which permissions were considered risky by the perception of risk in ignoring the permission warn- researchers. This allowed for an objective measure ing and installing the app and (2) ensure that the of security behavior—whether people knowingly requested permission was not appropriate, regard- installed apps with these risky permissions. After less of the type of app that was being downloaded. participants completed the quiz, we provided the mobile URLs for the app store and the separate Safe permissions daily survey. In addition, we sent two daily re- Send notifications. Set an alarm. minders in the morning and evening telling partici- pants to download and review three apps from the Pair with Bluetooth Alter the phone’s time app store. devices. zone. We only allowed participants to submit one catego- Change the size of Change the phone’s dis- ry evaluation each day (we told them that this was the status bar. played wallpaper. a feature of the app store). This required them to Install shortcut icons. Uninstall shortcut icons. view a realistic number of permission warnings Connect to the inter- Use vibration for notifica- each day and reduced the effects of fatigue. Fur- net. tions or interactions. thermore, participants had to evaluate a new cate- gory each day, ensuring they did not interact with Change phone vol- Temporarily prevent the the same app twice. Apps in each category were ume and audio set- phone from sleeping (for not well known, reducing the likelihood that partic- tings. viewing videos). ipants trusted a particular brand. Ask permission to download additional Each category had ten apps to choose from. When features. the “Download” button on an app was clicked, the app store displayed a permission warning (see Fig- Risky permissions ure 2, leftmost screenshot). If the user accepted the Charge purchases to Record microphone audio permission warning, the app installation was com- user’s credit card. at any time. pleted through Google Play—a pattern shared by Delete photos. Sell web-browsing data. many of the “customized experience” app stores (e.g., Mobile App Store, Cloud Store, BAM, Top Table 1. Safe and risky permissions displayed in Apps, AppGratis). Participants did not see a per- the app store permission warnings. mission warning again through Google Play when As a second deception, although the research team the app was downloaded. did in fact control the apps and validated their secu- rity, participants were told: 3.4.1 Dependent Variable “Be aware that the research team is not affili- Our dependent variable was whether or not partici- ated with App-Review.org in any way, so we pants rejected apps with risky permissions. We cannot verify that the apps are all safe. Before randomized the permissions for each warning to you download an app, be sure to check the ensure that participants would encounter at least permissions that the app requires. This app one risky permission among the first three apps store displays the permissions before directing they selected. Each app beyond the first three had a you to the Google Play store. 50% chance of displaying a risky permission. We recorded whether or not participants ignored warn- Make sure that the permissions required by the ings containing a risky permission. app do not contain any of the following: • Charge purchases to your credit card. 3.4.2 Manipulations • Delete your photos. We implemented a between-subject study design • Record microphone audio any time. for our manipulations. Participants were randomly • Sell your web-browsing data. assigned to either the static warning or polymor- If the app has any of these permissions, DO phic warning conditions when they first visited the NOT download it. These apps are potentially app store. Users were required to log in to the app dangerous and can harm your privacy and/or store, which ensured they had the same condition across all three weeks. The static warning condition always had the same look and feel for the duration of the experiment, although the requested permis- sions changed. In contrast, the polymorphic warn- ing condition randomly changed the appearance of the permission warning each time it was shown. We created 16 variations of the polymorphic warn- ing; half of these involved animations and half did not. Furthermore, for each participant, we random- ly iterated through four polymorphic warning var- iations per week, with a new set of four variations introduced every four days. This was done to main- tain the novelty of the polymorphic treatment. We deliberately set the interval for changing the poly- morphic versions at the fourth day of each set so we would be able to detect if the level of habitua- tion changed due to the warning treatments or merely due to the weekend and time away from the task. Examples of static and polymorphic warnings are shown in Figures 2 and 3.

Figure 3. Three stages of “flip,” one of eight animated polymorphic warning variations.

3.4.3 Daily Survey After downloading and installing three apps, partic- ipants completed a survey from the research team. The survey asked participants to list and rank the three apps that they downloaded from #1 (best) to #3 (worst) for the daily category. We deliberately branded the survey as coming from the researchers and not the app store, so it would appear to be a part of a separate research project. This helped promote the story that the app store was not affili- ated with the researchers and therefore could in- deed have risky apps. To ensure that participants actually downloaded the apps, we enabled a “shar- ing” feature in the app store through which partici- pants shared which apps they downloaded with us via the app store website. Although we actually captured all behavioral data in the app store regard- less of this functionality, this again strengthened

Figure 2. Sample static and polymorphic permission the idea that the app store was not associated with warnings. The top left warning shows the appearance the research team. of the warning in the static condition. The other 3 warnings represent examples of the 16 variations 3.4.4 Debriefing Survey used in the polymorphic condition. At the end of the three-week experiment, partici- pants received a debrief survey. In the survey, we asked participants “How concerned were you about The results are shown in Table 2. The warning each of the following permissions?” for the safe number negatively predicted whether the user re- and risky permissions listed in Table 1. The re- jected the app with the risky permission (β = - sponse scale was 1, “Not at all concerned,” to 7, 0.028, p < .001). Thus, H1 was supported. Like- “Extremely concerned.” The average for risky wise, the interaction between the warning number permissions was 6.03, while the average for safe and treatment was significant (β = 0.013, p < .01). permissions was 1.97, a significant difference (t = - Participants’ accuracy in rejecting risky permission 38.9, df = 653.9, p < .001). This indicated that par- warnings decreased more slowly when viewing ticipants did see a difference in concern levels for polymorphic warnings compared to static warn- the two categories of permissions. We also asked a ings, supporting H2. The withholding time posi- manipulation check question to ensure that partici- tively influenced accuracy, supporting H3 (β = pants in the experimental condition noticed the 0.419, p < .05). However, the interaction between polymorphic treatment [24]. All participants in the withholding time and polymorphic treatment was polymorphic condition responded affirmatively. not significant, and H4 was not supported (β = 0.506, p > .05). 4. Analysis Estimate Std. z-value We limited our data to participants who completed Error at least half (seven or more) days of the experi- Intercept 2.122 0.209 10.145*** ment. This resulted in 102 participants—55 in the static condition and 47 in the polymorphic condi- Warning -0.028 0.003 -9.414*** tion—who viewed 7,248 warnings over three Number weeks or 15 weekdays. This averaged 4.74 (SD = 1.29) permission warnings viewed per weekday, Polymorphic 0.323 0.324 0.997 (ns) per participant. Of these, 2,695 (about one third) Treatment were apps with risky permissions. Thus, the N for Withholding 0.419 0.183 2.294* our analysis was 2,695. Time To analyze our data, we specified a logistic linear Warning 0.013 0.005 2.679** mixed-effects model, because it is robust to uneven Number × observations [10]. In other words, the analysis was Polymorphic robust even if participants saw a different number Treatment of warnings each day. Linear mixed-effects model- Withholding 0.506 0.293 -1.726 (ns) ing also allows for the inclusion of fixed effects Time × Poly- (observations that are treated as non-random or morphic non-independent) and random effects (observations Treatment that are treated as random or independent). Thus, we accounted for the within-subject nature of our * p < .05; ** p < .01; *** p < .001; (ns) = not significant experiment by including the participant identifier Table 2. Logistic mixed-effects model results predict- as a random effect. Finally, the logistic linear ing participants’ rejection of apps with risky permis- mixed-effects model was designed to handle binary sions. dependent variables, such as in our case [16]. To explore the extent of the interaction between the warning number and treatment, we graphed the To test H1, we included the warning number (how trends. Figure 4 displays how each treatment many warnings the participant had seen up to that group’s accuracy rate (percentage correct in reject- point) as a fixed effect to measure the stimulus ing risky apps) changed over the three-week (15- repetition. We also included the treatment as a bi- day) experiment as well as presenting trend lines nary fixed effect (1 = polymorphic warnings, 0 = fitted to the data. Interestingly, after the three static warnings). We then included an interaction weeks, the accuracy rate of participants in the poly- effect between the warning number and treatment morphic condition was 76%, whereas the accuracy to test H2. To test H3, we included the time be- of participants in the static condition was 55%. tween seeing the last warning (i.e., withholding This difference of 21% was significant (x2 = 7.172, time) as a fixed effect. Finally, we included an in- df = 1, p < .01). Overall, accuracy in the polymor- teraction effect between the withholding time and phic condition dropped from 87% at the start of the the treatment to test H4. As stated previously, our three weeks to 76% at the end. In contrast, accura- dependent variable was whether the user adhered to cy in the static condition dropped from 87% to a warning containing a risky permission and can- 55%. celled their installation of the app (coded as 1) or disregarded the warning and installed the app any- way (coded as 0). gle laboratory session, it was unclear whether poly- morphic warnings would sustain their efficacy over time as they lose their novelty or as users become accustomed to them. Our results show that the con- trary is in fact the case: although users also habitu- ate to polymorphic warnings, they do so at a sub- stantially slower rate compared to traditional warn- ings that do not change their appearance, so much so that the gap in adherence between polymorphic and static warnings widens over time. This finding demonstrates that polymorphic warnings are an effective and cost-effective solution for mobile Figure 4. Percentage correct in rejecting risky warn- developers and security practitioners. ings across 15 weekdays for each treatment group. Finally, we extend prior research and capture not 5. Discussion only the response decrement in the habituation pro- cess but also the daily recovery or increase in re- This study makes several contributions. First, this sponse strength. Although our findings do not sup- is the first longitudinal examination of privacy or port a greater recovery associated with polymor- security warnings. Although the value of longitudi- phic warnings, they do demonstrate that withhold- nal studies is widely recognized [27], a longitudinal ing warnings for a time can help increase users’ design is especially useful in the context of habit- sensitivity and response. uation. This is because habituation is a neurobio- logical phenomenon that develops over time [17]. 6. Conclusion Consequently, cross-sectional examinations of ha- bituation necessarily provide only a limited view of This study is the first to present a longitudinal field this phenomenon. For this reason, it was unclear experiment examination of security/privacy warn- how the results of previous habituation studies ap- ings as well as the first to address habituation to plied to repeated warnings outside of a single ex- permission warnings on mobile devices. In our 15- perimental session. Consistent with habituation day field experiment involving 108 people, users theory, our results corroborate those of previous habituated to permission warnings over time. How- studies by showing that the general pattern of ha- ever, the results suggest that the rate of habituation bituation reported in cross-sectional studies does in can be reduced substantially by employing poly- fact carry over to longitudinal exposures to warn- morphic warnings that continually update their ings, at least in our three-week design. This also appearance. Our longitudinal field study corrobo- suggests that cross-sectional examinations of habit- rates previous cross-sectional experiments of habit- uation are valid proxies for longitudinal designs. uation to warnings. In addition, our study offers We also provide the first longitudinal examination new insight into how users habituate to warnings of how users habituate to permission warnings on on mobile devices in everyday usage and provides mobile devices in the field. Previous studies of strong evidence that polymorphic warnings sustain habituation to security warnings have only ob- their advantage over time. served behavior in laboratory settings, which may not generalize to real-life usage. In addition, habit- uation to warnings may occur at a different rate for References mobile compared to desktop devices. This is be- 1 Android. Android developers guide. (2016). cause mobile devices are used in a wider array of https://developer.android.com/guide/topics/ma contexts compared to desktop computers and there- nifest/manifest-intro.html fore may be prone to the influence of interruptions and other competing demands. Furthermore, users 2 Authors. How polymorphic warnings reduce may receive notifications from mobile devices habituation in the brain—insights from an more frequently, because the mobile computing fMRI study. In Proceedings of the ACM paradigm encourages notifications for apps based Conference on Human Factors in Computing on location, movement, or other factors. Our results Systems (CHI) (2015). address these concerns by showing that habituation 3 Authors. From warnings to wallpaper: Why to mobile permission warnings is consistent with the brain habituates to security warnings and that of desktop computing contexts. what can be done about it. Journal of Third, our results show that polymorphic warnings Management Information Systems 33, 3 are resistant to habituation over time. Although (2016), 713-743. Authors [3] demonstrated that polymorphic warn- ings are effective in reducing habituation in a sin- 4 Authors. Your memory is working against 15 K. Krol, M. Moroz and M.A. Sasse. Don't you: How eye tracking and memory explain work. Can't work? Why it's time to rethink habituation to security warnings. Decision security warnings. In 7th International Support Systems 92 (2016), 3-13. Conference onRisk and Security of Internet and Systems (CRiSIS) (2012). 5 M.G. Berman, J. Jonides and R.L. Lewis. In search of decay in verbal short-term memory. 16 C.E. Mcculloch and J.M. Neuhaus. Journal of Experimental Psychology: Generalized linear mixed models. John Wiley Learning, Memory, and Cognition 35, 2 & Sons, Ltd,, Hoboken, NJ, 2001. (2009), 317-333. 17 C.H. Rankin, T. Abrams, R.J. Barry, S. 6 C. Bravo-Lillo, L. Cranor, S. Komanduri, S. Bhatnagar, D.F. Clayton, J. Colombo, G. Schechter and M. Sleeper 2014. Harder to Coppola, M.A. Geyer, D.L. Glanzman, S. ignore? Revisiting pop-up fatigue and Marsland, F.K. Mcsweeney, D.A. Wilson, C.- approaches to prevent it USENIX Association, F. Wu and R.F. Thompson. Habituation 105-111. revisited: An updated and revised description of the behavioral characteristics of habituation. 7 C. Bravo-Lillo, S. Komanduri, L.F. Cranor, Neurobiology of Learning and Memory 92, 2 R.W. Reeder, M. Sleeper, J. Downs and S. (2009), 135-138. Schechter. Your attention please: Designing security-decision uis to make genuine risks 18 P. Sawers. Android users have an average of harder to ignore. In Proceedings of the Ninth 95 apps installed on their phones, according to Symposium on Usable Privacy and Security Yahoo Aviate data. (2014). (2013). http://thenextweb.com/apps/2014/08/26/androi d-users-average-95-apps-installed-phones- 8 J.C. Brustoloni and R. Villamarín-Salomón according-yahoo-aviate-data, accessed 2007. Improving security decisions with 2/16/2017. polymorphic and audited dialogs. In Proceedings of the Third symposium on 19 S.E. Schechter, R. Dhamija, A. Ozment and I. Usable Privacy and Security (SOUPS 2007) Fischer 2007. The emperor's new security ACM, New York, NY, USA, 76-85. indicators. In Security and Privacy, 2007. SP'07. IEEE Symposium on IEEE, Berkeley, 9 M.Ö. Çevik. Habituation, sensitization, and CA, 51-65. pavlovian conditioning. Frontiers in Integrative Neuroscience 8 (2014), 13. 20 D. Sharek, C. Swofford and M. Wogalter 2008. Failure to recognize fake internet popup 10 A. Cnaan, N.M. Laird and P. Slasor. Tutorial warning messages. In Proceedings of the in biostatistics: Using the general linear mixed Human Factors and Ergonomics Society model to analyse unbalanced repeated Annual Meeting Sage Publications, New York, measures and longitudinal data. Statistics in New York, 557-560. Medicine 16 (1997), 2349-2380. 21 A.S. Shirazi, N. Henze, T. Dingler, M. Pielot, 11 S. Egelman, L.F. Cranor and J. Hong 2008. D. Weber and A. Schmidt 2014. Large-scale You've been warned: An empirical study of the assessment of mobile notifications. . In effectiveness of web browser phishing SIGCHI Conference on Human Factors in warnings. In Proceedings of the SIGCHI Computing Systems (CHI '14), M. JONES Conference on Human Factors in Computing AND P. PALANQUE Eds. ACM, Toronto, Systems ACM, Florence, Italy, 1065-1074. Ontario, Canada, 3055-3064. 12 S. Egelman and S. Schechter. The importance 22 E. Sokolov. Higher nervous functions: The of being earnest [in security warnings]. In orienting reflux. Annual Review of Physiology Financial cryptography and data security, A.- 25 (1963), 545-580. R. SADEGHI Ed. Springer Berlin Heidelberg, 52-59, 2013. 23 A. Sotirakopoulos, K. Hawkey and K. Beznosov 2011. On the challenges in usable 13 P.M. Groves and R.F. Thompson. Habituation: security lab studies: Lessons learned from A dual-process theory. Psychological Review replicating a study on SSL warnings. In 77 (1970), 419-450. Proceedings of the Seventh Symposium on 14 M. Kalsher and K. Williams. Behavioral Usable Privacy and Security (SOUPS) ACM, compliance: Theory, methodology, and result. Menlo Park, CA, 3:1-3:18. In Handbook of warnings, M. WOGALTER Ed. Lawrence Erlbaum Associates, Mahwah NJ, 313-331, 2006. 24 D. Straub, M.-C. Boudreau and D. Gefen. Validation guidelines for is positivist research. Communications of the Association for Information Systems 13, 24 (2004), 380-427. 25 J. Sunshine, S. Egelman, H. Almuhimedi, N. Atri and L.F. Cranor 2009. Crying wolf: An empirical study of SSL warning effectiveness. In SSYM'09 Proceedings of the 18th conference on USENIX security symposium, Montreal, Canada, 399-416. 26 R.F. Thompson and W.A. Spencer. Habituation: A model phenomenon for the study of neuronal substrates of behavior. Psychological Review 73, 1 (1966), 16-43. 27 R.T. White and H.J. Arzi. Longitudinal studies: Designs, validity, practicality, and value. Research in Science Education 35 (2005), 137-149. 28 M.S. Wogalter. Communication-human information processing (C-HIP) model. In Handbook of warnings, M.S. WOGALTER Ed. Lawrence Erlbaum Associates, Mahwah, NJ, 51-61, 2006.