Does Stress Impact Technical Interview Performance? Mahnaz Behroozi Shivani Shirolkar Titus Barik Chris Parnin [email protected] [email protected] [email protected] [email protected] NC State University NC State University Microsoft NC State University Raleigh, NC, USA Raleigh, NC, USA Redmond, WA, USA Raleigh, NC, USA ABSTRACT 1 INTRODUCTION Software engineering candidates commonly participate in white- Most companies in the software industry administer a technical board technical interviews as part of a hiring assessment. During interview as a procedure for hiring a software developer [3, 49]. these sessions, candidates write code while thinking aloud as they Companies believe technical interviews offer a “reasonably consis- work towards a solution, under the watchful eye of an interviewer. tent evaluation of problem-solving ability, communication skills, While technical interviews should allow for an unbiased and inclu- and preparedness” [75]. Technical interviews can also give visibility sive assessment of problem-solving ability, surprisingly, technical to the personality of the candidate, how they interact with their interviews may be instead a procedure for identifying candidates future colleagues and how much they pay attention to details—such who best handle and migrate stress solely caused by being examined as checking all possible test cases—and knowledge of programming by an interviewer (performance anxiety). languages. Companies also find it important to reduce unwanted To understand if coding interviews—as administered today—can stress [8] during a technical interview, as “not everyone does their induce stress that significantly hinders performance, we conducted best work in fast-paced, high-pressure situations” [52]. In principle, a randomized controlled trial with 48 Computer Science students, companies should expect the technical interview process to be a comparing them in private and public whiteboard settings. We sound and fair assessment of a candidate’s ability, and thus yield a found that performance is reduced by more than half, by simply steady stream of qualified candidates for positions that open upin being watched by an interviewer. We also observed that stress and the company. cognitive load were significantly higher in a traditional technical in- Technical interviews can also introduce other effects on can- terview when compared with our private interview. Consequently, didates who report unexpectedly “bombing” [50], “freezing” [67], interviewers may be filtering out qualified candidates by confound- or “choking” [59] during this critical hiring procedure. Through a ing assessment of problem-solving ability with unnecessary stress. happy accident, the software industry has seemingly reinvented We propose interview modifications to make problem-solving as- a crude yet effective instrument for reliably introducing stress in sessment more equitable and inclusive, such as through private subjects, which typically manifests as performance anxiety [78]. A focus sessions and retrospective think-aloud, allowing companies technical interview has an uncanny resemblance to the trier social to hire from a larger and diverse pool of talent. stress test [41], a procedure used for decades by psychologists and is the best known “gold standard” procedure [1] for the sole pur- CCS CONCEPTS pose of reliably inducing stress. The trier social stress test involves • Software and its engineering; • Human-centered comput- having a subject prepare and then deliver an interview-style pre- ing ! Empirical studies in HCI; sentation and perform mental arithmetic, all in front of an audience. Alone, none of these actions consistently induce stress in a subject; KEYWORDS however, the unique combination of cognitive-demanding tasks with a social-evaluative threat (essentially being watched) is consis- cognitive load, eye-tracking, program comprehension, technical tent and powerful. If a technical interview is essentially a de facto interviews version of a trier social stress test, then the implications can be ACM Reference Format: profound. Rather than measuring the few that answer correctly in a Mahnaz Behroozi, Shivani Shirolkar, Titus Barik, and Chris Parnin. 2020. timely manner, companies are most likely measuring the ability of Does Stress Impact Technical Interview Performance?. In Proceedings of the the few who perform well under stress [31]. Rather than measuring 28th ACM Joint European Software Engineering Conference and Symposium explanation skills, companies are most likely measuring the ability on the Foundations of Software Engineering (ESEC/FSE ’20), November 8– of candidates to handle or mitigate stress (e.g. through practice [58]). 13, 2020, Virtual Event, USA. ACM, New York, NY, USA, 12 pages. https: Finally, rather than avoiding unwanted stress, technical interviews //doi.org/10.1145/3368089.3409712 may be inadvertently designed with the sole purpose of inducing it. To understand how to maintain the desirable goals of a technical Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed interview (e.g. measure time and correctness), while mitigating for profit or commercial advantage and that copies bear this notice and the full citation the undesirable effects of stress, we created a design probe[77] on the first page. Copyrights for components of this work owned by others than the which removed the social-evaluative threat component of a techni- author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission cal interview. To this end, we designed an interview format where and/or a fee. Request permissions from [email protected]. participants privately solved a technical problem on a whiteboard, ESEC/FSE ’20, November 8–13, 2020, Virtual Event, USA without any monitoring or interaction with an interviewer. We © 2020 Copyright held by the owner/author(s). Publication rights licensed to ACM. ACM ISBN 978-1-4503-7043-1/20/11...$15.00 evaluated this format in a randomized controlled trial to under- https://doi.org/10.1145/3368089.3409712 stand the impact of stress during a technical interview and whether ESEC/FSE ’20, November 8–13, 2020, Virtual Event, USA Mahnaz Behroozi, Shivani Shirolkar, Titus Barik, and Chris Parnin we could isolate and dampen its influence. We then compared this constructing data structures, implementing search and sorting al- to a typical interview format, where a candidate used talk-aloud gorithms, and characterizing running time and space requirements. to explain their problem-solving in front of a proctor. Participants Furthermore, students represent potential technical interview can- wore specialized eye-tracking glasses, which allowed us to obtain didates, as many were actively engaged in their initial job search measurements associated with high cognitive load and stress. We prior to graduation. To use head mounted eye-trackers as a part of then compared the correctness, self-reported experiences, and cog- our study, participants had to have normal or corrected-to-normal nitive load and stress levels of participants across the interview vision. They received extra course credit through an experiment settings. pool and could end the experiment at any time. The study was We found that participants reported high levels of stress and approved by the local ethics board. The first and second authors of often had difficulty coping with someone else being present while the paper conducted the study. they attempted to solve a cognitively-demanding task. The impact To estimate our population size for our study, we used the results on performance was drastic: nearly twice as many participants of our pilot study to obtain estimators for effect size. Through a failed to solve the problem correctly, and the median correctness power analysis of 2-sample 1-sided test with 80% power and 5% score was cut more than half, when simply being watched. In con- Type I error [19], we found that approximately 18 participants trast, participants in the private setting reported feeling at ease, would be required in each group. having time to understand the problem and reflect on their solution. Finally, measures of cognitive load and stress were significantly 2.2 Pilots and Tasks lower in the private setting, and majority of participants solved We ran pilot studies at the authors’ institutions. During our pilots, the problem correctly. We also observed that no women success- we experimented with tasks that would satisfy the following criteria: fully solved the problem in the public setting, whereas all women be solvable within the time limits of the experiment, demonstrate solved it correctly in the private setting. Standard NASA-TLX pro- sufficient difficulty such that cognitive load can be induced andis cedure [35] and various cognitive metrics, provided evidence that not trivial to solve, and have ecological validity—the problem should higher rates of stress could causally explain these differences across be similar to those used in actual interviews. Based on feedback and groups. observations during the pilot studies, we found some tasks were We suggest that private interview settings have considerable too complex, where participants would quickly give up after a few advantages for candidates, that both reduce their stress and allow minutes, and some tasks too ambiguous,