A Study on the Behavior Under Multitasking Conditions in A

ASTUDYONTHEBEHAVIORUNDER MULTITASKINGCONDITIONSINA DYNAMICTASKSCENARIOINTHE CONTEXTOF HUMAN-MACHINE-INTERACTION

vorgelegt von Dipl.-Psych. Jürgen Kiefer

von der Fakultät V - Verkehrs- und Maschinensysteme der Technischen Universität Berlin GRK PROMETEI zur Erlangung des akademischen Grades Dr. phil. genehmigte Dissertation

Promotionsausschuss: Berichterstatter: Prof. Dr. Ing. Leon Urbas Berichterstatter: Prof. Dr. Phil. Manfred Thüring Tag der wissenschaftlichen Aussprache: 05.10.2009

Berlin 2010 D 83 This document was created using the tools:

• MikTex 2.7 (Freeware)

• WinEdt v5.60 (Shareware)

• Ghostscript, Ghostview, GSview

Special thank to Tony Jameson for providing the style-ﬁles:

• sfb378.sty

• sfb378-apacite.sty

• sfb378-apacite.bst

Dipl.-Psych. Jürgen Kiefer: A Study on the Behavior under Multitasking Conditions in a Dynamic Task Scenario in the Context of Human-Machine- Interaction , vorgelegt von , © Tag der wissenschaftlichen Aussprache: 05.10.2009 pour JULES

ABSTRACT

The work presented here is focusing on the behavior of participants in situations of daily life, in which several demands apparently at the same time need to be dealt with. After introducing into the topic labeled as "human multitasking", embedded in situation of routine life, reasons for choosing the topic and approaching it are provided (chapter 1). In chapter 2, an overview of the history in human multitasking and task switching is given. First approaches starting at the beginning of last century up to recent approaches and ideas are presented and their impact for psychological science is displayed. Entering chapter 3, the empirical work is presented: study 1 portrays a driving simulation in a driving simulator, in which a primary task (driving) plus a concurrent task (a test of attention which was adapted for the in-car scenario) are applied. The secondary task featured three different levels. The first study gives an impression about how people manage such scenarios. Please note that the main task (driving in the simulator) was considered a dynamic task. The second study mimics study one and is a replication with the additional aspect of training and its impact on performance. With the help of study two, strategies how to handle the scenario are derived and a heuristic is described which is applied by a bunch of people. Setting of the study was taken from study one. As task configuration was expected to strongly moderate the behavior of the participants during the task scenario, main task (driving) and secondary task (test of attention) were varied: applying the lane change task (LCT), a computer-simulation for driving behavior, it was possible to better analyze the lane derivation during driving (which was taken as a measure of performance for the main task). As for the secondary task, the variations of the test of attention used in the previous two studies were systematically extended. Results of study 3 were used to illustrate the impact of task configuration for the scenario. In the last study, time pressure as additionally component was applied and its impact was measured on task performance both for main task (driving) and secondary task (test of attention). The last chapter (chapter 4) resumes the results and provides design recommendations. The work closes with a conclusion and mentions aspects that were not considered due to time constraints.

ich hatte nur diese zeit (rainer werner fassbinder)

ACKNOWLEDGMENTS thank you: • Dr. Dirk Schulze-Kissing ("Ulysses") • Joachim ("Jolle") Wutke • Prof. Leon Urbas • Prof. Manfred Thüring • Prof. Hartmut Wandke • Prof. Anthony Jameson • Prof. Werner H. Tack • Dipl.-Ing. Marcus Heinath • Dipl.-Psych. Robert Lischke • Daniel Doering • Dipl.-Psych. Tobias Katus • Dipl.-Psych., Dipl.-Ing. Holger Schultheis • Dipl.-Psych. Cordula Krinner ("Miss LateX") • Dipl.-Psych. Nicola Fricke • Dipl.-Psych. Diana Woelki (apt pupil) • Dipl.-Psych. Necla Soyak • (cand.) Dipl.-Psych. Bob Kaldasch • (cand.?) Dipl.-Psych. Michael Schulz • Antti Oulasvirta (PhD) • Adam Chuderski (PhD) • Jing Qian (PhD) • Joscha Bach (PhD) • Inessa Seifert (PhD) • Dipl.-Psych. Nadya Dich • Michal B. Paradowski (PhD) • Stefan Mattes (Daimler) • Dirk Weishaar (pour Messiaen) • Karin Scherinsky-Pingel • Birgit Trogisch • Elke Fadel • Mario Lasch • ... and a few others

vii

CONTENTS i introduction1 1 introductory note5 1.1 Preface 5 1.2 Multitasking and human-machine interaction 7 1.3 Technology can do multitasking 8 1.4 An excerpt of recent studies 9 1.5 An ability to multitask? 10 ii theory 11 2 theoretical background 15 2.1 A short historical survey 16 2.1.1 The 1920‘s 16 2.1.2 The 1930‘s 17 2.1.3 The 1940‘s 17 2.1.4 The 1950‘s 17 2.1.5 The 1960‘s 18 2.1.6 The 1970‘s 18 2.1.7 The 1980‘s 19 2.1.8 The 1990‘s 20 2.1.9 2001: The cognitive bottleneck 20 2.1.10 2005: A general multitasking component 21 2.2 Multitasking or task interruption? 23 2.2.1 Characteristics of task interruption 23 2.2.2 The task switching question 25 2.2.3 A grammar for task scheduling 26 2.3 Single or multiple resources? 26 2.4 Summary and criticism 28 2.4.1 Need for continuous tasks in multitasking studies 29 2.4.2 Training and task repetition 29 2.4.3 Cognitive heuristics - human multitasking 29 iii studies 31 3 empirical studies 35 3.1 Study I: Identification of multitasking heuristics 35 3.1.1 Method in study I 36 3.1.2 Hypothesis: D2-Drive under multitasking 39 3.1.3 Results of study I 40 3.1.4 Discussion of study I 41 3.2 Study II: Practice - multitasking heuristics 42 3.2.1 Method in study II 42 3.2.2 Hypotheses for study II 44 3.2.3 Results of study II 44 3.2.4 Discussion of study II 46 3.3 Study III: The role of task configuration 47 3.3.1 Method in study III 47 3.3.2 Hypotheses for study III 50 3.3.3 Results of study III 51 3.3.4 Discussion of study III 52 3.4 Study IV: Amplification via time pressure 52

ix x contents

3.4.1 Method in study IV 53 3.4.2 Hypotheses for IV 55 3.4.3 Results of study IV 56 3.4.4 Discussion of study IV 57

iv discussion 59 4 critical discussion 63 4.1 Scope and ﬁndings 63 4.2 Cognitive modeling 64 4.3 Design recommendations 65 4.4 Criticism and outlook 67 4.4.1 The role of memory in human multitasking 68 4.4.2 Domain independence 69 4.4.3 Need for a computational model of human multitasking 69 4.5 fMRI studies on multitasking 70 4.6 Popular stereotypes about multitasking 71 4.6.1 Multitasking and happiness 71

bibliography 73 a appendix 81 a.1 Appendix: Structured interview 81 LISTOFFIGURES

Figure 1 A sketch on MT 6 Figure 2 Human-machine-interaction in daily life 7 Figure 3 Advertisement, Berlin (2007) 8 Figure 4 Robert Rauschenberg: First Landing Jump 10 Figure 5 A cartoon on multitasking 15 Figure 6 Jersild (1927): task switching paradigm 16 Figure 7 Model of Rasmussen (1983) 19 Figure 8 Cognitive bottleneck (Pashler, 1993) 21 Figure 9 Multitasking models - Salvucci (2005) 22 Figure 10 Interruption scenario 25 Figure 11 A grammar for task scheduling 27 Figure 12 Wickens‘ Model of multiple resources 28 Figure 13 Study I: Scenario 36 Figure 14 Study I: D2 test of attention 37 Figure 15 Study I-IV: D2-Drive 38 Figure 16 Study I: performance D2-Drive 40 Figure 17 Study I-IV: the merge heuristic 41 Figure 18 Study II: performance D2-Drive 45 Figure 19 Study II: the impact of training 46 Figure 20 Study III: the lane change task 48 Figure 21 Study III: analyzing LCT 49 Figure 22 Study III: performance D2-Drive 51 Figure 23 Study IV: scenario 53 Figure 24 Study IV: performance D2-Drive 54 Figure 25 Study IV: the impact of time pressure 56 Figure 26 Study IV: performance D2-Drive 57 Figure 27 Study IV: time pressure and D2-Drive 57 Figure 28 Study IV: eye movements 58 Figure 29 Overview of cognitive modeling 64 Figure 30 Pattern processing in D2-Drive 65 Figure 31 Modeling of D2-Drive 66 Figure 32 Pandoras box (Source: www) 67 Figure 33 Modiﬁcation of LCT (Soyak, 2008) 68 Figure 34 General executive for multitasking 70

LISTOFTABLES

Table 1 Controlled vs. automatic processing 18

xi xii List of Tables

Table 2 Study II: Amount of attention 47

ACRONYMS

UCD User-centered design

LCT Lane change task

IRG Information-requirement grammar

AOI Areas of interest Part I

INTRODUCTION

INTRODUCTORYNOTE 1

1.1 preface

efficiency matters! In modern western society (and not only there), time is money, and the less time required to do a job or task the more efficient your work is considered to be. Even before the word multitasking itself was applied, psychological approaches towards the phenomenon of how to handle the demand of multiple tasks were reported. The intellectual debate goes back even to the ancient Greek times. "To do two things at once - is to do neither. to do two things at once is to do neither. With these words, " roman philosopher Publilius Syrus 1 describes a phenomenon which thousand of years later released a core discussion in psychology lasting almost a hundred years, and a plethora of studies investigate whether people in fact turn out to be able to perform several tasks concurrently or not. Multitasking madness, some scientists say, leads to a waste of time. The myth of human multitasking not only remains but even more gains popularity in the era of mobile computing and human computer interaction. Is there in fact an illusion of concurrency? This question does not define the main issue of this work albeit it plays a (minor) role. Moreover, the author aims to show how people handle the demands of multiple tasks at the same time, put in a real-life situation which might occur day by day. This work starts with the word "efficiency", meaning the degree of target achievement in relation to necessary costs, be them mental, physical, or financial. It is efficient to save time, but do we really save time via multitasking?

procrastination is a type of behavior which is characterized by defer- ment of actions or tasks to a later time. Psychologists often cite procrastination as a mechanism for coping with the anxiety associated with starting or completing any task or decision. (taken from: http://en.wikipedia.org/ wiki/Procrastination.

efficiency matters! According to Steel[ 2007], procrastination is closely related to perfectionism and workaholism. Ego syntonic perfectionists tend to be less likely to procrastinate than non-perfectionists. This is remarkable: employers more and more concentrate on efficiency and most of them assert the more we can do at the same time the better our job. On the opposite, researchers in the field of psychology find in their studies that multitasking seems to be counterproductive or inefficient. Fast switching between tasks leads not only to lower performance in each task individually but also takes more time, as, for instance, Rogers and Monsell[ 1995] report. A variety of empirical investigations on multitasking do already exist and one might ask: "Why more studies?" or "What is the benefit of

1 Publilius Syrus was a native of Syria and a Latin writer of maxims in the 1st century BC. The legacy of his work is a collection of sentences (Sententiae), a series of moral maxims in iambic and trochaic verse.

5 6 introductory note

Figure 1.: To do two things at once is to do neither? (Source: unknown)

this investigation?". To reply to these objections, let me analyze common problems of former studies on task switching, dual-task - and multitasking scenarios:

1. Most studies in the context of psychology are not applied studies. Systematic control as a precondition for proper research prevents from a direct connection to real life. Only recently (and with the help of the vast development in computer technology), more and more researchers dare to investigate applied studies in the ﬁeld.

2. Most studies lack task repetition and systematic analysis. In order to derive how humans perform several tasks which are presented concurrently, it is necessary to provide a large number of task execution. Otherwise, the conclusions drawn from the experiments are based solely on a small number of observations.

In his studies, Saluvicci[ 2005] uses cell phone dialing as a secondary task in a driving situation. Although the studies within his paradigm provide a deep inside into human multitasking in real life, the secondary task in his scenarios (dialing) misses issues which are included in the studies reported within my work. To properly investigate and analyze a multitasking scenario including two tasks, I claim that a secondary task should be

1. fully controllable 1.2 multitasking and human-machine interaction 7

2. context-independent

3. interruptable

4. observable

Finding a balance between proper, science-based research on the one side without losing touch to reality on the other side is the challenge of this work. Nowadays, multitasking takes place everywhere: most people walking on the street use their phone and speak to friends. Rapid switches from one task to another, even without being aware of it, occur. Most of us do not necessarily become aware of simultaneous "actions" (e.g., walking and speaking at the same time). To my mind, many situations closely relate to the use and application of modern technological systems. This is the starting point of the following work.

1.2 multitasking and human-machine interaction

Figure 2.: Human machine interaction in daily life

An increasing development of technological systems in the beginning of the 21st century puts us into situations in which we have the lure of choice when interacting with a mobile system, a cash machine or a portable data assistant (PDA). This is especially true for new technology features in cars Strayer and Johnston[ 2001], and McCarley et al.[ 2004] describe this phenomenon as "burgeoning popularity of in- vehicle technology". In the last decades, so-called in-vehicle information systems (IVIS) aim to support the driver by offering a multitude of possibilities while driving. Green[ 1999], for instance, postulates a "15-second rule for driver information systems", meaning 15 seconds to be the maximum time for a task duration while driving. According to Green, the time a task requires is strongly correlated to crash risk. Also, Green promotes to measure task time instead of applying eye tracking (e.g., eyes-off-the-road-time) due to its instability. His rule became an international standard and highly focuses on in-car security to prevent human errors. According to Green[ 1999], the "15-Second rule" 8 introductory note

• is consistent with existing national and trade association guide- lines • is consistent with accepted vehicle design practice • is a feature to minimizes harm to drivers

Technological development is inevitable. Using the availability of (more or less) intelligent systems can support us in daily routines, but it can also turn out to be a burden. Almost twenty years ago, this has been referred to as "techno stress".

Figure 3.: Advertisement, Berlin (2007)

1.3 technology can do multitasking

Technology can do multitasking forever but humans can not! Caig Brod, author of the ground-breaking bestseller Techno Stress: The Human Cost of the Computer Revolution Brod[ 1984], directed to the implicit danger of the vast development of modern technology. Rosen and Weil 1 provide an excellent explanation why nowadays, modern communication like email is so distracting. They call this phenomenon "multitasking madness": Human beings have brains that allow them to appear as though they can comfortably perform more than one task at a time. In reality, our brains have an excellent ﬁltering mechanism that helps switch our attention rapidly from one thought to the next. To overcome this problem and stop the multitasking madness, Rosen and Weil recommend the following:

1. Precise time estimation: typically, we underestimate time needed to perform or fulﬁll a task. This bias creates expectations we cannot meet. Realistic time estimation is a ﬁrst step towards handling the demand of several tasks (almost) concurrently. 2. External memory: letting go off memory traces reduces cognitive load and helps us to focus more intensively to the current task.

1 published on www.contextmag.com 1.4 an excerpt of recent studies 9

3. Task perseverance: a full focus on one task at a time without maintaining thoughts for other tasks decreases time needed to perform a task and increases task accuracy.

4. Down time: work will be more efﬁcient after a refresh, be it playing with children, watching TV, or reading a book.

A study by the Institute for the Future reported that employees of Fortune 1.000 companies send and receive 178 messages a day and are interrupted an average of at least three times an hour!

1.4 an excerpt of recent studies

In chapter two, an overview on the history of task switching and multitasking is given. But before that, let me mention a few exemplary studies to illustrate the importance nowadays.

• Rogers and Monsell[ 1995] point out that people are faster in repeating a task compared to task switching. This is also true for familiar tasks which can easily be anticipated. Given more time between the trials did not help to completely eliminate the switching costs. According to Rogers and Monsell, switch costs are explained by (a) the need for mental control for the new setting, and (b) carry-over effects from the previous trial. Proper preparation did not have any signiﬁcant improvements.

• Meuter and Allport[ 1999] did a study in which subjects were asked to name digits in their ﬁrst or second language (depending on the background color). Not surprisingly, response time for digits in the ﬁrst language was faster compared to the second language (in a repetitive task setting). But also, subjects were slower when the language changed (task switching).

• Rubinstein et al.[ 2001] showed in a serial of four famous studies using a variety of tasks (e.g., maths, geometry) that task switching causes tremendous time loss. Additionally they were able to show that performance was strongly inﬂuenced by task complexity.

• Yeung and Monsell[ 2003] present a modeling of experimental interactions between task dominance and task switching, illustrating the importance of so-called prospective memory (we will come back to that concept later within this work). It seems that remembering where to continue a task plays a key role in the context of task switching.

These four excerpts demonstrate that multitasking and task switching play a key role, both in scientiﬁc research as well as in real life. In context of the design of new technological products, insights how people handle several, apparently concurrent tasks, is of special importance: knowing the cognitive mechanisms behind allows to adapt human-machine-systems to the user‘s need already in early stages of the design process. This we call prospective design. Prospective design means to develop and integrate tools and meth- ods in order to investigate the human-machine interaction already in the early development stages of technical systems. Design includes aesthetic as well as functional aspects. User-centered design (UCD) is 10 introductory note

an approach to integrate knowledge about and aspects of the user, e.g. cognitive limitations. UCD includes multiple stages, such as analyzes, development, testing, or re-design. Norman[ 1999] simply describes UCD as design based on the needs of the user. The earlier a designer is familiar with these insights (i.e., the users‘ needs) before a product is ﬁnalized, the better (see also Chin et al.[ 1988]). This work aims to provide exactly this knowledge about the user. In four empirical studies, multitasking scenarios are applied to analyze how users interact with a system (i.e. doing a secondary task) while concentrating on a dynamic activity (i.e., the primary task). Findings will provide helpful recommendations for a prospective design of human-machine-systems.

1.5 an ability to multitask?

More than half a century ago, Cherry[ 1953] mentioned that we have a natural ability and predisposition to multitask. At that time, for sure, the technological demands remained rather limited. Almost at the same time, Robert Rauschenberg1 argued that "technology is contemporary nature". The author would rather go with the last quote than with the ﬁrst one. And to show how we adapt to this "nature" is the focus of "There’s been an this work. exponential explosion of available information. Part of the responsibility of people developing this technology - computer manufacturers, ’big Bill’ Gates out in Seattle - should be taking into account the multitasking limitations of people using it." David Meyer (2001, Interview)

Figure 4.: Combine painting: cloth, metal, leather, electric ﬁxture, cable, and oil paint on composition board, with automobile tire and wood plan (R. Rauschenberg, 1961)

1 In the 1950s, artist Robert Rauschenberg (1925-2008) created the concept known as "Combines": he put non-traditional materials and objects in innovative combinations, thus combining both painting and sculpture Part II

THEORY

THEORETICALBACKGROUND 2

For all but the most routine tasks (and few mental undertakings are truly routine) it will take more time for the brain to switch among tasks than it would have to complete one and then turn to the other. (David Meyer)1 In his LA Times article, David Meyer - an expert in the ﬁeld of research on human multitasking - communicates a strong and direct message, which is already summarized in the headline of his article: we are all multitasking, but what is the costs? Meyer points out that "costs" do not only refer to time but also to mental fatigue and a loss of attention, i.e. accuracy. His quote opens this chapter which is meant to provide an overview of the history of (human) multitasking, with other words, the theoretical background of this work. Before the journey starts, let us be aware that we are interested in human multitasking. However, the term multitasking has been used in several areas, such as:

computer multitasking - the apparent simultaneous performance of two or more tasks by a computer’s central processing unit.

media multitasking could involve using a computer, mp3, or any other media in conjunction with another.

human multitasking: the ability of a person to perform more than one task at the same time

Figure 5.: A cartoon on multitasking (Source: EDUCATION 2.0, http://atedu20.blogspot.com)

1 taken from LA Times, Monday, July 19, 2004

15 16 theoretical background

2.1 a short historical survey

Deﬁning the concept of "multitasking" has been challenging scientists from various disciplines for decades (or even centuries). Long time before the word itself was used, psychological approaches towards the phenomenon of how to handle the demand of multiple tasks were reported. Chapter 2.1 gives a short introduction into the history of task switching and multitasking, from the early beginning (Jersild and early task switching paradigms) to nowadays studies (Saluvicci[ 2005], Taatgen[ 2005], etc.). This summary is not meant to provide a full description of all studies in this area, moreover I mention and describe important steps towards the current state of the research by giving an overview of the history of human multitasking.

2.1.1 The 1920‘s

1927. Jersild[ 1927] confronted participants with a list of stimuli to investigate the ability to alternate between different tasks. He was interested in how people switch from one task to another. In his studies, two conditions had to be executed, one in which the same task was performed on each item (so-called pure task blocks) and a second one in which different tasks were performed (alternating task blocks).

PURE TASK BLOCK CONDITION TASK REPETITION TASK REPETITION

task A task A task A task A

STIM - 3 STIM - 3 STIM - 3 STIM - 3

r(A-A) r(A-A) r(A-A) TASK SWITCHING ALTERNATING TASK BLOCK CONDITION TASK SWITCHING

task A task B task A task B

STIM - 3 STIM + 6 STIM - 3 STIM + 6

r(A-B) r(B-A) r(A-B)

Figure 6.: Early task switching study by Jersild (1927)

In the pure task block condition, task A was "subtracting three from each number on the list". In the alternating task block condition, Task B was "adding six to the number on the list". As we can see, already in this early period of psychological studies, Jersild assumed that, though both tasks being mathematical operations, he assumed them to be different in terms of cognitive processing, long time before the term "cognitive" was used. A further, second distinction was between univalent (i.e., each stimulus is a potential input only for the appropriate task) and bivalent (i.e., every stimulus is a potential input for either task) item lists (see also Fagot, 1994). The tasks mentioned above imply a bivariate condition. Univariate lists, in contrast, contained words (input for task 2.1 a short historical survey 17

A) and numbers (input for task B). Switch costs (in terms of reaction times) were measured as the difference between a switch (r(A-B) or r(B-A)) and a no-switch (r(A-A)). To my best knowledge, Jersild was the ﬁrst one to introduce the term switch costs. Even nowadays, the notion "switch cost" still holds (Pashler[ 2000], Rubinstein et al.[ 2001]), as well as "mental set" (see Spector and Biederman[ 1976], Meiran[ 1996]). Main ﬁndings in the studies by Jersild[ 1927] are:

1. For bivalent item lists, performance time is slower in the alternating condition. 2. For univalent item lists, performance is slower in the pure condition

These early, surprising findings remark a first step towards the distinction between modalities required to properly execute a specific task. The definition of switching costs still nowadays is used in many studies on task switching. A few years after Jersild and his task switching paradigm, the aspect of concurrency became of deeper interest.

2.1.2 The 1930‘s

1931. Telford[ 1931] asked the question what happens if two tasks overlap, i.e. the second task appears with a temporal delay to the ﬁrst one. In contrast to reported studies in which tasks are presented sequentially (task switching), i.e. one after the other, this scenario is called the psychological refractory period (PRP). The PRP - paradigm is as follows: a stimulus S2 of a task T2 is presented shortly after the onset of a stimulus S1 of a task T1. The difference of these two onsets (S2- S1) is called "simulus onset asynchrony" (SOA). This extension of task coordination is a ﬁrst step into the investigation of task concurrency, i.e. the fact that for a short moment two tasks appear concurrently. Main results in the context of PRP is that the smaller the (temporal) distance between S1 and S2 (the shorter the stimulus onset asynchrony, SOA), the longer the reaction time (RT) for S2. RT is measured for both T1 and T2. Following the argumentation of Jersild[ 1927], RT for S2 should be shorter after T1 than T2. Unfortunately, many studies in the context of PRP do not include a pure task condition (task repetition) as applied in Jersild[ 1927].

2.1.3 The 1940‘s

Vince, M. (1949). The connection between the psychological refractory period and rapid response sequences (Vince[ 1949]) is investigated by Margaret Vince. She could conﬁrm PRP - results by Telford[ 1931].

2.1.4 The 1950‘s

1952. A processing bottleneck is proposed by Welford[ 1952]. Two decisions about two responses to two different stimuli at the same time, Welfold claims, is impossible. Imagine two visual stimuli and two necessary responses (e.g., button presses). Participants had to respond both stimuli by pressing a button, respectively. Welford found that the reaction time for the second stimulus is slower than reaction time for the ﬁrst stimulus. He called this delay psychological refractory period 18 theoretical background

controlled processes automatic processes

slow fast

ﬂexible use no easy modiﬁcation

reduce capacity do not reduce capacity

conscious (attention) unconscious (no attention)

Table 1.: Controlled vs. automatic processing

and argues that it is always present, even for quite different stimuli. Welford[ 1952], to my best knowledge, was the ﬁrst one to introduce the concept of a bottleneck. Many subsequent studies support his assumptions for a general bottleneck in dual-task processing, implying serial processing of cognitive steps. Welford did not disclaim that this bottleneck is sometimes small or can be reduced, e.g. by training. The impact of training (which is also a denotative feature within the presented studies) will get more attention later within this chapter, in context of assumptions about single vs. multiple resources in dual task performance.

2.1.5 The 1960‘s

1963. Borger[ 1963] investigated the refractory period and serial choice reactions. He found PRP - effects with visual and auditory stimuli. In the studies, some participants applied a queuing strategy, i.e. reaction to task one (R1) is buffered and given shortly before reaction to the second task (R2). Pashler[ 2000] calls this behavior grouping strategy: it can be avoided by giving appropriate instructions. Meyer and Kieras[ 1997a] instructed to produce R1 as fast as possible. This, one might object, potentially evokes time pressure but prevents from answer queuing (see also Meyer and Kieras[ 1997b]. Learning from instructions became even more important in recent years (Taatgen et al., Taatgen et al.[ 2006]). In the presented studies in chapter three, to come to the point, the applied main task will be instructed as priority task. By doing so, "grouping" effects of task response are implicitly excluded. Before further PRP - or task switching studies are presented, it is necessary to address our attention to the way a task is processed - a diminutive but not exiguous aspect in context of human multitasking.

2.1.6 The 1970‘s

1977.Schneider and Shiffrin[ 1977] emphasize the necessity of a distinction between controlled and automatic processes. Later in this chapter we will see why this distinction is of deeper interest for human multitasking and dual task performance. The price for ﬂexibility (controlled processes) is a reduce of speed. Of special interest related to the studies present in the next chapter and the issue (human multitasking), automatic processes are not necessarily consciously accessible. Automatic processing is the result of training and practice. Automatic processing is a typical feature of skill acqui- 2.1 a short historical survey 19 sition (e.g., Anderson[ 1982], Lee and Taatgen[ 2002]). One model to contribute to this phenomenon was proposed by Rasmussen.

Figure 7.: Skills, rules, and behavior (Rasmussen[ 1983])

2.1.7 The 1980‘s

1983. Rasmussen[ 1983] developed a model about skill, rules and knowledge to explain the essential features of human skilled behaviour. In his eyes, a skill is a combination of open- and closed-loop behavior. In his model, he claims three levels of behavior:

1. Skill-based behavior (SBB): Automatic processing, without conscious attention or control, relying on signals.

2. Rule-based behavior (RBB): Behavior is based on familiar rules and consists of a sequence of subroutines (e.g., mathematical problem solvin, driving)

3. Knowledge-based behavior (KBB): Relying upon a "mental model" (of the system), no rules needed.

With other words, behavior turns from a cognitive stage to an associative stage and ﬁnally to an autonomous stage. Conscious processing, thus, becomes unconscious processing, including human error behavior (from error-prone to error-free) and speed (from slow to fast processing). In contrast to the assumptions from Schneider and Shiffrin[ 1977] whose theoretical "explanation" remains more descriptive than explanatory, the approaches on skill acquisition from the decade of the 1980s provide a comprehensive and plausible framework for the question how processes become automatic and thus resource-saving. Later in this chaper, a model by Chris Wickens will be introduced. This model postulates multiple cognitive resources. But before that, let us have a deeper look into further relevant studies on task switching and dual tasking. 20 theoretical background

2.1.8 The 1990‘s

1995. Rogers and Monsell[ 1995] introduce task set reconﬁguration to explain the phenomenon of alternation costs even without item repetitions. Monsell[ 1967] claims that different processing modules are needed for different aspects of a task. Alternation costs are deﬁned as difference between so-called pure and alternating-task blocks. Monsell used an "alternating runs" procedure. Though the empirical approach is rather abstract and not intuitively transferable into a real-life context, Monsell also provides an illustrating example to explain what is meant by a task set:

a professor sits at a computer, attempting to write a paper. The phone rings, he answers. It is an administrator, demanding a completed module review form. The professor sighs, thinks for a moment, scans the desk for the form, locates it, picks it up and walks down the hall to the administrators ofﬁce, exchanging greetings with a colleague on the way. Each cognitive task in this quotidian sequence (sentence-composing, phone-answering, conversation, episodic retrieval, visual search, navigation, social exchange) requires an appropriate conﬁguration of mental resources, a procedural "schema" or "task-set". 1992-2000. Hal Pashler summarizes recent analyzes of dual-task studies in Pashler[ 2000] by putting them into two categories, namely

1. studies of task switching or mental sets

2. studies on divided attention or dual task performance

Pashler claims that people show limitations when they have to perform two tasks concurrently, and these limitations are strongest in central stages of decision, memory retrieval, and response selection, with other words, in cognitive aspects where tasks are "intellectually demanding" (p. 287). It is widely known that training and practice supports performance, especially in perception and motor response. Hazeltine et al.[ 2002], for instance, strongly promote a simultaneous dual-task performance with parallel response selection afer sufﬁcient training (see also Ruthruff et al.[ 2003]). Instead of practice or training, some authors (e.g., Meiran and Daichman[ 2005], Sohn and Anderson [2005]) use the expression advanced task preparation to emphasize the preparatory control. In their study, task switching produced a performance decrease ("task errors") which disappeared after "advanced task preparation" (i.e., extensive task practising). Doing two things at the same time (Pashler[ 1993]) is inevitably connected to restrictions of a so-called bottleneck (Pashler[ 1984], Greenwald and Shulman[ 1973], Greenwald[ 1972]). A few year ago, a discussion about the central bottleneck took place starting after a paper published by Meyer and Kieras[ 1997a] (see also: Meyer and Kieras[ 1997b]).

2.1.9 2001: Uncorking the central cognitive bottleneck

2001. A ground-breaking article published in Psychological Science reani- mated an old discussion about the simultaneous performance of two or more tasks involving perception, cognition and action. As mentioned by Pashler[ 2000], human multitasking is restricted and main reason for this restriction is a central bottleneck. The response-selection bottleneck 2.1 a short historical survey 21

S1 R1

RESPONSE SELECTION PERCEPTION EXECUTION AND PROGRAMMING

R2 SOA S2

RESPONSE SELECTION PERCEPTION EXECUTION AND PROGRAMMING

RT 1

RT 2

Figure 8.: Bottleneck theory (adapted from Pashler(1993)

(RSB) hypothesis assumes the steps "perception - response selection and programming - execution" and claims that response selection to a stimulus S2 from a Task T2 can only be executed after the response selection to a stimulus S1 from a Task T1 has been ﬁnished (see Pashler and his bottleneck theory, Pashler[ 1993]). According to that, parallel processing is possible during perception (early stage of information processing) and execution (late stage of information processing). How- ever, processing of response selection is serial. Schumacher et al.[ 2001] argue in their article that even after "moderate" training, people reach a state in which they perform two tasks in parallel and the authors call this virtually perfect time sharing. But they also mention that not all participants in their studies were able to reach this state (individual differences) and the question arises whether extensive practice would enable virtually perfect time sharing for all participants. Dual-task interference is explained by conservative executive control postponing one task while another one is not yet executed. Main claim in their approach, in sum, is that intensive training and practice allow human multitasking without dual tasking costs for switching or response selection time according to a central bottleneck. Based on these results, study II of my work investigates allows participants a large amount of practice in order to overcome limitations and to become skilled for the applied multitasking scenario.

2.1.10 2005: A general multitasking component

2005. Salvucci categorizes multitasking studies related to real-world tasks as illustrated in Fig. 9 (taken from Saluvicci[ 2005]). In contrast to many psychological approaches within this subject, he highlights the fact that in "real life", many situations should be understood as multitasking scenarios. While Lee and Taatgen[ 2002] deﬁne multitasking as "the ability to handle the demands of multiple tasks simultaneously", Saluvicci[ 2005] sees human multitasking as the "ability to integrate, interleave, and perform 22 theoretical background

Figure 9.: Examples of multitasking models developed in a cognitive architecture (from: Saluvicci[ 2005])

multiple tasks and/or component subtasks of a larger complex task". For a classiﬁcation, he divides discrete (duration < 10 s) and continuous (duration > 10 s) tasks and proposes four categories:

1. Models of discrete successive tasks

2. Models of discrete concurrent tasks

3. Models of elementary continuous tasks

4. Models of compound continuous tasks

Models of discrete successive tasks are task switching studies like those already examined in the 1920‘s. Alternating simple choice- reaction tasks are applied to investigate switching costs. In these scenarios, the aspect of concurrency is not given. For this reason, i do not consider them as multitasking studies per se. Models of discrete concurrent tasks include a temporal delay. PRP- studies in the context of dual task performance belong to this section. Stimulus onset asynchrony defines when the second task begins. As already mentioned before, Pashler[ 2000] and others assume a central bottleneck which does not allow absolute concurrency. Elementary continuous tasks build the bridge to multitasking in daily life: one continuous task (e.g., driving) is performed while at some points a discrete task (e.g., a simple choice reaction task) is presented. To the authors belief, integrating these aspects of concurrency is a first 2.2 multitasking or task interruption? 23 step into human multitasking in a realistic context. Even more important and relevant for this work are compound continuous tasks. As the former category refers to tasks with a duration shorter than 10 seconds, this last section captures many scenarios, be it in the context of air traffic control, driving, or mobile computing. Salvucci mentions multiple recent examples (see Fig. 9) such as a model of manual tracking by Meyer and Kieras[ 1997a] and Meyer and Kieras [1997b], a radar-operator model, identification if new aircraft on radar, or a scenario in which driving and phone dialing is modeled (Salvucci [2001]. According to Saluvicci[ 2005], "...all these efforts contribute to a broader understanding of multitasking through study of both overall measures of task performance and particular measures of multitasking performance".

2.2 multitasking or task interruption?

The following section focuses on task interruption, on its characteristic features and gives some example of task interruption studies. To the author‘s mind, task switching from an unﬁnished task (which later is resumed) to a secondary task is, strictly spoken, a task interruption of this primary task. For this reason, both cases directly refer to human multitasking.

2.2.1 Characteristics of task interruption

Already in 1927, Bluma Wulfovna Zeigarnik, a soviet psychologist and student of Kurt Lewin and Lev Vygotsky, showed that people remember interrupted tasks better than uninterrupted tasks (Zeigarnik effect, see Zeigarnik[ 1927] and Zeigarnik[ 1967]). She observed that waiters seem to have a better memory for unpaid orders (van Bergen [1968]). Similar to Zeigarnik, Maria Ovsiankina showed that people tend to resume unfinished, but not finished tasks (Ovsiankina[ 1928]. Her assumptions closely relate to Zeigarnik and this effect is referred to as Ovsiankina effect. A few years later,H.[ 1941] was interested in the impact of feedback (success and failure) on the resumption of a task (motivational component). However, these effects could not always be reproduced, but at least it shows that interrupting people affects their task performance, be it in terms of stress (Cohen, 1980), decrease in task performance (Gillie and Broadbent[ 1989]), producing mistakes (McFarlane and Latorella[ 2002]) or recalling information. McFarlane[ 1998] defines an interruption as "the process of coordinating abrupt changes in people’s activities" and in reference to this definition, McFarlane and Latorella[ 2002] classify interruptions using a taxonomy which is presented here with slight modifications modified by the author: source of interruption The interruption can be taken by the person who is doing a task (i.e., self, internal interruption), by another person (i.e., external interruption), or by a machine (e.g., computer, external interruption). In many classical studies on task switching, the source of the interruption can easily be controlled, for instance by stopping task one and allowing to fully focus on task two. However, in the context of human multitasking, the situation looks rather different when one ongoing task is not stopped even though a second task starts. 24 theoretical background

individual differences Humans are bounded and rely on personal limitations. Cognitive processing is limited, and so are processes of perception and motor response. Individual differences play an important role in the ﬁeld of human interruption, but are not of deeper interest within this work.

method of coordination Immediate interruptions occur without coordination, in contrast to negotiated interruption. Further meth- ods are (human- or machine-) mediated interruption and sched- uled interruption based on an explicit agreement or by convention for repetitive interruptions.

meaning of interruption We all know the most common meaning of interruptions in our daily life. Alarms clocks during a meeting remind us to stop the current activity (task) and turn to another task/appointment/activity. Simply spoken, we are reminded that now, starting with the alert, our attention has to be focused on a speciﬁc action. Interruptions can also beckon us to ultimately stop our current task.

method of expression Physical expression (verbal, paralinguistic, kinesic), expression for effect on face-wants (politeness),a signal- ing type (by purpose, availability, and effort), metal-level expres- sions to guide the process, adaptive expression of chains of basic operators, intermixed expression, expression to afford control.

channel of conveyance Face-to-face, other direct communication channel, mediated by a person, mediated by a machine, meditated by other animate object.

changed human activity Internal or external, conscious or sub- conscious, asynchronous parallelism, individual activities, joint activities (between various kinds of human and non-human participants), facilitation activities (language use, meta-activities, use of mediators).

effect of interruption An interruption can cause multiple changes in human activity. It can inﬂuence motor behavior but also memory, awareness and the focus of attention. Especially in the context of multitasking situations, an interruption might create a complete new situation to which the person who is interrupted must adapt.

2002. Beginning of this decade, Altmann and Trafton[ 2002] describe a sequence of actions within an interruption situation, as illustrated in Fig. 10. A primary task is performed and interrupted by an alert. The period between this alert and the start of a secondary task is the interruption lag. In their studies, Trafton et al.[ 2003] focus on two characteristics of an interruption lag:

• the availability of the primary task during the interruption period

• the duration of the interruption lag

Quite obviously, this time lag is a function of both the time the secondary task starts and a person‘s reaction time that deﬁnes when to start the secondary task. The secondary task itself is performed and ends at a certain moment in time. Before the primary task is resumed, 2.2 multitasking or task interruption? 25

BEGIN ALERT BEGIN END RESUME OF 1st FOR 2nd OF 2nd OF 2nd OF 1st TASK TASK TASK TASK TASK

INTERRUPTION RESUMPTION LAG LAG

Figure 10.: Interruption situation (taken from: Altman and Trafton(2002)) time passes. This interval is called the resumption lag. Fig. 10 clearly illustrates the complete sequence of a scenario in which two tasks are executed, but they lack the integration of one or even more continuous tasks which might be partially executed in parallel. While driving, for instance, people seem to be able to switch their visual attention to the phone for a split second and dial a number. Nevertheless, they continue driving (without visual awareness for that moment).

2.2.2 The task switching question

One of the core questions in the context of task switching and multitasking is: "when do people switch between tasks?" Kushleyeva et al. [2005] mention three criterions (referred to as "major skill sets") which have to be met for "satisfactory" multitasking performance, namely

1. the ability to create and schedule future intentions

2. the facility to remember and prioritize these intentions

3. the ability to switch from carrying out one to another task when the appropriate moment in time is ﬁnally reached

The ﬁrst and the second condition resemble a concept which gains more and more interest recently: memory for future intentions is often named prospective memory (remembering to remember, Winograd [1988]). McDaniel and Einstein[ 2000] distinguish between event-based prospective memory (cue is an event, e.g., pressing a button or answering a question) and time-based prospective memory (recalling to continue a task at a certain time, e.g. going to a meeting at 4pm). Following Smith and Bayen[ 2004] and Smith[ 2003], maintaining an intention always requires attention resources, whereas McDaniel and Einstein[ 2000] note that cue identiﬁcation can be automatic or effort- ful, depending on a variety of parameters. The connection between prospective memory and the area of interruption becomes quite visi- 26 theoretical background

ble: Altmann and Trafton[ 2002] highlight that in order to resume an interrupted task, two essential conditions have to be met:

1. prospective goal encoding ("what was I about to do?")

2. retrospective rehearsal ("what was I doing?")

In their eyes, prospective goal encoding constitutes "a key mechanism behind prospective memory". Retrospective rehearsal is connected to what Kushleyeva (Kushleyeva et al.[ 2005]) calls "facility to remember" and can be suppressed by tasks which prevent from rehearsal, such as the n-Back task (McElree[ 2001], Owen et al.[ 2005], Juvina and Taatgen [2007]). The third and last criterion mentioned above focuses on the moment in time when precisely a switch has to be executed. This decision, however, determines not only a switch of attention but also the activation of another task set, i.e. the task set which is necessary to perform the resumed task. In Chapter 3, I claim and provide evidence that this decision is both conscious (and reported) but also unconscious (and thus only available in eye tracking data). In their approach, Dario Salvucci and his team use a computational grammar (an algorithm) called "information requirements grammar" to describe when people switch from one task to another.

2.2.3 A grammar for task scheduling

Andrew Howes and his collegues propose a "theory of competence for tasks", so-called information requirements grammar, IRG (Howes et al.[ 2005]). IRG implies the assumptions that (a) information and control requirements constrain the execution of a task and (b) available resources constraint the performance of a task (i.e., of their component processes and the necessary information). Two different kinds of constraints deﬁne task scheduling, namely information and control constraints on the one side and resource constraints on the other side. This is illustrated in the following example: IRG, however, does not allow a delay in performance when the information necessary to execute a task or subtask is available. In addition, this grammar proposed perfect task switching.

2.3 single or multiple resources?

Single-resource theories assume one central, unique resource (General- Purpose-Limited-Capacity Central Processor). Nobel prize winner Daniel Kahneman claims that we have only one global resource, and if we reach this available capacity, e.g. by demands of multiple concurrent tasks, we feel cognitive load (Kahneman[ 1992]). Single-resource theories postulate a direct connection between number and difficulty of concurrent tasks on the one side and resulting cognitive load on the other side. Performance decreases with increasing number of tasks and directly influences our limited, cognitive resource. The more difficult a task, the more reduced the available cognitive resource and consequently the performance. Especially in dual task studies, this theoretical assumptions by Kahneman[ 1992] became popular. Two tasks can be performed concurrently until the limit is exceeded. If this happens, cognitive resources are no longer available and both tasks cannot be performed in parallel. Mainly, a decrease in reaction time 2.3 single or multiple resources? 27

Figure 11.: Howes (2005): a grammar for task scheduling results from that. Performance errors are expected to increase in this case. In contrast to theories on one central cognitive resource, so-called theories of multiple resources postulate different and speciﬁc moduls for information processing (see Fig. 12). Similar to assumptions by Kahneman[ 1992], a limitation of the cognitive system is not denied. But the main difference to single-resource theories is that the central capacity is a product of different, independent individual capacities. Wickens (Wickens[ 2002], Wickens[ 1984], Wickens and Liu[ 1988], and Wickens[ 2004]) proposes a model of multiple resources, as shown in Fig. 12 with the following categorial, dichotomic dimension:

1. processing stages (perception, cognition, responding)

2. perception modalities (visual vs. auditory)

3. cisual processing (focal vs. ambient)

4. processing codes (spatial vs. verbal)

Following Wickens and his model of multiple resources, it is generally possible to perform multiple tasks under the same conditions without distraction or loss in performance. Division of attention, for instance, turned out to be more robust under "cross-modal time-sharing" compared to "intra-modal time-sharing". The model by Wickens (Wickens [2004], Wickens[ 2002]) allows different, divided resources for individual stages of information processing: resources for perception and resources for response do not interfere,thus both processes theoretically run in parallel without performance loss. For the presented studies in the next chapter, these implications play a key role. 28 theoretical background

Figure 12.: Model of multiple resources, taken from: Wickens and Liu[ 1988] and Wickens[ 2004]

2.4 summary and criticism

The presented theoretical approaches towards human multitasking serve to give an short overview and can be summarized in the following main messages:

1. First empirical approaches to investigate human multitasking go back to the task switching studies (Jersild[ 1927]) in which simple choice reaction tasks were applied.

2. These studies were enriched by the aspect of overlapping tasks (Telford[ 1931]) and the concept of psychological refractory period was introduced.

3. In the following decades until the late 1980s, many variations of the early ﬁndings systematically analyzed human task switching.

4. Pashler[ 2000] emphasizes the role of a central bottleneck and aims to show the inevitability to fully parallelize two tasks.

5. With their Psychological Science article on how to reach virtually perfect time-sharing, Schumacher et al.[ 2001]

6. In recent studies (Saluvicci[ 2005], Taatgen[ 2005]), real-life scenarios in the context of human multitasking gain increasing attention and the importance of continuous tasks in such investigations is highlighted.

7. Core questions in explaining human multitasking still remain: a) Is it sufﬁcient to apply discrete tasks in multitasking studies? b) How much training do people need to optimize task scheduling? c) When and how do people decide to switch from one task to another? d) Which strategies do people apply in human multitasking situations? 2.4 summary and criticism 29

2.4.1 Need for continuous tasks in multitasking studies

In Pashler[ 2000], the main focus is on discrete tasks. This is a rather common handling in the context of studies on dual task performance. Altmann and Trafton[ 2002] use discrete tasks in their task switching scenario. As Saluvicci[ 2005] remarks, to draw conclusions about human multitasking behavior in real life, it is necessary to investigate continuous tasks in a dynamic environment. Driving as the most prominent example for a continuous task underlines this demand. But also various other situations in daily life like walking in the street and concurrently using a mobile phone support a call for applied studies including continuous tasks. For this reason, my studies rely on scenarios of "compound continuous tasks", as proposed by Saluvicci [2005].

2.4.2 The importance of training and task repetition

Schumacher et al.[ 2001] highlight the importance of task training in the context of multitasking studies. They claim that participants become skilled in that speciﬁc task set. Lee and Taatgen[ 2002] considers skill acquisition as a method to "learn" how to do multitasking. Within the cognitive architecture ACT-R (Anderson[ 2007]), Taatgen refers to a mechanism called "production compilation" as main explanation how to perform successful multitasking. The following nice example (taken from: Taatgen[ 2005]) helps to illustrate how this works: Put water in kettle, put water on stove until it boils, put tea leaves in teapot, pour boiling water in teapot, and wait 3 to 5 min. These ﬁve instructions for making tea can be stored almost literally in declarative memory. The sim- plicity of the representation explains why this is the starting point for a new skill: Declarative items of knowledge can be added as single items to memory. The disadvantage of declarative representations is that they cannot act by themselves; instead they need, according to Anderson’s theory, production rules to be retrieved from memory and interpreted. This explains why initially processing is slow, because the declarative representations must be retrieved before they can be carried out, and it is prone to errors because the right declarative fact might not be retrieved at the right time. In study II within chapter III, the role of training and practise is illustrated. We will see to what extend this will contribute to handle the demands of concurrent multiple tasks in a dynamically changing environment.

2.4.3 Cognitive heuristics under human multitasking

Even though the approaches by Taatgen[ 2005], Saluvicci[ 2005] and others provide (computational) models which accurately predict human performance under multitasking in their concrete task scenarios, I doubt that human task scheduling behavior follows a grammar like IRG (see Howes et al.[ 2005]) or a formal description. Instead, I claim that in a human multitasking scenario, people adapt to the environment and develop strategies to optimally "survive" in such situations. Addition- ally, people might not necessarily be aware of their applied strategies, i.e. they arise either consciously (strategic) or unconsciously. They do not need to be precise, either. Therefore, I use the word "heuristic" which comes from the same Greek root as Eureka! And means "to ﬁnd". 30 theoretical background

In my understanding, a heuristic is a "rule of thumb", with other words a rule which is simple, efficient and can easily be learned through experience and training. Kahneman and Tversky[ 1973] propose the availability heuristic as a a heuristic for judging frequency and probability where people base their prediction of the frequency of an event or the proportion within a population based on how easily an example can be brought to mind. With other words, the ease of imagining an example has more weight for the judgment than the actual statistical probability. Because an example is easily brought to mind or mentally "available", the single example is considered as representative of the whole rather than as just a single example in a range of data. Stuart Sutherland illustrates this heuristic using a plausible example (Stewart et al.[ 1994]). Asked whether there are more words with "r" as the first letter than with "r" in the third position and also whether there are more words beginning with "k" than with "k" as the third letter, most people tend to reply that in both cases there are more words with "r" on 1st position than on the 3rd position. The same counts for the "k" - example. Nevertheless, people make a mistake, because in both cases, there are more words with "r" (same for "k") on the 3rd position. Words are arranged according their initial letter. Retrieving from memory thus is facilitated for words starting with a letter, e.g. with "r" (road, run) whereas it is more difficult to retrieve words with this letter on the 3rd position (like street, care). The statistical frequency is completely ignored and people judge based on the availability of words in their mind. This heuristic is used to explain findings in the area of probability judgment and people in general are not aware that they make use of a cognitive heuristic. The unconscious application of a heuristic implies that cognitive resources should suffer less compared to a strict propositional processing. Heuristics furthermore are not, as some scientists state, a bias or a failure: According to Gigerenzer and Selten[ 2002], heuristics can be fast, frugal and accurate all at the same time by exploiting the structure of information in natural environments (page 9). With other words, the development is an adaptation mechanism to the environment. This characteristic is of main importance for the four empirical studies, now presented in the upcoming chapter. Part III

STUDIES

EMPIRICALSTUDIES 3

Purpose of the following four studies and scope of the entire work is to convince and give ample evidence that, in a real life scenario, people do not follow principles of pure optimization when doing multitasking. Multiple approaches (e.g., Brumby et al.[ 2007]; Saluvicci [2005]) propose a task switching behavior according to free resources (e.g., visual attention or manual action) and available information required for performing a specific task. In study I-IV, the author tries to illustrate how people do multitasking by adapting to the structure of the environment. In contrast to many (if not even most) psychological studies in the last century, a dynamic main task in man machine interaction will be used. Study I introduces the multitasking-scenario which was applied in all of the four studies. Main focus in the first study lies on investigating how people manage a multiple task situation in a real-life context. Study II goes one step further by concentrating on the impact of practice and thus analyzes multitasking strategies to a deeper extend. Study III shows the importance of task configuration: with a systematic variation of the involved secondary task, it becomes obvious how people manage the demand of several tasks according to their design. The last and final study (IV) refers to an aspect of daily life, i.e. time pressure: in dynamic man-machine-interaction, available time is often rather precious and especially high time pressure plays a key role in how we handle multitasking in everyday life. Study I and II were conducted in a driving simulator. In study III and IV, a driving simulation on a PC was used. Please note that for the first two studies, ecological validity turns out to be higher compared to the last two studies, but these studies (I and II) lack a complete systematic control of external influences. For this reason, study III and IV were run on a PC with a simulated driving environment and perfectly controlled conditions. Findings of the four studies provide helpful insight and consequently support the prospective design of human machine interaction already in early stages of system development. Understanding how people behave in a multitasking scenario, what kind of strategies they use and how to support their performance allows a direct intervention during the design process and helps to reduce human errors in daily life.

3.1 study i: identification of multitasking heuristics

Most studies in the context of human multitasking mainly include standard, PC-based psychological tasks. As illustrated in 2, psychologists have been analyzing multiple-task coordination for quite a long time, starting already in the 1920s. Task shifting studies (Altmann and Trafton[ 2002]) or PRP-studies (Pashler[ 2000], Pashler[ 1993]), however, lack a direct connection to human behavior in daily life. The catego- rization of multitasking studies by Saluvicci[ 2005] nicely reﬂects that only recently, researchers started to pay more and more interest in dynamic task environments. Study I, for this reason, is interested in the adaptation and the allocation of attention in a scenario in which

35 36 empirical studies

people have to perform a continuous task (driving) and concurrently somehow handle the demands of a secondary task which is connected to the main task. It is well-known that a driver is multitasking while driving: everywhere, everyday. Be this eating, reading email, or talking on the phone. Inattention, distraction, and mental fatigue still are the most dangerous contributing factors leading to an accident. The design of an in-car-system which requires less attention and is easy to use, thus, can prevent from vehicle crashes. To do this, it is necessary to understand how people behave in a naturalistic multitasking environment.

3.1.1 Method in study I

Participants in study I Twenty four male and female undergraduate students (age between 20 and 30) of Technical University Berlin participated the ﬁrst study. All participants had a driving license and were experienced in driving. Wearers of glasses were excluded from the study. Gender effects were of no interest for the study.

Figure 13.: Car interior in study I

Involved tasks in study I Primary task in study I was a driving task (with lane derivation being considered a dependent variable): participants were asked to drive with constant speed (130 km/h) in a driving simulation in a car. The task itself was quite trivial (keeping the lane): participants were instructed to keep the lane (a simulation which does not require a deeper spec- iﬁcation here). Performing this task (after being instructed to do so) turned out to be feasible for all subjects. Also, the focus on driving as primary task was understood by all subjects and they acted accord- ingly, considering driving with main importance. As secondary task, the MODYS research group, especially Marcus Heinath and Jeronimo Dzaack, build a derivation from the "D2 test of attention" (Brickenkamp 3.1 study i: identification of multitasking heuristics 37

[1992]), an in-car version adapted to be displayed on a 10inch screen in the car interior. In what follows, this test is hence referred to as "D2-Drive". In comparison to other "in-vehicle information systems" (IVIS), D2-Drive is a model of a secondary task on a screen-oriented driver information system and captures the following characteristics:

• Attention:D2-Drive requires full visual attention.

• Access:D2-Drive can easily be learned.

• Interruptability:D2-Drive can be interrupted and resumed.

• Resources:D2-Drive is a measuring tool for residual resources.

• Cognition:D2-Drive requires perception (read), cognition (decide) and action (motor response).

In this sense, D2-Drive is cultural independent, scalable in terms of complexity and serves as an optimal tool to measure individual attention. In this case, attention is needed for the secondary task for perception ("reading") and action (manual response). As a dependent variable, Dr-Drive performance was considered in study I and all upcoming studies.

Figure 14.: Original D2 test of attention (Brickenkamp[ 1992])

Similar to the original paper and pencil version of the D2-test developed by Brickenkamp (Brickenkamp[ 1992], in D2-Drive people have to judge whether a pattern contains the letter "d" and concurrently two strokes. However, in contrast to the original version, D2-Drive requires to press a button (Yes or No) instead of crossing the pattern out. This feature underscores the similarity to many situations in which we interact with systems in daily life. Based on the fact that D2-Drive requires full visual attention and asks for a decision (a cognitive evaluation), performing D2-Drive in context of a multitasking scenario means a visual interruption of a concurrent task. Three versions of D2-Drive were used in study I (see Fig. 15): d2-drive-v1.1: Presentation of a complete row of (ﬁve) patterns Fo- cus only on the pattern in the middle (third pattern) Execution only of pattern in the middle (1 pattern) d2-drive-v1.2: Presentation of a complete row of (ﬁve) patterns Fo- cus on complete row Execution of complete row (5 patterns) d2-drive-v1.3: Presentation of a matrix of patterns (rows and columns) Focus on the row whose number was presented Execution of this complete row

As can be seen, D2-Drive-v1.2 is close to the original version of the D2- test. D2-Drive-v1.3 contains an additional memory element, i.e. while 38 empirical studies

performing a complete row on the n-th screen, the row which needs to be performed on the next (i.e., n+1 - th) screen has to be read and kept in mind. The design of all three versions assumes increasing complexity and the demand of different resources. Whereas version D2-Drive-v1.1 requires visual ﬁxation, a cognitive decision process and a response, the other two versions further include a reading element. D2-Drive-v1.3 additionally relies on vertical as well as horizontal aspects.

Figure 15.:D 2-Drive (Urbas et al., 2005)

Design of study I Complexity of D2-Drive was treated as between-subjects factor (three groups with 8 participants per version) and condition (single- vs. multitasking) as within-subject factor. Measure of performance for main task (Driving) was lane derivation, for secondary task (D2-Drive) number of correct patterns per minutes (i.e., trial). Please note that the error rate in all three versions of D2-Drive approximated zero: hence, the decrease in performance is reﬂected in the number of performed patters (i.e., patterns per min).

Procedure in study I In study I, participants were ﬁrst introduced to the complete procedure and instructed to perform the primary task, i.e. driving in the simulator environment with constant speed (130 km/h) and main priority. First, participants trained the primary task and afterwards, baseline measures (single task condition) for driving were recorded. Subsequently the secondary task (D2-Drive) was explained, trained and performed under single task condition (pretest). Following, the multitasking condition started: while driving, at four different positions a sound indicated a D2-Drive test (duration of 60sec). As soon as participants heard this sound, they were requested to perform the secondary task (D2-Drive) without neglecting the primary task (driving). Within a lap, D2-Drive was presented four times. The study concluded with a post test for D2-Drive and a structured interview in which participants were asked questions about their experience while performing the study. For data analysis, a multivariate analysis of variance was conducted.

1. Welcome, introduction and instruction

2. Training "Driving"

3. Baseline "Driving"

4. Training "D2-Drive" 3.1 study i: identification of multitasking heuristics 39

5. Single-Task "D2-Drive" (Pretest)

6. Dual-task session (4 x "D2-Drive")

7. Single-Task "D2-Drive" (Posttest)

8. Structured interview

Hypotheses for study I In collaboration with the MODYS research group as well as from a explorative perspective, the following hypotheses, mentioned below, were tested.

Hypothesis: Driving under multitasking The primary task itself (driving) is a continuous tracking task and does not require deep cognitive processing. Driving is instructed to be considered as priority task. Therefore, no performance decrease is expected under multitasking compared to single tasking (baseline for driving).

3.1.2 Hypothesis: D2-Drive under multitasking

• For the three versions of the secondary task, complexity is assumed to increase from version 1 to 3 consecutively and hence performance to decrease. This assumption is based on a deeper cognitive processing (visual perception, cognitive processing in terms of decision making, and action via motor response), especially for D2-Drive-v1.2 and D2-Drive-v1.3. It can be expected that performance (patterns per minute) in D2-Drive-v1.2 is comparable to results from the paper and pencil version developed by Brickenkamp[ 1992]. Increasing complexity (D2-Drive-v1.1 < D2-Drive-v1.2 < D2-Drive-v1.2) should be reﬂected in performance data (correct patterns per minute) and result in signiﬁcant differences.

• D2-Drive requires visual attention, cognitive processing as well as motor action (response). Therefore performance in D2-Drive is expected to decrease under multitasking performance. Driving (main task) was instructed to be performed with priority and consequently a decrease in the primary task cannot be expected. In addition, the three different versions of D2-Drive require a different amount of visual attention. For D-Drive-v1.2, performance is expected to be best: in this version, visual-motor coordination (as described by Wickens[ 2002]) is possible. D2-Drive-v1.1 and D2-Drive-v1.2 should not differ in performance signiﬁcantly in the pretest but in the dual task condition and even more in the post-test due to stronger learning effects and development of a "multitasking skill" (Taatgen[ 2005]). D2-Drive-v1.3 is supposed to be the most complex and thus most demanding version, leading to worst performance. This is a result of the additional cognitive (memory) element in it. 40 empirical studies

3.1.3 Results of study I

Results: Driving under multitasking Not surprisingly, driving was not affected by an additional secondary task. Performance under multitasking was similar compared to single task performance. Recordings of baseline driving after training conﬁrm that under multitasking, participants do not improve performance, i.e. after single tasking, they reached their maximum and learning effects due to longer practice can be excluded. The results further support the assumption that participants follow the instructions and consider driving as main task with main priority.

Results: D2-Drive under multitasking

Figure 16.: Performance in D2-Drive in study I

Pre- and post-tests (single task condition) were applied to check whether learning effects (see Taatgen[ 2005]) occur. For D2-Drive- v1.1, performance did not change significantly whereas in both D2- Drive-v1.2 and D2-Drive-v1.3, participants improved significantly after multitasking (for both, p < 0.05). Overall, there was a strong significant difference between single and multitasking for all three versions (for D2-Drive-v1.2: p < 0.01, for D2-Drive-v1.2: p < 0.05, for D2-Drive-v1.3: p < 0.06). Surprisingly, in D2-Drive-v1.2 performance was best in all conditions and performance data are comparable to the paper and pencil version. After the complete scenario, participants were asked several questions concerning the manner in which they performed both tasks individually as well as concurrently. Based on these interviews, the most prominent "strategies" how people performed the secondary task were:

1. (1) For D2-Drive-v1.1, no speciﬁc strategy was applied. Partic- ipants ﬁxated the pattern in the middle and from time to time turned their visual attention to the street. The behavior remained constant during the complete procedure of the study. 3.1 study i: identification of multitasking heuristics 41

2. (2) For D2-Drive-v1.2, participants started to perform the same way they did in D2-Drive-v1.1, but over time, several patterns were read sequentially and responses were given sequentially as well. 14 out of 24 conﬁrmed to use this "strategy", some did so already in the beginning of the dual task condition while others "developed" this strategy during the multitasking scenario.

3.( 3) For D2-Drive-v1.3, the interviews state that participants used the same strategy. One additional "adaptation" was an external- ization of the memory element: 10 out of 24 participants reported that they did not keep this number in their mind but used the corresponding ﬁnger of their left hand (which was on the wheel).

Mostly, for the secondary task, participants apply a processing or a strategy which here in this context is referred to as "merge heuristic": at the beginning, participants perform the test pattern by pattern. After a while, participants seem to have "understood" that merging patterns together and replying them in a sort of block (e.g., reading 2 or 3 patterns, scanning the street, responding these 2 or 3 patterns by manually pressing keys) is a resource-friendly and clever adaptation to the environment, i.e. the multitasking situation.

Figure 17.: The "merge heuristic"

3.1.4 Discussion of study I

In sum, study I shows that performance under multitasking does not decrease in both tasks. One reason for this phenomenon is the impact of training (practice effects) as mentioned by the participants in the structured interviews. A second, maybe even more important reason, is the development and application of multitasking heuristics. These heuristics can potentially be derived not only from verbal reports (structured interviews). In study II, eye tracking data will be applied additionally. The observation that performance improves over time (effect of practice) implies a need for a second study in which (a) participants are given more time to practice both tasks concurrently and (b) eye tracking data are used to support assumptions about the "merge heuristic". Addition- ally to these aspects, a third extension is use of another 3 versions of D2-Drive with nine instead of ﬁve patterns. These version are referred 42 empirical studies

to as D2-Drive-v2.1(5), D2-Drive-v2.2(5), D2-Drive-v2.3(5) and consequently D2-Drive-v2.1(9), D2-Drive-v2.2(9), D2-Drive-v2.3(9), where the number in brackets indicates the corresponding pattern lengths.

3.2 study ii: practice and multitasking heuristics

In study I, participants used cognitive heuristics which developed without instruction and not necessarily consciously. The "merge heuristic" was detected and described in detail in the previous paragraph. Main critical issue of study I is based on the observation that participants increased in performance over time, especially in D2-Drive-v1.2. Thus, the question is whether the detected heuristic is moderated by practice and how practice supports multitasking heuristics. In many studies on skill acquisition, practice has a tremendous impact on performance. For this reason the same scenario was applied with some slight modifications. Does more intensive driving amplify the use of the "merge heuristic"? Will performance increase in a second driving lap with the help of the "merge heuristic"? Or will participants feel mental fatigue and produce worse performance under multitasking? And also, in analogy to the original paper and pencil test of D2, it is of further interest if longer patterns per version (9 instead of 5) even amplify the gap between the three versions. Especially D2-Drive-v1.2 is expected to benefit from such a modified configuration. Therefore, what is the impact of pattern length in the secondary task? Is there any connection between pattern length and the "merge heuristic"? These questions will be answered in this paragraph.

3.2.1 Method in study II

In study II, the same method was applied as in study I.

Participants in study II Thirty-six undergraduate students of Technical University Berlin took part in the second study. Gender was equally distributed. Age was of no further interest. As in study I, all participants of study II had a driving license and were experienced in driving. Because of the measuring of eye tracking data, all participants had to conﬁrm to wear neither glasses nor contact lenses. Participants were male and female, gender effects were not of any interest in this context.

Involved tasks in study II Primary task for study II was driving, using the same environment (driving simulator with identical lane) as in study I. Lane derivation and performance in D2-Drive are dependent variables in study II. In contrast to the ﬁrst study, this time participants were asked to drive two laps. In fact, study II is a slight variation and a design similar to the one applied in study I, with the following extensions:

• Providing more training by extensive driving (2 laps)

• Investigating the impact of pattern length in the secondary task (5 vs. 9 patterns) 3.2 study ii: practice - multitasking heuristics 43

As in the previous study, secondary task was D2-Drive. This is a necessary condition in order to compare participants behavior and to see whether any possible effect is in fact based on practise. d2-drive-v2.1 Same as in study I (for 5 as well as 9 patterns) d2-drive-v2.2 Same as in study I (for 5 as well as 9 patterns) d2-drive-v2.3 Same as in study I (for 5 as well as 9 patterns)

In study II, subjects were instructed to pay main attention to the primary task (driving in the simulation environment) and at the same time - without neglecting the driving - to perform D2-Drive as appropriate as possible. The three versions described in the previous section were presented with either five patterns (as in the first scenario) or with 9 patterns. This is denoted by an according number in brackets, e.g., D2- Drive-v2.2(9), indicating that the second version was presented using nine patterns. Please note that not coincidentally, the numbers refer to assumptions about working memory capacity: Cowan (2001) claims that people are able to maintain 4 (plus/mins 1) chunks, whereas former studies (Miller, 1957) promote a "magical number seven", implying that working memory capacity features a size of seven. Nevertheless, this work is not meant to focus on people‘s capacity limit when doing multitasking. Choosing a nine-patterns version just plays to the variance in working memory performance: participants with a size bigger than five "have the chance" to make use of their full capacity and do not find themselves restricted by a five-patterns version.

Design in study II Exactly like before, "complexity in D2-Drive" (3 versions) was treated as between-subject factor and "pattern length" (ﬁve vs. nine) as within- treatment.

Procedure in study II As in the previous study, participants were welcomed and introduced to the scenario. As in study I, it was important for them not to have par- ticipate in a similar driving study (e.g., study I) before (inexperienced). procedure for extended driving and d2-drive

1. Welcome, introduction and instruction

2. Training "Driving"

3. Baseline "Driving"

4. Training "D2-Drive"

5. Single-Task "D2-Drive" (Pretest)

6. Dual-task session (4 x "D2-Drive", lap 1)

7. Dual-task session (4 x "D2-Drive", lap 2)

8. Single-Task "D2-Drive" (Posttest)

9. Structured interview 44 empirical studies

3.2.2 Hypotheses for study II

Hypothesis: Intensive driving under multitasking In the previous study, driving performance was not inﬂuenced by multitasking. Therefore, in study II no signiﬁcant differences were expected under multitasking compared to driving in the single task condition. The instruction (i.e., considering driving as man task with main priority) was expected to fully work, meaning that participants should perform D2-Drive without neglecting the driving task. Mental fatigue should not occur.

D2-Drive under multitasking (pattern lengths and practice) Schneider and Shiffrin[ 1977] claim that tasks become "automized" after practice, thus no voluntary control is needed and no interference with mental operations appears, i.e. simultaneous performance without interference is possible. However, in their studies, discrete dual task situations were investigated. In this study, a dual task scenario with continuous tasks is used. With reference to results from Schumacher et al.[ 2001], Schneider and Shiffrin[ 1977] and Taatgen[ 2005], practice not only improves performance but also turns controlled processing of (part of) a task into automatic processing (from error-prone to error-free behavior with an increase in speed). Intensive training should thus promote using the "merge heuristic" (consciously as well as unconsciously) and thus lead to a better performance for D2-Drive-v2.2, but not for D2-Drive-v2.1 (due to its configuration not supporting the use of the described heuristic). This counts for the dual task condition (merging of action parts adaptively to the multitasking situation) as well as for the post-test (participants will have learned how to optimally use the "merge heuristic"). However, D2-Drive-v2.3 is not expected to show significantly better performance after training. The memory element in it prevents from automatically enter responses by demanding resources in working memory. The "merge heuristic" (described in the previous section) is a smart, cognitive tool to adapt to the multitasking environment. Similar to studies on perfect-time sharing by Schumacher et al.[ 2001], two tasks coalesce and almost become one (as much as possible). Visual attention, obviously, cannot be shared, and both tasks require visual perception: the driving task requires, even if it is only a minimum, control scans of the street, and by nature, D2-Drive is based on D2 (Brickenkamp [1992]) which is a test of attention. However, participants in the first study somehow managed to separate those elements of both tasks that allow parallel processing (control scans on the street and concurrently response in terms of manual action). Even more surprisingly, this all happens without instruction: people understand the structure of their (dynamically changing) task environment and automatically adapt to it.

3.2.3 Results of study II

Results: Intensive driving under multitasking Not surprisingly, driving remained constant and was not inﬂuenced by a secondary task. Even more, under dual task condition, driving 3.2 study ii: practice - multitasking heuristics 45

Figure 18.: Performance in D2-Drive in study II performance did not signiﬁcantly change from lap 1 to lap 2. This con- stancy of performance goes in line with results from the previous study and again shows that participants follow the instructions and consider driving as main task with main priority. It also shows that mental fatigue, in case participants felt it, did have no effect on performance.

Results: D2-Drive under multitasking: pattern lengths and learning The factor "pattern length" (five vs. nine patterns) did not have a significant influence on performance in the secondary task. In the nine-patterns version, participants were only slightly better. However, this result is not worth further mentioning. For this reason, all data is aggregated, subsumed and hence I refer to D2-Drive overall. As already confirmed by study I, D2-Drive-v2.2 outperforms D2-Drive-v2.1 as well as D2-Drive-v2.3. As can be seen, participants perform rather poor in D2-Drive-v2.3 (the obviously most difficult version). In this version, performance under dual task performance does not improve (in contrast to the other versions), only small, but not significant learning effects (pretest vs. post-test) were found. Comparing D2-Drive-v2.1 and D2- Drive-v2.2, the power of the "merge heuristic" becomes transparent: this version of the secondary task allows to optimally perform both tasks (or part of them) concurrently. Combining single parts of each of them (visual perception, cognition, motor action) is possible via the ability to separate individual parts of the secondary task and reconfigure them. Also, please note that the level of performance can be carried over to the posttest in D2-Drive-v2.2, i.e. learning effects are stronger in this version than in D2-Drive-v2.1. Remarkably, in D2-Drive-v2.3, performance in the post-test is even better Participants in study II passed eight trials, i.e. while driving they were presented eight times D2-Drive (two laps). 19 displays the performance over time and clearly shows the improvement from trial 1 until trial 8.Interestingly, this effect is strongest for D2-Drive-v2.2. A closer look at the trials gives insight about the zigzag - sequence in the performance data: even though it was not applied as a factor, a post observation gives insight about this phenomenon. Odd trials (i.e., trial one, trial 46 empirical studies

three, trial ﬁve (or trial one in the second lap), trial, seven (or trial two in the second lap)) correspond to a lane curve whereas even trial include a straight lane. There was no assumption about street behavior and thus the signiﬁcance and meaning of this "factor" is rather weak. Driving behavior on a curved vs. straight lane remains stable.

Figure 19.: Improvement over time in study II

Results: Cognitive heuristics under multitasking In 3.1, one main critical issue was the lack of eye-tracking data. Study II accounts for this measure, a comparison of visual time spend on the primary vs. secondary task was done. On average, participants spend 47 percent of visual attention (time of gaze in percent for the deﬁned area of interest, AOI, "lane" and for AOI "D2-DRIVE" during multitasking) on D2-DRIVE and 51 percent on the lane. The residual amount (2 percent) can be considered as noise. Interestingly, D2-Drive- v1.2 requires only 36 percent of eye gazes, whereas D2-Drive-v1.1 as well as D2-Drive-v1.3 both require 48 percent. In combination with the high performance in D2-Drive-v1.2, this supports the assumption that participants perform the secondary task while their visual attention is on the primary task. This adaptive task coordination can be explained with the help of the "merge heuristics" mentioned in paragraph 3.1. The explanations of participants of how they perform the individual tasks and both tasks together (structured interview) again support that most people use the "merge heuristic" (reported by 22 out of 36), and it can be assumed that even people who were not able to verbalize their applied strategy/heuristic, might have used them (as can be derived from eye tracking data).

3.2.4 Discussion of study II

In study II, the goal was to investigate the impact of intensive practise. Main ﬁndings of the second study are: 3.3 study iii: the role of task configuration 47

version of d2 gazes on d2-drive gazes on lane

Total 47 percent 51 percent

D2-Drive-v1.1 48 percent 51 percent

D2-Drive-v1.2 35 percent 62 percent

D2-Drive-v1.3 48 percent 51 percent

Table 2.: Study II: Amount of attention (eye gazes, AOI)

1. Practice supports the development of cognitive heuristics.

2. Task complexity seems to have an decisive inﬂuence.

3. Different task conﬁgurations require different amounts of visual attention.

4. The driving task (tracking) remains unaffected by the secondary task.

It seems that the more practice participants have, the stronger the impact of the task conﬁguration. Eye tracking analyzing supports the assumption that cognitive heuristics (in particular, the "merge heuristic") are applied. For a deeper analysis of human multitasking, study III uses a standardized driving simulation as primary task.

3.3 study iii: the role of task configuration

Study I and II provide ample evidence that:

1. People adapt to the environment with the help of cognitive heuristics.

2. This behavior is more pronounced under intensive training.

The applied secondary task in the previous studies varied in terms of complexity (three difficult levels) but also in terms of its task configuration. In other words, the task itself determines how people perform it. To further investigate the impact of task configuration, four versions of the "D2-Drive"-test were build which systematically offer the potential for cognitive heuristics to be applied. Even though the first two studies contain a rather high degree of ecological validity, the systematic control in a driving simulator is strictly limited. To further investigate human multitasking in a real life context, in the subsequent studies the "lane change task" (Mattes[ 2003]), hence LCT, was applied.

3.3.1 Method in study III

Study III has not been conducted in a driving simulator. Instead, a standardized tool for measuring lane derivation (LCT) was used. An additional feature of LCT is a (cognitive demanding) task involved in it (i.e., to change the lane on the appropriate moment in time). LCT is a standard tool in a dynamically changing task environment and has often been used in the context of human machine interaction in the area of driving (see Mattes[ 2003]). In september 2005, LCT 48 empirical studies

was submitted as measuring tool to the international organization for standards (ISO) and for the time being, LCT is in a testing routine with the title "ISO/DIS 26022: Simulated Lane Change Test To Asses Driver Distraction" (International Organization for Standardization, 2007). Based on the fact that LCT is "freeware", easy to install and manage, and also features a tool to analyze driving data (called LCTA), it is a perfect main task for investigating human multitasking like in study I and II.

Participants Forty students of Technical University Berlin (same characteristics as in study I and in study II) fulﬁlling the same premises (no glasses, age 20-30, experienced in driving) participated study III.

Involved tasks

Figure 20.: Lane Change Task (taken from Mattes, 2003)

Figure 20 illustrates LCT: participants drive on a simulated highway and are asked to change the lane according to predeﬁned road signs. These signs contain three columns, two with crosses (i.e., the letter "x") and one with an arrow. This arrow indicates the position on which participants have to switch. To do LCT properly, four steps are included:

1.( 1) Perception (road sign)

2.( 2) Action (start maneuver)

3.( 3) Action (perform lane change)

4.( 4) Action (keep lane)

LCT measures the lane derivation: in Fig. 21, the green (even) line is the optimal lane calculated from the system itself based on a large amount of driving samples. This optimal lane can be modiﬁed by changing individual parameters (for a deeper insight, please have a look at the manual of LCT). The red, wiggly line shows the driving 3.3 study iii: the role of task configuration 49 behavior of a participant. The density between the two lines constitutes a value for the lane derivation. Lane derivation is the dependent variable for the main task (driving).

LANE DERIVATION UNDER SINGLE TASKING

LANE DERIVATION UNDER DUAL TASKING

Figure 21.: How to calculate lane derivation in the LCT

LCT is a standardized tool often applied in the context of driving studies. For a further description of the LCT, please see Mattes[ 2003]. In study III and IV, primary task in the scenario is the LCT, secondary task is the D2-Drive. Secondary task in study III was a systematic variation of D2-Drive-v1.2 (i.e., D2-Drive-v2.2) with the following features: d2-drive-v3.1 Constant row, no visual support d2-drive-v3.2 Changing row, no visual support d2-drive-v3.3 Constant row and visual support d2-drive-v3.4 Changing row and visual support

Visual support means that the current pattern was highlighted. This helps to remember the current pattern and there is no need to keep that position in mind. Also, a constant row allows (technically) to apply the "merge heuristic", i.e. the answers for several patterns can be kept in mind and replied combined. Please note that for all four versions, different assumptions deﬁne the hypotheses for study III (as further described in section 3.3.2).

Design In contrast to the previous two studies, complexity of D2-Drive was treated as within-subjects factor. To avoid learning effects, the order of the D2-Drive versions was balanced. The condition "single- vs. multitasking" was also treated as within-subject factor. Measure of performance for main task (Driving) was lane derivation, for secondary task (D2-Drive) number of correct patterns per minutes (i.e., trial). As before, the error rate in all three versions of D2-Drive approximated 50 empirical studies

zero: hence, the decrease in performance is reﬂected in the number of performed patters (i.e., patterns per min). Pre- as well as post-tests were applied to investigate possible learning effects.

Procedure Each participant had to perform the following (sequence of) steps:

1. Welcome and general introduction

2. Introduction of LCT

3. Training "Driving" (LCT)

4. Single task "Driving" (LCT, Baseline)

5. Introduction and Training "D2-Drive-v3.1"

6. Single task "D2-Drive-v3.1" (Pretest)

7. Introduction and Training "D2-Drive-v3.2"

8. Single task "D2-Drive-v3.2" (Pretest)

9. Introduction and Training "D2-Drive-v3.3"

10. Single task "D2-Drive-v3.3" (Pretest)

11. Introduction and Training "D2-Drive-v3.4"

12. Single task "D2-Drive-v3.4" (Pretest)

13. Introduction to eye tracking measuring

14. Checking of eye tracking system

15. Dual task (session) (including "D2-Drive-v3.1" - "D2-Drive-v3.4"

16. Single task "D2-Drive-v3.1" (Posttest)

17. Single task "D2-Drive-v3.2" (Posttest)

18. Single task "D2-Drive-v3.3" (Posttest)

19. Single task "D2-Drive-v3.4" (Posttest)

20. Structured interview

3.3.2 Hypotheses for study III

The following hypotheses in this subsection were derived from results of the previous studies in 3. As in study I and II, hypotheses refer both to driving behavior and to performance in the secondary task, under single as well as dual tasking.

Hypothesis: Independence of primary task (stability) As in the previous two studies, in study III driving is expected to remain stable under multitasking as it is instructed as primary task. No performance decrease is expected. Same counts for lane change behavior: as an effect of proper instruction and training, participants should not produce errors (i.e., assuming the constantly change to the correct lane). 3.3 study iii: the role of task configuration 51

Hypothesis: Inﬂuence of task conﬁguration The investigated main task in study III contains higher cognitive demands than the main tasks in the previous two studies. For this reason, performance in D2-Drive under multitasking is expected to strongly decrease under multitasking in comparison to single task performance. In D2-Drive-v3.1 and D2-Drive-v3.3, the pattern row does not change which offers the possibility to apply the "merge heuristic": responses can be anticipated and entered in a unit of several patterns. During this action (motor response), visual resources are "free" and participants can scan the lane and (visually) resume the primary task, i.e. driving.

3.3.3 Results of study III

Results are divided into driving behavior and performance in D2-Drive.

Results: Performance in driving Under multitasking condition, driving was not affected by the secondary task. Performance in LCT while D2-Drive was presented did not differ signiﬁcantly from LCT - performance under single task condition.

Results: Performance in D2-Drive

Figure 22.: Performance in D2-Drive in study III

Figure 22 shows that performance with D2-Drive is better if the pattern row remains the same (anticipation and merging is possible) compared to a version in which the row changes after each response. With other words, D2-Drive-v3.1 and D2-Drive-v3.3 outperform D2- Drive-v3.2 and D2-Drive-v3.4. This effect is statistically signiﬁcant for single tasking as well as multitasking. 52 empirical studies

Results: Eye tracking

D2.Drive-vrs. Gazes at Gazes at D2- Gazes at En- LCT Drive vironment D2-Drive-v3.1 60 35 5 D2-Drive-v3.2 44 51 5 D2-Drive-v3.3 71 26 3 D2-Drive-v3.4 50 46 4 The presented eye tracking data in study III support the assumption of the applied heuristics when doing multitasking. However, a deeper analysis and interpretation of the precise meaning and implication of the eye tracking data is not provided due to a missing embedding within a theoretical context. For upcoming studies, i highly recommend to synchronize the formulated hypotheses with the corresponding assumptions on expected eye tracking results.

Results: Structured Interview The structured interviews from the previous two studies were adapted and applied in study III.

3.3.4 Discussion of study III

In study III,

• performance in primary task (LCT) is not affected by multitasking

• performance in secondary task (D2-Drive) is highly inﬂuenced by the presence of a higher demanding primary task (LCT)

• conﬁguration of secondary task (D2-Drive) effects performance, even though this inﬂuence reduces under multitasking

• conﬁguration of secondary task (D2-Drive) strongly inﬂuences multitasking heuristic

Especially in those versions of D2-Drive in which the "merge heuristic" could not be applied (changing row), participants reported a (subjective) feeling of time pressure which is an inherent component of the structure of the task itself. Performance under multitasking thus seems to depend (a) on the task environment as well as (b) on situational components such as time pressure. These two issues are considered in study IV, investigating the impact of time pressure and system design. For this reason, the next study focused on the impact of time pressure under human multitasking, applying the identical experimental scenario.

3.4 study iv: amplification via time pressure

As study III illustrates, the structure of the involved tasks seems to have a tremendous inﬂuence on participants performance under multitasking. The way D2-Drive is designed supports or hinders performance but nevertheless, driving performance remains untouched by the applied secondary tasks. 3.4 study iv: amplification via time pressure 53

Figure 23.: Scenario in study IV

3.4.1 Method in study IV

Participants Forty subjects participated in study IV. Due to technical problems, four participants were excluded from the data analysis. Therefore, sample size for this study reduced to thirty six, with an equal gender proportion. Age distribution (within a range from 20 to 40 years) was not analyzed as not being of deeper interest for the research questions.

Involved tasks As in study III, primary (main) task is LCT. Secondary task again is D2-Drive. With reference to the structure of D2-Drive in the previous studies, two versions of D2-Drive were designed by cand. Dipl.-Psych. Robert Lischke, with the following requirements:

1. Clear distinction in terms of ease of use (subjective impression)

2. Clear distinction in terms of required time (reaction time)

3. Clear distinction in terms of necessary visual attention (eye tracking data)

Based on these requirements, the design was as follows: d2-drive-v4.1 Constant row of patterns and constant reply buttons d2-drive-v4.2 Changing row of patterns and changing reply buttons

D2-Drive-v4.1 allows participants to use the "merge heuristic". Sev- eral patterns can be scanned and replied together. Also, after a while, entering the solution without visual attention is expected due to the fact that both reply buttons (yes, no) remain at the same position. D2-Drive-v4.2 does not allow the use of the "merge heuristic" due to its structure: after each reply (motor action), the patterns change, i.e., a 54 empirical studies

new row with patterns appears. For instance, if a person performs the ﬁrst pattern (at the beginning of the row), after pressing a button (yes or no) this row changes and the person now has to perform the second pattern (which has not been shown before). The same counts for the reply buttons: after each button press, the positions of the buttons (yes, no) change, either one of them or both. Four possible conﬁgurations for the reply buttons are possible, as is shown in Fig. 24. To check the three claimed preconditions (ease of use, required time, necessary visual attention), in a pretest of these two versions, eight participants of the graduate program prometei tested both versions under single task condition.

D2-DRIVE-v4.1 D2-DRIVE-v4.2

YES YES YES YES YES

NO NO NO NO NO

Figure 24.:D 2-Drive in study IV

Design Two independent variables (time pressure, complexity level in D2- Drive) were investigated, both as within-subject factors. Please note that complexity of D2-Drive is within the secondary task and should not lead to any confusion plus not be considered the same as doing multitasking. Complexity in D2-Drive it the label for the difﬁculty levels in D2-Drive. As in the previous studies, dependent measures were performance in main task (LCT) and performance in secondary task (D2-Drive). For each participant, four conditions were applied.

Procedure 1. Welcome

2. Technical preparation (for physiological data measurement)

3. Relaxation period for participants

4. Baseline (Physiology)

5. Calibration of eye tracking equipment

6. Training main task (LCT) 3.4 study iv: amplification via time pressure 55

7. Training secondary task (D2-Drive-v4.1)

8. Single task (D2-Drive-v4.1, Pretest)

9. Training secondary task (D2-Drive-v4.2)

10. Single task (D2-Drive-v4.2, Pretest)

11. Dual task session (driving and D2-Drive-v4.1): low time pressure condition

12. Dual task session (driving and D2-Drive-v4.2): low time pressure condition

13. Dual task session (driving and D2-Drive-v4.1): high time pressure condition

14. Dual task session (driving and D2-Drive-v4.2): high time pressure condition

15. Single task (D2-Drive-v4.1, Posttest)

16. Single task (D2-Drive-v4.2, Posttest)

17. Demographical questionnaire (age, driving experience, etc.) the application of time pressure: Time pressure was applied via instruction as follows: participants were asked to consider driving as main task (priority A), and at the same time to perform (1) as many lance changes as possible and (2) as many patterns in D2-Drive as possible. low time pressure Prioritize (safe) driving and perform D2-Drive without neglecting LCT. high time pressure Consider driving as main task, but at the same time aim to perform as many lane changes (LCT) and as many patterns (D2-Drive) as possible.

3.4.2 Hypotheses for IV

Hypothesis I: The impact of time pressure Hypothesis I claims that time pressure has a negative impact, both on driving as well as on D2-Drive. Reported statements from study III recommend to consider time pressure as one of the main situational inﬂuences.

Hypothesis II: The impact of task complexity As suggested from the quantitative and qualitative data of the previous studies, D2-Drive-v4.1 is expected to provide a better performance compared to D2-Drive-v4.2. This is based on (1) the possibility to apply the "merge heuristic". 56 empirical studies

Figure 25.: The inﬂuence of time pressure on driving

3.4.3 Results of study IV

Results: LCT and time pressure Performance in LCT was not influenced by the secondary task: no significant difference in lane derivation was found between single tasking and multitasking. This goes in line with previous results from study I-IV. However, time pressure highly effects the driving behavior (p < 0.01), as can be seen in figure 27: under time pressure, lane derivation is pronounced twice as much as without time pressure. This indicates that the instruction given to the participants was taken seriously. It is also remarkable that after the driving (single task), the performance in LCT did even imporve (though not statistically significant). Lane derivation was lower under multitasking without time pressure compared to the performance under baseline driving. This can be explained in terms of learning effects.

Results: D2-Drive An overall comparison shows that D2-Drive-v4.1 is permanently better (in terms of number of correct patterns performed) compared to D2- Drive-v4.2. Please also note that learning effects (comparing pre-test vs. post-tests) occur for D2-Drive-v4.1 (p < 0.05) but not for D2-Drive-v4.2. Also, for D2-Drive-v4.1, number of performed patterns is significantly lower under multitasking (p < 0.05), but for D2-Drive-v4.2 there is 8sta- tistically) not difference between single- vs. multitasking. Interestingly, under multitasking, D2-Drive-v4.1 seems to receive some losses. For this reason, a more detailed analysis of the interplay between time pressure (factor 1) and task difficulty (factor 2) is necessary. Overall, time pressure has a significant influence on performance of D2-Drive (p < 0.05). Additionally, time pressure shows a stronger impact on D2-Drive-v4.2 (p < 0.01). This is surprisingly, also considering the fact that multitasking does not negatively influence D2-Drive-v4.2 3.4 study iv: amplification via time pressure 57

Figure 26.: Performance in D2-Drive in study IV

Figure 27.: Time pressure and performance in D2-Drive to the same amount as for D2-Drive-v4.1. Taken together all results, we can conclude that under multitasking and concurrent time pressure, D2-Drive-v4.2 performs quite worse and participants seem to fail completely. A look at ﬁgure 25 conﬁrms this assumption.

Results: Eye tracking Eye tracking data in study IV did not deliver any additional information. However, as illustrated in ﬁgure 28, gaze analysis helped identifying the (development and) application of the "merge heuristic", which was also mentioned in verbal reports by the participants.

3.4.4 Discussion of study IV

The two main ﬁndings in study IV are:

1. Time pressure highly inﬂuences both primary as well as secondary task. 58 empirical studies

Figure 28.: Tracking eye movements in D2-Drive)

2. Time pressure accelerates the development of cognitive heuristics.

Participants in study IV reported a strong feeling of time pressure and felt highly motivated to "optimize" their performance in terms of (1) number of lane changes (LCT), and (2) number of performed patterns (D2-Drive).

in chapter iv the results of the four empirical studies will be summarized and a critical analysis is given. Part IV

DISCUSSION

CRITICALDISCUSSION 4

4.1 scope and findings

Observations of human behavior in real life gave birth to the idea of this work. Modern technology as the core domain in which multiple daily tasks occur at the same time was chosen as research area for the empirical studies presented in this work. An overall review of studies in the ﬁeld (see chapter II) of human multitasking showed that

• many (if not most) psychological studies remain artiﬁcial. Espe- cially the reported task switching or PRP approaches (an excellent overview provides Pashler[ 2000]) do not offer the possibility to draw conclusions about human behavior in daily life.

• applied studies in the ﬁeld of human machine interaction mainly lack task repetition and systematic control (as in the studies reported by Saluvicci[ 2005]). Also, theory embedment is missing quite often.

• contemporary approaches (e.g., Brumby et al.[ 2007]) mainly focus on optimization and do not fully integrate qualitative assumptions about human (conscious and unconscious) information processing.

• the aspect of dynamically changing task environments has hardly been considered.

These critical remarks serve to show the starting point of this work. After an introduction (chapter I) and a general overview of the history of human multitasking (chapter II), four applied studies in the ﬁeld of human machine interaction were presented (chapter III), illustrating

1. ... how people in general do human multitasking using heuristics instead of trying optimization (study I).

2. ... to what extend training and extensive task repetition impacts human multitasking and applied heuristics (study II).

3. ... the importance of task conﬁguration (study III).

4. ... what happens with human behavior given people have a subjective feeling of time pressure (study IV).

This work, for sure, is not the final cut in studying human multitasking. A specific domain (driving plus a secondary task) was chosen as research domain. Based on the empirical studies of this work, I would like to summarize the findings. I supported evidence for the following results: cognitive heuristics highly support human multitasking in dynamically changing environments (study I). This was shown for two different main tasks (a simple driving task in a simulator and using the lane change task).

63 64 critical discussion

extensive training does not only support the use of cognitive heuristics but also improves overall performance (study II). the configuration of a task turns out to be determining how people manage multiple, concurrent tasks (study III). time pressure tremendously inﬂuences multitasking performance (study IV). Both primary as well as secondary task highly suffer under instructed time pressure. As reported in the theoretical part (chapter II), recent approaches have been started to simulate human multitasking behavior in a cognitive architecture. To fully develop a multitasking mechanism for cognitive modeling was not the scope of this work. However, partially an implementation has been done, as will be reported in the next section.

4.2 cognitive modeling

To additionally conﬁrm theoretical assumptions about the "merge heuristic", a computational model within the cognitive architecture ACT-R (Anderson[ 2007]) was built, based on version D2-Drive-v1.2. ACT-R is a production system, information processing is simulated via a collection of rules. ACT-R contains "condition-action-pairs", i.e. if (a condition is met), then (do the following action) - pairs, and is a hybrid cognitive architecture: it features a symbolic level (production system, sequential processes) as well as a sub-symbolic level (parallel processes, utility-based selection). Main components of ACT-R are modules (e.g., the perceptual-motor (PM) module is the interface with the - simulation of - the real world), buffers, and a pattern matcher. Memory in ACT-R is either declarative (facts about the world) or procedural (knowledge about how we do things). Many successful models have been implemented in ACT-R, mainly in the areas of learning and memory, problem solving and decision making, language and communication, perception and attention, cognitive development and individual differences. Con- ceptual reasons for choosing ACT-R in my work are its assumptions on human memory, the psychological plausibility and the interaction with environment. Before I introduce results of the ACT-R model of D2-Drive, let me give a short explanation of cognitive modeling, as illustrated in Fig. 29:

Figure 29.: Cognitive modeling (following Taatgen[ 1999])

According to Werner H. Tack (personal communication), cognitive modeling is meant to be a "simulation of human problem solving". Tack 4.3 design recommendations 65

[1995] refers to it as "defining symbol structures for specific cognitive tasks". Cognitive modeling is not a metaphor for the mind. Its goal is the prediction of human behavior. Mental task processes are specified and cognitive tasks are performed by computing (Lewis[ 1999]). A cognitive architecture 1 contains theoretical assumption (theory) about cognitive processes. In combination with task knowledge, a task model is build and derived. This task model produces performance data (see Fig. 29). Traditional measures are time to perform the task, accuracy in the task, or neurological (fMRI) data. From another side, a (psychological) experiment produces data as well. This data is analyzed and compared to the data produced by the model. The matching between model data and empirical data defines the goodness of the model. Both on a micro-level (performing of a single pattern) as well as on a macro-level, the model describes the processing how participants performed D2-Drive.

d-pattern p-pattern

pattern processing READ middle READ middle at the beginning

READ upper READ upper

READ lower READ lower YES NO

REPLY (d) REPLY (p) implicit learning

READ middle READ middle pattern processing after training READ upper READ upper

NO READ lower READ lower

YES REPLY (d) REPLY (p)

Figure 30.: Processing of a single pattern in D2-Drive

Fig. 30 shows the basic assumptions for the corresponding ACT-R model (see also Kiefer et al.[ 2006], for more details). We generally distinguish "d-patterns" (patterns containing the letter "d") from "p- patterns" (patterns containing the letter "p"). In both cases, participants start on one position (i.e., the middle, which is at the same time the letter-component in the pattern). Next, they scan the upper part and then the lower part. After a while, implicit learning takes place, participants "understand" that with "p-patterns", the last two steps are not necessary. This leads to shorter reaction times for p-patterns (see left part of Fig. 31). The implemented Act-R model ﬁts the data pretty well, especially performance of "p-patterns".

4.3 design recommendations

Visiting national and international conferences to present preliminary results of my work, it happened quite often that after my presenta-

1 an algorithm that simulates a non-linear theory of cognition, based on Taatgen[ 1999] 66 critical discussion

Figure 31.: Modeling of D2-Drive

tion/talk, people coming from different disciplines (e.g., designers, engineers) came to me asking me about advice. "What would you recommend me if i tell you that i am planning to develop an in-vehile information system?" Or: "Which mistakes can i avoid when I design an information system which should be used in daily life?" Derived from the results described in the previous chapter and from the insight i won during the last years doing theoretical and practical work in that area, here is my advice. Some of the recommendations might be helpful ideas, with the hope to prevent from avoidable errors in multitasking situations:

do not force people to multitask! Giving advice how to perform two tasks concurrently not only prevents from developing individual strategies or heuristics, but also leads to a direction in processing which might be suboptimal for participants‘ individual style. Learning, from a psychological perspective, happens consciously or unconsciously. Remember that some participants in the previous four studies could not even report that they used the "merge heuristic", i.e., they learned implicitly i instead of rule-based. Instructions do not allow such a possibility.

reduce complexity! The more complex a system, i.e. the higher the functionality, the more difficult it is to comprehend. We saw in the first two studies that D2-Drive-v1.3 (and D2-Drive- v2.3 consequently) performed worst. Please keep in mind that a more demanding system requires a longer learning period until individual steps can be taken without mental load or cognitive demand. What is the benefit of a extremely functional mobile phone if it takes you months (or years) to understand the structure and the navigation? Cognitive demanding tasks produce human errors, so always ask yourself: is the benefit worth the effort?

support familiarity! As study III and IV show, a dynamically changing secondary task requires visual (re-)orientation as well as cognitive (resumption) costs. Part of a task needs to be resumed, but the mental set has changed (i.e., the positions of the answer- buttons). Familiarity does not require many cognitive resources. Therefore, a system which is easy to understand, learn and become familiar with, can be accessed immediately and promotes human multitasking.

apply prospective design! Do not mainly focus on results from studies in literature. Even if prominent theories advice you to 4.4 criticism and outlook 67

do this or that (this work is not an exception, by the way), try to investigate the human-machine interaction already in the early development phases of your planned technical systems.

These recommendations mainly result from an overall impression of the studies in the previous chapter, supported by quantitative (performance data) as well as qualitative (interview data) measures.

4.4 criticism and outlook

Aim of the presented work was to approach human behavior in multitasking scenarios from a human machine interaction perspective. In- cluding four copious empirical studies, it can only give an extract of relevant issues in context of human multitasking. When i started in 2005, i shortly realized that investigating human multitasking and the necessary, connected domains feels like opening Pandoras box1.

Figure 32.: Pandoras box, found on: www.icarusgirl.blogspot.com

For this work can never be complete by going into all details, three issues are critically discussed:

1. The role of (prospective) memory in human multitasking (theoretical aspect)

2. Domain (in-)dependence (practical aspect)

3. Need for a (computational) model of human multitasking (modeling aspect)

The following sections concentrate on these three issues.

1 In Greek mythology, Pandora was the ﬁrst woman. Each god helped create her by giving her unique gifts. Zeus ordered Hephaestus to mold her out of Earth as part of the punishment of mankind for Prometheus theft of the secret of ﬁre, and all the gods joined in offering this beautiful evil seductive gifts. According to the myth, pandora opened a jar in modern accounts referred to as Pandoras box, releasing all the evils of mankind (greed, vanity, slander, envy, pining) leaving only hope inside once she had closed it again. 68 critical discussion

4.4.1 The role of memory in human multitasking

Each task interrupted becomes a prospective task? In study I-IV, design and complexity of the applied secondary task (i.e. D2-Drive, developed by Kiefer et al.[ 2006]) have been systematically analyzed and varied. However, in study I and II, the same main task (lane keeping on a rather monotonous street) was applied. To increasing complexity and cognitively enrich the main task, study III and IV focus on LCT as main task. In a recent study by Soyak, a modiﬁcation of LCT based on the hypotheses in this study throws some new light to the scenario. Soyak points out the importance of prospective memory ("remembering to remember", Winograd[ 1988]). In the area of task interruption, Dodhia and Dismukes claim that "a task interrupted becomes a prospective task".

Figure 33.: Modiﬁcation of LCT in a prospective task study

McDaniel and Einstein[ 2000] distinguish between two kinds of prospective memory (PM), namely:

event-based pm Recalling an action or an intention triggered by a stimulus ("event"), e.g. receiving a reminder-email ("cue") reminds to submit a paper ("intention")

time-based pm Recalling an action or an intention triggered by a time, e.g. watching the news in television at 8pm.

Based on McDaniel and Einstein[ 2000], Soyak investigated the impact of disruptions on prospective memory performance. Main task in her study was a modiﬁed version of LCT (see Fig.33: participants were asked to keep in mind a verbally given city name (Hamburg, Berlin, München, Stuttgart, Köln, Leipzig) and change the lane at the moment the road sign with this name appears. Soyak showed a negative inﬂuence of disruptions on successful prospective task performance. Delayed disruptions require longer reaction times on the target cue. In her study, Soyak used a more demanding, cognitively enriched main task and concurrently the two versions of D2-Drive mentioned in study IV of the previous chapter. Eye tracking data were recorded but not yet analyzed (purpose of her work was not on cognitive heuristics under multitasking). Nevertheless, it would be of interest to look 4.4 criticism and outlook 69 for participants processing in the two tasks individually and under multitasking. We can assume the following:

1. Modiﬁed LCT require to many cognitive resources and heuristics for D2-Drive cannot be applied.

2. Participants use the "merge heuristic" in a reduced frequency.

3. Due to the new task conﬁguration, participants develop new heuristic(s) adaptively.

In context of human multitasking, the author highlights the importance to take memory aspects under deeper consideration. The importance of the prospective task seems to be a core aspect in each multitasking scenario.

4.4.2 Domain independence

All of the four presented studies in this work include a driving task (as main task) in the multitasking scenario. The question, hence is whether this multitasking behavior, the application of heuristics and information processing, can be transferred to other situations in real life in which people use technical systems while doing a continuous "task" (e.g., walking on the street). Studies by Antti Oulasvirta from the Helsinki Institute of Technology (HIIT) (Oulasvirta and Blom[ 2008], Oulasvirta et al.[ 2007], Oulasvirta and Saariluoma[ 2006], Oulasvirta et al.[ 2005], Oulasvirta and Saariluoma[ 2005]) show that in fact, in various situations, an ongoing task needs to be interrupted and resumed. Oulasvirta provides both empirical data (eye tracking) as well as a qualitative analysis to explain how multitasking in these scenarios is explained. Especially the area of mobile computing is a promising ﬁeld for further research. Modern technology for us is a constant challenge to develop fast and frugal (conscious or unconscious) "heuristics" to adapt to a dynamically changing environment. This direction in the area of human machine interaction studies in this direction will become of increasing importance in future research.

4.4.3 Need for a computational model of human multitasking

In this chapter, a computational model of the applied secondary task was introduced and explained. However, a general model for human multitasking is still missing (though it was not meant to be part of this work). Saluvicci[ 2005] proposes an approach in which he aims to incorporate human multitasking in cognitive modeling, namely within the ACT-R architecture (Anderson et al.[ 2004], Anderson[ 2005]. Salvucci proposes an general executive which is

• an architectural mechanism

• dependent on time

• sensitive to goal representations

Fig34 represents core mechanisms within this idea. Main features of the general executive proposed by Saluvicci[ 2005] are: 70 critical discussion

Figure 34.: Overview of multitasking general executive proposed by Saluvicci [2005]

• A cognitive processor manages the concurrency of multiple goals at the same time.

• As in earlier frameworks of ACT-R, only one goal can be executed at the same time.

• Goal switching is moderated by heuristics.

• Urgency deﬁnes when to switch to which goal.

Salvucci mentions "natural breaking points" necessary for interleav- ing tasks. He proposes two core heuristics to decide when to switch between goals, namely

the iterating heuristic Salvucci gives the example of a task with a duration of 100 sec, assuming 50msec per production rule, so in sum 2.000 rules to ﬁre. When the model returns to a previous ﬁred rule, task switching should be proposed at this point. The next iteration is initiated by a new goal. Especially for models with a long duration in terms of execution time, this heuristic becomes plausible.

the blocking heuristic Salvucci mentions "signiﬁcant time" and the problem how to decide about that. He illustrates that for perceptual motor actions (PM) in particular, ACT-R has to wait until an action is done. In such a case, the blocking heuristic proposes to create a new goal which gives permission to a secondary task to intercede.

4.5 fmri studies on multitasking

The reported work did not consider functional magnetic resonance imaging (fMRI) scans. However, some studies (e.g., Leber et al.[ 2008]) have revealed that superior multitasking performance is correlated 4.6 popular stereotypes about multitasking 71 with higher basal ganglia, anterior cingulate cortex, prefrontal cortex, and parietal cortex activity. Philippe Peigneux, Professor of Clinical Neuropsychology in Brussels, even talks of a multitasking mind (published in April 2006 on seedmagazine.com), meaning that even when we sleep, our mind works constantly, processing several tasks concurrently. In future, neuro-scientiﬁc studies will dive deeper into the area of multitasking.

4.6 popular stereotypes about multitasking

4.6.1 Multitasking and happiness

Multitasking has been criticized as a hindrance to completing tasks or feeling happiness. Timothy Ferriss argues that one should rarely do multitasking and should instead devote full attention to completing a very small set of deﬁned goals (taken from the interview "I receive 500 to 1000 emails per day", published in The Economist on 2008-04-04). Barry Schwartz has noted that, given the media-rich landscape of the Internet era, it is tempting to get into a habit of dwelling in a constant sea of information with too many choices, which has been noted to have a negative effect on human happiness.

BIBLIOGRAPHY

E.M. Altmann and J.G. Trafton. Memory for goals: An activation-based model. Cognitive Science, 26:39–83, 2002. (Cited on pages 24, 26, 29, and 35.)

J. R. Anderson. Human symbol manipulation within an integrated cognitive architecture. Cognitive Science, 29(3):313–341, 2005. (Cited on page 69.)

J. R. Anderson, D. Bothell, M. D. Byrne, S. Douglass, C. Lebiere, and Y . Qin. An integrated theory of the mind. Psychological Review, 111(4): 1036–1060, 2004. (Cited on page 69.)

J.R. Anderson. Acquisition of cognitive skill. Psychological Review, 89: 369–403, 1982. (Cited on page 19.)

J.R. Anderson. How Can the Human Mind Occur in the Physical Universe? Oxford University Press, New York, 2007. (Cited on pages 29 and 64.)

R. Borger. The refractory period and serial choice reactions. Quarterly journal of Experimental Psychology, 15:1–12, 1963. (Cited on page 18.)

R. Brickenkamp. Test D2, Aufmerksamkeits-Belastungs-Test. Hogrefe Verlage, Schweiz, 1992. (Cited on pages 36, 37, 39, and 44.)

C. Brod. Technostress: The human cost of the computer revolution. Addison- Wesley Reading, MA, 1984. (Cited on page 8.)

D.P. Brumby, D.D. Saluvicci, and A. Howes. Dialing while driving? A bounded rational analysis of concurrent multi-task behavior. In Proceedings of the 8th International Conference on Cognitive Modeling, Michigan, 2007. Ann Arbor, Michigan, USA. (Cited on pages 35 and 63.)

E. C. Cherry. Some experiments on the recognition of speech ears. Journal of the Acoustical Society of America, 25, 1953. (Cited on page 10.)

J.P. Chin, V.A. Diehl, and K.L. Norman. Development of a tool measuring user satisfaction of the human-computer interface. In ACM SIGCHI, volume 88, pages 213–218, 1988. (Cited on page 10.)

G. Gigerenzer and R. Selten. Bounded rationality: The adaptive toolbox. the MIT Press, 2002. (Cited on page 30.)

T. Gillie and D. Broadbent. What makes interruptions disruptive? a study of length, similarity, and complexity. Psychological Research, 50: 243–50, 1989. (Cited on page 23.)

P. Green. Estimating compliance with the 15-second rule for driver- interface usability and safety. In Proceedings of the Human Factors and Ergonomics Society 43rd Annual Meeting, Santa Monica, 1999. Human Factors and Ergonomics Society. (Cited on page 7.)

A. G. Greenwald. On doing two things at once: I. time-sharing as a function of ideomotor compatibility. Psychological Review, 100:52–57, 1972. (Cited on page 20.)

73 74 bibliography

A. G. Greenwald and H. Shulman. On doing two things at once: Ii. elimination of the psychological refractory period. Psychological Review, 101:70–76, 1973. (Cited on page 20.)

Nowlis H. H. The inﬂuence of success and failure on the resumption of an interrupted task. Journal of Experimental Psychology: General, 28(4): 304–325, 1941. (Cited on page 23.)

E. Hazeltine, D. Teague, and R. B. Ivry. Simultaneous dual-task performance reveals parallel response selection after practice. Journal of Experimental Psychology: Human Perception and Performance, 28(3): 527–545, 2002. (Cited on page 20.)

A. Howes, R.L. Lewis, A. Vera, and J. Richardson. Information Re- quirements Grammar: A theory of the structure of competence for interaction. In Proceedings of the 27 th Annual Meeting of the Cognitive Science Society, Stresa, Italy, 2005. (Cited on pages 26 and 29.)

T. Jersild. Mental set and shift. Archives of Psychology, 1927. (Cited on pages 16, 17, and 28.)

I. Juvina and N.A. Taatgen. Modeling control strategies in the N-back task. In Proceedings of the eight International Conference on Cognitive Modeling, pages 73–78, New York, 2007. Psychology Press. (Cited on page 26.)

D. Kahneman. Attention and effort. NJ: Prentice-Hall, Englewood Cliffs, 1992. (Cited on pages 26 and 27.)

D. Kahneman and A. Tversky. On the psychology of prediction. Psy- chological Review, 80:237–251, 1973. (Cited on page 30.)

J. Kiefer, M. Schulz, D. Schulze-Kissing, and L. Urbas. Multitasking- Strategien in der Mensch-Maschine-Interaktion. MMI-Interaktiv, 11: 26–42, 2006. (Cited on pages 65 and 68.)

Y. Kushleyeva, D.D. Saluvicci, and F.J. Lee. Deciding when to switch tasks in time-critical multitasking. Cognitive Systems Research, 6:41–49, 2005. (Cited on pages 25 and 26.)

A.B. Leber, N.B. Turk-Browne, and M.M. Chun. Neural predictors of moment-to-moment ﬂuctuations in cognitive ﬂexibility. Proceedings of the National Academy of Sciences, 105(36):13592, 2008. (Cited on page 70.)

F.J. Lee and N.A. Taatgen. Multi-tasking as Skill Acquisition. In Pro- ceedings of the twenty-fourth annual conference of the cognitive science society, Fairfax, VA, 2002. ahwah, NJ: Erlbaum. (Cited on pages 19, 21, and 29.)

R.L. Lewis. Cognitive modeling, symbolic. MIT Press, Cambridge, MA, 1999. (Cited on page 65.)

S. Mattes. The lane-change-task as a tool for driver distraction evaluation. In Proc. of IGfA, 2003. (Cited on pages 47 and 49.)

J.S. McCarley, M. Vais, H. Pringle, A.F. Kramer, D.E. Irwin, and D.L. Strayer. Conversation disrupts scanning and change detection in complex visual scenes. Human Factors, 46, 2004. (Cited on page 7.) bibliography 75

M.A. McDaniel and G.O. Einstein. Strategic and automatic processes in prospective memory retrieval: A multiprocess framework. Applied Cognitive Psychology, 14:127–144, 2000. (Cited on pages 25 and 68.)

B. McElree. Working memory and focal attention. Journal of Experimental Psychology: Learning, Memory and Cognition, 27:817–835, 2001. (Cited on page 26.)

D. C. McFarlane and K. A. Latorella. The scope and importance of human interruption in human-computer interaction design. Human- Computer Interaction, 17(1):1–61, 2002. (Cited on page 23.)

D.C. McFarlane. Interruption of People in Human-Computer Interaction (Dissertation). George Washington University, Washington, 1998. (Cited on page 23.)

N. Meiran. Reconﬁguration of processing mode prior to task performance. Journal of Experimental Psychology: Learning, Memory, and Cognition, 22:1423–1442, 1996. (Cited on page 17.)

N. Meiran and A. Daichman. Advance task preparation reduces task error in the cuing task-switching paradigm. Memory and Cognition, 33(7):1272–1288, 2005. (Cited on page 20.)

R.F.I. Meuter and A. Allport. Bilingual language switching in naming: Asymmetrical costs of language selection. Journal of Memory and Language, 40:25–40, 1999. (Cited on page 9.)

D. E. Meyer and D. E. Kieras. A computational theory of executive cognitive processes and multiple-task performance: Part 1. basic mechanisms. Psychological Review, 104:3–65, 1997a. (Cited on pages 18, 20, and 23.)

D. E. Meyer and D. E. Kieras. A computational theory of executive cognitive processes and multiple-task performance: 2. accounts of psychological refractory-period phenomena. Psychological Review, 104:749–791, 1997b. (Cited on pages 18, 20, and 23.)

S. Monsell. Control of mental processes, pages 93–148. Erlbaum (UK), Hove, E. Sussex, 1967. (Cited on page 20.)

D.A. Norman. Affordance, conventions, and design. interactions, 6(3): 38–43, 1999. (Cited on page 10.)

A. Oulasvirta and J. Blom. Motivations in personalisation behavior. Interacting with Computers, 20(1):1–16, 2008. (Cited on page 69.)

A. Oulasvirta and P. Saariluoma. Long-term working memory and interrupting messages in human-computer interaction. Behavior and Information Technology, 23(1):53–64, 2005. (Cited on page 69.)

A. Oulasvirta and P. Saariluoma. Surviving task interruptions: In- vestigating the implications of long-term working memory theory. International Journal of Human-Computer Studies, 64(10):941–961, 2006. (Cited on page 69.)

A. Oulasvirta, L. Kärkkäinen, and J. Laarni. Expectations and memory in link search. Computers in Human Behavior, 21(5):773–789, 2005. (Cited on page 69.) 76 bibliography

A. Oulasvirta, R. Petit, M. Raento, and S. Tiitta. Interpreting and acting on mobile awareness cues. Human-Computer Interaction, 22 (1,2):97–135, 2007. (Cited on page 69.)

M. Ovsiankina. Die wiederaufnahme unterbrochener handlungen. Psychologische Forschung, 11(1):302–379, 1928. (Cited on page 23.)

A. M. Owen, K. M. McMillan, A. R. Laird, and E. Bullmore. N-back working memory paradigm: A meta-analysis of normative functional neuroimaging studies. Human Brain Mapping, 25:46–591, 2005. (Cited on page 26.)

H. Pashler. Task switching and multitask performance. MA: MIT Press, Cambridge, 2000. (Cited on pages 17, 18, 20, 22, 28, 29, 35, and 63.)

H. Pashler. Processing stages in overlapping tasks: Evidence for a central bottleneck. Journal of Experimental Psychology: Human Perception and Performance, 10:358–377, 1984. (Cited on page 20.)

H. Pashler. Doing two things at the same time. American Scientist, Jan-Feb:47–56., 1993. (Cited on pages 20, 21, and 35.)

J. Rasmussen. Skills, rules, and knowledge; signals, signs and symbols, and other distinctions in human performance models. IEEE Transac- tions on Systems, Man, and Cybernetics, 13(3):257–266, 1983. (Cited on page 19.)

R. Rogers and S. Monsell. The costs of a predictable switch between simple cognitive tasks. Journal of Experimental Psychology: General, 124:207–231, 1995. (Cited on pages 5, 9, and 20.)

J. S. Rubinstein, D. E. Meyer, and J. E. Evans. Executive control of cognitive processes in task switching. Journal of Experimental Psychol- ogy: Human Perception and Performance, 27(4):763–797, 2001. (Cited on pages 9 and 17.)

E. Ruthruff, J. C. Johnston, M. Van Selst, S. Whitsell, and R. Rem- ington. Vanishing dual-task interference after practice: Has the bottleneck been eliminated or is it merely latent? Journal of Exper- imental Psychology-Human Perception and Performance, 29(2):280–289, 2003. (Cited on page 20.)

D.D. Saluvicci. A multitaksing general executive for compound continuous tasks. Cognitive Science, 29:457–492, 2005. (Cited on pages 6, 16, 21, 22, 23, 28, 29, 35, 63, 69, and 70.)

D. D. Salvucci. An integrated model of eye movements and visual encoding. Cognitive Systems Research, 1(4):201–220, 2001. (Cited on page 23.)

W. Schneider and R.M. Shiffrin. Controlled and automatic human information processing: I. detection, search, and attention. Psychological Review, 84(1):1–66, 1977. (Cited on pages 18, 19, and 44.)

E. H. Schumacher, T. L. Seymour, J. M. Glass, D. E. Fencsik, E. J. Lauber, D. E. Kieras, and D. E. Meyer. Virtually perfect time sharing in dual-task performance: Uncorking the central cognitive bottleneck. Psychological Science, 12(2):101–108, 2001. (Cited on pages 21, 28, 29, and 44.) bibliography 77

R. E. Smith. The cost of remembering to remember in event-based prospective memory: Investigating the capacity demands of delayed intention performance. Journal of Experimental Psychology: Learning Memory and Cognition, 29(3):347–361, 2003. (Cited on page 25.)

R. E. Smith and U. J. Bayen. A multinomial model of event-based prospective memory. Journal of Experimental Psychology: Learning Memory and Cognition, 30(4):756–777, 2004. (Cited on page 25.)

M. H. Sohn and J. R. Anderson. Task preparation and task repetition: Two-component model of task switching. Journal of Experimental Psychology: General, 130:764–778, 2005. (Cited on page 20.)

A. Spector and I. Biederman. Mental set and mental shift revisited. American Journal of Psychology, 89:669–679, 1976. (Cited on page 17.)

P. Steel. The nature of procrastination: A meta-analytic and theoretical review of quintessential self-regulatory failure. Psychological Bulletin, 133(1):65, 2007. (Cited on page 5.)

B.S. Stewart, C.F. Liaw, C.C. White III, U.S.W. Technol, and CO Denver. A bibliography of heuristic search research through 1992. IEEE Trans- actions on Systems, Man and Cybernetics, 24(2):268–293, 1994. (Cited on page 30.)

D. L. Strayer and W. A. Johnston. Driven to distraction: Dual-task studies of simulated driving and conversing on a cellular phone. Psychological Science, 12:462–466, 2001. (Cited on page 7.)

N. Taatgen. Learning without Limits: From Problem Solving towards a Uniﬁed Theory of Learning (PhD Thesis). Rijksuniversiteit Groningen, Groningen, 1999. (Cited on pages 64 and 65.)

N. A. Taatgen, D. Huss, D. Dickison, and J. R. Anderson. The acquisition of robust and ﬂexible cognitive skills. Journal of Experimental Psychology: General. (Cited on page 18.)

N. A. Taatgen, D. Huss, and J. R. Anderson. How cognitive models can inform the design of instructions. British Journal of Psychology, 40: 23–40, 2006. (Cited on page 18.)

N.A. Taatgen. Modeling parallelization and speed improvement in skill acquisition: from dual tasks to complex dynamic skills. Cognitive Science, 29:421–455, 2005. (Cited on pages 16, 28, 29, 39, 40, and 44.)

W. H. Tack. Wege zu einer differentiellen kognitiven Psychologie, volume 2, pages 172–185. Hogrefe, Göttingen, 1995. (Cited on page 64.)

C.W. Telford. The refractory phase of voluntary and associative responses. Journal of Experimental Psychology, 14:1–36, 1931. (Cited on pages 17 and 28.)

J. G. Trafton, E. M. Altmann, D. P. Brock, and F. E. Mintz. Preparing to resume an interrupted task: Effects of prospective goal encoding and retrospective rehearsal. International Journal of Human-Computer Studies, 58:583–603, 2003. (Cited on page 24.)

L. Urbas, S. Dzaack J. Schulze-Kissing, D.and Leuchter, J. Kiefer, and M. Heinath. [Programmbeschreibung D2-Drive-Aufmerksamkeitstest (Manual for D2-Drive Test of Attention). 2005. 78 bibliography

A. van Bergen. Task Interruption. North-Holland, Amsterdam, 1968. (Cited on page 23.)

M. Vince. Rapid response sequences and the psychological refractory period. British Journal of Psychology, 40:23–40, 1949. (Cited on page 17.)

A.T. Welford. The psychological refractory period and the timing of highspeed performance. British Journal of Psychology, 43:2–19, 1952. (Cited on pages 17 and 18.)

C.D. Wickens. Multiple resources and performance prediction. Theoreti- cal Issues in Ergonomics Science, 3(2):159–177, 2002. (Cited on pages 27 and 39.)

C.D. Wickens. Multiple Resource Time Sharing Mode, pages 77–105. CRC Press, London, 2004. (Cited on pages 27 and 28.)

C.D. Wickens. Multiple Resource Model of Human Performance: implications for Display Design. In AGARD/NATO Proceedings, Williams- burg, VA., 1984. AGARD. (Cited on page 27.)

C.D. Wickens and Y. Liu. Codes and modalities in multiple resources: a success and a qualiﬁcation. Human Factors, 30:599–616, 1988. (Cited on pages 27 and 28.)

E. Winograd. Some observations on prospective remembering, volume 2, pages 348–353. MA: MIT Press, Chichester: Wiley, 1988. (Cited on pages 25 and 68.)

N. Yeung and S. Monsell. Switching between tasks of unequal familiarity: The role of stimulus-attribute and response-set selection. Journal of Experimental Psychology-Human Perception and Performance, 29(2):455–469, 2003. (Cited on page 9.)

B. Zeigarnik. Das behalten erledigter und unerledigter handlungen. Psychologische Forschung, 9:1–85, 1927. (Cited on page 23.)

B. Zeigarnik. On ﬁnished and unﬁnished tasks. Humanities press, New York, 1967. (Cited on page 23.) DECLARATION

I hereby declare that:

• I autonomously carried out the PhD-thesis entitled "Multitasking in HMI". All third party assistance has been enlisted.

• My submission as a whole is not substantially the same as any that I have previously made or am currently making, whether in published or unpublished form, for a degree, diploma, or similar qualiﬁcation at any university or similar institution.

The thesis has not been submitted elsewhere for an exam, as thesis or for evaluation in a similar context.

Berlin 2010 D 83, Tag der wissenschaftlichen Aussprache: 05.10.2009

Dipl.-Psych. Jürgen Kiefer

APPENDIX A a.1 appendix: structured interview

The following questions (qualitative interview) were put to the participants after the experiment:

• Which of the two tasks did you perform with prioritization?

• Which of the two tasks did you experience as more difﬁcult?

• Please explain how you proceeded the both tasks?

• On a scale from 1 to 5 (where 1 is the lowest and ﬁve is the highest value), how much mental fatigue during the complete experiment?

• Can you in detail describe how you performed the driving task as a single task?

• Can you in detail describe how you performed the pattern task as a single task?

• Can you in detail describe how you performed the combination of both tasks?

In addition to these questions, for all of the four reported studies, a verbal qualitative interview on the task processing of all participants was applied.