Cognitive Work Analysis: an Influential Legacy Extending Beyond Human

Applied Ergonomics xxx (2016) 1e13

Contents lists available at ScienceDirect

Applied Ergonomics

journal homepage: www.elsevier.com/locate/apergo

Cognitive work analysis: An inﬂuential legacy extending beyond human factors and engineering

Neelam Naikar

Centre for Cognitive Work and Safety Analysis, Joint and Operations Analysis Division, Defence Science and Technology Group, 506 Lorimer St, Fishermans Bend, Victoria 3207, Australia article info abstract

Article history: Jens Rasmussen’s multifaceted legacy includes cognitive work analysis (CWA), a framework for the Received 1 April 2015 analysis, design, and evaluation of complex sociotechnical systems. After considering the framework’s Received in revised form origins, this paper reviews its progress, predictably covering experimental research on ecological 25 April 2016 interface design, case studies of the application of CWA to human factors and engineering problems in Accepted 7 June 2016 industry, and methods and modelling tools for CWA. Emphasis is placed, however, on studying the nexus Available online xxx between some of the recent results obtained with CWA and the original ﬁeld studies of human problem- solving that motivated the framework’s development. Of particular interest is a case study of the use of Keywords: ﬁ Cognitive work analysis CWA for military doctrine development, a problem commonly regarded as lying outside the elds of Problem-solving human factors and engineering. It is concluded that the value of CWA, even for such diverse problems, is Reasoning likely to result from its conceptual grounding in empirical observations of patterns of human reasoning Sociotechnical systems in complex systems. Jens Rasmussen Crown Copyright © 2016 Published by Elsevier Ltd. This is an open access article under the CC BY-NC-ND license (http://creativecommons.org/licenses/by-nc-nd/4.0/).

1. Introduction activity analysis (Naikar et al., 2006) and the diagram of work organisation possibilities for social organisation and cooperation In the design of complex sociotechnical systems, cognitive work analysis (Naikar and Elix, 2015, 2016a, 2016b) are more recent ad- analysis (CWA) provides a distinctive perspective. Described as ditions that are consistent with the conceptual foundations of CWA. formative, this framework considers the constraints on actors’ The work of Jens Rasmussen, which is embodied in part by the behaviour to be the primary unit of analysis for design, rather than concepts of CWA, has been lauded as representing “nothing less the details of those behaviours (Rasmussen, 1986; Rasmussen et al., than a paradigm shift” (Moray, 1988, p.12). Moray’s discourse fo- 1994; Vicente, 1999). While these constraints limit the possibilities cuses on the skills, rules, and knowledge taxonomy, as he sees this for action available to actors, within these constraints actors still tool as having had the most widespread impact in discussions of have many remaining degrees of freedom for behaviour. Therefore, complex human-machine systems. This observation may still be designs based on constraints can promote worker adaptation to relevant, as the skills, rules, and knowledge taxonomy continues to local contingencies, which cannot always be predicted a priori, motivate discussions in such diverse contexts as mental workload, within the boundaries on effective performance. human error, accident analysis, display design, and operator Table 1 summarises the essential features of CWA. This frame- training, often independently of CWA. However, it is also true that work has five distinct dimensions, which focus on different kinds of other aspects of CWA have significantly greater prominence now constraints. In addition, each dimension has special modelling tools than they had twenty or thirty years ago. for constructing representations of constraints. Many of these Arguably, the primary shift in thinking that CWA represents in modelling tools were part of the original descriptions of CWA the study of human-machine systems is the change in focus of (Rasmussen, 1986; Rasmussen et al., 1994), arising out of the work design from actors’ behaviours to system constraints, as intimated of Jens Rasmussen and his colleagues at the Risø National Labora- earlier. The prevailing thinking when CWA was conceived was that tory in Denmark. However, the contextual activity template for the design objective should be to support workers in executing specific behaviours. CWA fosters a different mindset. This mindset arises from the recognition that complex sociotechnical systems, being open systems, are exposed to unforeseen events and that E-mail address: [email protected]. http://dx.doi.org/10.1016/j.apergo.2016.06.001 0003-6870/Crown Copyright © 2016 Published by Elsevier Ltd. This is an open access article under the CC BY-NC-ND license (http://creativecommons.org/licenses/by-nc-nd/ 4.0/).

Table 1 CWA dimensions, constraints, and modelling tools.

Dimensions Constraints Modelling tools

Work domain analysis Work domaindconstraints placed on actors by the physical, social, and cultural Abstraction-decomposition space, abstraction environment, including the system’s purposes, values and priorities, functions, hierarchy and physical resources Activity analysis Activitydconstraints placed on actors by the activities necessary in the system Contextual activity template, decision ladder to achieve the system’s purposes, values and priorities, and functions with the available resources Strategies analysis Strategiesdconstraints placed on actors by the cognitive strategies that can be Information ﬂow map utilised for achieving the activities necessary in the system Social organisation and Work organisationdconstraints placed on actors by the ways in which work can Diagram of work organisation possibilities cooperation analysis be allocated, distributed, and coordinated in the system Worker competencies analysis Workersdconstraints placed on actors by the ways in which the work demands Skills, rules, and knowledge taxonomy of the system can be met given human cognitive capabilities and limitations

these events pose significant threats to system effectiveness. included 29 cases in the nuclear domain and 100 cases in the air Consequently, the objective of promoting successful worker adap- transportation domain (Rasmussen, 1968a, 1968b, 1969). This study tationdirrespective of the circumstancesdis critical. Given this revealed that human error accounted for approximately three intent, designs cannot be based on descriptions of the cognitive quarters of accidents, and that these errors arose when workers tasks or actions carried out by workers, as these cannot be fully were confronted with unfamiliar events. As these events could not anticipated a priori. Instead, to support adaptive problem-solving, have been foreseen by analysts or designers, workers could not designs must be specified in terms of the constraints that shape have been provided with pre-planned procedures for handling the behaviours that are possible in any situation. these occurrences. However, the analysis also showed that, in The rest of this paper traces the evolution of CWA, from its or- nearly all cases, workers could have formulated an appropriate igins to its subsequent development and impact. While its progress response if the actual state of the system at the time had been has been documented before (Naikar, 2012), this account places known to them. These findings signified the importance of emphasis on the outcomes of the last four or five years. Moreover, providing workers with the information they needed about the this account examines the relationship of some recent advances in system to adapt their behaviour to the demands of a wide range of CWA to the conceptual origins of the approach in studies of human situations, including unforeseen events, and thus finish the design problem-solving in complex systems. Of particular note is a new (Rasmussen and Goodstein, 1987). application of CWA to military doctrine development (Brady et al., In the 1970s, therefore, the focus of the Risø group shifted to 2013; Naikar et al., 2014). Juxtaposed against the foundations of human-machine reliability. A number of empirical studies were CWA and preceding applications of this framework within the conducted with the intent of establishing a sound basis for fields of human factors and engineering, it becomes possible to designing safer human-machine systems. Many of these studies examine the relevance of earlier insights on human reasoning, as involved examining the problem-solving strategies of experienced revealed in the context of interface design particularly, to this novel workers as they carried out a variety of tasks that were represen- problem area. tative of their normal jobs. One such study was concerned with electronic troubleshooting. 2. Foundations In this field study, Rasmussen and Jensen (1973) investigated the way in which professional technicians troubleshoot complex faults CWA was conceived in the 1960s and 1970s by Jens Rasmussen in commercially available electronic equipment. Their study and his colleagues at the Risø National Laboratory (Vicente, 1999, involved the participation of six professional technicians, and it 2001). The aim of this organisation was to conduct research in included eight different types of instruments, each with a particular support of the implementation of nuclear power in Denmark. CWA fault. The investigation utilised verbal protocol methodology, so the evolved specifically from a program in the Electronics Department, technicians were required to verbalise their problem-solving pro- which was headed by Jens Rasmussen. The challenge this group cesses as they set about their troubleshooting tasks. A total of 45 faced, which led ultimately to the development of CWA, was how to cases were recorded and transcribed, although only 30 were sub- advance the safety of nuclear power plant operations at a time jected to detailed analysis. when hardware systems were already performing with extremely The data analysis involved the development of a preliminary high reliability. coding scheme, which was used to analyse the technicians’ verbal In the early 1960s, the focus of Rasmussen’s group had in fact protocols. All of the verbal protocols were then reviewed to been on hardware reliability. That is, the group was concerned with determine if the information in them was captured effectively by examining the reliability of nuclear reactor equipment and instru- the coding scheme. Changes were made to the coding scheme to mentation (Jensen et al., 1963; Rasmussen and Timmerman, 1962). eliminate any discrepancies with the verbal protocols, and the Their research investigated such issues as the probability of whole process was repeated until the coding scheme stabilised. The equipment failure and the degree of redundancy needed in backup verbal protocols were then re-analysed with the final coding systems to achieve high levels of safety. However, on the basis of scheme. Fig. 1 shows some of the results of this process. Although empirical data collected in their research facilities, they found that the details are unclear, the illustrations provide a sense of the although they could design hardware systems with extremely high complexity of the analyses and the painstaking nature of Rasmus- reliability, accidents still occurred. What followed was a program sen and Jensen’s (1973) work. that is inspiring for its vision, depth, and meticulousness. The results of this field study of electronic troubleshooting, and First, Rasmussen sought to understand precisely why accidents other such studies, revealed a number of patterns in how workers eventuated despite increasingly high levels of hardware reliability. reason about complex systems during problem solving in a range of An analysis of major industrial accidents was conducted, which situations (Rasmussen, 1979, 1985), which are now well known.

Fig. 1. Illustrations of Rasmussen and Jensen’s (1973) analyses.

First, it was found that workers reason at different levels of by the preceding discussion, many of this framework’s key concepts abstraction while performing their jobs, spontaneously shifting are embodied in its modelling tools. A common theme is that the their view of the system from purposive properties to physical tools are concerned with representing system properties, or the properties in line with their task or situational demands (Fig. 2). In problem characteristics, in a way that constrains or shapes effective addition, workers reason at different levels of decomposition, work practice, thereby reflecting the empirical findings of the Risø moving between coarse and fine-grained views of the system. group about how experienced workers are able to reason Third, when workers shift their view of the system from purposive aboutdand meetdthe demands of their jobs successfully. Thus the to physical properties, the resolution at which they view the system decision ladder is concerned with the constraints on the recurring also changes from coarse to fine-grained. Consequently, workers decision tasks in the system, the information flow map is con- tend to adopt models of the system along the diagonal of this cerned with the constraints on the strategies suitable for decision reasoning space. making in the system, and the skills, rules, and knowledge taxon- Finally, this reasoning space is event-independent. That is, omy is concerned with the constraints on the modes of cognitive regardless of whether workers adopt similar or diverse problem- reasoning workers can employ in interacting with the system. solving paths, in the same or in different situations, all of these These modelling tools were conceived largely with the intent of trajectories can be mapped onto, or can be explained by, this providing a suitable basis for designing interfaces for workers in reasoning space. Hence, Rasmussen proposed the abstraction- complex sociotechnical systems, as the succeeding developments decomposition space as a framework that can be used to repre- highlight. sent information about the properties of the system, which is the In the 1980s, Rasmussen produced several seminal publications object of workers’ activities, in a way that is both compatible with on CWA. Among these were journal papers on the skills, rules, and the characteristics of human problem-solving and event- knowledge taxonomy (Rasmussen, 1983) and the abstraction- independent. Provided with this information, workers should be decomposition space (Rasmussen, 1985). His first book inte- able to reason effectively about the system state in a variety of grating the foundational ideas of CWA (Rasmussen, 1986) followed. circumstances, including those that haven’t been anticipated, and take appropriate action to preserve safety and performance. Thus 3. Ecological interface design these insights led to the concepts and modelling tools of work fi domain analysis, the rst dimension of CWA. In the late 1980s, Jens Rasmussen collaborated with Kim Vicente In much the same way, the various empirical studies conducted from the University of Toronto to develop the ecological interface by the Risø group in the 1970s led to other conceptual de- design (EID) framework (Rasmussen and Vicente,1989; Vicente and velopments that were ultimately integrated into CWA. As indicated Rasmussen, 1990, 1992). Although the Risø group had already been

Low resolution High resolution (coarse) (fine-grained) DECOMPOSITION

Whole Subsystem Functional Circuit-stage Component System Unit

Functional No communication meaning; to tape or disk Test program Purpose Purposive

Program Abstract Joint node in Flags and and flag conditions Function information paths conditions

Instructions Generalized Function

Multiplexer ABSTRACTION Physical Solenoide circuit Function Interface to Transistor DEC-writer Not Manipulation Solenoide short-circuit keyboard of power supply. driver but- Study function Physical Form Is here Looks burnt Physical

Fig. 2. The abstraction-decomposition space.

employing CWA concepts in designing interfaces for complex et al., 1996; Pawlak and Vicente, 1996) and by Reising and Sand- sociotechnical systems, EID develops and formalises some of these erson (e.g., 1998, 2000a, 2000b) utilised process control micro- ideas. To design an ecological interface, the abstraction- worlds, which represented relatively small-scale problems. decomposition space is used to develop a model of a work However, Ham and Yoon’s (2001b) work, which involved a simu- domain and thus define what information is necessary on a display lation of a secondary cooling system of a nuclear power plant, to support human problem-solving in that system. The skills, rules, represented a medium-scale problem. Furthermore, the studies by and knowledge taxonomy guides, or influences, the form that in- Jamieson (2007) and by Lau et al. (2008) were concerned with formation takes, or how that information is presented on a display, industrial-scale problems in the petrochemical and nuclear power so that the interface supports all three modes of cognitive domains, respectively. reasoning, namely skill-based, rule-based, and knowledge-based Studies of EID in relation to systems other than process control behaviour (also see Bennett and Flach, 2011; Burns and have also met all three criteria. As shown in Table 2, the systems Hajdukiewicz, 2004; Vicente, 2002). investigated include information retrieval (Xu et al., 1999), medical Since EID was developed, extensive research has been con- (Sharp and Helmicki, 1998), network management (Burns et al., ducted on its utility. In 2002, Vicente published a review of this 2003), aviation (Borst et al., 2006), and military command and body of work in which he proposed three criteria for assessing this control (Bennett et al., 2008; Hall et al., 2012) systems. All of these framework. The proof of principle criterion is concerned with studies were concerned with medium-scale or large-scale prob- establishing whether EID can in fact be applied to produce in- lems. However, at present, there are only one or two cases of terfaces for complex sociotechnical systems. The analytical criterion empirical evaluation for each class of system. is concerned with examining whether EID reveals new information Nevertheless, collectively, these studies demonstrate that EID requirements compared with those addressed in existing in- can be applied to a range of systems and that, for those systems, EID terfaces. Lastly, the empirical criterion seeks to verify whether EID can uncover novel information requirements not met by existing promotes better performance by workers relative to that obtained interfaces. Moreover, in comparison with existing interfaces, EID with existing interfaces. can lead to significantly better performance by workers. These three criteria may be used to summarise the value of EID Three sets of experimental results are worth discussing in more for designing interfaces for complex sociotechnical systems, as detail. First, studies have shown that EID generally leads to better illustrated in Table 2. In all cases shown in this table, the empirical performance than existing interfaces on non-routine or complex assessment involved controlled experimental investigations. The tasks, with no disadvantage on routine or simple tasks (e.g., Duez shaded cells indicate some of the results obtained since Vicente’s and Vicente, 2005; Ham and Yoon, 2001b; Lau et al., 2008; (2002) review. Pawlak and Vicente, 1996; Xu et al., 1999). This finding is consis- Specifically, Table 2 shows that studies of EID in relation to tent with the framework’s emphasis on supporting workers in process control systems have satisfied all three criteria. The foun- situations requiring adaptive problem-solving. Second, studies dational studies by Vicente and his colleagues (e.g., Christoffersen have shown that the benefit of EID is due to its information content,

Table 2 Value of EID for a range of systems as established against proof of principle, analytical, and empirical criteria. The letter ‘Y’, which stands for ‘yes’, signiﬁes positive ﬁndings.

System Example publications Proof-of-principle Analytical Empirical

Process control Christoffersen et al., 1996; Pawlak and Vicente, 1996 YYY Process control Reising and Sanderson, 1998, 2000a, 2000b YYY Process control Ham and Yoon, 2001b YYY Process control Jamieson, 2007 YYY Process control Lau et al., 2008 YYY Information retrieval Xu et al., 1999 YYY Medicine Sharp and Helmicki, 1998 YYY Network management Burns et al., 2003 YYY Aviation Borst et al., 2006 YYY Military command and control Bennett et al., 2008; Hall et al., 2012 YYY

not just its visual format (Ham and Yoon, 2001b; Xu et al., 1999). developing team designs (Naikar et al., 2000; Naikar et al., 2002, That is, when the visual format of ecological and existing interfaces 2003), defining software specifications (Leveson, 2000), devel- is controlled, such that both present information in an alpha- oping strategies for managing human error (Naikar and Saunders, numeric format, for example, EID still leads to better performance. 2002, 2003), and formulating recommendations for automation Third, studies have demonstrated that higher levels of abstrac- and role allocation (Bisantz et al., 2001, 2003). tion and structural means-ends relations in the abstraction- Aside from highlighting the utility of CWA for problems other decomposition space play a key role in the improved perfor- than interface design, another outcome of some of these early mance obtained with EID displays (e.g., Burns, 2000; Ham and studies was the concrete demonstration of the value of CWA in an Yoon, 2001a; Ham et al., 2008; Janzen and Vicente, 1998; Vicente industrial context. These cases involved the application of CWA on et al., 1995). Progressively adding information from higher levels projects for the Australian Department of Defence. In one project, of abstraction to an EID display leads to increasingly better levels of the best design concept out of three competing proposals for an performance. In addition, grouping information according to Airborne Early Warning and Control aircraft was selected based on structural means-ends relations fosters better performance the results of CWA (Naikar and Sanderson, 2001). Furthermore, the compared with when the information is not organised in this way. team design that was adopted for this future military system was In sum, this body of research provides significant support for the developed with CWA (Naikar et al., 2003). In the third case, CWA validity of some of the CWA concepts developed by the Risø group. was used as a basis for formulating training-system requirements Specifically, the experimental studies on EID substantiate the for a fighter aircraft (Naikar and Sanderson, 1999). earlier field observations that workers reason about system prop- Given that experimental investigations were not possible in erties at different levels of abstraction and decomposition, these industrial cases, Naikar (2013) assessed the value of CWA for engaging skill-based, rule-based, and knowledge-based behaviours these applications in terms of three criteria: impact, uniqueness, for problem-solving. Moreover, these work patterns are evident and feasibility. Impact concerns the ability of CWA to influence across a range of systems and problem scales. Hence, by providing a practice, uniqueness relates to the ability of CWA to lead to basis for representing system properties in a way that is consistent distinctive outcomes relative to standard industrial techniques, and with empirical observations of patterns of human reasoning in feasibility refers to the ability of CWA to be implemented within the complex systems, the abstraction-decomposition space and the schedule, staff, and financial constraints of an industrial project. As skills, rules, and knowledge taxonomy can lead to the design of discussed by Naikar, in all three cases for the Australian Depart- interfaces that foster effective performance by workers. ment of Defence, CWA satisfied these criteria. Alongside the developments in EID, the 1990s was significant Despite the fact that these industrial case studies of CWA have for the publication of two other seminal monographs on CWA, been described in some detail previously, the relationship of the further to Rasmussen’s (1986) foundational text. The first of these outcomes obtained to the empirical foundations of the framework by Rasmussen et al. (1994) expanded the concepts of CWA to in studies of human problem-solving in complex systems has not encompass a wide variety of systems. While Rasmussen’s (1986) been examined. Therefore, in the following discussion, this rela- text had focused on causal systems, the newer formulation of CWA tionship is investigated in one case. The selected application is the incorporated intentional systems. The second monograph was first of the three projects to be conducted, which involved the use of authored by Vicente (1999). This influential text can be credited for CWA for the evaluation of system design concepts (Naikar, 2013; situating CWA within a scholarly context and bringing the frame- Naikar and Sanderson, 2001). work to the attention of a considerably broader audience, among In this case study, work domain analysis was utilised to evaluate other worthy contributions. Arguably, together with the experi- three design proposals for an Airborne Early Warning and Control mental evidence for EID, these publications motivated the forth- system, which were submitted by competing aircraft manufac- coming advances in CWA. turers. An abstraction hierarchy was developed of this future system, defining its purposes, value and priority measures, purpose- 4. Beyond ecological interfaces related functions, object-related processes, and physical objects. Using this model, the three design concepts were evaluated in Since the late 1990s, there has been growing recognition of the terms of how well their respective solutions for the technical suitability of CWA for addressing a broad range of problems. components of the aircraft, which could be mapped onto the Accordingly, the application of this framework has been extended physical objects and object-related processes levels of abstraction, beyond interface design to a variety of other human factors and satisfied the higher-level functions, values and priorities, and pur- engineering concerns. Studies conducted early during this period poses of the system. demonstrated the relevance of CWA for defining training needs and The first demonstration of impact on this project was that work training-system requirements (Naikar and Sanderson, 1999), eval- domain analysis was adopted as an additional method for evalua- uating system design concepts (Naikar and Sanderson, 2000, 2001), tion by the Australian Department of Defence. Prior to that, the

Please cite this article in press as: Naikar, N., Cognitive work analysis: An influential legacy extending beyond human factors and engineering, Applied Ergonomics (2016), http://dx.doi.org/10.1016/j.apergo.2016.06.001 6 N. Naikar / Applied Ergonomics xxx (2016) 1e13 organisation had been planning to use only standard systems en- constraint-based method for the problem of evaluation can be gineering and operations analysis techniques. The systems engi- established with greater certainty would be valuable. neering technique involved assessing the proposals against pre- Finally, in closing this section, two further observations are specified functional requirements, whereas the operations anal- worth making. The first is that since the initial case studies ysis technique involved comparing the performance of computa- demonstrating the value of CWA for applications beyond interface tional models of the three aircraft designs in six mission scenarios design, the use of this framework, or adaptations of it, has been using Monte Carlo simulation. considered for an even broader range of human factors and engi- In contrast, the nature of the evaluation promoted by work neering concerns. For example, CWA has been used to consider domain analysis was constraint-based. That is, the problem of problems of organisational design (Durugbo, 2012; Xiao and identifying the best design proposal was characterised in terms of Sanderson, 2013) and website design for supporting social net- the constraints of the system at multiple levels of abstraction, works (Euerby and Burns, 2014). In addition, this framework has comparable to the way in which an ecological interface represents been considered as a basis for supporting the development of the problem demands of a system to workers. Given this problem measures for evaluating system effectiveness (Crone et al., 2007; representation, the three design proposals could be considered, or Jenkins et al., 2008) and for the identification of system design reasoned about, in terms of the impact of their respective technical problems and enhancements for their resolution (Xu, 2007). solutions on the physical and purposive properties of the system. Second, while many applications of CWA have focused on the Moreover, the propagation of effects of each design solution could design of particular system components, such as the interfaces, be traced throughout the system from lower to higher levels of teams, training, or automation, there is growing recognition of the abstraction. Accordingly, in using the abstraction hierarchy for need for an integrated approach, whereby the designs of multiple problem-solving, the evaluators were considering the question of elements are coordinated based on a philosophy of promoting the best design proposal at multiple levels of abstraction, shifting worker adaptation. The initial call was put forward by Vicente their view of this problem between physical and purposive prop- (2002) in his review of ecological interface design and reiterated erties of the system, arguably analogous to the way electronic by Naikar (2012) in relation to discussions of team design. Since technicians were reasoning about faults in computer equipment in then, Read et al. (2015) have reported a case study in which CWA Rasmussen and Jensen’s(1973)field study. was used in combination with a template requiring design features The second demonstration of impact on this project was that the associated with multiple elements to be documented. While it design concept that was identified as best using work domain seems that such a template could help ensure that designs of analysis was selected by the Australian Department of Defence as multiple elements are considered concurrently, the findings of the the winning proposal. It is important to emphasise that the oper- case study suggest that this is not a guaranteed result. ations analysis technique also produced the same result and that In contrast, Naikar and Elix (2015, 2016a, 2016b) have the decision about the winning proposal was also based on factors approached the problem of integrated system design from a other than the results of these techniques. Nevertheless, in a brief compatibility rather than concurrency perspective. That is, their given to the Deputy Secretary of the Department of Defence on the concern is with how the designs of multiple elements can be co- evaluation process and outcomes, work domain analysis was cited ordinated such that they all promote adaptations in workers’ for its usefulness by the Air Force personnel leading the evaluation. behaviour and structure, or work organisation, in a coherent The fact that work domain analysis was adopted as an additional manner. They also recognise that, to support workers in dealing framework for evaluation, alongside standard techniques, suggests with the unanticipated, the system design must accommodate the that a constraint-based representation of the problem was seen as full range of opportunities for behavioural and structural adapta- having validity by subject matter experts, particularly by the Air tion, irrespective of the circumstances. To achieve this level of Force personnel who made the decision to use this framework for integration, they propose that the designs of multiple elements evaluation. Moreover, over a hundred evaluators, including Air must be coordinated around the organisational constraints of the Force personnel and defence scientists and engineers, documented system, as these constraints circumscribe the full set of possibilities the results of their evaluation of the three design proposals using for action available to any actor, in either novel or recurring situa- this constraint-based method, suggesting that the problem could tions. Accordingly, Naikar and Elix introduce the diagram of work be considered in these terms. Finally, the fact that the design organisation possibilities as a tool for modelling the constraints on concept identified as being best by work domain analysis was the organisational structures that are possible in a system regard- selected as the winning proposal suggests that the result of the less of the circumstances, and they demonstrate how the designs of constraint-based reasoning process itself was judged as having multiple elements can be anchored to these constraints. This validity by subject matter experts, including by senior defence modelling tool is described further in the next section. personnel who had not participated in the evaluation process but carried significantly responsibility for the project. 5. Methods and modelling tools While this case study provides some evidence for the value of constraint-based representations and reasoning processes for the Alongside research investigating the applicability of CWA to a problem of evaluating design concepts, it does not demonstrate variety of problem areas, the methods and modelling tools of this conclusively that this method does in fact lead to the selection of framework have been refined and extended. Initial work in this the best design concept in terms of performance. The fact that the area concentrated predominantly on the first two dimensions, operations analysis technique, which evaluated the performance of namely work domain analysis (Naikar, 2013; Naikar et al., 2005) computational models of the three aircraft designs under and activity analysis (Ashoori and Burns, 2010; Elix and Naikar, controlled conditions in a range of scenarios using Monte Carlo 2008; Lamoureux and Chalmers, 2009; Lamoureux and Sartori, simulation, identified the same design concept as being best does 2007; Naikar et al., 2006). However, over time, increasingly provide reasonably strong validation of the results of the greater attention has been paid to the other dimensions, specifically constraint-based method. However, the computational models strategies analysis (e.g., Cornelissen et al., 2013; Hassall and were only partial models of the aircraft designs, albeit of critical Sanderson, 2014), social organisation and cooperation analysis components. Such a practical approach was necessary in this in- (e.g., Naikar and Elix, 2015, 2016a, 2016b; Pfautz and Pfautz, 2009), dustrial project. Future research considering how the validity of the and worker competencies analysis (e.g., Kilgore et al., 2009).

This line of work on methods and modelling tools appears to have two main motivations. One impetus is that the texts by Rasmussen (1986), Rasmussen et al. (1994), and Vicente (1999), though greatly enhancing the visibility of CWA, focus on articu- lating a conceptual perspective of the framework. As Vicente (1999) discusses, this scope was chosen deliberately because CWA was still relatively unknown at the time. However, since then, the outcomes obtained with CWAdboth in the laboratory and in indus- trydprovide strong incentives for pursuing the development of a complementary methodological perspective, with the intent of increasing the accessibility of this framework. Another reason for this area of investigation derives from ongoing observations of how work can be, and is, accomplished effectively in complex sociotechnical systems. These observations Fig. 3. Diagram of work organisation possibilities. have stimulated extensions to CWA, which are particularly apparent in its suite of modelling tools, so that the constraints on effective work practice are properly accounted for by this frame- Second, the WOP diagram depicts the constraints on the pos- work. Two such developments are the contextual activity template sibilities for work organisation rather than describing the possi- (Naikar et al., 2006) and the diagram of work organisation possibilities themselves. As an example, Fig. 3 shows that Actors A and C bilities (Naikar and Elix, 2015, 2016a, 2016b). The rest of this section can take responsibility for Work Demand 1. These are the con- concentrates on describing the latter as it is a more recent straints on the possibilities. Given these constraints, the possibil- development. ities are: Actor A has this responsibility, Actor C has this The diagram of work organisation possibilities (WOP) was responsibility, or Actors A and C share this responsibility. Which developed to account for the observation that complex socio- possibility is adopted depends on the details of the situation, which technical systems are comprised of multiple workers and that these may not always be known a priori, such that the problem can only workers deal with the problem demands of such systems by be resolved online and in real time by actors. This is one reason the adapting not just their individual behaviours but also their group possibilities are regarded as emergent, despite the fact they may be structure or organisation (Naikar and Elix, 2015, 2016a, 2016b). computed. Moreover, the adaptations in structure are necessary as a response Finally, depending on the scale or complexity of the system and to changing circumstances as well as to novel events (e.g., Bigley the level of granularity at which the work demands are modelled, and Roberts, 2001; Bogdanovic et al., 2015; Rochlin et al., 1987). the number of possibilities may be so large that it is not feasible for This means that designs for complex sociotechnical systems must them all to be considered meaningfully by analysts or designers. ’ accommodate adaptations in both workers behaviour and struc- Certainly this was found to be the case in a study of a military ture in a wide range of situations, including unforeseen events. system for airborne maritime surveillance (Naikar and Elix, 2015, The WOP diagram provides a tool for circumscribing the set of 2016a). However, to support adaptation in complex sociotechnical possibilities for work organisation in a system, as these organisa- systems, it is not necessary for all of the possibilities to be tional constraints delineate the full range of opportunities for computed and comprehended upfront. Rather, it is only necessary structural and behavioural adaptation in the workplace (Naikar and for the designs to be based on the constraints on the possibilities. Elix, 2015, 2016a, 2016b). This diagram is created by identifying the That is, as long as a design considers the set of work demands for boundaries placed on the distribution of work demands across which each actor can be responsible, actors will be able to handle actors by a set of criteria that have been observed to govern shifts in those work demands effectively if and when the need arises. work organisation dynamically (Rasmussen et al., 1994; Vicente, The WOP diagram, and its associated concepts, significantly 1999), as these criteria limit the organisational structures that are extend the capacity of CWA to support adaptation in complex possible. These criteria relate to the competencies of actors, the sociotechnical systems (Naikar and Elix, 2015, 2016a, 2016b). Prior access actors have to information or the means for action, the re- to this development, the analysis of organisational structures quirements for safety and reliability, the need for compliance with within the CWA framework was restricted to recurring classes of policies and regulations, the requirements for feasible coordination situation and to structures that are observable or regarded as demands, and the requirements for manageable workloads. If the reasonable (Rasmussen et al., 1994; Vicente, 1999). Importantly, the focus of the design effort is on supporting workers in dealing with a structural possibilities in a system constrain the behavioural op- wide range of situations, irrespective of whether these are novel or portunities available to actors. Thus, by providing a means for recurring, the work demands could be derived solely from a work defining the set of possibilities for work organisation in a system domain analysis. In contrast, if the intent is to provide fuller or irrespective of the situation, the WOP diagram can support the richer support for workers in recurring classes of situation, which development of designs that accommodate the full range of op- are circumstances they are required to deal with more commonly, portunities for structural and behavioural adaptation in the the work demands could be derived from an activity analysis, workplace. strategies analysis, or both. Fig. 3 depicts the WOP diagram in a generic form to emphasise some key features of this formative representation. The first point 6. Beyond human factors and engineering to appreciate is that this diagram results in an understanding of the set of work demands for which an actor can be responsible. Which Despite the preceding advances in CWA, arguably the potency of work demands an actor will be responsible for at any point in time this framework is still far from being fully realised. One reason for is situation-dependent, such that the responsibilities of actors this outlook is that only recently has CWA been applied substan- could vary over time. Therefore, initially Actor A could be respon- tively to problems that would be commonly regarded as lying sible for Work Demand 2 but subsequently this responsibility could outside the fields of human factors and engineering. Specifically, be assumed by Actors B or C. CWA was used to contribute to the development of military

Please cite this article in press as: Naikar, N., Cognitive work analysis: An influential legacy extending beyond human factors and engineering, Applied Ergonomics (2016), http://dx.doi.org/10.1016/j.apergo.2016.06.001 8 N. Naikar / Applied Ergonomics xxx (2016) 1e13 doctrine (Brady et al., 2013; Naikar et al., 2014) and future operating A fundamental reason for the problems observed in the codifi- concepts (Brady et al., 2015) with effective results. The case of cation of military doctrine may be found in how it is created. military doctrine is discussed in more detail below. While this Conventionally, doctrine is developed by military experts. Thus the study has been reported previously, emphasis is placed here on process relies innately on military experience, as indeed must be considering how the fundamental observations of human problem- the case. However, as is widely appreciated, being highly experi- solving in complex systems, which led to the development of CWA, enced at executing an action or series of operations does not are relevant to military doctrine development. necessarily mean the expert can relate easily how that performance is achieved. This observation has given rise to the classic distinction 6.1. Military doctrine between declarative and procedural knowledge (Anderson, 1980, 1983), which highlights that some types of knowledge are more Military doctrine is essential for exercising military power suc- open to articulation than others. A military expert, therefore, might cessfully. The codification of a military force’s fundamental phi- be very good at carrying out military operations but less adept at losophy, or system of ideas by which it governs its operation, explaining how good performance is achieved, particularly in an provides its members with a common conceptual framework for abstract sense. Yet, abstraction is necessary for the codification of organizing and executing military action. Without this framework, military doctrine as an effective ideology is one that is generalisable military activities are likely to be disorderly and inefficient, and across situations. ultimately disastrous (Drew and Snow, 1988; Schubert and Kavanagh, 1989; Sloan, 2012). 6.2. Relevance of CWA The system of ideas constituting doctrine can encompass a force’s overarching aims and missions, its attitudes and values, and The CWA framework provides a means for addressing such its strategies, techniques, and practices. This body of ideas is challenges in military doctrine development. As discussed earlier, believed to be effective predominantly on the basis of empirical this framework comprises a set of concepts for representing system evidence drawn from prior military experience, whether historical properties, or the problem characteristics, in ways that are or contemporary. Consequently, doctrine represents a set of central, consistent with how humans reason. The problem characteristics, and accepted, beliefs about the best way of conducting military which are in the form of a set of constraints that must be respected affairs, which continues to guide present and future military ac- by workers across a wide range of circumstances, shape the pos- tions until those beliefs are revised or invalidated (Drew and Snow, sibilities for action available to them. Thus human problem-solving 1988; Schubert and Kavanagh, 1989; Sloan, 2012). involves reasoning about these constraints or boundaries on suc- The codification or formalisation of military doctrine is a chal- cessful performance as a basis for generating effective, typically lenging goal, as is apparent in the problems of logic, rigour, and adaptive, courses of action, as was evident in the verbal protocols of coherence typical of many existing formulations. The central issue the electronic technicians in Rasmussen and Jensen’s (1973) field is that the fundamental concepts underpinning doctrine, such as study. purposes, goals, values, principles, functions, tasks, missions, and That military experts are also reasoning about such system roles, tend to be conceptualised inconsistently and in overlapping, properties is evident in their writings on military doctrine. Table 3 circular, and disjointed ways, precluding a meaningful account of shows how a number of abstract concepts referred to in doctrine the military ideology from being derived. Such incongruities are can be matched to the concepts that form the various dimensions of particularly problematic in the case of formal communications of CWA. This mapping highlights that military doctrine is comparable doctrine, such as military publications, as shared meaning cannot to CWA in that it attempts to capture relatively stable or enduring be refined, clarified, or created as is possible in conversation. problem characteristics in a way that can guide effective military As an example, the concept of function features recurrently in action across a range of circumstances. In fact, this is much the publications of military doctrine. In an Australian context, functions stated intent of doctrine, as noted earlier. Once again, though, this are defined as the “appropriate or assigned duties, responsibilities, does not mean that military experts are necessarily adept at missions, or tasks of an individual, office, or organisation” stringing together abstract concepts into a logical, rigorous, and (Department of Defence, 2014). In this definition, the concept of a coherent narrative, especially within tight timeframes, as demon- function is seen as signifying what persons or groups in a military strated by the problems with the codification of military doctrine establishment do. In addition, the concept of a function is likened to discussed above. Indeed it is unlikely that the electronic technicians such concepts as mission, task, and responsibility. However, in Rasmussen and Jensen’s(1973)field study could have readily doctrinal publications may refer to functions and missions in ways provided a formal description of their work. that suggest they are different ideas. For example, distinct lists of The CWA framework, therefore, can provide a strong conceptual functions and missions may be presented. Another such problem is basis for analysing and refining military doctrine. Its five distinct that the concept of a function may be equated with various other dimensions cleanly demarcate the fundamental orientations from concepts such as that of a role. Potentially, then, roles could be seen which a system’s problem characteristics may be viewed, while at as being the same as missions, as well as functions, but then roles the same time integrating these perspectives into a unified and functions may be described as being achieved through mis- framework, such that the relationships between the problem di- sions. Moreover, disparate sets of roles, functions, and missions mensions are clear. Consequently, the framework offers a powerful may be presented. Such conceptual problems are also evident in the lens for deciphering the distinctions and interrelationships be- doctrinal publications of other nations. tween a variety of military concepts, thereby enabling a more Aside from the confusion such incongruities could create among credible and compelling ideology to be created. military personnel, the lack of sufficient differentiation and inte- For example, CWA provides a systematic framework for differ- gration of concepts is problematic because it makes it difficult to entiating and integrating the concepts of a system’s work domain capture the complexities and subtleties inherent in the application or physical, social, and cultural context (e.g., functions); activity of military power. As a result, cultivating a sophisticated appreci- (e.g., missions); and work organisation (e.g., roles). Thus, based on ation of the military ideology guiding action may be far from the ideas encapsulated within this framework, the concepts of a straightforward for members of a military force, potentially limiting military force’s functions, missions, and roles may be elucidated. the effectiveness with which military power can be employed. Functions may be viewed as representing the utility of objects or

Table 3 Mapping of some of the military concepts referred to in doctrine to CWA.

CWA dimensions CWA concepts Military concepts

Work domain analysis Work domain e.g., Purposes, values, principles of war, laws of armed conflict, functions, capabilities, characteristics, resources Activity analysis Activity e.g., Goals, missions, operations, campaigns, activities, tasks, conflict situations Strategies analysis Strategies Strategies (e.g., influence, deterrence, coercion, punishment, destruction) Social organisation and cooperation analysis Work organisation e.g., Command structure, roles, centralised control and decentralised execution Worker competencies analysis Workers e.g., Competencies

devices independently of actions, whereas missions may be terms in Table 4 are further defined in a glossary. More compre- considered an activity-based concept involving sequences of ac- hensive descriptions of the air power and air combat models, and tions for achieving specific goals with particular objects. Further- how they were produced, are provided by Brady and Naikar (2015) more, roles may be viewed as signifying the distribution of work, and by Treadwell and Naikar (2014), respectively. including functions or missions, across individuals, groups, or or- The overarching concepts of CWA and the two work domain ganisations. In this way, the conceptual distinctions and relations models were used to refine the logic, rigour, and coherence of the inherent in CWA can provide a basis for organizing military con- Air Force’s doctrine, with a clear demonstration of impact. Of 150 cepts into a coherent doctrinal framework. recommendations, approximately ninety percent were incorporated into the sixth edition of the Air Power Manual (Royal 6.3. Case study Australian Air Force, 2013), the Air Force’s foundational strategic- level doctrine text. Some of the recommendations could not be Recently, CWA was used to assist the Royal Australian Air Force adopted for practical reasons, and input from other stakeholders with their doctrine development (Brady et al., 2013; Naikar et al., led to changes to the Air Power Manual following the analysts’ 2014). Aside from the overarching concepts of CWA, two work involvement. However, for the first application of CWA to doctrine domain models were useful: one of Australia’s military air power development, reasonable outcomes were achieved. Greater con- system and the other a more focused analysis of the air combat tributions may be possible with succeeding editions of the text. system. A sample of the abstraction hierarchy of the air power One example of the contribution of CWA to doctrine develop- system is shown in Fig. 4. This figure also provides a more detailed ment is that the distinctions and interrelationships between func- mapping of some of the military concepts in Table 3, specifically tions, missions, and roles, as specified earlier, were respected in the those associated with the work domain, to the levels of abstraction. discussions of military ideology in the Air Power Manual. Similarly, Table 4 illustrates the depth at which the nodes in the abstraction in line with other relationships inherent in CWA, connections were hierarchies are defined, and thus the depth of knowledge that was established and preserved between such concepts as the air power developed of the air power and air combat systems; the underlined system’s objectives, principles of war, and characteristics. Fig. 4

Functional To secure Australia and its interests from threats by providing the ability to exploit the earth’s e.g., Purposes Purposes atmosphere and space for manoeuvre, including for the application of lethal force when necessary, given the nation’s unique geographical, population, economic, ethical, and legal constraints.

Value and e.g., Values, Priority Stability Humanity Readiness principles of Measures war, laws of armed conflict

Purpose- e.g., Functions related Threat Target Target Target Threat Functions Detection Monitoring Localisation Identification Prediction

Object- Ground-to- Ground-to- Ground-to- Space-to- e.g., Capabilities, Ground-to-air Space-to-air Air-to-sea Air-to-ground Air-to-air related sea ground space ground information information information information information characteristics information information information information Processes gathering gathering gathering gathering gathering gathering gathering gathering gathering

GROUND SURVEILLANCE AND RECONNAISSANCE SPACE SURVEILLANCE AND RECONNAISSANCE AERIAL SURVEILLANCE AND RECONNAISSANCE

Physical e.g., Resources Objects Surveillance Tracking Meteorological Imaging Communication Navigation Meteorological AP-3C P-8A Heron E-7A radars radars radars radars satellites satellites satellites Orion Poseidon UAV Wedgetail

SURVEILLANCE AND RECONNAISSANCE NETWORKED SYSTEMS SURVEILLANCE AND RECONNAISSANCE PLATFORMS

Fig. 4. Sample of the abstraction hierarchy of the air power system.

Table 4 Deﬁnitions of some of the purpose-related functions in the air power model.

Purpose-related functions Deﬁnitions

Threat detection The ability of the Air Power system to establish the presence of targets with the potential to inflict harm on friendly assets and/or the civilian population and its infrastructure. Target monitoring The ability of the Air Power system to survey the activities of specified targets. Target localisation The ability of the Air Power system to determine the exact location of specified targets. Target identification The ability of the Air Power system to establish the class and identity of specified targets. Threat prediction The ability of the Air Power system to predict the intentions of specified targets.

shows that these military concepts can be matched to the func- these characteristics onto the abstraction hierarchy, as demon- tional purposes, value and priority measures, and object-related strated in the middle section of the figure, it could be shown that processes levels of an abstraction hierarchy, respectively. Thus, the characteristics fall at multiple levels of abstraction. This sug- consistent with the view that a system’s value and priority mea- gested that these air power characteristics, although viewed by sures can shape how its object-related processes may be utilised to military personnel as distinct properties, could in fact be dupli- fulfil its purposes, attention was drawn to the fact that the princi- cating some of the same concepts, albeit at different levels of ples of war can guide how the air power characteristics may be abstraction. exploited to achieve the system’s objectives. While such relation- Close examination of the concepts substantiated this observa- ships may be recognised implicitly by military experts, they are tion. For example, the concepts of responsiveness and penetration generally not drawn out in accounts of doctrine, perhaps because (which map onto the value and priority measures and purpose- they are not readily open to articulation, although they might seem related functions levels, respectively) may be conceptualised as obvious to experts in hindsight. Consequently, accounts of doctrine properties emerging from the synergy of various other properties, can manifest as inventories of loosely connected or disjointed including speed and reach (which map onto the object-related concepts, potentially making it difficult for military personnel, processes level). Based on the conceptual distinctions inherent in especially those less experienced, to garner a credible and the abstraction hierarchy, the case could be made that logically air compelling view of air power. power characteristics should be conceived as object-related pro- Another example of the contribution of CWA to doctrine cesses as these characteristics are derived from the special qualities development is that the existing set of air power characteristics was of the air environment and the unique parameters of flight, thereby altered. Air power characteristics may be regarded as those prop- signifying the functional capabilities and limitations of the system. erties of the air domain that either can be limiting or can be Therefore, as shown at the right of Fig. 5, the set of air power exploited in the employment of military power. The original set of characteristics was altered. Specifically, consistent with the work air power characteristics is shown at the left of Fig. 5. By mapping domain models that had been produced, the concepts of

Original air Revised air power power Functional characteristics Purpose characteristics Perspective Perspective

Speed Speed Value/Priority Value/Priority Reach U W Reach

Penetration Penetration

Responsiveness Responsiveness Purpose-related Purpose-related Function X Function Z Flexibility Flexibility

Versatility Versatility

Dependency Dependency

Object-related Object-related Payload Process A Process C Payload

Impermanence Impermanence

Physical Object Physical Object Physical Object Precision A B C Fragility

Fig. 5. The original air power characteristics, their mapping to the abstraction hierarchy, and the revised characteristics.

7. Conclusion References

Based on decades of imaginative, careful, and purposeful Anderson, J.R., 1980. Cognitive Psychology and its Implications. W. H. Freeman, San research, Jens Rasmussen has left us an influential legacyda robust Francisco. Anderson, J.R., 1983. The Architecture of Cognition. Harvard University Press, and powerful approach to work analysis. This paper has demon- Cambridge, MA. strated that CWA provides a convincing basis for designing Ashoori, M., Burns, C., 2010. Reinventing the wheel: control task analysis for ecological interfaces, which can promote high levels of perfor- collaboration. In: Proceedings of the 54th Annual Meeting of the Human Factors and Ergonomics Society. Human Factors and Ergonomics Society, Santa Monica, mance; addressing a variety of human factors and engineering CA, pp. 274e278. concerns in industry, such as the evaluation of system design Bennett, K.B., Flach, J.M., 2011. Display and Interface Design: Subtle Science, Exact concepts; and developing military doctrine. Given its widespread Art. CRC Press, Boca Raton, FL. and continuing influence for over fifty years, the shift in perspective Bennett, K.B., Posey, S.M., Shattuck, L.G., 2008. Ecological interface design for military command and control. J. Cognitive Eng. Decis. Mak. 2 (4), 349e385. this framework represents, from the emphasis on actors’ behaviour Bigley, G.A., Roberts, K.H., 2001. The incident command system: high-reliability to system constraints, may be confidently described as a significant organizing for complex and volatile task environments. Acad. Manag. J. 44 e creative breakthrough. The cases considered in this paper also (6), 1281 1299. Bisantz, A.M., Roth, E., Brickman, B., Gosbee, L.L., Hettinger, L., McKinney, J., 2001. suggest that the outcomes achieved with CWA, even in very diverse Integrating Cognitive Analyses into a Large Scale System Design Process. In: problem areas, are likely to be attributable to the compatibility of Proceedings of the 45th Annual Meeting of the Human Factors and Ergonomics its concepts with empirical observations of human problem-solving Society. Human Factors and Ergonomics Society, Santa Monica, CA, pp. 444e448. in complex systems. Bisantz, A.M., Roth, E., Brickman, B., Gosbee, L.L., Hettinger, L., McKinney, J., 2003. Despite its power, many avenues remain for future research. In Integrating cognitive analyses in a large-scale system design process. Int. J. relation to EID, further work could include expanding this approach Human Comput. Stud. 58, 177e206. Bogdanovic, J., Perry, J., Guggenheim, M., Manser, T., 2015. Adaptive coordination in for interface design by incorporating concepts from additional CWA surgical teams: an interview study. BMC Health Serv. Res. 15 (1), 128. http:// dimensions, and establishing whether these extensions lead to dx.doi.org/10.1186/s12913-015-0792-5. improvements in performance. In particular, the new modelling Borst, C., Suijkerbuijk, H.C.H., Mulder, M., van Paassen, M.M., 2006. Ecological interface design for terrain awareness. Int. J. Aviat. Psychol. 16 (4), 375e400. tool for social organisation and cooperation analysis, the diagram of Brady, A., Naikar, N., 2015. Work Domain Analysis of Australia’s Air Power System: work organisation possibilities, should be investigated as a means Purpose-related Functions of Combat; Transport; and Intelligence, Surveillance, for developing interface concepts that support actors in adapting and Reconnaissance Subsystems. DSTO Technical Report (DSTO-TR-3085). not just their individual behaviours but also their group structure Fishermans Bend, Australia: Joint and Operations Analysis Division. Available from: http://www.dsto.defence.gov.au/sites/default/files/publications/ or organisation under a range of conditions. documents/DSTO-TR-3085.pdf. Applications beyond interface design should seek to advance Brady, A., Naikar, N., Treadwell, A., 2013. Organisational storytelling with work CWA by providing more demonstrations of utility. In particular, domain analysis: case study of air power doctrine and strategy narrative. In: Ahamed, S., Piantadosi, J., Agrawal, M., Boland, J. (Eds.), MODSIM 2013, 20th proof-of-concept studies suggesting new applications of CWA International Congress on Modelling and Simulation. Adelaide, Australia. should be followed up with thorough investigations of value. For Brady, A., Naikar, N., Treadwell, A., 2015. Future operating concept development: a example, the value of CWA for specific applications could be further application of cognitive work analysis beyond human factors and engineering. In: Proceedings of the 19th Triennial Congress of the International demonstrated by reliable judgements of validity by subject matter Ergonomics Association. Available from: http://ergonomics.uq.edu.au/iea/ experts, by impact on practice, or by experimental studies. proceedings/Index_files/papers/471.pdf.

Burns, C.M., 2000. Putting it all together: improving display integration in ecolog- Naikar, N., Drumm, D., Pearce, B., Sanderson, P.M., 2000. Designing new teams with ical displays. Hum. Factors 42 (2), 226e241. cognitive work analysis. In: Proceedings of the 5th International Symposium of Burns, C.M., Hajdukiewicz, J.R., 2004. Ecological Interface Design. CRC Press, Boca the Australian Aviation Psychology Association. Australian Aviation Psychology Raton, FL. Association, Manly, NSW, Australia. Burns, C.M., Kuo, J., Ng, S., 2003. Ecological interface design: a new approach for Naikar, N., Elix, B., 2015. Designing for adaptation with cognitive work analysis: visualizing network management. Comput. Netw. 43, 369e388. initial conceptualisation of an approach for integrated sociotechnical system Christoffersen, K., Hunter, C.N., Vicente, K.J., 1996. A longitudinal study of the effects design. In: Proceedings of the 19th Triennial Congress of the International Er- of ecological interface design on skill acquisition. Hum. Factors 38 (3), 523e541. gonomics Association. Available from: http://ergonomics.uq.edu.au/iea/ Cornelissen, M., Salmon, P.M., Jenkins, D.P., Lenne, M.G., 2013. A structured proceedings/Index_files/papers/468.pdf. approach to the strategies analysis phase of cognitive work analysis. Theor. Naikar, N., Elix, B., 2016a. Integrated system design: promoting the capacity of Issues Ergonomics Sci. 14 (6), 546e564. sociotechnical systems for adaptation through extensions of cognitive work Crone, D., Sanderson, P., Naikar, N., Parker, S., 2007. Selecting sensitive measures of analysis. Front. Psychol. 7, 962. http://dx.doi.org/10.3389/fpsyg.2016.00962. performance in complex multivariable environments. In: Proceedings of the Naikar, N., Elix, B., 2016b. A Consideration of Design Approaches Based on Cognitive 2007 Simulation Technology Conference. Simulation Industry Association of Work Analysis: System Design and Integrated System Design (Manuscript Australia, Rundle Mall, SA, pp. 4e7. submitted for publication). Department of Defence, 2014. Australian Defence Glossary (Version 6.2.0). Naikar, N., Hopcroft, R., Moylan, A., 2005. Work Domain Analysis: Theoretical Retrieved from. http://adg.eas.defence.mil.au/default.asp. Concepts and Methodology (DSTO-TR-1665). Defence Science and Technology Drew, D.M., Snow, D.M., 1988. Making Strategy: an Introduction to National Security Organisation, Fishermans Bend, VIC, Australia. Processes and Procedures. Maxwell Air Force Base. Air University Press, Naikar, N., Moylan, A., Pearce, B., 2006. Analysing activity in complex systems with Alabama. cognitive work analysis: concepts, guidelines, and case study for control task Duez, P., Vicente, K.J., 2005. Ecological interface design and computer network analysis. Theor. Issues Ergon. Sci. 7 (4), 371e394. management: the effects of network size and fault frequency. Int. J. Human Naikar, N., Pearce, B., Drumm, D., Sanderson, P.M., 2002. A formative approach to Comput. Stud. 63, 565e586. designing teams for first-of-a-kind, complex systems. In: Johnson, C.W. (Ed.), Durugbo, C., 2012. Work domain analysis for enhancing collaborations: a study of Proceedings of the 21st European Annual Conference on Human Decision the management of microsystems design. Ergonomics 55 (6), 603e620. Making and Control. Department of Computing Science, University of Glasgow, Elix, B., Naikar, N., 2008. Designing safe and effective future systems: a new Glasgow, Scotland, pp. 162e168. approach for modelling decisions in future systems with cognitive work anal- Naikar, N., Pearce, B., Drumm, D., Sanderson, P.M., 2003. Designing teams for first- ysis. In: Proceedings of the 8th International Symposium of the Australian of-a-kind, complex systems using the initial phases of cognitive work analysis: Aviation Psychology Association. Australian Aviation Psychology Association, case study. Hum. Factors 45 (2), 202e217. Sydney, NSW, Australia. Naikar, N., Sanderson, P.M., 1999. Work domain analysis for training-system defi- Euerby, A., Burns, C.M., 2014. Improving social connection through a communities- nition and acquisition. Int. J. Aviat. Psychol. 9 (3), 271e290. of-practice inspired cognitive work analysis approach. Hum. Factors 56 (2), Naikar, N., Sanderson, P.M., 2000. Evaluating design proposals with work domain 361e383. analysis. In: Proceedings of the Joint 14th Triennial Congress of the Interna- Hall, D.S., Shattuck, L.G., Bennett, K.B., 2012. Evaluation of an ecological interface tional Ergonomics Association and the 44th Annual Meeting of the Human design for military command and control. J. Cognitive Eng. Decis. Mak. 6 (2), Factors and Ergonomics Society (IEA2000/HFES2000). Human Factors and Er- 165e193. gonomics Society, Santa Monica, CA, pp. 202e205. Ham, D.eH., Yoon, W.C., 2001a. Design of information content and layout for pro- Naikar, N., Sanderson, P.M., 2001. Evaluating design proposals for complex systems cess control based on goal-means domain analysis. Cognit. Technol. Work 3 (4), with work domain analysis. Hum. Factors 43 (4), 529e542. 205e223. Naikar, N., Saunders, A., 2002. Crossing the boundaries of safe operation: training Ham, D.eH., Yoon, W.C., 2001b. The effects of presenting functionally abstracted for error detection and error recovery. In: Johnson, C.W. (Ed.), Proceedings of information in fault diagnosis tasks. Reliab. Eng. Syst. Saf. 73, 103e119. the 21st European Annual Conference on Human Decision Making and Control. Ham, D.eH., Yoon, W.C., Han, B.T., 2008. Experimental study on the effects of Department of Computing Science, University of Glasgow, Glasgow, Scotland, visualized functionally abstracted information on process control tasks. Reliab. pp. 8e15. Eng. Syst. Saf. 93, 254e270. Naikar, N., Saunders, A., 2003. Crossing the boundaries of safe operation: an Hassall, M.E., Sanderson, P.M., 2014. A formative approach to the strategies analysis approach for training technical skills in error management. Cognit. Technol. phase of cognitive work analysis. Theor. Issues Ergonomics Sci. 15 (3), 215e261. Work 5, 171e180. Jamieson, G.A., 2007. Ecological interface design for petrochemical process control: Naikar, N., Treadwell, A., Brady, A., 2014. Cognitive work analysis beyond human an empirical assessment. IEEE Trans. Syst. Man Cybern. e Part A Syst. Humans factors and engineering: application to military doctrine and strategy devel- 37 (6), 906e920. opment. In: Proceedings of the 58th Annual Meeting of the Human Factors and Janzen, M.E., Vicente, K.J., 1998. Attention allocation within the abstraction hierar- Ergonomics Society. Human Factors and Ergonomics Society, Santa Monica, CA, chy. Int. J. Human Comput. Stud. 48, 521e545. pp. 330e334. Jenkins, D.P., Stanton, N.A., Salmon, P.M., Walker, G.H., 2008. Using work domain Pawlak, W.S., Vicente, K.J., 1996. Inducing effective operator control through analysis to evaluate the impact of digitization on command and control. In: ecological interface design. Int. J. Human Comput. Stud. 44, 653e688. Proceedings of the 52nd Annual Meeting of the Human Factors and Ergonomics Pfautz, J.D., Pfautz, S.L., 2009. Methods for the analysis of social and organizational Society, 52(4). Human Factors and Ergonomics Society, Santa Monica, CA, aspects of the work domain. In: Bisantz, A.M., Burns, C.M. (Eds.), Applications of pp. 438e442. Cognitive Work Analysis. CRC Press, Boca Raton, FL, pp. 175e228. Jensen, A., Rasmussen, J., Timmerman, P., 1963. Analysis of Failure Data for Elec- Rasmussen, J., 1968a. Characteristics of Operator, Automatic Equipment and tronic Equipment at Risø. Danish Atomic Energy Commission, Research Estab- Designer in Plant Automation (Risø-M-808). Risø, Roskilde, Denmark. lishment Risø, Roskilde, Denmark (Report No. 38). Rasmussen, J., 1968b. On the Reliability of Process Plants and Instrumentation Kilgore, R.M., St-Cyr, O., Jamieson, G.A., 2009. From work domains to worker Systems (Risø-M-706). Danish Atomic Energy Commission, Research Estab- competencies: a five-phase CWA. In: Bisantz, A.M., Burns, C.M. (Eds.), Appli- lishment Risø, Roskilde, Denmark. cations of Cognitive Work Analysis. CRC Press, Boca Raton, FL, pp. 15e47. Rasmussen, J., 1969. Man-machine Communication in the Light of Accident Records Lamoureux, T.M., Chalmers, B.A., 2009. Control task analysis: methodologies for (Report No. S-1e69). Danish Atomic Energy Commission, Research Establish- eliciting and applying decision ladder models for command and control. In: ment Risø, Roskilde, Denmark. Bisantz, A.M., Burns, C.M. (Eds.), Applications of Cognitive Work Analysis. CRC Rasmussen, J., 1979. On the Structure of Knowledge e a Morphology of Mental Press, Boca Raton, FL, pp. 95e127. Models in a Man-machine System Context (Risø-M-2192). Risø National Labo- Lamoureux, T., Sartori, J., 2007. Two methods for control task analysis. In: Pro- ratory, Roskilde, Denmark. ceedings of the 51st Annual Meeting of the Human Factors and Ergonomics Rasmussen, J., 1983. Skills, rules, and knowledge; signals, signs, and symbols, and Society. Human Factors and Ergonomics Society, Santa Monica, CA, other distinctions in human performance models. IEEE Trans. Syst. Man Cybern. pp. 293e297. 13 (3), 257e266. Lau, N., Jamieson, G.A., Skraaning Jr., G., Burns, C.M., 2008. Ecological interface Rasmussen, J., 1985. The role of hierarchical knowledge representation in deci- design in the nuclear domain: an empirical evaluation of ecological displays for sionmaking and system management. IEEE Trans. Syst. Man Cybern. 15 (2), the secondary subsystems of a boiling water reactor plant simulator. IEEE Trans. 234e243. Nucl. Sci. 55 (6), 3597e3610. Rasmussen, J., 1986. Information Processing and Human-machine Interaction: an Leveson, N.G., 2000. Intent specifications: an approach to building human-centered Approach to Cognitive Engineering. North-Holland, New York. specifications. IEEE Trans. Softw. Eng. 26 (1), 15e35. Rasmussen, J., Goodstein, L.P., 1987. Decision support in supervisory control of high- Moray, N., 1988. Ex Risø semper aliquid antiquum: sources of a new paradigm for risk industrial systems. Automatica 23 (5), 663e671. engineering psychology. In: Goodstein, L.P., Andersen, H.B., Olsen, S.E. (Eds.), Rasmussen, J., Jensen, A., 1973. A Study of Mental Procedures in Electronic Trouble Tasks, Errors and Mental Models: a Festschrift to Celebrate the 60th Birthday of Shooting (Risø-M-1582). Danish Atomic Energy Commission, Research Estab- Professor Jens Rasmussen. Taylor & Francis, London, pp. 12e17. lishment Risø, Roskilde, Denmark. Naikar, N., 2012. Cognitive Work Analysis: Foundations, Extensions, and Challenges. Rasmussen, J., Pejtersen, A.M., Goodstein, L.P., 1994. Cognitive Systems Engineering. DSTO General Document (DSTO-GD-0680). Defence Science and Technology John Wiley & Sons, New York. Organisation, Fishermans Bend, Australia. Rasmussen, J., Timmerman, P., 1962. Safety and Reliability of Reactor Instrumen- Naikar, N., 2013. Work Domain Analysis: Concepts, Guidelines, and Cases. Taylor tation with Redundant Instrument Channels (Risø Report No. 34). Danish and Francis Group, LLC, Boca Raton, FL. Atomic Energy Commission, Research Establishment Risø, Roskilde, Denmark.

Rasmussen, J., Vicente, K.J., 1989. Coping with human errors through system design: Ergonomics Society, Santa Monica, CA, pp. 350e354. implications for ecological interface design. Int. J. Man Machine Stud. 31 (5), Sloan, G., 2012. Military doctrine, command philosophy and the generation of 517e534. fighting power: genesis and theory. Int. Aff. 88 (2), 243e263. Read, G.J.M., Salmon, P.M., Lenne, M.G., Jenkins, D.P., 2015. Designing a ticket to ride Treadwell, A., Naikar, N., 2014. The Application of Work Domain Analysis to Defining with the cognitive work analysis design toolkit. Ergonomics 58 (8), 1266e1286. Australia’s Air Combat Capability. DSTO Technical Report (DSTO-TR-2958). Joint http://dx.doi.org/10.1080/00140139.2015.1013576. and Operations Analysis Division, Fishermans Bend, Australia. Available from: Reising, D.V.C., Sanderson, P.M., 1998. Designing displays under ecological interface http://www.dsto.defence.gov.au/sites/default/files/publications/documents/ design: towards operationalizing semantic mapping. In: Proceedings of the DSTO-TR-2958%20PR.pdf. Human Factors and Ergonomics Society 42nd Annual Meeting. Human Factors Vicente, K.J., 1999. Cognitive Work Analysis: toward Safe, Productive, and Healthy and Ergonomics Society, Santa Monica, CA, pp. 372e376. Computer-based Work. Lawrence Erlbaum Associates, Mahwah, NJ. Reising, D.V.C., Sanderson, P.M., 2000a. Testing the impact of instrumentation Vicente, K.J., 2001. Cognitive engineering research at Risø from 1962e1979. In: location and reliability on ecological interface design: control performance. In: Salas, E. (Ed.), Advances in Human Performance and Cognitive Engineering Proceedings of the Joint 14th Triennial Congress of the International Ergo- Research. Elsevier Science, Oxford, UK, pp. 1e57. nomics Association and the 44th Annual Meeting of the Human Factors and Vicente, K.J., 2002. Ecological interface design: progress and challenges. Hum. Ergonomics Society (IEA2000/HFES2000), 1. Human Factors and Ergonomics Factors 44 (1), 62e78. Society, Santa Monica, CA, pp. 124e127. Vicente, K.J., Christoffersen, K., Pereklita, A., 1995. Supporting operator problem Reising, D.V.C., Sanderson, P.M., 2000b. Testing the impact of instrumentation solving through ecological interface design. IEEE Trans. Syst. Man Cybern. 25 location and reliability on ecological interface design: fault diagnosis perfor- (4), 529e545. mance. In: Proceedings of the Joint 14th Triennial Congress of the International Vicente, K.J., Rasmussen, J., 1990. The ecology of human-machine systems II: Ergonomics Association and the 44th Annual Meeting of the Human Factors mediating “direct perception” in complex work domains. Ecol. Psychol. 2 (3), and Ergonomics Society (IEA2000/HFES2000), 3. Human Factors and Ergo- 207e249. nomics Society, Santa Monica, CA, pp. 591e594. Vicente, K.J., Rasmussen, J., 1992. Ecological interface design: theoretical founda- Rochlin, G.I., La Porte, T.R., Roberts, K.H., 1987. The self-designing high-reliability tions. IEEE Trans. Syst. Man Cybern. 22 (4), 589e606. organization: aircraft carrier flight operations at sea. Nav. War. Coll. Rev. 40 (4), Xiao, T., Sanderson, P., 2013. Evaluating the generalizability of the organizational 76e90. constraints analysis framework: a hospital bed management case study. Cognit. Royal Australian Air Force, 2013. The Air Power Manual (AAP 1000-D), sixth ed. Air Technol. Work 16 (2), 229e246. Power Development Centre, Department of Defence, Canberra, ACT. Xu, W., 2007. Identifying problems and generating recommendations for enhancing Schubert, D., Kavanagh, B., 1989. The RAAF writes its doctrine. Def. Force J. 75, complex systems: applying the abstraction hierarchy framework as an analyt- 13e18. ical tool. Hum. Factors 49 (6), 975e994. Sharp, T.D., Helmicki, A.J., 1998. The application of the ecological interface design Xu, W., Dainoff, M.J., Mark, L.S., 1999. Facilitate complex search tasks in hypertext by approach to neonatal intensive care medicine. In: Proceedings of the Human externalizing functional properties of a work domain. Int. J. Human Comput. Factors and Ergonomics Society 42nd Annual Meeting. Human Factors and Interact. 11 (3), 201e229.