There Is No Interface (Without a User). a Cybernetic Perspective on Interaction

INTERFACE CRITIQUE JOURNAL – VOL. I – 2018

THERE IS NO INTERFACE (WITHOUT A USER). A CYBERNETIC PERSPECTIVE ON INTERACTION

By Lasse Scherffig

“Interaction is seen as a one-way street, conveying a design model to a user, who is acting by that model either because they adapted to it, or because the model replicates their given structure. This is the cognitivist heritage of the HCI discourse responsible for the idea that interfaces can actually be designed.”

Suggested citation: Scherffig, Lasse (2018). “There is no Interface (without a User). A cybernetic Perspective on Interaction.” In: Interface Critique Journal Vol.1. Eds. Florian Hadler, Alice Soiné, Daniel Irrgang DOI: 10.11588/ic.2018.0.44739

This article is released under a Creative Commons license (CC BY 4.0).

SCHERFFIG: THERE IS NO INTERFACE

The interface in itself does not exist. rected development from the compu- This is not to say that any phenome- ting machinery of the past towards a non must be perceived in order to ex- future of interaction. ist, but rather that interfaces quite This trajectory constitutes the literally only come into being if they standard account of the history of in- are used. They are effects of interac- teraction. Often the field is seen as tion and thus they are ultimately cre- following a teleological development ated by their users. of progress, during which computers Of course, academic and pro- became more and more interactive, fessional disciplines like human- and interaction became more intui- computer interaction and interaction tive, rich, and natural. This develop- design assume the opposite: namely ment is often explicated as a that interfaces are designed (and ex- genealogy. Depending on the focus ist) before they are used, possibly and goals of their narrators, there are even creating their users. This article1 genealogies of interaction focusing traces the development of this view, on a succession of hardware genera- as well as offering an alternative to it tions, interaction paradigms, theo- that fundamentally understands any retic frameworks, or visionaries interface as “cybernetic interface.”2 pushing the field to the next level. An early and paradigmatic account that focuses on hardware is John Walker’s genealogy of five “User GENEALOGIES OF Interaction Generations”4 published in the early 1990s. This account starts INTERACTION with the “plug boards” and “dedicated setups” of early computing.5 These When during the late 1990s, the new were followed by the (in)famous era millennium prompted countless ret- of batch processing – a time when rospectives and outlooks, Terry Wino- programming meant punching holes grad contributed a chapter to a book into cards, handing batches of these about the next fifty years of computer cards to a mainframe operator and science titled, “From Computing Ma- 3 waiting for hours to be handed back a chinery to Interaction Design.” Fol- printed result. lowing an idea of evolutionary It is only the third of these progress, this title described a goal di- generations that was interactive. As

–––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––– 1 A German language discussion of similar 3 Terry Winograd, “From Computing questions, that is much broader in scope, can be Machinery to Interaction Design,” in Beyond found in Lasse Scherffig, “Feedbackmaschinen. Calculation: The Next Fifty Years of Computing, ed. Kybernetik und Interaktion” (Dissertation, Peter Denning and Robert Metcalfe (Berlin and New Kunsthochschule für Medien Köln, 2017). York: Springer, 1997). 2 Søren Bro Pold, “Interface Perception: The 4 John Walker, “Through the looking glass,” in Cybernetic Mentality and Its Critics: The Art of Human-Computer Interface Design, ed. Ubermorgen.com,” in Interface Criticism: Aesthetics Brenda Laurel (Redding, MA: Addison-Wesley, 1990), Beyond Buttons, ed. Christian Ulrik Andersen and 439. Søren Bro Pold (Aarhus: Aarhus University Press, 5 Ibid. 439-440. 2011), 91.

INTERFACE CRITIQUE JOURNAL – VOL. I – 2018

to be expected from a proper genera- defined by “electrical interaction” (us- tion, it was a child of the previous one ing cables, plugs, and the soldering and generated by it: When, this ac- iron) with Walker’s dedicated set-ups. count goes, the algorithms allocating This was followed by the era of “sym- a mainframe’s computing time to bolic interaction” that was marked by several batch jobs got more and more the use of punch cards and batch pro- advanced, it became clear that it cessing – which were often pro- would be possible to divide the com- grammed using the symbols of puting time of a mainframe even fur- assembler languages instead of the ther. Divided into small enough raw zeros and ones of machine code. pieces, that follow each other in rapid This generation, in turn, led to the succession, it would seem to several “textual interaction” with the termi- people that they would have exclu- nals of time sharing systems. “Graph- sive control over the whole machine. ical interaction” here again marks the This idea, named “time sharing,” did final step in an evolution starting not divide a computer’s resources be- with machinery and ending with to- tween several batch jobs but between day’s interactive surfaces. several humans (re-defining these, as Apart from their implicit as- we will see below, as “users”) who sumption that interactivity progres- could now engage in “conversational sively increases, these, and most interactivity” with the machine.6 other histories of human-computer While the conversations of the time interaction (HCI) have one thing in sharing generation happened as ex- common: They all assume an origin changes of written text, the fourth of of interaction. While operating a com- Walker’s generation of interaction in- puter during Walker’s second genera- troduced graphical displays that con- tion meant batch processing, the centrated textual commands into third generation introduced time visual menus. The fifth and final gen- sharing and with it, interaction. In eration then spawned the graphical Dourish’s terminology, this transition user interfaces of personal compu- corresponds to the shift from “sym- ting that, in various iterations, keep bolic” (based on assembler language accompanying us on our desktops, and punchcards) to “textual interac- laptops, and phones until today.7 tion” (based on conversational inter- Paul Dourish’s “History of In- activity via command line). Only teraction,”8 which is much more con- when computers, after time-sharing temporary in style, follows a very was introduced, started to react similar path “from soldering to (seemingly) exclusively and directly mouse.” Focusing on the mode of in- to human input, did they become interaction instead of hardware gener- teractive: “Arguably, this is the origin ations, the first generation here was of »interactive« computing.”9

–––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––– 6 Ibid. 441. 8 Paul Dourish, Where the action is. The 7 Ibid. 441-442. foundations of embodied interaction (Cambridge, MA: MIT Press, 2001), 1-17. 9 Ibid. 10.

SCHERFFIG: THERE IS NO INTERFACE

Of course, history is not that ing with analog (i.e. continuous) val- simple. A closer look behind the nar- ues, it rather implied an entirely dif- ratives postulating a teleological de- ferent approach toward calculation: It velopment of interaction instead relied on building electrical and me- reveals contradictory and asynchro- chanic systems, that, as analogues or nous developments, as well as chron- analogies, could stand in for the sys- ological overlaps.10 Especially the tems they were built to simulate. Be- origin of interactive computing itself cause building such analog can be described differently, for in- computers entailed accurately fol- stance, by looking at the first interac- lowing and amplifying changing tive computer ever built – which physical signals, it largely depended happens to be one of the first comput- on another development: the rise of ers at all. It is, as this look reveals, the the use of negative feedback as the de very first generation of computing facto standard method for handling machinery that defined interactivity electro-mechanical systems. In fact, up to this day, including its problems. during the early twentieth century, negative feedback became so important in both control and communication engineering that both INTERACTIVE disciplines merged into one feedback COMPUTERS AS based control theory – in a paradigm shift that yielded the era of “classi- FEEDBACK cal”12 control. This development, in turn, constituted the nucleus of what MACHINES Norbert Wiener would later call cybernetics – a science of “control and During the 1940s, the MIT Servomech- communication in the animal and the anisms Lab started to build a flight machine”13 that would become think- simulator. As the leading paradigm able mainly because the application for automatic computation at that of negative feedback and the associ- time was analog computing, the flight ated mathematical formalisms simulator was planned to be based on seemed to be powerful enough to that: “a cockpit or control cabin con- tackle any form of “behavior” – of liv- nected, somehow, to an analog com- ing and non-living systems.14 Be- puter.”11 “Analog computing” in this cause feedback implies using the context did not only imply calculat- output of a system as its own input, the systems of cybernetics exhibited –––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––– 10 Hans Dieter Hellige, “Krisen- und Innovati- 12 Stuart Bennett, A History of Control onsphasen in der Mensch-Computer-Interaktion,” in Engineering 1930–1955 (Hitchin: Peter Peregrinus Mensch-Computer-Interface. Zur Geschichte und Zu- Ltd., 1993), 17. kunft der Computerbedienung, ed. Hans Dieter Hellige 13 Norbert Wiener, Cybernetics or: Control and (Bielefeld: Transcript, 2008), 15-20. Communication in the Animal and the Machine 11 Kent C. Redmond and Thomas M. Smith, (Cambridge, MA: MIT Press, 1961). Project Whirlwind: the history of a pioneer computer 14 Arturo Rosenblueth, Norbert Wiener, and (Bedford, MA: Digital Press, 1980), 32. Julian Bigelow, “Behavior, Purpose and Teleology,” Philosophy of Science 10 (1943): 18–24.

INTERFACE CRITIQUE JOURNAL – VOL. I – 2018

“circular causality”15 – a circular inter- were trying to build a flight simulator. dependence of input and output, en- This was especially problematic, as tailing that agency within the system the ASCA’s cockpit still was the ana- is distributed and cannot be pinned log machine the project started with. down to specific agents. Whereas coupling the motion and In building the simulator, continuously changing electrical sig- moving axes and disks, and changing nals of an analog computer to the an- voltages and currents were used as alog instruments of a cockpit did not analogies to the complex dynamics of pose a categorical problem, this had a plane in flight. As these analogies changed with digital computing. The constituted electro-mechanical mo- digital and discrete state changes of tion, coupling them to the moving the new computer had to be trans- controls of a cockpit and the motion lated into continuous motion of the of their human operators was self-ev- instruments, while the reactions of ident. However, during the develop- the operators on these instruments, ment of this “Aircraft Stability and in turn, had to be translated into digi- Control Analyzer” (ASCA)16 the first tal states.19 “These problems were not digital computers were under con- impossible, but neither did estab- struction as well. The engineers at lished solutions exist. The digital MIT observed this development and computer was too new,”20 one of the Jay Forrester, one of the project leads, engineers in the project later wrote. In became more and more interested in consequence, the project manage- digital computation – so interested, ment acknowledged that it was not in fact, he sacrificed the core of the about building a flight simulator any- project (building a flight simulator) to more and the cockpit of the ASCA was his new interest (building a digital scrapped.21 The computer was re- computer): The development of the named “Whirlwind”22 and became a analog computer was halted, and a general-purpose digital computer not “general purpose, high speed”17 digital usable for flight simulation anymore. computer was built. As it was one of As it thus became a computer without the first of its kind, the engineers application, it later would be turned building it were constantly “pushing from flight simulation to air defense the state of the art,”18 developing new and become the foundation for SAGE, building blocks for digital computa- the “Semi-Automatic Ground Envi- tion, such as memory mechanisms. ronment” air defense system – the Caught up in this task, however, they largest computer built to date that increasingly lost sight of the fact they was in use until 1983.23 –––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––– 15 For a detailed discussion, see Heinz von 18 Robert Everett, “Whirlwind,” in A History of Foerster, “Cybernetics of Epistemology,” in von Computing in the Twentieth Century, ed. J. Howlett, Foerster, Understanding Understanding: Essays on Gian Carlo Rota, and Nicholas Metropolis (Orlando: Cybernetics and Cognition (New York: Springer, Academic Press, 1980), 365. 2003): 229–246. 19 Redmond and Smith, Project Whirlwind, 49. 16 Redmond and Smith, Project Whirlwind, 51. 20 Ibid. 49. Emphasis by author. 17 MIT, Whirlwind I: A high-speed Electronic 21 Ibid. 60. Digital Computer, promotional brochure (Cambridge, 22 Ibid. 43-44. MA: MIT, 1951), 6. 23 Ibid. 206.

SCHERFFIG: THERE IS NO INTERFACE

What set Whirlwind apart was both: a machine and interactive from the other first-generation digital – a feedback machine. computers of its time was its heritage in analog computing and flight simulation: It was conceived as a machine that reacts to changes in an environ- INTERRUPTION ment (the cockpit) by incorporating any change happening here into its AND COUPLING: A calculating. In addition, it would have BLACK ART the results of these calculations directly, and in real-time affect the en- Even after having scrapped the cock- vironment. In other words; it was a pit, Whirlwind was still a machine to digital computer that was to function be used by human operators in real- like the control systems of analog time and as such posed two problems: computing and cybernetics – as a How to integrate real-time input from digital computer that can react to its the environment into an ongoing dig- environment in real-time. ital computation, and how to couple This is remarkable, given that the process of digital computation to theoretical computer science oper- the action and perception of human ates with a conception of “machine,” operators. The engineers of Whirl- explicated as with the Turing ma- wind approached these novel (or even chine, that does not know time or any “too new”) problems pragmatically. reciprocal interaction between calcu- The fundamental problem of lation and its environment. Only rela- having the machine react to its envi- tively recently did theoretical ronment was tackled introducing a computer science start to basic technique into computer engi- acknowledge, that the actual compu- neering whose heritage is alive until ting machines we have been using today: Whirlwind could interrupt from the very beginning had done what it was working on, turn to any something that goes beyond Turing’s new data that may have arrived in the definition of computation – by incor- meantime, integrate that data (by porating interaction with an environ- 24 copying it into memory), and con- ment. 25 tinue where it had left off. Coupling Whirlwind thus was a strange the machine to the environment thus hybrid: A digital computer that also became a function of interruption – tried to be a cybernetic feedback sys- which, as hardware interrupt, later tem, in constant dialog with the envi- became a core feature of any interac- ronment it controlled. If we follow tive computer. Winograd’s juxtaposition of compu- The problem of coupling com- ting machinery and interaction, it putation to human operation was instead approached by introducing –––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––– 24 Peter Wegner, “Why interaction is more 25 Everett, “Whirlwind,” 377. powerful than algorithms,” Communications of the ACM 40, no. 5 (1997): 83.

INTERFACE CRITIQUE JOURNAL – VOL. I – 2018

what later would be defined as one of back the computer’s visual output to the teleological ends of the develop- its own interrupt: The “gun” was de- ment of interactivity: Whirlwind pro- signed not to shoot but to pick up duced graphical representations that light. Pointed at a visual representa- could be touched. This was made pos- tion on screen, it would pick up the sible when the engineers in the pro- light emitted when the computer ject coupled memory registers of the drew this very representation. If an computer to the x/y-control of the operator now pressed a button, the magnetic fields of a cathode ray tube computer was interrupted while (CRT). Whirlwind could thus paint drawing it. It thus “knew” which item symbolic representations of data onto was selected and could take this se- screen: “One of the things that I think lection into account for further com- we did first was to connect a visual putation.29 Even the light gun, display to a computer.”26 This great although pioneered here as an inter- leap into our screen-based present action device, had technically already happened with the pragmatic natu- been built before it became part of the ralness of something “I think we did configuration of interactive compu- first,” simply because all prerequisites ting – as it was originally used to test for it were already in place: The se- the Williams Tube memory devices cond world war had established vari- for errors.30 ous modes of coupling (analog) radar Coupling Whirlwind to people was data to CRTs. Project Whirlwind could thus both: the pragmatic problem- build on this foundation and even use solving of engineers using parts and the leftover CRTs of the war.27 In addi- components at hand, and a revolution, CRTs had already been coupled tionary prototype for most interactiv- to digital computers: In the “Williams ity to come. But while it offered the Tube,” the afterglow of the light basic capability of having human ac- painted onto a screen was used as a tion become part of an ongoing com- short-term memory device that was putation, it did not solve any not meant to be looked at by humans, problems of how exactly this setup but nevertheless constituted com- should be used. Instead, computer puter control of light on a screen.28 science had unexpectedly introduced In order to close the loop be- a new class of problems, as the repre- tween representation and action, the sentations and couplings it made images painted by Whirlwind onto its possible now had to be designed. It CRTs were accompanied by a device became a field of design, a “black art” to touch them: a “light-gun” (figure 1). The device realized this by feeding

–––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––– 26 Ibid. 375. 30 Michael Friedewald, Der Computer als Werk- 27 Ibid. 379. zeug und Medium: Die geistigen und technischen Wur- 28 Claus Pias, “Computer Spiel Welten” (Disser- zeln des Personal Computers (Berlin and Diepholz: tation, Bauhaus-Universität Weimar, 2000), 55-56. Verlag für Geschichte der Naturwissenschaften und 29 C. R. Wieser, Cape Cod System and der Technik, 1999), 103. Demonstration, Technical Report (Cambridge, MA: Lincoln Laboratory – Division 6, 1953), 2.

SCHERFFIG: THERE IS NO INTERFACE

in which “engineering design,” “crea- teraction”34 by Stuart Card, Thomas tive design,” and scientific methods Moran, and Allen Newell. The role of came (and still come) together.31 the latter in establishing HCI is remarkable, as he serves as a link back to the first interactive computer as well as pointing towards the future of (IN)HUMAN the field. FACTORS: THE Early in his career, Newell worked at RAND’s Systems Research Labora- USER AS NEW tory. Here, he was in charge of training the operators of the SAGE system HUMAN – and thus the first professional operators of interactive computers.35 This In spite of Whirlwind, the narratives work was conducted together with of the progressive incline of interac- Herbert Simon, with whom Newell tivity are not plainly wrong. Although would continue working on a number interactive computing existed before of subsequent projects. While build- time-sharing,32 MIT’s Whirlwind was ing a training environment for the a singular development and most of SAGE operators, Newell used com- computer science for a long time puter modeling to simulate the input stuck to building machines running into the training system, consisting algorithms that produce answers of human operators and simulated without being interrupted. computer consoles. His simulation Important early developments, such created sequences of “radar blips,” as as Ivan Sutherland’s Sketchpad33 and they would have shown up on the real especially Douglas Engelbart’s NLS, screens of the SAGE air defense sys- were running against this main- tem. stream that was so dominant it took The realization that computers could the field until the 1980s to do something like this, and thus acknowledge “interaction” as an inde- “more than arithmetic”36 would prove pendent area of inquiry. One of the highly influential for Newell and Si- first books carrying human-computer mon. The fact that in training these interaction (HCI) in its title was “The computer operators, computer mod- Psychology of Human-Computer In- eled input data shown on computer

–––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––– 31 Hellige, “Krisen- und Innovationsphasen,” 16. Interaction (Hillsdale, NJ and London: Lawrence 32 In fact, “time-sharing” is an after-the-fact Erlbaum Associates, 1983). conceptualization of what was done in the project, as 35 Douglas D. Noble, “Mental Materiel. The the term was first used by an engineer working on militarization of learning and intelligence in US the already interactive SAGE system. See Friedewald, education,” in Cyborg Worlds. The Military Information Der Computer als Werkzeug und Medium, 128. Society, ed. Les Levidow and Kevin Robins (London: 33 Which was programmed on a TX-2 – a direct Free Association Books, 1989), 19. descendant of Whirlwind. See Friedewald, Der 36 Herbert A. Simon, “Allen Newell. 1927-1992,” Computer als Werkzeug und Medium, 110-118. National Academy of Sciences Biographical Memoirs 34 Stuart K. Card, Thomas P. Moran, and Allen (1997): 146. Newell, The Psychology of Human-Computer

INTERFACE CRITIQUE JOURNAL – VOL. I – 2018

CRT and light-gun in action Copyright: The MITRE Corporation

SCHERFFIG: THERE IS NO INTERFACE

screens37 would be perceived and in- able to manipulate symbols – be it hu- terpreted by human observers, led man or machine.39 This argument, at Newell and Simon to the far-reaching the time, was part of the development conclusion that all participants of the of a new scientific field of studying system were essentially involved in the human mind that, at least for a the same task: processing infor- long time, understood thinking as mation. Just as Newell’s digital simu- rule-based information processing: lation processed information in order cognitive science.40 The field from to produce the fake radar blips, the the very beginning “subsumes vari- human operators looked at these ous computational theories of mental blips and perceived them as infor- phenomena. Their computational na- mation to be processed and acted ture is what unifies the multiple dis- upon. In other words: “Within the sim- ciplines in the field and may count for ulated training environment, Newell much of its success in recent years.”41 came to view the human operators In this sense, the human trained to too as »information processing sys- perform in front of the computer be- tems« (IPS), who processed symbols came the model for the thinking hu- just like his program »processed« the man in general – a human acting as a symbols of simulated radar blips.”38 computer. This is the crucial outcome of This is what Newell brought the training for the first interactive back to working with interaction: He computers. Subsequently, Newell and proposed to XEROX PARC an “Applied Simon authored a number of papers Information-processing Psychology that took this idea further, developing Project (AIP)”42 that promised to apply an understanding of human thinking cognitive science to the black art of that was driven by the verdict that it designing interaction. The project is a form of the symbolic information started in 1974, led by Card and Mo- processing exhibited by computers. ran, who were consulted by Newell. This culminated with the “Physical One of its results was the publication Symbol System Hypothesis,” declar- of “The Psychology of Human-Com- ing intelligence to be a feature of all puter Interaction” by the three. forms of physical systems that are –––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––– 37 Which, ironically, in the training system were standard account of the history of the field see simulated by complex analog display machinery Howard Gardner, The Mind’s New Science. A History showing sequences of pre-rendered screens. See of the Cognitive Revolution (New York: Basic Books, Robert L. Chapman, John L. Kennedy, Allen Newell, 1985), 28, and George A. Miller, “The cognitive and William C. Biel, “The Systems Research revolution: a historical perspective,” Trends in Laboratory’s Air Defense Experiments,” Management Cognitive Sciences 7, no. 3 (2003): 141-144. Science 5, no. 3 (1959): 256-262. 41 Frank Schumann, “Embodied Cognitive 38 Noble, “Mental Materiel,” 19. Science: Is it Part of Cognitive Science? Analysis 39 Allen Newell and Herbert A. Simon, within a Philosophy of Science Background,” PICS. “Computer science as empirical inquiry: symbols and Publications of the Institute of Cognitive Science 3 search,” Communications of the ACM 19, no. 3 (2004): 12. (1976): 116. 42 Stuart K. Card and Thomas P. Moran, “User 40 Newell and Simon presented a Logic Theory Technology: From Pointing to Pondering,” in Machine at the famous Symposium on Information Proceedings of the ACM Conference on The History of Theory at MIT in 1956, which often is understood as Personal Workstations (ACM: 1986), 183. the founding event of cognitive science. For this

INTERFACE CRITIQUE JOURNAL – VOL. I – 2018

The center of this project was wrote: “There is emerging a psychol- not longer the computer operator. In- ogy of cognitive behavior that will stead, it was the “user” of the com- permit calculation of behavior in new puter interface. Card, Moran, and situations and with new humans…”46 Newell stated: “But the user is not an Since this user was to be subject to operator. He does not operate the the technical understanding pro- computer, he communicates with it vided by computational theories of to accomplish a task.”43 This attribu- mental phenomena, what emerged tion of agency to the computer (as an here was a view of the human being equal partner in communication) using the computer, as a computer. probably followed from the nature of The human factors of human- the interactive computer as feedback computer interaction, and human- or machine that exhibits circular cau- user-centered design thus become sality between machine and (human) readable as the “inhuman factors”47 of environment. For the authors, how- thinking humans as machines – and ever, the relationship of user and making them act accordingly. The computer was defined solely by the training required to become the new postulated equivalence of all infor- human that an interface demands, in mation processing systems. this sense, can be seen as a “subtle en- In proposing the project to slavement”, and a “total, unavowed XEROX, Newell suggested to marry disqualification of the human in favor the empirical methods of human fac- of the definitive instrumental condi- tors with the formal (and computa- tioning of the individual.”48 tional) models of cognitive science, creating “a technical understanding of the user himself and of the nature of human-computer interaction.”44 COGNITIVE This would be a “science of the user ENGINEERING rooted in cognitive theory.”45 In doing so, he seemed to be VERSUS aware that this user was not a given, but something that was created by CONCRETE the systems being used – after all, it was a training environment for early THINKING computer users that gave rise to the idea of the human as information It is this convergence of computer processing system. In the memo pro- and cognitive science that served as posing the AIP to XEROX he thus the “origin myth” of human-computer interaction.49 The field did, for a long

–––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––– 43 Ibid. 7. 47 Anthony Dunne, Hertzian tales. Electronic 44 Ibid. 183. products, aesthetic experience, and critical design 45 Ibid. (Cambridge, MA: MIT Press), 2008, 21. 46 As quoted in Card und Moran, “User 48 Paul Virilio as cited in Dunne, Hertzian tales, Technology,” 183. 21. 49 Dourish, Where the action is, 61.

SCHERFFIG: THERE IS NO INTERFACE

time, embrace cognitive science and communicated in a way yielding the its methods, effectively becoming a appropriate mental model. In terms of form of “cognitive engineering” as information processing, this means Donald Norman defined it in a semi- that by its design a system must pro- nal paper: “neither Cognitive Psychol- vide the information that, once per- ogy, nor Cognitive Science, nor ceived and processed, leads to the Human Factors. It is a type of applied appropriate actions that fulfill a given Cognitive Science, trying to apply goal. what is known from science to the According to Norman, there design and construction of ma- are two ways of achieving this: chines.”50 “(M)ove the system closer to the user; Being based on the cognitive move the user closer to the system.”55 science idea of what a human is, cog- Of course, user-centered design nitive engineering was seen as a form wants to move the system closer to of “user-centered” design. At the cen- the user, by creating systems whose ter of this idea stands a juxtaposition physical states behave in an “intui- of the mental and the physical. Inter- tive” or “natural” way, close to the action, the argument goes, is an act of mental intentions of their users. This, mediating between a user’s mental of course, implies that the latter can goals and the physical states of a sys- be formulated in terms of the former. tem. This mediation happens in a A user’s non-physical goals and in- loop of “execution” and “evaluation,” tentions must be translatable into while execution is based on action se- physical actions and system states, quences a user formulates according thus reproducing the assumption to their goals.51 Formulating these ac- that computer users ultimately can be tion sequences is possible because understood on the same ground as users possess a “mental model” of the computers they use. how they assume a system func- The relationship between the tions.52 psychological states of a user and the The task of the interface de- physical states of a system has been signer as cognitive engineer now is to described as “directness.”56 This term make sure that this mental model is entered HCI discourse when Ben correct – so that an action sequence Shneiderman in 1983 was puzzled by will lead to the expected and intended “[c]ertain interactive systems” which results. They must bridge the gulf be- “generate glowing enthusiasm tween execution and evaluation.53 Creating an interface thus becomes an act of communication where a de- signer’s “design model”54 must be

–––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––– 50 Donald A. Norman, “Cognitive Engineering,” 52 Ibid. 46. in User-Centered System Design: New Perspectives on 53 Ibid. 38. Human-Computer Interaction, ed. Donald A. Norman 54 Ibid. 46. and Stephen Draper (Hillsdale, NJ: Lawrence Erlbaum 55 Ibid. 43. Associates, 1986), 31. 56 Ibid. 52. 51 Norman, “Cognitive Engineering,” 41.

INTERFACE CRITIQUE JOURNAL – VOL. I – 2018

among users.”57 What set these sys- cooperation with James Hollan and tems apart was the interactivity al- Edwin Hutchins.58 This argument ready introduced by Whirlwind: started with the assertion that, “[w]e “(D)irect manipulation” of graphical see promise in the notion of direct representations without the need to manipulation, but as of yet we see no type text – the origin (as constructed explanation of it.”59 here) of interaction thus once more Trying to formulate this ex- got reinterpreted as a milestone of the planation as a full-fledged “cognitive progressive incline of the field. But as account”60 of direct manipulation, opposed to the pragmatic engineer- they reformulate Shneiderman’s dis- ing behind Whirlwind’s early inter- tance of syntax and semantics as an faces, Shneiderman’s discussion of “information processing distance” be- direct manipulation followed a cogni- tween human intentions and ma- tivist pattern, having a clear idea of chine states61 – a distance that direct the human user as rational problem manipulation is minimizing. These solver in mind: Shneiderman repro- interfaces, in this view, are easier to duced Norman’s idea of interaction as use because what we want do with psychophysical mediation by identi- them corresponds to the way it is fying a “problem domain” of “seman- done. tic” intentions and a “program do- This, however, may not be main” of “syntactic” manipulations at enough to explain the “glowing en- the interface. Direct manipulation, he thusiasm” described by Shneider- argued, enables users to interact di- man. Instead, the authors acknow- rectly with the objects of the problem ledged that direct manipulation domain – by, for instance, enabling a seems to entail an experiential com- writer to directly interact with para- ponent that can not be explained by graphs of text, instead of having to information processing alone. It fea- deal with the commands meant to tures a feeling of “engagement,”62 that manipulate these paragraphs. Direct is hard to come by: “Although we be- manipulation would hence be a (or lieve this feeling of direct engage- maybe the) realization of Norman's ment to be of critical importance, in “move the system closer to the user” fact, we know little about the actual by minimizing the distance of the requirements for producing it.“63 Re- problem and program domain. ferring to Brenda Laurel’s work that Not surprisingly, Norman applied Aristotelian poetics to HCI, himself later joined the discussion, they concluded that a feeling of “first- expanding Shneiderman’s work in personness”64 must be responsible for –––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––– 57 Ben Shneiderman, “Direct Manipulation: A 60 Ibid. Step Beyond Programming Languages,” in The New 61 Ibid. 311. Media Reader, ed. Noah Wardrip-Fruin and Nick 62 Ibid. 332. Montfort (New York, NY and London: W. W. Norton & 63 Ibid. 332-333. Company, 2001), 486. 64 Ibid. 318. See also Brenda K. Laurel, 58 Edwin L. Hutchins, James D. Hollan, and “Interface as mimesis,” in User-Centered System Donald A. Norman, “Direct Manipulation Interfaces,” Design: New Perspectives on Human-Computer Human-Computer Interaction 1 (1985): 311–338. Interaction, ed. Donald A. Norman and Stephen 59 Ibid. 316.

SCHERFFIG: THERE IS NO INTERFACE

the feeling of engagement. For Laurel, and other forms of non-screen-based this feeling was based on the inter- interactivity emerged. When, for in- play of user and interface, as “[a]n instance, physical objects in research terface [...] is literally co-created by its projects at MIT and elsewhere be- human user every time it is used.“65 came phicons67 – physical icons that Direct manipulation hence represent data and computational seems to contain a playful compo- processes – researchers at XEROX nent and a residue of the non-ra- PARC coined the term, “interfaces for tional. It is not about a cognitive dis- really direct manipulation.”68 If tangi- tance between mental intention and ble user interfaces use real-world ob- physical representation and action jects as representations of compu- alone, it also is about a subjective ex- tation, the hope was, they would feel perience that is created through the ultimately natural and the infor- cyclic dependence of user action and mation processing distance would be machine response. This non-rational reduced to zero. (or non-cognitivist) residue, however, This, however, makes two things ap- seemed to deeply bother Hutchins, parent: First, if the mouse and screen Hollan and Norman, who stated: felt natural during the 1980s and tangible user interfaces felt more (or re- On the surface, the fundamen- ally) natural during the early 2000s, tal idea of a direct manipulation naturalness itself must be understood interface to a task flies in the as a fluid category depending on what face of two thousand years of feels natural for the “new human” of development of abstract for- each era. Interfaces like the touch malisms as a means of under- screen in this light must be understanding and controlling the stood as being products of a naturali- world. Until very recently, the zation creating the very human for use of computers has been an which they feel natural. Second, the activity squarely in that tradi- whole discussion of tangible interaction. So the exterior of direct tion neglects the fact that all inter- manipulation, providing as it faces in one form or another have does for the direct control of a been tangible: We have never “di- specific task world, seems rectly” manipulated a paragraph of somehow atavistic, a return to text but always had to deal with pens 66 concrete thinking. and marks on paper, keyboard and This return to concrete thinking sub- screen, fingers on a touchscreen. The sequently became even more promi- atavistic syntax of executing manual nent when “tangible user interfaces” –––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––– Draper (Hillsdale, NJ: Lawrence Erlbaum Associates, Controls for Online Media,” in Proceedings of 1986), 67–85. SIGGRAPH (ACM: 1998), 379. 65 Laurel, “Interface as mimesis,” 73. 68 Kenneth P. Fishkin, Anuj Gujar, Beverly L. 66 Hutchins, Hollan and Norman, “Direct Harrison, Thomas P. Moran, and Roy Want, Manipulation Interfaces,” 337. “Embodied user interfaces for really direct 67 Brygg Ullmer, Hiroshi Ishii, and Dylan Glas, manipulation,” Communications of the ACM 43, no. 9 “mediaBlocks: Physical Containers, Transports, and (2000): 74–80.

INTERFACE CRITIQUE JOURNAL – VOL. I – 2018

actions always existed and was al- conditioning that ultimately natural- ways different from any semantic izes a non-human mode of action and goal or intention. perception, we presuppose the inter- What, instead, differentiates face as the agent of this process. If in- tangible user interfaces from graph- terfaces are seen as forming the new ical user interfaces and these from human after their own image (by the command line is something moving them closer to the system) or much more profane: It is the simple if they supposedly assist a given hu- spatio-temporal distance of human man by modeling their non-physical action and computer reaction as well goals or semantics, they are assumed as their perceived similarity. The in- to be sources of information that are cremental progress of interaction, perceived, processed and acted upon. postulated by the genealogies of in- Interaction is seen as a one-way teractivity, is another clue suggesting street, conveying a design model to a that what is really interesting about user, who is acting by that model ei- interactivity is the closure of the gap ther because they adapted to it, or be- in space and time between human cause the model replicates their and computer action. In particular, given structure. This is the cognitivist the theoretical reflection on non- heritage of the HCI discourse respon- screen based interfaces had under- sible for the idea that interfaces can stood this at an early stage. Already in actually be designed. 2000, Michel Beaudouin-Lafon has When, however, the engineers concluded that what is really im- in project Whirlwind coupled digital portant about the experience of com- computation to symbolic representa- puter interfaces is the “spatial and tion and human action back to com- temporal offset,” the “ratio between putation, they not only wrapped its the number of degrees of freedom” human operators in the feedback loop and the “similarity between the phys- of the circular systems of cybernetics: ical actions of the users on the instru- They also created a setting in which ment and the response of the the representations would be object.”69 wrapped in a loop of human action and perception. The motion on a computer screen is not real motion but a cin- PERCEIVING ema-like sequence of still images, ACTION which psychology denotes as “apparent motion.” Apparent motion has

been a subject of experimental psy- No matter if we hail the natural or in- chology since the cinematograph and tuitive interface as bridging the gap cinema rendered it ubiquitous, pro- between user and system, or if we viding the experimental systems for condemn the interface as a form of studying it and for using it as a tool to

–––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––– 69 Michel Beaudouin-Lafon, “Instrumental WlMP User Interfaces,” in Proceedings of CHI (ACM: Interaction: An Interaction Model for Designing Post- 2000), 446–453.

SCHERFFIG: THERE IS NO INTERFACE

study perception in general.70 One about how form, color and initial po- seminal early work was Max sition of ambiguous motion stimuli Wertheimer’s experimental studies of influence the way we perceive the perception of movement,71 which them.74 During his experiments, how- today is seen as one of the founding ever, he did make one especially re- texts of Gestalt psychology.72 While markable observation: His subjects Werheimer pioneered the experi- were able to actively control the per- mental investigation of apparent mo- ceived direction of motion if they tion, Gestalt psychologists like Paul moved their heads or hands. This was Linke and later Paul von Schiller a case, he remarked in a footnote studied the phenomenon with a focus only, where motor activity shapes the on a fringe case of it: The perception Gestalt of optical perception.75 of “ambiguous motion,” which is present whenever the direction of an apparent motion stimulus can not be decided objectively (figure 2). Such stimuli are interesting because they afford more than one possible perceptual interpretation, while subjectively only one direction of motion is perceived at a time. They thus reveal how the sensory system is treating Ambiguous motion as described by stimuli in deciding how they are to be Linke: Rotating a cross by steps of perceived, making them “invaluable 45° can be perceived as clockwise or tools for the study of the neural basis counterclockwise motion. of visual awareness, because they al- low us to distinguish neural re- sponses that correlate with basic About sixty years later, this sensory features from those that cor- little noted observation was con- relate with perception.”73 firmed by modern psychology: In a Trying to establish the “laws” brief article in the journal “Investiga- of how visual perception deals with tive Ophthalmology & Visual Sci- ambiguous motion, von Schiller pre- ence,” Ishimura and Shimojo report empted a number of results of con- temporary experimental psychology

–––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––– 70 Christoph Hoffmann, “φ-Phänomen Film. Der discovery launched the Gestalt revolution,” Vision Kinematograph als Ereignis experimenteller Psycho- Research 40 (2000): 2257–2264. logie um 1900,” in Die Adresse des Mediums, ed. Ste- 73 David M. Eagleman, “Visual illusions and fan Andriopoulos, Gabriele Schabacher, and Eckhard neurobiology,” Nature Reviews Neuroscience 2 Schumacher (Cologne: DuMont, 2001), 236. (2001): 922. 71 Max Wertheimer, “Experimentelle Studien 74 Paul von Schiller, “Stroboskopische Alterna- über das Sehen von Bewegung,” Zeitschrift für Psy- tivversuche,” Psychologische Forschung 17 (1933): chologie und Physiologie der Sinnesorgane 61 (1912). 180. 72 Robert M. Steinman, Zygmunt Pizlo, and 75 von Schiller, “Stroboskopische Filip J. Pizlo, “Phi is not beta, and why Wertheimer’s Alternativversuche,” 196,

INTERFACE CRITIQUE JOURNAL – VOL. I – 2018

that “Voluntary Action Captures Vis- action merely is planned, but not exe- ual Motion.”76 In a series of studies,77 cuted. It could also be changed they had shown that hand move- through training: After using a mouse ments capture (as in: influence) the whose control of the cursor on screen way we perceive visual motion. Their was inverted such that a motion to experiments, of course, were con- the right yielded an on-screen motion ducted with a computer, coupling mo- to the left, subjects exhibited a corre- tion on a physical interface to visual sponding change in action capture, so representation on screen. A few years that a motion to the left influenced later, Andreas Wohlschläger contin- ambiguous motion to the right. Ap- ued this research, analyzing more parently, the effect takes into account features of the effect.78 Later, it has the expected results an action has. also been shown that this does not Action capture thus demonstrates only hold for the relation of hand and that the more an action is related to eye, but that auditory and tactile per- the reaction it provokes (in terms of ception can be influenced by motion spatio-temporal distance, orientation, of the hands, eyes, head or feet as and its expected results), the more it well.79 influences perception of that action. What these studies showed is From the point of view of not only that our motor actions di- physiology, it has long been known rectly influence what we perceive. that the neural activity causing mo- They also showed that this influence tion, which originates in the motor is stronger, the closer action and per- cortex of the brain, is not only com- ception happen in space and time municated to the muscles executing and the more their features (like their motion, but also to sensory areas. Mo- spatial orientation) align. The strong- tor signals are accompanied by an “ef- est influence was measured when ference copy”80 or “corollary dis- manual motion and computer reac- charge”81 that relays them to parts of tion happened simultaneously and the brain responsible for perception. overlapped each other. More im- This is thought to be part of a process portantly even, they also showed that in which the expected results of an the effect is even present if a motor action are compared to what actually –––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––– 76 The note only covers one sixth of a page in Moore, “Tactile Rivalry Demonstrated with an the issue. G. Ishimura and S. Shimojo, “Voluntary Ambiguous Apparent-Motion Quartet,” Current action captures visual motion,” Investigative Biology 18 (2008): 1050–1054; Yoshiko Yabe and Ophthalmology and Visual Science (Supplement) 35 Gentaro Taga, “Treadmill locomotion captures visual (1994): 1275. perception of apparent motion,” Experimental Brain 77 Continued with G. Ishimura, “Visuomotor Research 191, no. 4 (2008): 487–494. factors for action capture,” Investigative 80 Erich von Holst and Horst Mittelstaedt, “The Ophthalmology and Visual Science (Supplement) 36 Principle of Reafference: Interactions Between the (1995): 357. Central Nervous System and the Peripheral Organs,” 78 Andreas Wohlschläger, “Visual motion in Perceptual Processing: Stimulus Equivalence and priming by invisible actions,” Vision Research 40 Pattern Recognition, ed. Peter C. Dodwell (New York: (2000): 925–930. Appleton-Century-Crofts, 1971), 41. 79 Bruno H. Repp and Günther Knoblich, “Action 81 Roger W. Sperry, “Neural Basis of the Can Affect Auditory Perception,” Psychological Spontaneous Optokinetic Response Produced by Science 18, no. 1 (2007): 6–7; Olivia Carter, Talia Visual Inversion,” Journal of Comparative and Konkle, Qi Wang, Vincent Hayward, and Christopher Physiological Psychology 43, no. 6 (1950): 482–489.

SCHERFFIG: THERE IS NO INTERFACE

is perceived, in a feedback loop re- representations seem almost comi- sembling the one of cybernetic con- cally impoverished alongside the trol systems.82 richness of the moment-by-moment Motor activity thus is directly in- engagement of an experienced body scribed into the perception of its re- with a culturally constituted world.”84 sults. The reactions we expect an He thus shifted his focus on how the activity to have is driving its percep- “actions of the hands”85 drive insight tion, based on their spatio-temporal and even constitute the physical relation and perceived similarity. symbols or representations we are working with: “To apprehend a material pattern as a representation of something is to engage in specific FACTORING THE culturally shaped perceptual pro- HUMAN BACK IN: cesses.”86 This view corresponded to the CYBERNETIC way enactive cognitive science understands how our actions are ulti- INTERACTIONS mately responsible for the perceived features of objects, such as their Years after Hutchins worked with shape. As Kevin O’Regan and Alva Norman on a cognitive account of di- Noë wrote in a seminal text on how rect manipulation, he diverged from an action-centric view of cognitive classical cognitive science. As if he science could look like: “The idea we could not longer ignore the “concrete wish to suggest here is that the visual thinking” conducted by the hands on quality of shape is precisely the set of the physical interface, he turned to all potential distortions that the “embodied” and “enactive” cognitive shape undergoes when it is moved science, trying to understand think- relative to us, or when we move rela- ing as a process involving bodies en- tive to it.”87 gaged in the culturally structured This also holds for the inter- world surrounding them.83 Analyzing face. Although very few research has the reasoning and actions of humans been devoted to studying how an in- performing nautical navigation, he terface is perceived while it is used, observed that “[t]he traditional »ac- there is a remarkable PhD thesis by tion-neutral« descriptions of mental Dag Svanæs titled “Understanding In- teractivity.”88 In explicit tradition of

–––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––– 82 For a thorough discussion see Lasse 84 Hutchins, “Enaction, Imagination, and Scherffig, “Moving into View: Enacting Virtual Insight”, 445. Reality,” Mediatropes 6, no. 1 (2016). 85 Ibid. 443. 83 For his own introduction to “embodiment” 86 Ibid. 429-430. and “enaction” see Edwin Hutchins, “Enaction, 87 Kevin O’Regan and Alva Noë, “A Imagination, and Insight,” in Enaction: Towards a New Sensorimotor Account of Vision and Visual Paradigm for Cognitive Science, ed. John Robert Consciousness,” Behavioral and Brain Sciences 24 Stewart, Olivier Gapenne, and Ezequiel A. Di Paolo (2001): 940. (Cambridge, MA: MIT Press, 2010), 428. 88 Dag Svanæs, “Understanding Interactivity: Steps to a Phenomenology of Human-Computer

INTERFACE CRITIQUE JOURNAL – VOL. I – 2018

Gestalt psychology and its qualitative the sequence of their interactions. methods, Svanæs, as part of this the- Users clicked, observed, clicked again sis, conducted experiments in which and at some point would formulate a subjects (or users) interacted with ab- conceptualization of what they were stract minimalist systems of black dealing with, by saying, for instance, and white squares called “Square “It is a switch.”91 Notably, the switch World.”89 was not there from the beginning. Users interacted with these There was no symbolic representa- worlds using a mouse, clicking on the tion of a switch to be seen and inter- squares and observing the subse- preted as such. Instead, it appeared to quent changes in the world on screen. be encapsulated into the action se- Governed by more or less complex quence: state-transition-diagrams, the squares in the world changed their color When the subjects said »It is a (from black to white or back, see fig- switch«, they did not come to ure 3). Svanæs recorded the user ac- this conclusion from a formal tions while correlating these with analysis of the State Transition their verbal descriptions of what, ac- Diagram of the example. Nor did cording to them, was happening. they conclude it from the visual Among his observations was appearance of the square, as the an interesting shift in his users’ per- squares all looked the same. ceived “locus of agency,” which The switch behavior slowly moved from describing actions in the emerged from the interaction Square World (“it gets colored”) to- as the square repeated its re- wards locating oneself as acting in it sponse to the subject’s ac- 92 (“I turn it on”).90 He understood these, tions. using Merleau-Ponty’s terminology, as a gradual extension of the users body space by which the interface became incorporated. This, according to him, is direct manipulation: An extension of the perceived locus of agency into an interface, which would explain why interaction can feel direct although it is mediated by physical interfaces (like the mouse) that are distant from their effects (apparent motion on screen). A “switch” in the Square World.93 For his subjects, with this in- corporation came an “understanding” of the Square Worlds that grew from –––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––– Interaction” (Dissertation, Norges Teknisk- 91 Ibid. 206. Naturvitenskapelige Universitet Trondheim, 2000). 92 Ibid. 89 Svanæs, “Understanding Interactivity,” 128. 93 Ibid. 147. 90 Ibid. 159.

SCHERFFIG: THERE IS NO INTERFACE

By physically engaging in the “syn- and the response of the object.”96 tax” of moving a mouse and pressing These factors supporting di-rectness its buttons, the subjects established of interaction turn out to be the same the “semantics” of the Square World factors supporting the influence of our by literally enacting it: co-creating actions on the perception of their re- the perceived objects in the world sults. Interaction thus seems to de- through their actions. As these ob- pend on how closely action and jects existed only through being used, perception are fused by an interface, Svanæs described them as having while this fusing is subject to their Gestalt properties, naming them “In- physical qualities and our acquired ex- teraction Gestalts.”94 In Svanæs's pectations. Wrapped in their recipro- words: “At the perceptual level closest cal dependence, they create the to the computer are the rapid mouse Gestalt of that very interface. movements and button clicks that It is the circular causality of the subjects did when they explored cybernetic feedback, inherent to inter- new examples. At the cognitive level active computing since the very be- above emerge the Interaction Ge- ginning, that encapsulates user and stalts that result from the interac- interface in a loop within which ob- tions.”95 jects emerge through the process of In light of this, direct manipu- acting with them. No matter how sup- lation, in all its instances from Whirl- posedly natural the latest interface wind’s light-gun to mouse and might be, in the very moment when keyboard, tangible user interfaces computers became feedback ma- and today’s ubiquitous touch-screens, chines they set the stage for creating can be seen as not the reduction of a naturalness and its user in the recipro- psychophysical distance of material cal interplay of action, computer reac- syntax and mental semantics. It can tion and perception. rather be understood as an interplay Any button we touch on our of syntax and semantics, perceptual phones and tablets is, just like the level and cognitive level that together switch in Svanæs’ experiments, a but- create the Gestalt of the interface. ton only because it is used as such. As interfaces exhibit different The interface in itself therefore only levels of interactivity (few would disa- exists subjectively and is quite literally gree that a touchscreen somehow co-created, or enacted, every time it is feels more interactive than a key- used. While interaction design con- board), they also exhibit different de- stantly creates new humans, it never grees of what really makes their has them or its interfaces fully under interactivity direct: “spatial and tem- control. It may hence be time to start poral offset,” the “the ratio between the rethinking human-computer interac- number of degrees of freedom” and the tion as something that is, and always “similarity between the physical ac- has been, fundamentally participa- tions of the users on the instrument tory.

–––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––– 94 Ibid. 218. 96 Beaudouin-Lafon, “Instrumental Interaction,” 95 Ibid. 206. 446–453.

INTERFACE CRITIQUE JOURNAL – VOL. I – 2018

Friedewald, Michael. Der Computer als Werk- REFERENCES zeug und Medium: Die geistigen und technischen Wurzeln des Personal Computers. Beaudouin-Lafon, Michel. “Instrumental Inter- Berlin/Diepholz: Verlag für Geschichte der action: An Interaction Model for Design- Naturwissenschaften und der Technik, ing Post-WlMP User Interfaces.” In: 1999. Proceedings of CHI, 446-453. ACM, 2000. Gardner, Howard. The Mind’s New Science. A Bennett, Stuart. A History of Control Engineer- History of the Cognitive Revolution. New ing 1930–1955. Hitchin: Peter Peregrinus York: Basic Books, 1985. Ltd., 1993. Hellige, Hans Dieter. “Krisen- und Innovati- Card, Stuart K., and Thomas P. Moran. “User onsphasen in der Mensch-Computer-In- Technology: From Pointing to Pondering.” teraktion.” In: Mensch-Computer- In: Proceedings of the ACM Conference Interface. Zur Geschichte und Zukunft der on The History of Personal Workstations, Computerbedienung, edited by Hans Die- 183-198. ACM, 1986. ter Hellige, 11-92. Bielefeld: Transcript, Card, Stuart K., Thomas P. Moran, and Allen 2008. Newell. The Psychology of Human-Com- Hoffmann, Christoph. “φ-Phänomen Film. Der puter Interaction. Hillsdale, NJ and Lon- Kinematograph als Ereignis experimentel- don: Lawrence Erlbaum Associates, ler Psychologie um 1900.” In: Die Adresse 1983. des Mediums, edited by Stefan Andriop- Carter, Olivia, Talia Konkle, Qi Wang, Vincent oulos, Gabriele Schabacher, and Eckhard Hayward, and Christopher Moore. “Tactile Schumacher, 236-252. Cologne: DuMont, Rivalry Demonstrated with an Ambiguous 2001. Apparent-Motion Quartet.” In: Current Bi- Hutchins, Edwin L., James D. Hollan, and ology 18 (2008): 1050–1054. Donald A. Norman. “Direct Manipulation Chapman, Robert L., John L. Kennedy, Allen Interfaces.” In: Human-Computer Interac- Newell, and William C. Biel. “The Systems tion 1 (1985): 311–338. Research Laboratory’s Air Defense Exper- Hutchins, Edwin. “Enaction, Imagination, and iments.” In: Management Science 5, no. 3 Insight.” In: Enaction: Towards a New Par- (1959): 250-269. adigm for Cognitive Science, edited by Dourish, Paul. Where the action is. The foun- John Robert Stewart, Olivier Gapenne, dations of embodied interaction. Cam- and Ezequiel A. Di Paolo, 425-450. Cam- bridge, MA: MIT Press, 2001. bridge, MA: MIT Press, 2010. Dunne, Anthony. Hertzian tales. Electronic Ishimura, G. “Visuomotor factors for action products, aesthetic experience, and criti- capture.” In: Investigative Ophthalmology cal design. Cambridge, MA: MIT Press, and Visual Science (Supplement) 36 2008. (1995): S357. Eagleman, David M. “Visual illusions and neu- Ishimura, G., and S. Shimojo. “Voluntary ac- robiology.” In: Nature Reviews Neurosci- tion captures visual motion.” In: Investiga- ence 2 (2001): 920-926. tive Ophthalmology and Visual Science Everett, Robert. “Whirlwind.” In: A History of (Supplement) 35 (1994): 1275. Computing in the Twentieth Century, ed- Laurel, Brenda K. “Interface as mimesis.” In: ited by J. Howlett, Gian Carlo Rota, and User-Centered System Design: New Per- Nicholas Metropolis, 365-384. Orlando: spectives on Human-Computer Interac- Academic Press, 1980. tion, edited by Donald A. Norman and Fishkin, Kenneth P., Anuj Gujar, Beverly L. Stephen Draper, 67-85. Hillsdale, NJ: Harrison, Thomas P. Moran, and Roy Lawrence Erlbaum Associates, 1986. Want. “Embodied user interfaces for re- Miller, George A. “The cognitive revolution: a ally direct manipulation.” In: Communica- historical perspective.” In: Trends in Cog- tions of the ACM 43, no. 9 (2000): 74–80. nitive Sciences 7, no. 3 (2003): 141-144.

SCHERFFIG: THERE IS NO INTERFACE

MIT. Whirlwind I: A high-speed Electronic Digi- Schumann, Frank. “Embodied Cognitive Sci- tal Computer. Promotional brochure. ence: Is it Part of Cognitive Science? Cambridge, MA: MIT, 195. Analysis within a Philosophy of Science Newell, Allen, and Herbert A. Simon. “Com- Background.” In: PICS. Publications of the puter science as empirical inquiry: sym- Institute of Cognitive Science 3 (2004). bols and search.” In: Communications of Shneiderman, Ben. “Direct Manipulation: A the ACM 19, no. 3 (1976): 113-126. Step Beyond Programming Languages.” Noble, Douglas D. “Mental Materiel. The mili- In: The New Media Reader, edited by tarization of learning and intelligence in Noah Wardrip-Fruin and Nick Montfort, US education.” In: Cyborg Worlds. The Mil- 486-498. New York, NY and London: W. itary Information Society, edited by Les W. Norton & Company, 2001. Levidow and Kevin Robins, 13-41. Lon- Simon, Herbert A. “Allen Newell. 1927-1992.” don: Free Association Books, 1989. In: National Academy of Sciences Bio- Norman, Donald A. “Cognitive Engineering.” graphical Memoirs (1997): 139-173. In: User-Centered System Design: New Sperry, Roger W. “Neural Basis of the Sponta- Perspectives on Human-Computer Inter- neous Optokinetic Response Produced by action, edited by Donald A. Norman and Visual Inversion.” In: Journal of Compara- Stephen Draper, 31-61. Hillsdale, NJ: tive and Physiological Psychology 43, no. Lawrence Erlbaum Associates, 1986. 6 (1950): 482–489. O’Regan, Kevin, and Alva Noë. “A Sensorimo- Steinman, Robert M., Zygmunt Pizlo, and Filip tor Account of Vision and Visual Con- J. Pizlo. “Phi is not beta, and why sciousness.” In: Behavioral and Brain Wertheimer’s discovery launched the Ge- Sciences 24 (2001): 939-1031. stalt revolution.” In: Vision Research 40 Pias, Claus. Computer Spiel Welten. Disserta- (2000): 2257–2264. tion, Bauhaus-Universität Weimar, 2000. Svanæs, Dag. Understanding Interactivity: Pold, Søren Bro. “Interface Perception: The Steps to a Phenomenology of Human- Cybernetic Mentality and Its Critics: Uber- Computer Interaction. Dissertation, Nor- morgen.com.” In: Interface Criticism: Aes- ges Teknisk-Naturvitenskapelige Universi- thetics Beyond Buttons, edited by tet Trondheim, 2000. Christian Ulrik Andersen and Søren Bro Ullmer, Brygg, Hiroshi Ishii, and Dylan Glas. Pold, 91-113. Aarhus: Aarhus University “mediaBlocks: Physical Containers, Press, 2011. Transports, and Controls for Online Me- Redmond, Kent C., and Thomas M. Smith. dia.” In: Proceedings of SIGGRAPH, 379- Project Whirlwind: the history of a pioneer 386. ACM, 1998. computer. Bedford, MA: Digital Press, von Foerster, Heinz. “Cybernetics of Episte- 1980. mology.” In: Understanding Understand- Repp, Bruno H., and Günther Knoblich. “Ac- ing: Essays on Cybernetics and Cognition. tion Can Affect Auditory Perception.” In: Heinz von Foerster, 229–246. New York: Psychological Science 18, no. 1 (2007): Springer, 2003. 6–7. von Holst, Erich and Horst Mittelstaedt. “The Rosenblueth, Arturo, Norbert Wiener, and Jul- Principle of Reafference: Interactions Be- ian Bigelow. “Behavior, Purpose and Tele- tween the Central Nervous System and ology.” In: Philosophy of Science 10 the Peripheral Organs.” In: Perceptual Pro- (1943): 18–24. cessing: Stimulus Equivalence and Pattern Scherffig, Lasse. Feedbackmaschinen. Kyber- Recognition, edited by Peter C. Dodwell, netik und Interaktion. Dissertation, 464-476. New York: Appleton-Century- Kunsthochschule für Medien Köln, 2017. Crofts, 1971. Scherffig, Lasse. “Moving into View: Enacting von Schiller, Paul. “Stroboskopische Alterna- Virtual Reality.” In: Mediatropes 6, no. 1 tivversuche.” In: Psychologische For- (2016): 1-29. schung 17 (1933): 179-214.

INTERFACE CRITIQUE JOURNAL – VOL. I – 2018

Walker, John. “Through the looking glass.” In: The Art of Human-Computer Interface De- sign, edited by Brenda Laurel, 439-448. Redding, MA: Addison-Wesley, 1990. Wegner, Peter. “Why interaction is more powerful than algorithms.” In: Communica- tions of the ACM 40, no. 5 (1997): 80-91. Wertheimer, Max. “Experimentelle Studien über das Sehen von Bewegung.” In: Zeit- schrift für Psychologie und Physiologie der Sinnesorgane 61 (1912): 161-265. Wiener, Norbert. Cybernetics: or Control and Communication in the Animal and the Machine. Cambridge, MA: MIT Press, 1961. Wieser, C. R. Cape Cod System and Demon- stration, Technical report. Cambridge, MA: Lincoln Laboratory – Division 6, 1953. Winograd, Terry. “From Computing Machin- ery to Interaction Design.” In: Beyond Cal- culation: The Next Fifty Years of Computing, edited by Peter Denning and Robert Metcalfe, 149-162. Berlin and New York: Springer, 1997. Wohlschläger, Andreas. “Visual motion priming by invisible actions.” In: Vision Re- search 40 (2000): 925–930. Yabe, Yoshiko and Gentaro Taga. “Treadmill locomotion captures visual perception of apparent motion.” In: Experimental Brain Research 191, no. 4 (2008): 487–494.