sign in sign up
<<
Home , Donald Michie, Jack Copeland, Max Newman, Turochamp

What Turing Did after He Invented the Universal Author(s): B. and Diane Proudfoot Source: Journal of , Language, and Information, Vol. 9, No. 4, Special Issue on and (Oct., 2000), pp. 491-509 Published by: Springer Stable URL: http://www.jstor.org/stable/40180239 Accessed: 04/11/2009 17:35

Your use of the JSTOR archive indicates your acceptance of JSTOR's Terms and Conditions of Use, available at http://www.jstor.org/page/info/about/policies/terms.jsp. JSTOR's Terms and Conditions of Use provides, in part, that unless you have obtained prior permission, you may not download an entire issue of a journal or multiple copies of articles, and you may use content in the JSTOR archive only for your personal, non-commercial use.

Please contact the publisher regarding any further use of this work. Publisher contact information may be obtained at http://www.jstor.org/action/showPublisher?publisherCode=springer.

Each copy of any part of a JSTOR transmission must contain the same copyright notice that appears on the screen or printed page of such transmission.

JSTOR is a not-for-profit service that helps scholars, researchers, and students discover, use, and build upon a wide range of content in a trusted digital archive. We use information technology and tools to increase productivity and facilitate new forms of scholarship. For more information about JSTOR, please contact [email protected].

Springer is collaborating with JSTOR to digitize, preserve and extend access to Journal of Logic, Language, and Information.

http://www.jstor.org ^£ Journalof Logic,Language, and Information9: 491-509,2000. ^gi ^rr © 2000 KluwerAcademic Publishers. Printed in the Netherlands.

What TuringDid after He Inventedthe Universal TuringMachine

B. JACKCOPELAND and DIANE PROUDFOOT The TuringProject, ,Private Bag 4800, , E-mail: bjcopeland@canterbury. ac. nz, [email protected]. ac. nz; http://www.AlanTuring, net

(Received 1 June 1999; in final form 15 April 2000)

Abstract Alan Turing anticipated many areas of currentresearch in and cognitive science. This article outlines his contributions to Artificial Intelligence, connectionism, hypercomputation, and Artificial Life, and also describes Turing's pioneering role in the development of electronic stored-programdigital . It locates the origins of Artificial Intelligence in postwar Britain. It examines the intellectual connections between the work of Turing and of Wittgenstein in respect of their views on cognition, on machine intelligence, and on the relation between provability and truth. We criticise widespread and influential misunderstandings of the Church-Turing thesis and of the halting theorem. We also explore the idea of hypercomputation, outlining a number of notional machines that "compute the uncomputable."

Key words: Artificial Intelligence, Artificial Life, Automatic Computing Engine (ACE), Church- Turing thesis, Colossus, connectionism, Halting theorem, history of computing, hypercomputation, Turing, Wittgenstein

1. The Race to Build the First Computer It is often said that, apart from specifying the in 1935, Turingplayed little or no role in the development of computers. The reality is very different. In 1945 Turing produced a detailed design for his proposed Automatic Computing Engine or ACE (Turing, 1945; Copeland, 1999a; Copeland, 2000b). This design was the first relatively complete specification of an electronic stored- program general-purpose digital computer. The slightly earlier "First Draft of a Report on the EDVAC,"produced by the Moore School group at the University of Pennsylvania (Von Neumann, 1945), contained little engineering detail, in partic- ular concerning electronic hardware.Turing, on the other hand, supplied detailed circuitdesigns and specifications of hardwareunits, specimen programsin machine code, and even an estimate of the cost of building the machine (£1 1,200). Turing saw that speed and memory were the keys to computing. His design had much in common with today's RISC architecturesand called for a high-speed memory of roughly the same capacity as an early Macintosh computer (enormous by the standardsof his day). Had Turing'sACE been built as planned it would have 492 B.J.COPELAND AND D. PROUDFOOT been in a different league from the other early computers, but his colleagues at the National Physical Laboratory(NPL) thought the engineering work too difficult to attempt, and a considerably smaller machine was built. Known as the Pilot Model ACE, this machine ran its first program on May 10, 1950. With a clock speed of 1 MHz it was for some time the fastest computer in the world. Sales of DEUCE, the production version of the Pilot Model ACE, exceeded 30 (confounding a prediction by a top adviser to the NPL that Britain's computing needs would be satisfied by a total of 3 digital computers). The fundamentals of Turing's ACE design were employed by (at Wayne University, Detroit) in the Bendix G15 computer. The G15 was arguably the first personal computer and over 400 were sold worldwide. DEUCE and the G15 remained in use until about 1970. Another computer derived from Turing's ACE design, the MOSAIC, played a role in Britain's air defences during the Cold Warperiod; other derivatives include the Packard-BellPB250 (1961). Unfortunately,delays beyond Turing's control resulted in the NPL's losing the race to build the world's first electronic stored-programdigital computer- an hon- our that went to the , where, in the Royal Society Com- puting Machine Laboratory,the "" ran its first program on 21 June 1948. As its name implies, the Baby was a very small computer,and the news that it had run what was only a tiny program - just 17 instructions long - for a mathematically trivial task was "greeted with hilarity" by the team building the sophisticated Pilot Model ACE.* Turingdesigned the programmingsystem for the Mark I, the production version of the Baby's successor.**Completed in 1951, the Ferranti Mark I was the first commercially available electronic digital computer;about 10 were sold (in Britain, Canada, Holland and Italy). Both the ACE and the Manchester computer came out of research that nobody would have guessed might have a practical application. In a lecture in Cambridge in 1935 the mathematicianMax Newman - whose own role in the development of computers has been insufficiently emphasised - introduced Turing to the concept that led directly to the Turing machine: Newman defined a constructive process as one that a machine can carry out (Copeland, 1999b).* Turingtook Newman's words literally and the result was a typescript, which Turing showed to Newman in April 1936, setting out the concept of the universalTuring machine.** According to New- man, Turingwas interestedright from the startin the possibility of actually making

* Michael Woodger (Turing's assistant at the NPL from May 1946) in personal communication with Copeland (1998). ** ' A digital facsimile of Turing's typewritten Programmers Handbookfor Manchester Electronic Computer is in the Turing Archive for the History of Computing and may be viewed on-screen at http://www.AlanTuring.net * Newman in interview with Christopher Evans in 1976 ("The Pioneers of Computing: an Oral History of Computing," Science Museum: London). ** The Turing machine concept was announced by Turing in an address to the London Math- ematical Society on 12 November 1936, "On Computable Numbers, with an Application to the " (Turing, 1936). TURINGAFTER THE UTM 493 a machine of the sort described in the paper.*In 1937-1938 Turing designed and wired up a simple binary multiplier,but it was not until the development during the war of high-speed electronic switching that the dream of building a miraculously fast general-purposecomputing machine really took hold. Following decades of secrecy, Turing's wartime work as leading codebreaker at the GovernmentCode and Cypher School, , is now well known. F.H. Hinsley, official historian of British Intelligence in the Second World War, has estimated that the Bletchley Park codebreakers shortened the war in Europe by as much as two years. Turing's main work concerned the 'Enigma' code (his "Treatiseon the Enigma" has recently been declassified). Newman, in a different section, attackedthe 'Fish' codes. Based on binary teleprintercode, Fish was used in preference to Morse-based Enigma for the encryption of high-level signals, for example messages from Hitler and other members of the German High Command. Newman realised that the attack on Fish could be mechanised (Turing had already mechanised the attackon Enigma, with enormous success) and at Turing's sugges- tion sought the help of electronic engineer Tom Flowers.**The history of electronic data-processingbegins with Flowers' pre-warwork at the Post Office Research Sta- tion at in London: during the period 1934-1939 Flowers experimented with high-speed electronic data-storageand designed electronic digital telephone equipmentthat used 3,000-4,000 vacuum tubes runningcontinuously. Flowers has remarkedthat at the outbreakof war with Germanyhe was possibly the only person in Britain who realized that vacuum tubes could be reliably used on a large scale for high-speed digital computation. During 1943, in consultation with Newman, Flowers designed and built Colossus, the first large-scale fully-functioning elec- tronic digital computer.Colossus firstworked on 8 December 1943, some two years before the first comparable U.S. machine, the ENIAC. (ENIAC was constructed for the Army Ordnance Department by J. Presper Eckert and John Mauchly at the Moore School of Electrical Engineering, University of Pennsylvania.) Both Colossus and ENIAC were special-purpose computers (ENIAC's function was to calculate tables used by gunners when aiming artillery). Neither had internally stored programs- setting up these machines for a new task involved reconfigur- ing them by means of plugs and switches. During the construction of Colossus, Newman had tried to interest Flowers in Turing's 1936 paper - the origin of the stored-programconcept - but Rowers "didn'treally understandmuch of it." There can be little doubt that by 1943 Newman had firmly in mind the possibility of build- ing a universal Turing machine using electronic technology. Colossus, involving approximately2400 thermionic valves, established decisively and for the first time that large-scale electronic computing machinery was practicable. In 1945, the war in Europe over, Turing and Newman followed different paths, Turingto the NPL to build the ACE and Newman to the Fielden Chair of Mathem-

* Newman, op. cit. ** All material herein relating to T.H. Flowers derives from personal communications from Flowers to Copeland (1996-1998). 494 B.J.COPELAND AND D. PROUDFOOT atics at the University of Manchester.Shortly after his arrivalat Manchester,New- man wrote to von Neumann, a prominent figure in the development of computing in the U.S. (and strongly influenced by Turing): I am ... hoping to embark on a computing machine section here, having got very interested in electronic devices of this kind during the last two or three years. By about eighteen months ago I had decided to try my hand at starting up a machine unit when I got out. ... I am of course in close touch with Turing.* Newman applied to the Royal Society for a grant to establish his Computing Ma- chine Laboratory,and the rest is history. Not that history has been particularly kind either to Newman or to Turing. Their logico-mathematical contributions to the triumph at Manchester have been neglected, and the Manchester machine is nowadays remembered as the work of electronic engineers Williams and Kilburn. Fortunatelythe words of the late Freddie Williams are preserved (Williams, 1975: 328): and I knew nothing about computers, but a lot about circuits. Professor Newman and Mr. A.M. Turing . . . knew a lot about computers and substantially nothing about electronics. They took us by the hand and explained how numbers could live in houses with addresses and how if they did they could be kept track of during a calculation.** Turing's own phrase "the mechanic who . . . constructed the machine" (Turing, 1951a) perhaps reveals his attitudeto Kilburnand Williams.

2. Artificial Intelligence Turing was the first to carry out substantial research in the field now known as Artificial Intelligence. During the wartimeyears, Turinggave considerable thought to the issue of machine intelligence. Colleagues at Bletchley Park recall numerous off-duty discussions with him on the topic, and at one point Turing circulated a typewrittenreport (now lost) setting out some of his ideas. One of these colleagues, (who in the 1960s founded the Departmentof Machine Intelligence and Perception at the ),remembers Turing often talking about the possibility of computing machines (1) learning from experience and (2) solving problems by means of searchingthrough the space of possible solutions, guided by (what are now called) heuristic principles.* * Letter from Newman to von Neumann, 8 February 1946. ** Concerning Turing's knowledge of electronics, opinion among the engineers at Manchester seems to have been divided. G.C. Tootill - who bore much of the responsibility for liaising between the Computing Machine Laboratory and the engineers at FerrantiLtd who built the FerrantiMark I in collaboration with the University - spoke approvingly of Turing's "ability as a circuit designer" (letter from Tootill to Simon Lavington, 1 July 1975). (Tootill's "Informal Report on the Design of the Fer- ranti Mark I Computing Machine" (Royal Society Computing Machine Laboratory,November 1949) contains an interesting technical appendix by Turing entitled "Generationof Random Numbers") * Donald Michie in personal communication with Copeland (1995). TURINGAFTER THE UTM 495

At Bletchley Park Turing illustrated his ideas on machine intelligence by ref- erence to chess. He experimented with two heuristics that later became common in AI: minimax (assume that your opponent will move in a way that maximises their gain and make your move in such a way as to minimise the loss that your op- ponent's expected move will cause you) and best first (rankthe moves available to you according to a rule-of-thumbscoring system and examine their consequences, beginning with the highest-scoring move).* In London in 1947 Turinggave what was, so far as is known, the earliest public lecture to mention computer intelligence, saying "Whatwe want is a machine that can learn from experience" and "[t]he possibility of letting the machine alter its own instructions provides the mechanism for this" (Turing, 1947: 123). In 1948 he wrote (but did not publish) a report entitled "Intelligent Machinery."This was the first manifesto of Artificial Intelligence and in it Turing brilliantly introduced many of the concepts that were later to become central to the field, in some cases after reinventionby others. These included the concept of a genetic and of training an unorganised network of artificial neurons to perform specific tasks (see Section 3 below), the idea that "intellectualactivity consists mainly of various kinds of search"(Turing, 1948: 23), and the theorem-provingapproach to problem- solving. His 1950 paper, introducing the , is of course well known. In 1951 Turinggave a lecture on machine intelligence on British radio and in 1953 he published a classic early article on chess programming(Turing, 1951b, 1953). Both during and after the war Turing experimented with machine routines for playing chess: in the absence of a computer,the machine's behaviourwas simulated by hand, using paper and pencil. The first true AI programhad to await the arrival of a working general-purposeelectronic digital computer. The earliest AI programs**ran at Manchester University in the Royal Soci- ety Computing Machine Laboratory,of which Turing was appointed Deputy Dir- ector in 1948 (there was never a Director).* The first was written by , at the time a schoolmaster at Harrow and an amateurprogrammer (and later Director of the ProgrammingResearch Groupat Oxford University). Strachey chose the board game of draughts (or checkers) as the domain for his experiment in machine intelligence. He wrote a draughts program for the Pilot Model ACE as early as February 1951, but was dissatisfied with the method employed in the programfor evaluatingboard positions, and wrote an improved version for the Fer- ranti Mark I at Manchester(with Turing's encouragementand utilising the latter's recently completed Programmers9Handbook for the machine, (Turing, 1950b)).** * Ibid. ** See further Copeland (1993, 2000c). * In May of 1948 the vigour of the Manchester project and Newman's offer of a job had lured a "very fed up" Turing away from his position at the NPL, where work on the ACE had drawn almost to a standstill (the quoted words are those of Turing's close friend Robin Gandy, personal communication with Copeland, 1995). ** It seems that the version for the Pilot Model ACE never ran successfully (personal commu- nication from Michael Woodger to Copeland, 1999). An attempt in July 1951 foundered due to 496 B.J. COPELANDAND D. PROUDFOOT

By the summer of 1952 the Manchester version could play a complete game of draughts at a reasonable speed. (The first AI program to run in the U.S. was also a draughts program, written in 1952 by Arthur Samuel of IBM for the IBM 701, IBM's first mass-produced electronic stored-programcomputer. Samuel took over the essentials of Strachey's program,which Stracheyhad described at a computing conference in Canada in 1952, and over a period of years considerably extended it.) Prinz, who worked for the engineering firm of Ferranti Ltd., wrote the first chess program to be implemented. It ran in November 1951 on the FerrantiMark I. Prinz's program was for solving simple problems of the mate-in-two variety. The program would examine every possible move until a solution was found. On average several thousand moves had to be examined in the course of solving a problem, and the program was considerably slower than a human player. Turing had started to program his own Turochampchess-playing routine for the Ferranti Mark I but never completed the task. Unlike Prinz's program,the Turochampcould play a complete game and operatednot by exhaustive searchbut underthe guidance of heuristics. Prinz also used the FerrantiMark I to solve logical problems, and in 1949 and 1951 Ferrantibuilt two small experimentalspecial-purpose computers for theorem-provingand other logical work. This was several years before the program by Newell, Simon and Shaw known as the "Logic Theorist"- often incorrectlyde- scribed as being the first AI program- made its debut at the Dartmouthconference in 1956. Oettinger's Shopper was the earliest programto incorporaterote-learning (de- tails of the program were published in 1952).* The program ran on Wilkes's EDSAC computer at the University of Cambridge. Oettinger was considerably influenced by Turing's views on machine learning, and suggested that a program of the Shopper type could pass a highly restrictedversion of the TuringTest. Shop- per's simulated world was a mall of eight shops. When sent out to purchase an item, Shopper would if necessary search for it, visiting shops at random until the item was found. While searching, Shopper would memorise a few of the items stocked in each shop visited (as a human shopper might). Next time Shopper was sent out for the same item, or for some other item that it had already located, it would go immediately to the correct shop. (Strachey had also investigated aspects of machine learning, taking the game of NIM as his focus, and in 1951 he reported a simple rote-learning scheme in a letter to Turing.)

3. Connectionism Connectionists look back on Hebb and Rosenblatt as the originators of their ap- proach, but in fact both were preceded by Turing,who anticipatedmuch of modern programming errors. A major hardware change later in 1951 meant that the program would never work without extensive recoding. * "Shopper" is our name for what Oettinger terms "the shopping programme." TURINGAFTER THE UTM 497 connectionism in his 1948 paper "IntelligentMachinery" (see Proudfoot and Cope- land, 1994; Copeland and Proudfoot, 1996, 1999a). Here Turingintroduces what he calls "unorganisedmachines," giving as examples networks of neuron-likeboolean elements connected together in a largely random manner (we call these "Turing Nets", Copeland and Proudfoot, 1999b). He describes a certain form of Turing Net as "the simplest model of a nervous system" and he hypothesises that "the cortex of the infant is an unorganisedmachine, which can be organised by suitable interferingtraining" (Turing, 1948: 10, 16). The idea that an initially unorganised neural network can be organised by means of "interferingtraining" is undoubtedly the most significant aspect of Turing'sdiscussion (and is not prefiguredin the clas- sic McCulloch-Pitts paper, 1943). In Turing's model, the training process renders certainneural pathways effective and others ineffective. He anticipatedthe modern procedure of simulating neural networks and the training process by means of an ordinarydigital computer,saying (Turing, 1948: 21) quite definite teaching policies' . . . could also be programmedinto the [com- puter]. One would then allow the whole system to run for an appreciable period, and then break in as a kind of 'inspector of schools' and see what progress had been made. Turingclaimed a proof (now lost) of the propositionthat an initially unorganised TuringNet with sufficient neurons can be organised to become a universal Turing machine with a given storage capacity (Turing, 1948: 15). This proof first opened up the possibility, noted by Turing (1948: 16), that the human cognitive system is a universal symbol-processor implemented in a neural network.

4. Hypercomputation Turing'saim in his 1936 paper was to specify a machine, as simple as possible, that can performany calculation that can be performedby a humanmathematician with unlimited time, working with paper and pencil in accordance with some "rule-of- thumb"or rote method. (It is precisely because the universal Turing machine can carry out all such calculations that it is said to be "universal.")Modern digital computers, which are universal up to some resource limit, likewise possess the property of being abstract models of human beings engaged in rote calculation. As Turing wrote in his programmingmanual for the FerrantiMark I, "Electronic computersare intendedto carryout any definite rule-of-thumbprocess which could have been done by a human operator working in a disciplined but unintelligent manner" (Turing, 1950b: 1). An intriguing question arises: if we set aside the traditionalidea that digital information-processingmachines are to be modelled on humanbeings working in a rule-of-thumbmanner, can we describe machinery that is capable of accomplishing a wider variety of tasks than the universal Turing ma- chine? The answer to this question is that such machines can be specified on paper, but at the present time nobody knows whether it will be possible to build one. We call such machines hypercomputers(Copeland and Proudfoot, 1999a). A small but 498 B.J. COPELANDAND D. PROUDFOOT growing internationalgroup of researchersis exploring the possibility of achieving hypercomputation(Copeland and Sylvan, 1999, surveys this emerging field). It is a matter for speculation whether the human brain itself is a naturally-occurring instantiationof a hypercomputer. Turing was the first to investigate the idea of machines that are able to perform mathematicaltasks too difficult for the universal Turing machine (this was in his Ph.D. thesis (Turing, 1938), supervised by Church, published as Turing, 1939). Turing described these as "a new kind of machine," calling them "O-machines" (Turing, 1939: 173). The primitive operations of an (ordinary) Turing machine are six in number: (i) move the tape left one square; (ii) move the tape right one square; (iii) read (i.e., identify) the symbol currently under the head; (iv) write a symbol on the square of tape currently under the head (after first deleting the symbol already written there, if any); (v) change state; (vi) halt. These primitive operations are made available by unspecified subdevices of the machine - "black boxes." (The question of what mechanisms might appropriatelyoccupy these black boxes is not relevant to the machine's logical specification.) An O-machine is a Turing machine augmented with a primitive operation that returns the values of some function (on the natural numbers) that is not Turing-machine-computable. This additionalprimitive operation is made available by a black box. Turingrefers to this box as an "oracle" (Turing, 1939: 172). In other respects O-machines are similar to ordinary computing machinery. A digitally encoded program is fed in and the machine produces digital output from digital input by means of a step-by- step procedure. This step-by-step procedure consists of repeated applications of the machine's primitive operations, one of which is to pass data to the oracle and register the oracle's response (see furtherCopeland, 1997, 1998a, 2000a). According to Turing's specification, each oracle returns the values of a two- valued function. Let these values be written 0 and 1. Let p be the additional primitive operation, p is called into action by means of a special state x» the call state. The machine inscribes the symbols that are to form the input to p on any convenient block of squares of its tape, using occurrencesof a special symbol /z, the markersymbol, to indicate the beginning and the end of the input string. As soon as an instructionin the machine's programputs the machine into state x, the input is delivered to the subdevice that effects /?, which then returnsthe corresponding value of the function. On Turing's way of handling mattersthe value is not written on the tape. Rathera pair of states, the 1-state and the 0-state, is employed to record values of the function. A call to p ends with a subdevice placing the machine in one or other of these two states, as the value of the function is 1 or 0. (When a function g is computable by an O-machine whose oracle serves to returnthe values of a function /, then g is sometimes said to be computable relative to /.) One particularO-machine, the halting function machine, has as its "classical" part the universal Turing machine specified by Turing in 1936 and as its "non- classical" part a primitive operation that returns the values of Turing's famous halting function H(x, v). The halting function is defined thus: H(x, v) = 1 if TURINGAFTER THE UTM 499 and only if the xth Turingmachine eventually halts if set in motion with the integer y inscribed on its tape, say in binary code (think of y as being the machine's input); and H(x,y) = 0 otherwise. The O-machine concept has been neglected in the of mind and cog- nitive science. Sadly, Turing'spioneering theoretical work has been forgotten even by those pursuingthe goal of building hypermachines.Researchers talk of carrying out information-processing"beyond the Turing limit" and describe themselves as attemptingto escape from "the Turingtar-pit" and to "breakthe Turingbarrier." Turing said nothing about what might be "inside" the logically specified black boxes, saying only that an oracle works by "unspecified means" and that "we shall not go any furtherinto the nature of [an] oracle" (Turing, 1939: 172-173). However, notional machinery that fulfils the specifications of an oracle is not hard to concoct.

5. Notional Oracles and the Import of the Halting Theorem Many textbooks on the fundamentals of computer science offer examples of information-processingtasks that are, it is claimed, absolutely uncomputable, in the sense that no machine can be specified to carry out these tasks. For example, it is said that no machine can respond in accordance with the following rules to inputs of arbitrarilyselected finite strings of binary digits: (1) Answer "1" if the input stringis a programthat will cause a universalTuring machine to execute only a finite number of actions (such programs are called "terminating").(2) Answer "0" if the string is not a terminatingprogram - i.e., if the string is either a Turing machine program that does not terminate or is not a well-formed Turing machine programat all. (An example of a terminatingprogram is one for finding the highest factor of a given integer, terminating on producing the answer. An example of a nonterminatingprogram is one for calculating the digits of n .) It is false that no conceivable machine can carry out the task just described (which we refer to as the terminatingprogram or TP task). An AUTM, or Acceler- ating Universal Turing Machine, can carry out this task (Copeland, 1998b, 1998c; Stewart, 1991). An AUTM executes the program on its tape at an accelerating rate, performing each primitive operation that the program calls for in half the time that was taken for the immediately preceding primitive operation. So if the machine takes one unit of time to perform the first primitive operation, the second is performedin half a unit, the third in one quarterof a unit, and so on. Since 111 1 is less than 2, the AUTM requires less than two units of running time to do everything that the program on its tape instructs it to do. This is true even in the case of a program that does not terminate, for example a program that runs on forever calculating each successive digit of n: each of the infinite number of 500 B.J.COPELAND AND D. PROUDFOOT operations that the nonterminatingprogram instructs the machine to perform will be completed before the end of the second unit of running time.* (Copeland and Hamkins (forthcoming) discuss the issue of the physical plausibility of AUTMs with respect to Newtonian physics, relativistic physics, and quantumtheory.) Consider an AUTM that is set up to perform the TP task. The AUTM has a one-way infinite tape on which is inscribed the string that is to be tested. The initial square of the tape is used to display the result of the computation;at the startof the computationthis square contains 0. The AUTM first determines whether or not the string being tested is a well-formed Turing-machineprogram. If the string is not a well-formed programthen the AUTM halts. If the string is a well-formed program then the AUTM simulates it. If the string is a terminatingprogram then, once the simulation is complete, the head of the AUTM moves to the initial square of the tape and replaces the 0 that was written there during the setting-up procedure by 1. The AUTM then halts. If the string is a nonterminatingprogram, the head of the AUTM never returns to the start of the tape. Either way, at the end of the second unit of operating time the initial square contains the digit required by the above rules. It is essential to distinguish between two senses in which a function may be said to be computable by a given machine, which we refer to respectively as the internal sense and the external sense (see Copeland, 1998b). A function is computable by a machine in the internal sense just in case the machine can produce values from arguments (for all arguments in the domain), "halting"once any value has been produced, and where what counts as "halting"can be specified in terms of features internal to the machine and without reference to the behaviour of some device or system - e.g., a clock - that is external to the machine. (This condition on the nature of halting behaviour will be referred to as the "internalist"condition.) Numerous behaviours on the part of a machine can satisfy this condition, for example com- plete cessation of activity, or playing the British National Anthem,**or writing any sequence of digits in a certain location. A function is computable by a machine in the external sense just in case the machine can produce values from arguments(for all argumentsin the domain), displaying each value at a designated location some

* This temporal patterning of operations seems to have been first described by Russell, in a lecture given in Boston in 1914. In a discussion of Zeno's paradox, Russell said "If half the course takes half a minute, and the next quarter takes a quarter of a minute, and so on, the whole course will take a minute" (Russell, 1915: 172-173). Later he said that although it may be "medically impossible" for a person to run through "an infinite number of operations" by this means, it is nevertheless not "logically impossible" to do so (Russell, 1936: 143-144). Boolos and Jeffrey (1980: 14) envisage Zeus being able to act so as to exhibit the Russell temporal patterning.By an extension of terminology (which Boolos and Jeffrey do not make) a Zeus machine is any machine exhibiting this temporal patterning. An AUTM is a Zeus machine, but not every Zeus machine is an AUTM. ** The first program of any significant size to run on the Ferranti Mark I - written by Strachey at Turing's behest - brought its activity to a close by playing the National Anthem on the machine's "hooter." TURINGAFTER THE UTM 501 prespecified number of time units after the corresponding argumentis presented. The machine may or may not "halt"once a value has been displayed. For example, it is in the external sense that a given function may be computable by a logic circuit. The value of the function is displayed at some designated node n time units after the argumentis presented at the input nodes (mutatis mutandis for functions of more than one argumentand functions whose values require more than one binary node for their expression). Before and after that critical moment, the activity of the output node may afford no clue as to the desired value. Even where the logic circuit never stabilises (in the sense of eventually producing an output signal that remains constant until such time as the input signal alters) the circuit neverthelesscomputes values of a function in the external sense if it displays them at the designated location at the prespecifiedtimes. The same is true of neural networks.A particularnetwork may compute the values of a certain function in the external sense even though the network never stabilises (a network stabilises, or "halts,"if and only if after some point there is no further change in the activity level of any of its units). In some cases, a machine may compute the values of a function in both senses, there being an n such that whenever an argumentis presented, the machine 'halts,' displaying the correspondingvalue, within n time units. A machine that computes a function in the external sense can readily be converted into one that computes the function in both senses by the addition of a bell triggeredby an internal clock. The bell rings when the value (or, as appropriate,the last digit of the value) is produced, and the machine's ringing the bell constitutes its "halting."Of course, adding a clock to a machine may result in a machine not of the same type. A Turingmachine plus a clock is not a Turingmachine. Turing's Halting Theorem of 1936 - in essence the proposition that the TP task is uncomputable- speaks only of computabilityby Turingmachine in the internal sense, not of computability in the external sense: a Turing machine cannot carry out the TP task if it is required to produce the answer and then halt. But as the earlier example of the AUTM shows, a Turing machine can carry out the TP task if the internalist condition is lifted. While it may grate on one's ear to say so, it is nevertheless perfectly true that (even) a Turing machine can compute (in the external sense) that which is uncomputable(in the standard,internal sense). The claim that no machine can carry out the TP task is evidently false, but is this weaker form true: no machine that delivers each of its answers after only a finite number of atomic operations (we call these finitely-operating machines) can carry out the TP task? The answer is that this is not true. It is possible to specify a machine that is finitely-operatingin this sense and which carries out the TP task. Since each integer corresponds to a finite string of binary digits and vice versa (2 corresponding to 10, 3 to 11, 4 to 100, and so on), the TP task can be reformulatedlike this: given any integer, the machine is to output "1" if the binary string correspondingto the integer is a terminatingprogram, and is to output "0" if the binary string correspondingto the integer is not a terminatingprogram. Let us 502 B.J.COPELAND AND D. PROUDFOOT write an for the correct output when integer n is the input (an is always 0 or 1). Now consider the following decimal specification of a number:0.a\a2a3 .... We call this number r, for Turing (Copeland, 1998c; Copeland and Proudfoot, 1999a). Chaitin (1988) defines a number £2 that is analogous to, but not quite the same as, r. Like 7r, t is an irrationalnumber. Perhaps the first few digits of r are 0.00000001 1 Let us imagine a device S that stores exactly r units of some physical quantity.A measuring device M is able to measure the quantity stored in S to any specified number of significant figures. (Just as a hypothetical perfect frequency meter will measure the frequency in Hertz of a given signal to any desired number of signi- ficant figures.) M and S together form an oracle for the TP task (Copeland, 1997, 2000a). When any integer n is input into M, M determines an in a finite number of steps by measuring the quantity in S to an appropriatenumber of significant figures and outputting the nth digit of the result - which is an. (The first person to consider notional machines able to store arbitraryreal numbers appearsto have been Abramson. Abramson's "extended Turing machines" are able to store a real number on a single square of tape (Abramson, 1971).) Of course, a TP-task oracle designed in this way would not work very well in practice, since once n becomes very large, randomnoise would obstructM's efforts to determine an accurately. No one yet knows whether it is possible to produce a realistic design for an oracle. But the search is on for a physically realisable archi- tecture capable in principle of computing more than a finitely-operatinguniversal Turingmachine.

6. The Church-Turing Thesis The statement that it may be possible to mechanise tasks that cannot be performed by the universal Turing machine is often met with bafflement and incredulity.For there is a myth that Turing and Church set out a principle concerning the limits of mechanisability,to the effect that the universal Turingmachine can simulate the behaviourof any other machine. This (hypothesised) principle is commonly known as the Church-Turingthesis (see furtherCopeland, 2000a). The thesis that Turing and Church actually put forward is quite different: the universal Turing machine can perform any calculation that can be done by a human clerk working by rote with paper and pencil. This, the real Church-Turing thesis, does not entail that hypermachinesare impossible devices. This myth concerning the work of Turing and Church is widespread. For example: Paul and PatriciaChurchland, writing on Artificial Intelligence (Churchlandand Churchland,1990: 26): Turing's results entail something remarkable,namely that a standarddigital computer, given only the right program,a large enough memory and sufficient time, can compute any rule-governed input-outputfunction. That is, it can display any systematic patternof responses to the environmentwhatsoever. TURINGAFTER THE UTM 503

An O-machine carrying out the TP task responds to its input in a systematic, rule- governed manner. Turing had no result entailing that the systematic pattern of responses which this machine displays to its environment can be displayed by a standarddigital computer- exactly the reverse, in fact. Searle, writing on the question whether the operations of the brain can be simulated by the universalTuring machine (Searle, 1992: 200-201): Given Church's thesis that anything that can be given a precise enough char- acterizationas a set of steps can be simulated on a digital computer,it follows trivially that the question has an affirmativeanswer. An O-machine carries out the step-by-step proceduredictated by its program.So if the brainis an O-machine, it is true that the brain's processing can be characterised as a set of steps but it need not be true that the brain can be simulated by the universalTuring machine. Churchnever advanced the thesis that Searle ascribes to him, and nor did Turing.The thesis is false. Langton, writing on foundationalissues in Artificial Life (Langton, 1989: 12): There are certain behaviours that are incomputable' - behaviours for which no formal specification can be given for a machine that will exhibit that behaviour. The classic example of this sort of limitation is Turing's famous : can we give a formal specification for a machine which, when provided with the description of any other [sic] machine together with its initial state, will . . . determine whether or not that machine will reach its halt state? Turingproved that no such machine can be specified. Even Turing'sbiographer, Hodges, says this about Turing's work of 1935-1936 (Hodges, 1992: 109): Alan had . . . discovered something almost . . . miraculous, the idea of a universal machine that could take over the work of any machine. The sooner philosophy and cognitive science are free of this myth the better.

7. Turing and Wittgenstein Both Fellows of Cambridge colleges, Turing and Wittgenstein had a "wary re- spect" for each other.*Turing attended Wittgenstein's seminars from as early as 1935.**An apparentlywell-thumbed offprint of Turing's 1936 article "On Com- putable Numbers" was found among Wittgenstein's effects (Nedo and Ranchetti, 1983: 308). The published notes of Wittgenstein's lectures on the foundations of mathematicsin 1939 include lengthy discussions with Turing(Wittgenstein, 1976). (This record and Turing'scontributions to the now published radio broadcast"Can Automatic Calculating Machines Be Said To Think?"(Turing, 1952b) are the only * The quoted words are Gandy's, in personal communication with (1995). ** Copeland Hodges' biography of Turing suggests that prior to 1937 Turing had seen Wittgenstein only at the Moral Sciences Club (Hodges, 1992: 136). But see Nedo and Ranchetti (1983: 357-358). 504 B.J.COPELAND AND D. PROUDFOOT examples we have of Turing in discussion.) Although it is now impossible to de- terminethe precise natureof any influence between the two men, there is significant overlap in the work of Turing and Wittgenstein. Concerning , it is interesting that during the early and mid 1930s, Wittgensteinand his studentswere considering the question "Is every mathematical problem solvable?" (Ambrose, 1935: 186, 188). Ambrose says of the follow- ing view that "[t]his, or a view similar to it, has been put forward by Dr. L. Wittgenstein"in lectures delivered at Cambridge in 1932-1935 (Ambrose, 1935: 333): To say that a form has meaning is to say that in the symbolic system an answer to the question whether it is true, or false, is provided for ... And if the form . . . *acquiresmeaning' ... it has become the conclusion of a proof which is a new bit of mathematics. And this proof, on which the meaning of the form depends, provides the answer to the question whether the form is true or false. Turing's views on these and related matters are nicely summarised in a letter to Newman:

You say "If all this whole formal outfit is not about finding proofs which can be checked on a machine it's difficult to know what it is about". When you say "on a machine" do you have in mind that there is (or should be or could be, but has not been actually described anywhere) some fixed machine on which proofs are to be checked, and that the formal outfit is as it were about this machine? If you take this attitude . . . there is little more to be said: we simply have to ... resign ourselves to the fact that there are some problems to which we can never get the answer. On these lines my ordinal would make no sense. However, I don't think you really hold quite this attitudebecause you admit that in the case of the Godel example we can decide that the formula is true i.e. you admit that there is a fairly definite idea of a true formula which is quite different from the idea of a provable one. Throughout my paper on ordinal logics [Turing, 1938, 1939] I have been assuming this too. ... If you think of various machines I don't see your difficulty. One ima- gines different machines allowing different sets of proofs, and by choosing a suitable machine one can approximate 'truth' by 'provability' better than with a less suitable machine, and can in a sense approximate it as well as you please. The choice of a proof checking machine involves intuition, which is interchangeable with the intuition required for finding an Q if one has an ordinal logic A , or as a third alternativeone may go straightfor the proof and this again requires intuition. Or one may go for a proof finding machine. I am ratherpuzzled why you drew this distinction between proof finders and proof checkers. It seems to me rather unimportantas one can always get a proof * finder from a proof checker

* Turing to Newman, undated (probably written in 1939 or 1940). TURINGAFTER THE UTM 505

Concerning the nature of the brain, Wittgenstein, like Turing, anticipated as- pects of connectionism. Wittgenstein's objections to the representationaltheory of mind and his emphasis upon samples, training and "family resemblance" in concept-formation(Wittgenstein, 1965: 130ff, 1953: §65ff, §§208-210) prefigure connectionistaccounts of mind. As early as 1946-1948 he conjectured,as do many modern connectionists, that the brain does not process representations(Wittgen- stein, 1967b: §608ff). On cognition, however, Wittgenstein and Turing appear to have held very different views. In his later work (e.g., Wittgenstein, 1953: §152ff) Wittgenstein argued that understanding,thinking, intending, and other examples of cognition are not processes (because, for example, of the difficulty of speaking of the duration of understandingor intending in the way in which we speak of the durationof a process). It follows that the computationaltheory of mind is false: cognition cannot consist in computation,since computationis a process (for further discussion see Proudfoot, 1997). On Turing machines, Wittgenstein argued for a distinction between merely behaving in accordancewith a rule (of meaning, arithmetic,inference, etc.) and fol- lowing a rule. Cognition, he claimed, requiresgenuine rule-following. Wittgenstein argued,further, that "reading-machines"(his term: these include Turingmachines) as a matter of fact merely behave in accordance with a rule and so do not really read, calculate, and so on (Wittgenstein, 1967a: 1 19).* For Wittgenstein,to say that a Turingmachine does read or calculate is to engage in make-believe. AI tradition- ally has been anthropocentric(for example, the Turing Test holds that a computer is a genuine thinker if it resembles a human being to the degree that someone in- terviewing both the computer and a human by teletype cannot tell which is which). An effect of this anthropocentrismis to encourage the make-believe that computers possess human qualities. For example, Turing said that the P-type Turing machine which he "trained"by hand (Turing, 1948; Copeland and Proudfoot, 1996) received "reward"and "punishment"and that machines should be taught "discipline" and "initiative"(Turing, 1948: 17-21). The P-type is a "child-machine"and a child- machine could not be sent to school "withoutthe other children making excessive fun of it" (Turing, 1950a: 456-457). Its education, however, "shouldbe entrustedto some highly competent schoolmaster"(Turing, 1951a). Ongoing make-believe of this kind (see Proudfoot, 1999) has entrenchedin AI the aim of building machines that resemble human beings, even if the human qualities mimicked are irrelevant to the actual AI project. Wittgenstein's argumentspresent a challenge to Turing's romantic aim of pro- ducing an artificial "res cogitans." Wittgenstein argued that psychological states and their representationalcontents are individuated in terms of a subject's beha- viour, history and social environment,irrespective of internalstates. He also argued that ordinary(belief-desire) psychological explanation is not causal (Wittgenstein, 1965: 15, 110), and that, using such explanation, we can give different accounts of the behaviourof individuals who are physical duplicates of each other but have * Wittgenstein's remarks here anticipate Searle's argument, see Proudfoot (2000). 506 B.J. COPELANDAND D. PROUDFOOT different histories or environments. Computers do not as a matter of fact have the history necessary for genuine psychological states. This leads to the question: if giving a computer the requisite history is either impossible or too costly in time and money, and if the computer minus this requirementnevertheless does all that we want it to do, then why should we care if it does not really think?*

8. Morphogenesis In his final years Turing worked on what would now be called Artificial Life or A-Life, using the FerrantiMark I to model biological growth (Turing, 1952a). In February 1951 he wrote in a letter to a colleague at the NPL: Our new machine [the FerrantiMark I] is to start arriving on Monday. I am hoping as one of the firstjobs to do something about 'chemical embryology.' In particularI think one can account for the appearanceof Fibonacci numbers in connection with fir-cones.** Turingused the computerto simulate a chemical mechanism by which the genes of a zygote may determine the anatomical structureof the resulting animal or plant. During this period Turing achieved the distinction of being the first to engage in the computer-assistedexploration of non-lineardynamical systems (his theory used non-linear differential equations to express the chemistry of growth). Turingwrote concerning his work on neuralcomputation and on morphogenesis in a letter to the biologist Young: I am afraidI am very far from the stage where I feel inclined to startasking any anatomicalquestions [about the brain]. According to my notions of how to set about it that will not occur until quite a late stage when I have a fairly definite theory about how things are done. At present I am not working on the problem at all, but on my mathematical theory of embryology . . . This is yielding to treatment, and it will so far as I can see, give satisfactory explanations of i) Gastrulationii) Polyogonally symmetrical structures,e.g., starfish,flowers iii) Leaf arrangement,in particularthe way the Fibonacci series . . . comes to be involved iv) Colour patterns on animals, e.g., stripes, spots and dappling v) Patternson nearly spherical structuressuch as some Radiolaria,but this is more difficult and doubtful. I am really doing this now because it is yielding more easily to treatment. I think it is not altogether unconnected with the other problem. The brain structurehas to be one which can be achieved by the genetical embryological mechanism, and I hope that this theory that I am now working on may make clearer what restrictions this really implies. What you tell me about growth of neurons under stimulation is very interesting in this

* See Proudfoot (forthcoming), "Why did Wittgenstein think that computers don't follow rules, and does it matter?" ** Turing to Michael Woodger, undated, received 12 February 1951. TURINGAFTER THE UTM 507

connection. It suggests means by which the neurons might be made to grow so as to form a particularcircuit, ratherthan to reach a particularplace.* In June 1954 Turing died, while in the midst of this groundbreakingwork. He left a large pile of handwrittennotes and some programs.**This materialis still not fully understood.

Acknowledgements Research on which this article draws was supportedin part by University of Can- terburyResearch GrantNo. U6271 (Copeland) and Marsden Fund Research Grant No. UOC905 (Copeland and Proudfoot).

References

Abramson, EG., 1971, "Effective computation over the real numbers,"in TwelfthAnnual Symposium on Switching and Automata Theory, Northridge, CA: Institute of Electrical and Electronics Engineers. Ambrose, A., 1935, "Finitism in mathematics (I and II)," Mind 35, 186-203 and 317-340. Boolos, G.S. and Jeffrey, R.C., 1980, Computabihty and Logic, 2nd edn., Cambridge: Cambridge University Press. Chaitin, G.J., 1988, "Randomness in arithmetic,"Scientific American 259, 80-85. Churchland, P.M. and Churchland, P.S., 1990, "Could a machine think?" Scientific American 262, 26-31. Copeland, B.J., 1993, Artificial Intelligence, Oxford: Blackwell. Copeland, B.J., 1997, "The broad conception of computation," American Behavioral Scientist 40, 690-716. Copeland, B.J., 1998a, 'Turing's O-machines, Penrose, Searie, and the brain,"Analysis 58, 128-138. Copeland, B.J., 1998b, "Even Turing machines can compute uncomputable functions," pp. 150-164. in Unconventional Models of Computation, C. Calude, J. Casti, and M. Dinneen, eds., London: Springer-Verlag. Copeland, B.J., 1998c, "Super Turing-machines," Complexity 4, 30-32. Copeland, B.J., ed., 1999a, "The Turing-Wilkinson lectures on the Automatic Computing Engine," pp. 381-444 in Machine Intelligence 15, K. Furukawa, D. Michie, and S. Muggleton, eds., Oxford: . Copeland, B.J., ed., 1999b, A lecture and two radio broadcasts by Alan Turing, pp. 445-475 in Machine Intelligence 15, K. Furukawa, D. Michie, and S. Muggleton, eds., Oxford: Oxford University Press. Copeland, B.J., 2000a, "Narrowversus wide mechanism," Journal of Philosophy 97, 1-32. Copeland, B.J., ed., 2000b, Alan Turing's Automatic Computing Engine, London: Springer-Verlag. Copeland, B.J., 2000c, "Artificialintelligence," Encyclopedia Britannica (in press). Copeland, B.J. and Proudfoot, D., 19%, "On Alan Turing's anticipation of connectionism," Synthese 108, 361-377. Copeland, b.J. and Proudtoot, D., l*wa, Alan luring s forgotten ideas in computer science, Scientific American 280 (April), 76-81. Copeland, B.J. and Proudfoot, D., 1999b, "The ,"Mind 108, 187-195. * Turing to Young, 8 February 1951. ** These are in the Modern Archive Centre, King's College, Cambridge. 508 BJ. COPELANDAND D. PROUDFOOT

Copeland,B.J. and Sylvan,R., 1999, "Beyondthe universalTuring machine," Australasian Journal of Philosophy77, 46-66. Hodges,A., 1992,Alan Turing:The Enigma, London: Vintage. Langton,C.R., ed., 1989,Artificial Life, Redwood City, CA: Addison-Wesley. McCulloch,W.S. and Pitts, W., 1943, "Alogical calculusof the ideasimmanent in nervousactivity," Bulletinof MathematicalBiophysics 5, 115-133. Nedo, M. and Ranchetti,M., 1983, Wittgenstein:Sein Leben in Bildern und Texten,Frankfurt: SuhrkamnVerlac. Proudfoot,D., 1997, "OnWittgenstein on cognitivescience," Philosophy 72, 189-217. Proudfoot,D., 1999, "Howhuman can they get?" Science284, 745. Proudfoot,D., 2000, "Wittgenstein'santicipation of the ChineseRoom," in Viewsinto the Chinese Room,J. Prestonand M. Bishop,eds., Oxford:Oxford University Press (in press). Proudfoot,D. and Copeland,B.J., 1994, "Turing,Wittgenstein and the science of the mind," AustralasianJournal of Philosophy72, 497-519. Russell, B.A.W., 1915, Our Knowledgeof the ExternalWorld as a Fieldfor ScientificMethod in Philosophy,Chicago, IL: Open Court. Russell,B.A.W., 1936, "Thelimits of empiricism,"Proceedings of theAristotelian Society 36, 131- 150. Searle,J., 1992, TheRediscovery of the Mind,Cambridge, MA: MITPress. StewartI.. 1991. "Decidinethe undecidable."Nature 352. 664-665. Turing,A.M., 1936, "Oncomputable numbers, with an applicationto the Entscheidungsproblem," Proceedingsof the LondonMathematical Society, Series 2 42 (1936-37), 230-265. Turing,A.M., 1938, "Systemsof logic based on ordinals,"Dissertation presented to the facultyof PrincetonUniversity in candidacyfor the degreeof Doctorof Philosophy. Turing,A.M., 1939, "Systemsof logic basedon ordinals,"Proceedings of theLondon Mathematical Society,Series 2 45, 161-228. Turing,A.M., 1945, "Proposalfor developmentin the mathematicsdivision of an AutomaticCom- putingEngine (ACE)," National Physical Laboratory, London, pp. 2(M5 in AM. Turing'sACE Reportof 1946 and OtherPapers, B.E. Carpenterand R.W.Doran, eds., Cambridge,MA.: MIT Press. Turing,A.M., 1947, "Lectureto the LondonMathematical Society on 20 February1947," pp. 106- 124 in A.M. Turing'sACE Report of 1946 and OtherPapers, B.E. Carpenterand R.W.Doran, eds., Cambridge,MA: MITPress. Turing,A.M., 1948, "Intelligentmachinery," National Physical Laboratory, London, pp. 3-23 in MachineIntelligence 5, B. Meltzerand D. Michie,eds., Edinburgh:Edinburgh University Press. Turing,A.M., 1950a,"Computing machinery and intelligence," Mind 59, 433-460. Turing,A.M., 1950b,Programmers' Handbook for ManchesterElectronic Computer, Royal Society ComputingMachine Laboratory, University of Manchester. Turing,A.M., 1951a,"Intelligent machinery, a hereticaltheory," in Copeland(1999b). Turing,A.M., 1951b,"Can digital computers think?," BBC RadioBroadcast, in Copeland(1999b). Turing,A.M., 1952a, "Thechemical basis of morphogenesis,Philosophical Transactions of the RoyalSociety of London,Series B 237, 37-72. Turing,A.M., 1952b,"Can automatic calculating machines be saidto think?"BBC Radio Broadcast, in Copeland(1999b). Turing,A.M., 1953,"Digital computers applied to games,"typescript, reproduced as partof a longer articleof the same name by Turinget al. in Faster than Thought,B.V. Bowden, ed., 1953, London:Pitman. Von Neumann,J., ed., 1945, "Firstdraft of a reporton the EDVAC,"Moore School of Electrical Engineering,University of Pennsylvania. Williams, F.C., 1975, "Earlycomputers at ManchesterUniversity," The Radio and Electronic Engineer45, 237-331. TURINGAFTER THE UTM 509

Wittgenstein, L., 1953, Philosophical Investigations, G.E.M. Anscombe, trans., Oxford: Blackwell. Wittgenstein, L., 1965, The Blue and Brown Books, New : Harper. Wittgenstein, L., 1967a, Remarks on the Foundations of Mathematics, G.H. von Wright, R. Rhees, and G.E.M. Anscombe, eds. (2nd edn.), Oxford: Blackwell. Wittgenstein, L., 1967b, Zettel, G.E.M. Anscombe and G.H. von Wright, eds., G.E.M. Anscombe, trans., Oxford: Blackwell. Wittgenstein, L., 1976, Lectures on the Foundations of Mathematics, C. Diamond, ed., Ithaca:Cornell University Press. Wittgenstein, L., 1980, Remarks on the Philosophy of Psychology, Vol. I, G.E.M. Anscombe and G.H. von Wright, eds., G.E.M. Anscombe, trans., Oxford: Blackwell.