A Fundamental Property of All-Or-None Models, Binomial Distribution of Responses Prior to Conditioning, with Application to Concept Formation in Children

The Stanford Institute for Mathematical Studies in the Social Sciences

APPLIED MATHEMATICS AND STATISTICS LABORATORIES STANFORD UNIVERSITY

Reprint No. 70 A Fundamental Property of All-or-None Models, Binomial Distribution of Responses Prior to Conditioning, with Application to Concept Formation in Children

PATRICK SUPPES AND ROSE GINSBERG

Reprinted from Psychological Review Vol. 70, No.2, 1963 1963, Vol. 70, No.2, 139-161 A FUKDAl\IENTAL PROPERTY OF ALL-OR-i\ONE ::\IODELS, BE\O::\IIAL DISTRIBUTION OF RESPONSES PRIOR TO CONDITIONING, \VITH APPLICATION TO CONCEPT FORlVIATION IN CHILDREN 1

I. PATRICK SUPPES A~D ROSE GINSBERG Stanford University

A basic ~sumption of the simple all-or-none conditioning model is that the probability of a correct response remains constant over trials before conditioning. 4 implications of this assumption were tested: (a) prior to the'last error there will be no evidence of learning, (b) the sequence of responses prior to the last error forms a sequence of BerJif\oulli trials, (c) responses prior to the last error exhibit a binomial distribution and (d) specific sequences of errors and successes are dis tributed in accordance with the binomial hypothesis. These 4 tests were performed on the data from 7 experiments concerned with concept formation in children, paired-associate learning and probability learning in adults, and T maze learning in rats. The statistical evidence from these various experimental groups provided substantial support of the all-or-none model. However, when Vincent curves were constructed for responses prior to the last error, some of the learning curves showed significant departures from stationariness.

In the past year or two there has that the single stimulus element will been extensive application of a single be conditioned to the correct response. stimulus element conditioning model \Ve consider only those situations in' to paired-associate learning (Bower, which the subject is always informed 1961; Estes, 1961) and to concept of the correct response so that the formation in children (Suppes & Gins correct association may be learned on berg, 1962). In a paired-associate any trial. experiment the single stimulus element This all-or-none conditioning model represents a stimulus item from a list may be viewed as resulting from im of paired associates; in a concept posing special restrictions on more formation experiment the stimulus general models of stimulus sampling element represents a concept, or some theory. The statistics of this model aspect of a concept. The two essential have been analyzed in great detail in assumptions of the model are the Bower (1961). Supplementary sta following. First, until the single tistics for a finite number of trials at stimulus element is conditioned, there the end of which not all subjects are is a constant guessing probability, p, conditioned have been given by Estes that the subject responds correctly (1961) and Suppes and Ginsberg (the probability of an error on every (1962). trial is q = 1 - P). Second, on each The point of the present paper is to trial there is a constant probability, c, make explicit a simple but funda mentally important fact about the all 1 This research was performed pursuant to a or-none conditioning model: the as contract with the United States Office of Education, Department of Health, Education, sumption of a constant guessing and Welfare. probability on each trial before condi- 139 140 PATRICK SUPPES A;,-[D ROSE GIKSBERG tioning implies that there is a binomial Consider now how much simpler this distribution, with parameter p, of quantity is if we know whether or not responses prior to the last error. 2 This the subject is conditioned. Let Un observation has three important con stand for the unconditioned state on sequences for the analysis of experi Trial nand Cn for the conditioned state mental data. First, it implies that the on that trial, etc. Then the condi sequence of responses prior to the last tional probabilities are simply3 error forms a sequence of Bernoulli trials. This null hypothesis admits at P ,,(11/ U,,+I) = p2 [2J once the possibility of applying the P n (l1/ U n Cn+1) = p [3J many powerful statistics that are not applicable in the usual learning situa P,,(l1/C,,) =1 [4J tion for which the theory postulates dependence of responses from trial to Moreover, except for a few trials after trial. Second, the consideration of the last error when the subject may be response sequences prior to the last unconditioned but guessing correctly, error makes possible a deeper analysis we know what state he is in. In of response data than do statistics particular, on all trials prior to the which are averaged over subjects and last error we know he is in the un are a function of the conditioning conditioned state and thus that the parameter c. \Vhen statistics are ex probability of two successes in a row should be Relative to the third pressed as a function of c and the data pz. are analyzed in terms of all subjects point above, it may be noted that if the data are summed over subjects, regardless of whether or not they are test of Equation 1 requires the conditioned, then it is often the case that the large number of correct re assumption that all subjects have the same conditioning parameter sponses occurring after conditioning c, bias the statistics very favorably in whereas test of Equation 2 does not, and is compatible with the assumption terms of the model. Third, the ob of individual differences in condition servation that the distribution of re ing "propensi ty. " sponses prior to the last error should be binomial permits generalization of the model to admit individual differ STATISTICAL TESTS OF THE MODEL ences in the conditioning parameter c, Once the observation has been made while retaining a uniform guessing that according to the model responses parameter p. prior to the last error have a binomial These points may be emphasized by distribution, it is possible to consider considering just one example of a a variety of goodness of fit tests for familiar statistic for the model. Let this assumption. The virtue of these P" (11) be the joint pro babili ty of a goodness of fit tests is that in contra success on Trial n and on Trial n + 1. distinction to the many statistics It is easily shown that considered by Bower they permit a genuine statistical evaluation of the Pn(l1) null hypothesis that the model fits the = 1 - [1 - pZ (1 - c) - pc J data. There are four goodness of fit X (1 - c)n-l [lJ 3 There are only three cases to consider, 2 It is easy to demonstrate that it is sta namely, Un+1, UnC"+I, and Cn, because Un+1 tist~cally incorrect actually to include the last implies Un with probability one and Cn error in the analysis of response data. implies Cn+1 with probability one. A FUNDAMENTAL PROPERTY OF ALL-OR-NoNE MODELS 141 tests we believe to be of particular where i = 0, 1; n, (t) is the number of importance. In introducing these correct (i = 1) or incorrect (i = 0) four tests, we want to emphasize that responses in Block t; n (t) is the total we are not suggesting they are the number of responses in Block t; lh is only tests or that they are the only the number of correct (or incorrect) interesting ones. It seems to us, how responses summed over all blocks; ever, that they do ask the four most and N is the total number of responses important questions suggested by the summed over all blocks. The x2 "guessing" assumption of the model. statistic has the usual limiting dis The statistical properties of these four tribution with T - 1 degrees of tests are well known in the literature freedom, where T is the number of and do not need to be discussed here. blocks of trials. If there are m > 2 A good reference for the first two on responses, the number of degrees of stationarity and order is Anderson freedom is (m - l)(T - 1). Under ami Goodman (1957). the restriction to two responses, the Stationarity. Perhaps the most expression for X2 may be simplified to striking feature predicted is that if X2 = L [NnICt) - nln(t)]2/nln2n(t) data summed only over responses I made prior to the last error are con thus eliminating the summation over i. sidered, then there will be no evidence Order. The second property follow of learning over trials. Statistically ing from the guessing assumption this means the model predicts a which it is critical and significant to binomial distribution of responses test is that the sequence of responses with the constant parameter p. From the standpoint of learning theory this prior to the last error does indeed is a particularly interesting prediction form a sequence of Bernoulli trials, that is, that there is statistical in because of the classical emphasis on the mean learning curve. If the dependence in the responses made binomial assumption holds, the mean from trial to trial. There are various learning curve, when estimated over ways of testing this assumption but responses prior to the last error for it seems to us that the simplest and each subject, will be a horizontal line. most direct is to test the null hy Empirical tests of this prediction in pothesis that the dependence is zero experiments concerned with children's order versus the hypothesis that the concept formation, animal learning, dependence is first order. Acceptance probability learning, and paired-as of the null hypothesis has the strong sociate learning in human adults, are implication that we cannot predict given below. The appropriate sta responses better if we know whether tistical test for stationarity may be the preceding response was correct or formulated in terms of the null incorrect. The application of this hypothesis that there is no change in test to many other sets of learning the proportion of correct responses data has led to rejection of the null over trials. In order to obtain hypothesis at extremely high levels of 5 adequate data it is necessary to con significance (often p < 10- ). Many sider blocks of trials. Letting, then, results of this sort are to be found in the variable t run over blocks of trials Suppes and Atkinson (1960). In terms of other experimental evidence the appropriate x2 test is as follows: this must be regarded as a sensitive

2 test of the assumption of the statistical x = t1 net) (:ig;J - ~ r/~ independence of responses. H2 PATRICK SUPPES AKD ROSE GINSBERG

The appropriate formulation of the preuicteu frequencies. On the null x2 test is as follows. hypothesis that responses are statisti cally independent a standard x 2 test 2 n" 11 ,)2/ 11 ' for goodness of fit of the obtained and x = L: ni ( ..-2!. - --2 --2 i,i ni N N predicted frequencies is appropriate. Distribution of sequences of re where j as well as i is 0 or 1; n,j is the sponses. In addition to considering number of transitions from State i to the distribution of responses, we may Statej; ni = L: nij; nj = L: nij; and analyze the data in a still more refined i way by considering the distribution of N is the total number of responses, as sequences of responses. As in the case before. Again, x2 has the usual of the distribution of responses, the limiting distribution with (m - 1)2 practical approach is to consider degrees of freedom, where m is the blocks of trials of relatively small number of states; here, m = 2. length and look at the sequence of Distribution of responses. Granted responses within those blocks. For the assumption that responses prior to example, if we look at blocks of four the last error are binomially dis trials, then 0111 would represent a tributed, it is natural to ask if these sequence 011 which the first response responses do indeed exhibit a bino was incorrect and the subsequent mial distribution. Because the num three responses were correct. In four ber of responses prior to the last error trials there are, of course, 16 different varies from subject to subject and possible sequences of errors and suc because, unless the number of subjects cesses. If we consider the relative is very large, insufficient data will be frequency of every possible sequence obtained by grouping subjects accord of responses of this length, a x 2 test of ing to the number whose last error goodness of fit may then be applied occurs on the same trial, the natural in exactly the manner appropriate to and practical way to test the hy the distribution of responses them pothesis that the distribution is bino selves. In connection with this test, mial seems to be the following. For it is important to remark that the each subject consider blocks of, say goodness of fit of the distribution of four, trials taken up to the highest these sequences provides a goodness multiple of four equal to or less than of-fit test for the kind of run statistics the total number of responses prior much studied in the literature of to the last error. So that, for example, learning theory. The difficulty with if the last error for a subject occurred the usual statistics derived for runs on Trial 28, we would include in this is that they occur in the context of analysis the first six blocks of four statistical dependence in response trials. Over the total of such blocks, sequences and, therefore, a simple summed for all subjects, the frequency goodness of fit test is not valid. of occurrence of k errors, where Homogeneity of individual condition k = 0,1,2,3,4, provides our obtained ing parameters. On the assumption frequencies. The proportion of cor that all subjects have the same condi rect responses over the blocks of trials tioning parameter c, Bower (1961) included in this analysis is the derived the following distribution for maximum likelihood estimate of p. the trial 17' on which the IastCerror Using this estimate we may obtain occurs (essentially this distribution from the binomial distribution the was derived earlier by Bush & A FLNDAMEXTAL PROl'ERTY OF ALL-OR-NoNE MODELS 143

Mosteller, 1959). trials after conditioning, where the probability of a correct response is Pr(n' = k) unity, statistics such as the expected _ {bP for k 0 errors before the first success, the - b(1 - p)(i for k > 0 variance of this random variable, the expected number of success runs, the where expected number of alternations of c b=----- errors and successes, etc., are totally 1 P c) trivial and uninteresting and are The test of the null hypothesis is therefore best evaluated on the data simply a test of the goodness of fit prior to the last error where they do of this predicted distribution. Be not depend upon c. In itself the cause of the relative complexity of the goodness of fit test for the distribu expressions for this distribution we tion of last errors is a test of the null have found it convenient to estimate hypothesis that subjects have a homo c by a minimum x2 method. The geneous conditioning propensity. obtained minimum enables us to evaluate at once the goodness of fit ApPLICA TIONS TO COXCEPT of the assumption of homogeneity. FOR"IfATION IN CHILDREN When the number of subjects is small, a more sensitive test of significance Experiment on identity of sets. We needs to be used. first apply the tests considered in the The empirical distribution of the preceding section to some unpublished trial of the last error is a sufficient data of our own on the formation of statistic for estimating c and is the the concept of identity of sets in 48 only statistic that needs to be COIl children of first grade age. On each sidered in which the conditioning trial the child's task was to indicate parameter c enters. For sequences of whether two sets-each consisting of

1.0

., .9 '" -- all trials "0 before last 0 c. error J: .7

.0 1 2 4 5 b 7 12 13 14 BI Qcks of 4 trial s FIG. 1. Proportion of correct responses prior to last error and mean learning curve (Identity of Sets e.xperiment). 144 PATRICK SUPPES AND ROSE GIKSBERG

110 l''\. I /

100 / "\. ~---~"\.-"\.-"\.-'1 90 o

:>, u c: Q.J

0- '"Q.J 60 Lt

I 40 /

30 0---<) theoretical frequency 0- - -0 observed frequency

0 - - 2 3 4

5 uccesses FIG. 2. Empirical and predicted histogram of binomial distribution of corrcct responses in blocks of four trials (Identity of Sets experiment). one, two, or three elements-were on any trial was repeated for an identical or not. Specifically he was individual subject. instructed to press one of two buttons Because young children make oc when the stimulus pairs presented casional errors that are not necessarily were "the same" and the alternative an indication of incomplete learning,4 button when they were "not the we adopted a criterion of 16 successive same." A total of 48 subjects were correct responses as evidence of learn run through individual sessions of S6 ing. Errors occurring after this cri trials on 28 of which the stimulus dis terion was met are ignored in the plays showed identical sets, and the 4 An explicit model that accounts for these remaining 28 nonidentical sets. In occasional errors after conditioning is needed, this experiment no stimulus display but will not be pursucd here. -

A FUNDA::-IENTAL PROPERTY OF ALL-OR-NoNE )'10DELS 145

The number 16 was chosen TABLE 1 because the stimuli were randomized FREQUE:'{CY DISTRIBUTION OF SEQUENCES in blocks of eight trials with respect OF ERRORS AND SUCCESSES OVER to of ordered sets, identity BLOCKS OF F01;R TRIALS of nonordered sets, equipollence of nonidentical sets, and nonequipollent sets. In 1 is shmvn the proportion 0000 0 0.35 of correct responses prior to the last 1000 0 1.41 error in blocks of four trials. For com 0100 0 1.,1,2 the mean learning curve is 0010 4 1.42 shown in this figure. Note that 0001 0 1.42 1100 5 5.78 the mean learning curve is obtained in 1010 0 5.78 the usual way by summing over all 1001 5 5.78 trials regardless of where the last 0011 5 5.78 error occurred. The x" test of sta 0101 7 5.78 over blocks of four trials 0110 7 5.78 1110 22 23.56 leads to acceptance of the null hy 1101 30 23.56 UV~l.l\_"'l" (x2 4.95, df = 9, P > .80), 1011 31 23.56 and confirms the hypothesis of a 0111 29 23.56 constant probability of a correct re- 1111 86 96.06 over trials before conditioning. Note.~Identity of St.:ts experirnent. test for order described above yields a X2 of 1. 78 which, with 1 df is not significant (p > .10). This im four trials, as described above, also plies that we do have statistical yielded a nonsignificant x2 value independence of successive responses. (x 2 = 11.06, 6, > .05). The This result is particularly impressive predicted and quantities are the total number of responses shown in Table of the considered in the test is N = 937. small entries in some rows, the follow To examine the hypothesis that ing rows were combined to yield a net responses are binomially distributed of 6 df: Rows 1~6, Rows 7-8, Rows before the~final error we 9-11. the empirical frequency of The statistical for the dis- occurrence of zero, one, two, three, tribution of the last error were as and four successes in blocks of four follows. When the c value estimated trials over all subjects, with the pre frem mean total errors was used, the dicted binomial distribution. The fit of the distribution to the ob maximum likelihood estimate of p is served data was very poor (c = .061, simply the proportion of successes X2 = 26.71, df =(2, P < .001). When prior to the last error and is p = .803. c was estimated by a minimum X2 result of the goodness of fit test method directly from the empirical (x2 5.99, df 2, P = .05) supports data on the distribution with four the binomial assumption. The pre frequency classes in the histogram, and obtained histograms are the results were just at the shown in Figure 2. .02 level (c .035, X2 6.26, df =1). The goodness of fit test on the dis 'vVe return to these results later. tribution of specific sequences of Experiment on geometric forms. This successes and failures over blocks of experiment is reported in detail in 146 PATRICK SerrES AND ROSE GINSBERG 1.Or------.

VI '"<: ~ .7 e:::'"0.>

0- ----0____ _---0, (5 .6 - -0- -- " <: .... o , , 5 '0 Q, E c.. .5 all trials before last error

.3L-----~1------2~----~3------4~----~5------6L------J7 Blocks of 6 trials

FIG. 3. Proportion of correct responses prior to last error and mean learning curve (Quadrilateral and Pentagon concepts, Stoll experiment).

Stoll (1962), and we present some of was ever repeated for anyone subject the data here with her permission. and the stimulus displays representing The subjects were 32 kindergarten each form were randomized over children who were divided into two. experimental trials. The subjects equal groups. For both gmups the were run to a criterion of nine correct experiment was a successive dis responses in anyone session. cri mination, three-response si tua tioll, For the quadrilaterals and pen with one group discriminating be tagons the guessing probabilities prior tween triangles, quadrilaterals, and to the last error were essentially the pentagons, and the other group dis same, p = .609 and p = .600, re criminating between acute, right, and spectively, and the proportions of obtuse angles. For all subjects a correct responses for the combined typical case of each form was shown data are presented in blocks of six immediately above the appropriate trials, together with the mean learning response key. As in the previous curve, in Figure 3. experiment no single stimulus display Figure 4 presents the same curves A FUNDAMENTAL PROPERTY OF ALL-OR-NoNE MODELS 147

1.0

'" '".::: '"c "'- .9 '" a::" t) c; t: 0 u '0 .8 5 '';:; 1; "'- 0 ai! tr;a!s Q: .7 beFore last error

B:ocks cf b trials FIG. 4. Proportion of correct responses prior to last error and mean learning curve. (Acute, right, and obtuse angle concepts, Ston e."{periment.) for the combhled data for the three constant guessing probability prior types of angles, although in this to conditioning. case the probabilities varied For each concept with enough ob between the angles. Both figures servations before the last error to strongly support the hypothesis of a permit statistical good ness of-fit tests were performed for sta TABLE 2 tionarity over blocks of six trials, binomial distribution over of STATIONARITY, ORDER, AND BINOMIAL DISTRIBUTION RESULTS four trials, and order. The results of these tests are presented in Table 2. In general, there were too few observa dj p> x' tions to permit us to perform either 1.68 4 .70 the analysis of sequence of responses, 0.65 1 AD 65) 0.92 2 .60 or the test for homogeneity of con 2.40 4 .60 ditioning parameters, and these were 1.76 1 .IS 65) 2.07 2 .35 omitted in all cases. The results given in Table 2 primarily support 4 .05 1 .05 the constant guessing assumption 2 .25 of the a1l-or-none conditioning models . 4 . 10 1 _:0 Only two of the statistics in the table 2 .001 are significant at the .01 level, the 4 .85 1 .001 binomial distribution result for the 2 .20 right angles and the order test for the 4 .90 2 .40 obtuse angles. The total Ns on which 0.97 4 .90 the tests are based seem sufficiently large not to attribute the null results Geometric For::~s. to insufficient observations. To em- 148 PATRICK SUPPES AKD ROSE GINSBERG

.9 ,------_._------,------

.7 /~~ a::I u c " , \ 0 .6 ,0, , u , , , a , , , '0 _ , cf 1 "0 , -- 0 , ? .:; / , ,0, 8. , , , 0 .5 I '0' , ct I aii trlais 0 before iast error

.4 I 2 3 4 8 9 10 11 12 13 14 "Tnals FIG. 5. Proportion of correct responses prior to last error and mean learning curve (Binary )J umber experiment).

phasize this point, some additional stimuli was displayed on 16 trials, results for combined concepts are randomized over the experimental given at the bottom of the table. sequence of 96 trials. On each trial There is one systematic tendency the subject responded by placing one in the data that is not made evident of two response cards upon the by Table 2. Although the order tests stimulus display. are not significant except in the case From test trial responses, after each of the obtuse angle, in every case the experimental session, it seemed evi probability of a success following a dent that whereas some subjects success is slightly greater than the learned the concepts as such, others probability of a success following an learned only some of the specific error. The data are presented in Table 3. We shall discuss this point TABLE 3 later. PROBABILITY OF A SUCCESS FOLLOWI~G A Binary number experiment. This SUCCESS AKD OF A SL'CCESS FOLLOWIKG experiment is reported in detail in AN ERROR FOR RESPOKSES PRIOR Suppes and Ginsberg (1962). Five TO THE LAST ERROR and 6-year-old subjects were required to learn the concepts of the Numbers Problem of success following: Concept 4 and 5 in the binary number system, Success Error each concept represented by three different stimuli. There were two Quadrilateral .63 .58 groups of subjects but we consider Pentagon .63 .55 Acute angle .70 .61 here only the one group of 24 subjects Right angle .55 .46 who were required to make an overt Obtuse angle .77 .60 correction response following an in correct response, Each of the six Note.-Stoll's experiment on Geometric Forms. ~------

A FUNDAMENTAL PROPERTY OF ALL-OR-NoNE MODELS 149

stimuli representing the concepts so (concept and paired-associate learning that, in effect, there were two sub subjects) if the guessing probabilities groups of subjects. If the data from of the two groups are the same. both subgroups combined is analyzed Using the criterion that the concept in terms of paired-associate learning had been learned if the responses to of six independent items, responses the last three presentations of each of before the final error for each item the stimuli representing it were cor will represent the unconditioned state rect, we divided the data into two of that item for both kinds of subject, parts. The data from the group and the stationarity assumption meeting the criterion were arranged should still hold. In Figure 5 are for concept learning analysis (in this shown the proportion of correct re case a two-item learning situation), sponses prior to the last error and the the remaining data were assumed to mean learning curve, both presented represent paired-associate learning in from the poin t of view of paired volving six items. The proportion of associate learning. correct responses over all trials prior The data points are for individual to the last error for the two sub trials. Because a total of only 16 groups, with a criterion of six succes trials was run on each stimulus we sive correct responses maintained for have adopted a somewhat weaker the paired-associate learning sub criterion-six successively correct re group, was .498 for the latter, and .688 sponses-than was used in the analysis for the concept learning subgroup. of the two preceding experiments. \Ve were therefore not able to perform The proportion of correct responses the goodness of fit tests upon the prior to the last error are, therefore, combined data. However both sub only shown in Figure 5 for the first 10 groups provided, individually, suffi trials. The test of stationarity over cient observations for four of the five blocks of single trials for the:first 10 tests described-order, stationarity, trials again supports the null hy binomial distribution, and sequences pothesis (x 2 = 8.00, df = 9, P > .30, of successes and failures-but not for N = 844). the distribution of the trial on which The remaining goodness of fit tests the last error occurs. For the paired can only be performed on the com associate group over the:first 10 trials bined data from the two subgroups we had 81 cases, for the concept

TABLE 4

GOODNESS-OF-FlT TESTS OX RESPONSES BEFORE THE FIc-.:AL ERROR FOR COXCEPT LEARXIKG (Two ITEMS) S"CBGRO"CP, A:-!D PAIRED ASSOCIATE (SIX hE:Vl) S"CBGROl:P

Two-item subgroup Six-item subgroup Goodness-of-fit tests I l\Ta. x' dj I p> I x' I dj p> N ------I ------Stationarity 8.36 9 I .40 357 11.26 8 .10 570 I Order .57 I 1 .30 427 1.06 1 .20 476 Binomial _93 I 2 .50 141 3.21 1 .05 277 Sequence of successes and errors 1.28 I, 5 .90 141 7.22 2 .02 277 I I I I Note.-Binary Number experiment. a Xumber of observations on which test is based. 150 PATRICK AND GINSBERG

T.-\BLE 5

FREQUENCY DISTRIBUTION OF SEQUENCES OF ERRORS AND St;CCESSES FOR CO~'KEPT LEARNING (Two hE~f) SUBGROt;P, AND PAIRED·.-\SSOCIATE (SIX lTE~f) SUBGROUP

Tv.a-item subgroup Six-item subgroup

Sequenced- ! Obtained I',::dlc.'ed_ c. Obtained i Predicted ~t',~ee trial bloebl freQaency (tw~ti;;"F-' :reqael1cy f:-equency . 000 4 4.29 00 78 68.70 001 9 9.45 10 50 69.25 010 10 9.45 01 7-! 69.25 100 11 9.45 11 75 69.80 110 17 20.82 011 23 20.82 101 21 20.82 111 46 45.90

Xote.-Binary Number experiment. 1\ 0 :::; error, 1 = success. formation group we had 21 cases with mental research, we have examined 48 trials in each. The test results for from four published experiments each subgroup are reported in Table 4. of a different sort to give some in Of the eight test results listed, seven dication of the model's possible range. are nonsignificant and one, the good first two experiments were run ness of fit test for the distribution of with adult human subjects and are specific sequences, in the paired-asso here very briefly. The last ciate learning subgroup, is a borderline two are T maze studies with rats, and case (.05 > p > .02). The predicted we analyze the data in more detail. and obtained frequencies of specific Goodnow "two-armed bandit" experi sequences of successes and errors for ment. The apparatus used in this both subgroups are presented in Table and the general proced ure 5. The probabilities of a success employed are described in Goodnow following a success and following an (1955) and Goodnow and Pettigrew error for the paired-associate subgroup (1955). We make use here of the are .55 and .50, respectively. For the response data from this experiment as concept learning subgroup the same reproduced in Sternberg (1959), who conditional probabilities are .68 describes the experiment as follows: and .72. The apparatus was a "two-armed bandit" with a choice on each trial between pressing OTHER ApPLICATIO::-iS the left and right key. The reward schedule \Ye remarked earlier that the aU-ar was 100 :0, with the response on the left key rewarded, and the response on none-conditioning model we are con never rewarded. The sub sidering may be placed in the more undergraduates, were told general framework of stimulus sam particular trial one and only one pling theory. The applications to off. A subject in this experi- concept formation in children just -~.-;.:~.:-~ until he reached a criterion consecutive choices of the left key discussed represent only one of many 355). possible areas of applications. Al though concept formation has been a the criterion of 15 correct particular focus of our own experi- responses adopted by Goodnow, we A FUNDAMENTAL PROPERTY OF ALL-OR-NoNE MODELS 151 found that only 33 of the 77 subjects bias for the right side after the first made their last error after Trial 7. five trials. Primarily for this reason \Ye therefore terminated the analysis we have not analyzed this experiment of stationarity at this point and the ill greater detail. proportion of correct responses prior Bower paired-associate experiment. to the last error and the mean learning The present experiment is described curve shown in Figure 6 are for the in Bower (1961); in addition, Bower first seven trials. Although there is a has made available to us the data slight tendency for the proportion of pertinent to the statistical tests used correct responses prior to the last error to increase over these seven trials, the in this paper. The experiment is result is not statistically significant described by Bower as follows: (N = 382,)(2 = 9.97, df = 6, P > .10). Twenty-nine undergraduates learned a list In comparison the results for the mean of ten items to a criterion of two consecutive learning curve itself are highly signifi errorless cycles. The stimuli were different cant when the same stationarity test pairs of consonant letters and the responses were the integers 1 and 2, each response is applied on single trial blocks for the assigned as correct to a randomly selected first seven trials (N = 539, )(2 = 34.54, five stimuli for each subject. A response was df = 6, P < .001). According to obtained from the subject on each presenta Sternberg's description, all subjects tion of an item and he was informed of the correct answer following his responses. The were rewarded on the left side. The deck of ten stimulus cards was shuffled be data given in Figure 6 indicate that tween trials to randomize the presentation subjects had only overcome a guessing order of the stimuli (p. 258).

.80r------,

.70

Vl Q} Vl C o ~ .60 Q} cr:

'0 .50 ./ c 0/ / / ~o / c. 0- - / o / \ c: / \ / .40 o / all trials / beFore last / error \ / / .30 / 0/ .29L-----~------~------~------J------L-----~------~ 1 2 3 4 5 6 7 Trials

FIG. 6. Proportion of correct responses prior to last error and mean learning curve (Goodnow experiment). 152 PATRICK SUPPES AND ROSE GIKSBERG

1.00r------~

.90

.80

.70 all trials before last error

t:: :3 .60 5 C o _ __ 0 ..... _ c': ~ __ 0------0-- _------0 .... . 50

.40~----~------~------~------T------~------~6L-----~7

Trials

FIG. 7. Proportion of correct responses prior to last error and mean learning curve (Bower experiment).

The proportion of correct responses case of some of the experiments al prior to the last error and the mean ready considered, the results approach learning curve for the first seven trials, significance. Bower gives the ob averaged over subjects and items, are tained and predicted distributions of shown in Figure 7. As is evident, at last error in his article, and the agree the end of seven trials the learning ment between the two is excellent. was nearly perfect--the proportion of The other goodness of fit tests were overall correct responses being .96, not performed because of insufficient but for those items not yet condi observations. tioned, the natural guessing prob Galanter-Bush T maze experiments. ability has remained close to .50 (the Four T maze experiments, together drop below .50 of the last point is not with the complete response data, are significant for it is based on only 11 reported in Gat-anter and Bush (1959). observations). As would be expected Vle analyze here the initial acquisition from the figure, the x 2 test of sta data for Experiments III and IV for tionarity is not significant (N = 549, which the authors report improved x2 = .97, df = 6, P > .95). apparatus and methodology based 011 The test for order also was not the results of the first two experiments. significant (N = 417, x 2 = 3.41, In both experiments the animals were df = 1, P > .05), although as in the run under a noncorrection procedure A FUNDAMENTAL PROPERTY OF ALL-OR-NoNE MODELS 153

Exp. ill (21 5s) Exp.IT (19 5s) .7r------,------

~ " o "c. ~ '"" ~ .5 .3

"o .i J: .4

Blocks of 2 trials Blocb of 2 trials

FIG. 8. Proportion of correct responses prior to last error (Experiments III and IV, Galanter-Bush T maze) .

. 8 t OJ '"Vl '"0 .6 "- ~ Vl c::'" ~u ~ 0 u '0 .5 '"0 is Q. 0 ct

.4 2 9 Blocks of 2 trials

FIG. 9. Proportion of correct responses prior to last error on the combined data of Experiments III and IV (Galanter-Bush T maze). 154 PATRICK SUPPES AND ROSE GINSBERG and on each trial food .was always TABLE 7 located in the right-hand goal cup. FREQUE:-ICY DISTRIBUTION OF SEQUE;\fCES OF For both experiments we used 10 ERRORS AND SUCCESSES IN BLOCKS OF successively correct responses as a FOUR TRIALS criterion of learning. The probability Response sequence of satisfying this criterion before (0 = error, Obtained Predicted frequency frequency conditioning has occurred is extremely 1 = success) small and, as in the case of young 0000 10 8.89 children, occasional errors con tin ue to 1000 14 10.51 occur even after long series of correct 0100 11 10.51 responses, it seems psychologically un 0010 9 10.51 realistic to impose a stricter criterion. 0001 11 10.51 1100 13 12.45 The proportion of correct responses 1010 10 12.45 prior to the last error over the first 1001 12 12.45 18 trials are presented in Figure 8 for 0011 9 12.45 the 21 animals in Experiment III and 0101 15 12.45 0110 6 12.45 the 19 in Experiment IV. The mean 1110 13 14.73 learning curves are omitted because 1101 15 14.73 they are nearly identical for the first 1011 20 14~73 18 trials with the curves given. As 0111 17 14.73 the observed proportion of correct 1111 17 17.45 responses before the final error was Note.-T maze experiments, III and IV combined . .459 for Experiment III and .447 for Experiment IV, we combined the data tionarity tests for the two experi from both experiments and Figure 9 mental groups and for the combined shows the same curve for these data, data are performed for single trial again over the first 18 trials. The blocks over the first 18 trials; beyond graphic evidence of stationarity for this point there are too few subjects responses prior to the last error is left on each trial to permit statistical strongly supported by the statistical analysis. tests of stationarity which, together The statistical tests for binomial with the results of the tests for order, distribution of responses before the are shown in Table 6. The sta- final error (x 2 = 2.24, df = 3, P > .50) and for specific sequences of errors TABLE 6 and successes (x 2 = 9.35, df = 14, RESULTS OF STATIONARITY AND ORDER P > .80), estimated on the combined TESTS FOR EXPERnlE~TS I II AND IV data of Experiments III and IV with 202 observations, are also nonsignifi -----I-x-'I~I~ cant. The frequency of specific Experiment III I sequences of successes and errors in Stationarity 19.89 16 .20 blocks of four trials for the combined Order . .89 1 I .30 Experiment IV data are listed in Table 7. The Stationarity 7.53 161.95 probabilities of a success following a Order .90 1 .30 success and following an error for Experiments I I I and I V Experiment III are .57 and .52, re (combined) spectively. For Experiment IV the Stationarity 20.11 16 .20 Order 1.97 1 .10 same conditional probabilities are .59 and .54. :\'ote.-Galanter and Bush T maze. Of the goodness of fit test results iiillln

A FUNDA:\{ENTAL PROPERTY OF ALL-OR-NoNE MODELS 155

reported above for the two T maze \Ve applied the above tests to the experiments, none approaches signifi data from seven experiments in vari, cance so that the assumption of a ous areas-three in children's concept constant probability before formation, two in adult human learn the final error is strongly supported. ing, and two in animal learning. Six As for the test for homogeneity of of the experiments were two response conditioning parameters, as the aver situations, one-in children's concept age trial number on which the last formation-involved three responses. error occurred was 21.14 for Experi Because there were not in every case ment III and 23.84 for Experiment IV, sufficient observations, we were not we were able to use the combined data able to apply all the tests to the data which gave us sufficient observations from every experiment. In general to perform this test. 'Ye found the we were able to analyze the animal fit of the distribution of the last error and children's concept learning ex to the observed data, with the condi periments quite thoroughly, and the tioning parameter c estimated by a adult learning experiments rather minimum x2, to be very poor (8 = .03, more superficially. Two of the chil x2 = 29.50, df = 1, P < .001). In dren's 'concept formation experiments spection of the actual distribution involved subgroups, and wherever indicates that the observed variance possible the tests listed above were is much too small in relation to the applied to the data of each subgroup observed mean trial of the last error separately. for the model to fit very well. The test for stationarity was applied to the data from every experimental PRELDIIXARY DIscuSSrox group and subgroup. In all we per formed 16 such tests--3 of which 'Ye initiated the extensive analyses were on combined data of subgroups of the preceding pages to investigate which were also tested individually. two basic assumptions of the simple In no case was the result of the good all-or-none conditioning model; first, ness of fit test significant. In so far the assumption that correct responses as this test is concerned, the evidence before the final error are binomially that there is no change in the propor distributed, and second, that all tion of correct responses over trials subjects have the same conditioning that the process before the final error parameter. To examine the first hy is ill fact stationary-appears to be pothesis we suggested four goodness substantiaL 'Ye shall return to these of fit tests to be applied to responses results later. before the final error; these ,vere for 'Ye were able to perform the good, stationarity, order-a test of statis ness of fit test for order in 11 cases, tical independence of responses from none of "'hich were Oil data from one trial to the next-binomial dis combined groups. Of these test re- .tribution of responses, and finally the ID were not significant and 1 distribution of sequences of was highly significant (.01> P > .001). responses over smull blocks of trials. The latter result was from a subgroup To test the second assumption, that of the children's three·response con conditioning parameters are homo cept formation experiment (Geometric geneous, we proposed a goodness of Forms). The foregoing results in fit test for the distribution of trials on general provide quite good evidence which the last error occurs. to support the hypothesis that re- 156 PATRICK SUPPES AKD ROSE GIKSBERG sponses are independent over trials independence of trials and a binomial before the final error. However, we distribution of responses, is not un earlier pointed out a systematic tend reasonable. ency in the data of at least one of the As far as the test for homogeneity experiments reported which was not of learning parameters is concerned, in accord with the statistical evidence the results are by no means un of trial independence. In the six sub equivocal. \Ve were able to consider groups of the Stoll experiment (see this test for only three of the experi Table 3), the probability of a success mental groups-with each of the following a success was in every case groups from a different area; one slightly greater than the probability involving animals; one adult human of a success following an error. The subjects ;and the third, young children. same slight tendency is shown in the Bower reports that the fit of the concept learning subgroup of the observed and predicted distribution Binary Numbers experiment and in of trials on which the last error both animal learning experiments. occurred was very good for the adult On the other hand the inequality was humans in the paired-associate learn reversed for the paired-associate group ing experiment. On the other hand, of the Binary Numbers and the we found the fit to be very poor Identity of Sets experiments. For the for the animal learning experiment latter the probability of a success (p < .001~ and for the children's following a success was .81, the prob concept formation experiment (Iden ability of a success following an error tity of Sets) it was highly significant .85. All the other probabilities re when we estimated the conditioning ferred to are given earlier in the paper. parameter value from the total errors, \Ye do not have available the two (p < .001) and only just significant conditional probabilities for the Good at the .02 level when the parameter now Two-Armed Bandit or the Bower was estimated by a minimum x 2 Paired-Associate experiments. method. The binomial goodness of fit tests were applied to 10 groups and of the STATIONARITY RECONSIDERED 10 test results 9 were not significant The data from the seven experi and 1-from a subgroup of a children's ments we have examined indicate that concept formation experiment (Geo the simple all-or-none conditionipg metric Forms)-was again highly model is a good first approximation significant. Insofar as the test for to the actual response behavior in a distribution of specific sequences of reasonably wide class of situations. responses is concerned, we were able Ofthe various properties of the model to apply this test in only four cases, we have statistically examined above, of which three gave nonsignificant the property of stationarity of re results and one, again from a sub sponse probability prior to the last group of one of the children's concept error is the most crucial for supporting formation experiment (Binary Num the basic assumption that condition bers), approached significance (.05 ing occurs on an all-or-none rather > p > .02).· From these tests it ap than incremental basis. pears that the assumption of a con A rather consistent phenomenon in stant probability of a correct response respect to the stationary learning over responses before the final error, curves we present above is that of a with the consequent implication of persistent, though not statistically A Fu::mAME~TAL PROPERTY OF ALL-OR-)JONE MODELS 157 significant, tendency for the prob dividual differences in trial numbers ability of a correct response prior to of last errors we have constructed the last error to increase over trials. Vincent-type learning curves for trials This observation naturally gives rise prior to the last error. In Figures 10 to the question of whether the method and 11 the proportion of correct of statistical analysis used may not responses is presented over percentiles have been such as to miss what is, in of trials prior to the last error, in fact, a genuine change of probability stead of the usual blocks of trials. over trials. Gne possibility is the Five of the experiments are shown in following. The individual subject Figure 10 and 5 of the subgroups may actually be making a higher pro from the Stoll experiment are pre portion of correct responses towards sented in Figure 11. (The raw data the end of the sequence of trials prior from the Bower experiment were not to his last error. A.t the same time, available for this analysis.) To con as the data indicate, individual sub struct these curves the responses made jects are becoming conditioned at by each subject, prior to his final different rates. \Yhen the mean error, were divided into quartiles. stationarity curve is constructed by The fir.st data painton each curve averaging over a fixed block of trials represents the proportion of correct for all subjects no account is taken of responses in the first 25% of the re this individ ual difference. The result sponses of all subjects. The second, may be that the subjects who meet third, am! fourth data points similarly criterion early-and are therefore represent the further quartiles. At making more correct responses the far right of each figure is shown favorably \yeight the total proportion the criterion point C, where the re of correct responses in the early trials sponse proportion is of course, one. and thus appreciably support the null As the mean percentile of each of the hypothesis of stationarity, in spite of four quartiles is 12.5%, 37.5%, 62.5%, the actual fact of individual change and 87.5%, respectively, and C repre over trials before the last error. sents the 100% point, the distance To avoid this possible bias there between Points 4 and C on the fore, and to take into accoullt in- abscissa is one half of that between the quartiles themselves. LO" In the case of the Identity of Sets experiment the curve in Figure 10 is .. for the 38 subjects in the group who reached criterion, and in the case of the Binary Numbers experiment the

J curve is for those subjects who /' achieved concept mastery as defined 1Il the earlier discussion of this experi men t.

.5 i Nonstationarity and concavity are the two striking characteristics of all ~,;..~ five curves in Figure 10. Two of the .( --j -2 . five curves in Figure 11 also exhibit

FIG. 10. \"incent learning curves in quar these properties, whereas the other tiles for proportion of correct responses prior three are approximately stationary. to last error for li\"e experiments. The actual frequencies of correct 158 PATRICK SCPPES A:\D ROSE GI:\SBERG

1.0 r------

Oble,e ;;;'" .8 ";;- '"" "8 "-, , .7 , , 8 , , , '0 c __ '~\. __ ~ " .. -...... -8, ~ , 0 , 0: .6 ,

.4 ______-L ______~ ______L______~ ____~ 2

FIG. 11. Yincent learning curves ill quartiles for proportioll of correct responses prior to last error for Stoll's experiment. responses in each quartile for all ex has three of freedom. The periments are shown in Table 8, results exactly support the qualitative together with the results of the X Z test summary of the curves just given of statiollari ty--the test being per with the single exception of the formed with the quartiles as the newly Identity of.Sets experiment, for which clefilled trial blocks, so that each test the initial guessing probability in the

TABLE 8 RAW DATA AXD STATIO;';ARY TESTS FOR VIXCEXT LEAR"IXG CURVES PRIOR TO LAST ERROR, WHERE S = TOTAL OB3ERVATIOXS, n(l) = TOTAL OBSERV.UIO"S IX EACH BLOCK t, 111 (t) SUCCESSES r" BLOCK t FOR t 1,2,3, 4

EX1:eriment .Y n(l) ", (1) ndJ) ",(3) n1(-1)

I dent;t". of sets 56-10 141 117 110 ILl 126 7.66 Binary numbers 416 104 64 69 it 85 11.01 Goounow 436 109 49 49 59 86 31.09 Gabnter-Btl~h 111 396 99 41 ..U 56 71 .BAO Galanter-Bush IV 408 101 52 51 58 73 12.39 Quadrilateral5 260 65 43 41 36 41 1.75 Pentagon 260 65 40 35 .J.l 39 LB angles 320 80 41 H 38 42 .95 252 63 -15 34 48 51 12.63 340 85 50 49 54 71 16..13 A FUNDAMENTAL PROPERTY OF ALL-OR-NoNE MODELS 159

first quartile is already close to one. as follows. 5 There are two stimulus The Vincent curve for this latter group elements or patterns associated with is in appearance like the rest of the each experimental situation. With curves in Figure 10, that is concave equal probability exactly one of the upward. The statistical significance two elements is sampled on every of the nonstationarity test for six trial. Let us call the elements !J' and r. of the curves is in sharp contrast to \Vhen either element is unconditioned the nonsignificant results presented there is associated with it a guessing earlier for the same experiments. In probability g.or gT' as the case may the earlier case the stationarity test be, that the correct response wiII be was performed on an initial segment of made when that unconditioned stim trials and in that segment responses ulus is sampled. The assumption of from the last quartile of subjects who particular importance to the present conditioned in the early trials were model-and one that is not familiar averaged with responses from those in the literature-is that the prob who conditioned after a relatively ability of the sampled stimulus ele large number of trials. It will be ment becoming conditioned is not noted from Figures 10 and 11 that the necessarily the same when both nonstationary curves are relatively elements are unconditioned as it is stationary for the first two or three when the nonsampled element is quartiles, which we suggest explains already conditioned. \Ve call the first the excellent statistical results for the probability a and the second b' (the one-element model considered earlier. reason for the prime wiII become A rough but intuitive way of putting evident in a moment). it is that the one-element all-or-none Under these assumptions, together conditioning model seems to be ac with appropriate general independence counting for about two-thirds to of path assumptions as given, for three-fourths of the data. example, in Suppes and Atkinson On the other hand it is equally (1960, p. 5), the basic learning process evident that the nonstationary con may be represented by the following cave curves or- Figures 10 and 11 four state Markov process, where the cannot be accounted for by the simple four states (!J', r), !J', r, and 0 represent all-or-none conditioning model, which the possible states of conditioning of predicts a horizontal straight line. the two stimulus elements. It is not immediately clear what kind (11', r) r o of model will fit these curves with any accuracy. The problem is com (!J', r) 1 0 o o plicated by the fact that a percentile !J' b'/2 1 - b'/2 o o scale on the abscissa is not the kind of r b'/2 0 1 - b'/2 o scale ordinarily used in plotting learn o o a/2 a/2 1 - a lI1g curves. The remainder of this Because we do not attempt experi paper is devoted to a brief examina mentally to identify the stimuli !J' and tion of this problem in which we show r, this Markov process may be col that a' certain sort of two-element lapsed into a three-state process, stimulus sampling model can account whose states are simply the number of for the obtained empirical cures. ti The intuitive idea of this model originated The two-element model we consider in a conversation between the first author, may be psychologically conceptualized Gordon Bower, and Frank Restle. 160 PATRICK SUPPES AXD ROSE GmsBERG stimuli conditioned to the correct 6, but it yields the same response. Here it is convenient to concavity results for b > a, with replace b' /2 by b and we obtain the convexity for b < a. On the other transition matrix hand, it is not difficult to prove that at least for the simplest incremental 2 1 o model, the one-parameter linear model in which the increment is constant 2 1 0 0 independent of the response made, no 1 b 1 - b 0 [5J such concavity results can be obtained. 0 l-a o a Further detailed analysis of data Moreover, accompanying the states 0 will be required to determine the and 1 we have the guessing probabili adequacy of the two-element model ties go and gl defined"in the obvious proposed here. It is unfortunately manner in terms of the sampling not easy to make a good estimate of probability ! and the guessing prob the three parameters. ::\1 ore over, the abilities gff and gr statistical tests reported for the one-element model are not valid go = !gff + !gr for the two-element model. Evalua gl = tgu + tgr + ~ = !go + t tion of the goodness of fit of the two element model is consequently a The probabilities g" and gr are not ob difficult and is not pursued in servable, but go is, and gl is a simple this paper. function of it. This means that we If a two-element model does turn have a process with three free pa a good account of the kind rameters, the conditioning parameters data analyzed in this a and b, and the guessing probability paper, it may be thought of as a go. conceptual compromise between in To obtain a simple expression cremental and all-or-none condition- roughly corresponding to the concave models. The conditioning is curves of Figures 10 and 11, we give all-or-none for each of the two stim here the probability of a correct re ulus elements, but the probability of a sponse on Trial j (Event A I,j) given correct response prior to the last error that State 2-both stimuli conditioned will be at two different levels, go and -was entered on Trial N (Event' 2*N, gl, the sequence of trials prior j < N). to ,..,.,1'",,., "n

P(A 1,ii2*N) SnlMARY (gl go) [:.:=: := ~ ] + go [6J A assumption of the simple all-or-none conditioning model is that the probability of a correct response wereh a = 1 - _ ab . I't IS eaSl'J y sownh remains constant over trials before that the learning curve derived from In this paper we have Equation 6 is concave upward when examined assumption and some b > a, and convex upward-the of its implications, in some detail. standard result-when b < a. The implications we have specifically The expression for the probability data from a number of of a correct response on Trial j given different experimental areas to do so that the last error occurred on Trial N are, over data prior to the last error is considerably more complicated than there will be no evidence of learning A FC::-;DAc,IEXTAL PROPERTY OF ALL-OR-~ONE MODELS 161

over trials, responses prior to the last REFEREXCES form a sequence of Bernoulli trials, A:>DERSO:>, T. \Y., & GoomlA:-<, L. A. responses prior to the last error ex Statistical inference about :\!arkov chains . hibit a binomial distribution, and .1nl1. math. Statist, 1957,28,89-110. specific sequences of errors and suc BOWER, G. H. Application of a model to cesses are distributed in accordance paired-associate learning. Psycizollletrika, 1961,26,255-280. with the binomial hypothesis. FOllr Bl'SH, R. R., & :\!OSTELLER, F. A com goodness of fit tests were used to parison of eight models. In R. R. Bush evaluate the above implications and & \Y. K. Estes (Eds.), Studies in mathe these are described in detail in the maticallearning theory. Stanford: Stanford preceding pages. The four tests were Univer. Press, 1959. Pp.293-307. performed on the data from seven ESTES, \V. K. New developments in sta experiments concerned with concept tistical behavior theory: Differential tests of axioms for associative learning. Psycho- formation in children, paired-associate 111etrika, 1961,26,73-84. learning and probability learning in G.UA:-

1. "A Note on the Menger·Wieser Theory of Imputation." Hirofumi Uzawa, Zeitschrift lur NationaliJkonomie, Vol. XVIII, No.3, 1958, pp. 318-34. 2. "On a Class of Capacitated Transportation Problems," Harvey M. Wagner, Management Science, Vol. 5, No.3, April 1959, pp. 304-18. 3. "The Case for 'Revealed Preference'," Harvey M. Wagner, The Review of Economic Studies, Vol. XXVI, No.3, June 1959, pp. 178-89. 4. "Chains of Infinite Order and Their Application to Learning Theory," John Lamped and Patrick Suppes, Pacific Journal of Mathematics, Vol. 9, No.3, 1959, pp. 739-54. 5. "Competitive Stability Under Weak Gross Substitutability: The 'Euclidean Distance' Approach," Kenneth J. Arrow and Leonid Hunvicz, lnternan.(mat Economic Review, Vol. 1, No.1, January 1960, pp. 38-49. 6. "Prices of the Factors of Production in International Trade," Hirofumi Uzawa, Econo· metrica, Vol. 27, 3, July 1959, pp. 448-{'i8. 7. "Stability and Non-Negativity in a Walrasian Tiitonnement Process," Hukukane Nikaido and Hirofumi Uzawa, International Economic Review, Vol. 1, No.1, January 1960, pp. 50-59. B. "The Capacity Method of Quadratic Programming," H. S. Houthakker, Econometrica, Vol. 28, 1, January 1960, pp. 62-87. 9. "Optimization, Decentralization, and Internal Pricing in Business Firms," Kenneth J. Arrow, Stanford University, Contributions to ScientifiC Research in 11,lanagement, pp. 9-18. 10. "Classification Procedures Based on Dichotomous Response Vectors," Herbert Solomon, Stanford University, Contributions to Probability and Statistics, 1960, pp. 414-23. Stan· ford University Press. 11. "The Work of Ragnar Frisch, Econometrician," K. ], Arrow, Econometrica, Vol. 28, 2, April 1960, pp. 175-92. 12. "Best Linear Index Numbers of Prices and Quantities," H. Theil, Econometrica, Vol. 28, 2, April 1960, pp. 464-80. 13. "Walras' Tatonnement in the Theory of Exchange," H. Uzawa, The Review of Economic Studies, Vol. XXVTI, No. pp. ,182-94. 14. "Stability of the Gradient in n-Person Games," K. J. Arrow and Leonid Hurwicz, J. Soc. Indust. Appl. Math., Vol. B, No.2, June 1960. 15. "Decision Theory and the Choice of a Level of Significance for the t-test," K. J. Arrow, Stanford University, Contributions to Probabaity and Statistics, 1960, pp. 70-78. Stan ford University Press. 16. "Some Asymptotic Properties of Luce's Beta Learning Model," John Lamperti and Patrick Suppes, Psychometrika, Vol. 25, No.3, September 1960, pp. 233-4l. 17. "Some Remarks on the Equilibria of Economic Systems," K. J. Al;row and L. Hurwicz, Econometrica, Vol. 28, No.3, July 1960, pp. 640-46. 18. "Group and Individual Performance in Problem Solving Related to Previous Exposure to Problem, Level of Aspiration, and Group Size," Irving Lorge and Herbert Solomon. Behavioral Sciences, Vol. 5, No.1, January 1960, pp. 2B-38. ]9. "Some Examples of Global Instability of the Competitive Equilibrium," Herbert Scarf, International Economic Review, Vol. I, No.3, September 1960, pp. 157-73. 20. "Decentralization and Computation in Resource Allocation," Kenneth J. Arrow and Leonid Hurwicz, "Essays in Economics and Econometrics," 1960, pp. 34-104, University of North Carolina Press. 21. "Locally Most Powerful Rank Tests for Two-Sample Problems," Hirofumi Uzawa, The Annals of Mathematical Statistics, Vol. 31, No.3. September 1960, pp. 685--702. 22. "Price-Quantity Adjustments in Multiple Markets with Rising Demands," Kenneth J. Arrow, Stanford University, Mathematical Methods in the Social Sciences, 1959, pp. 3--15, Stanford University Press. 23. "Optimality and Informational Efficiency in Resource Allocation Processes," Leonid Hurwicz, University of Minnesota, Mathematical Methods in the Social Sciences, 1959, pp. 27-46, Stanford University Press. 24. "Preference and Rational Choice in the Theory of Consumption," Hirofumi Uzawa, Stanford University, Mathematical Methods in the Social Sciences, 1959, pp. 129-48, Stanford University Press. 25. "A Stationary Inventory Model with Markovian Demand," Samuel Karlin, Stanford University, Augustus J. Fabens, Dartmouth College, Mathematical Methods in the Social Sciences, 1959, pp. 159-75, Stanford University Press. 26. "The Optimality of (S,s) Policies in the Dynamic Inventory Problem," Herbert Scarf, Stanford University, Mathematical Methods in the Social Sciences, 1959, pp. 196-202, Stanford University Press. 27. "A Random·Walk Model for Choice Behavior," W. K. Estes, Indiana University, Mathe matical Methods in the Social Sciences, 1959, pp. 265-76, Stanford University Press. 28. "Measures of Worth in Item Analysis and Test Design," Herbert Solomon, Stanford University, Mathematical Methods in the Social Sciences; 1959, pp. 330-47, Stanford University Press. 29. "Stimulus-Sampling Theory for a Continuum of Responses," Patrick Suppes, Stanford University, Mathematical Methods in the Social Sciences, 1959, pp. 348-65, Stanford University Press. 30. "Optimal Policies for a Multi-Echelon Inventory Problem," Andrew J. Clark and Her bert Scarf, Management Science, Vol. 6, No.4, July 1960, pp. 475-90. 31. "Market Mechanisms and Mathematical Programming," Hirofumi Uzawa, Econometrica, Vol. 28, No.4, October 1960, pp. 872-81. 32. "On the Formation of Prices," Takashi Negishi, International Economic Review, Vol. 2, No.1, January 1961, pp. 122-26. 33. "Application of Stimulus Sampling Theory to Situations Involving Social Pressure," Patrick Suppes and Franklin Krasne, Psychological Review, Vol. 68, No.1, 1961, pp. 46-59. 34. "Test of Stimulus Sampling Theory for a Continuum of Respon,es with Unimodal Noncontingent Determinate Reinforcement," Patrick Suppes and Raymond W. Frank· mann, Journal of Experimental Psychology, Vol. 61, No.2, 1961, pp. 122-32. 35. "A Comment on Newman's 'Complete Ordering and Revealed Preference,''' Hirofumi Uzawa, The Review of Economic Studies, Vol. XXVIII, No.2, February 1961, pp. 140·-41. 36. "Neutral Inventions and the Stability of Growth Equilibrium," Hirofumi Uzawa, The Review of Economic Studies, VoL XXVIII, No.2, February 1961, pp. 117-24. 37. "Additive Logarithmic Demand Functions and the Slutsky Relations," Kenneth J. Arrow, The Review of Economic Studies, Vol. XXVIII, No.3, August 1961, pp. 176-81. 38. "Behavioristic Foundations of Utility," Patrick Suppes, Econometrica, Vol. 29, No.2, April 1961, pp. 186-202. 39. "A Generalization of Stimulus Sampling Theory," Richard C. Atkinson, Psychometrica, Vol. 26, No.3, September 1961, pp. 281-90. 40. "Capital-Labor Substitution and Economic Efficiency," K. J. Arrow, H. B. Chenery, B. S. Minhas, and R. M. Solow, The Review of Economics and Statistics, Vol. XLIII, No.3, August 1961, pp. 225-50. 41. "The Observing Response in Discrimination Learning," Richard C. Atkinson, Journal of Experimental Psychology, Vol. 62, No.3, 1961, pp. 253-62. 42. "The Philosophical Relevance of Decision Theory," Patrick Suppes, Journal of Phi losophy, Vol. LVIII, No. 2J, October 12, 1961, pp. 605-14. 43. "Monopolistic Competition and General Equilibrium," Takashi Negishi, Review of Eco nomic Stndies, Vol. 28. No.3, 1961, pp. 196-201. 44. "Constraint Qualifications in Maximization Problems," Kenneth J. Arrow, Leonid Hur· wicz, and Hirofumi Uzawa, Naml Research Logistics Quarterly, Vol. 8, No.2, June 1961, pp. 175-91. 45. "Advertising Without Supply Control: Some Implications of a Study of the Advertising of Oranges," Marc Nerlove and Frederick V. Waugh, Journal of Farm Economics, Vol. XLIII, No.4, Part I, November 1961, pp. 813-37. 46. "Stochastic Learning Theories for a Response Continuum with Non-Determinate Rein. forcement," Patrick Suppes and Joseph 1. Zinnes, Psychometrika, Vol. 26, No.4, De cember 1961, pp. 373-90. 47. "On a Two-Sector Model of Economic Growth," by Hirofumi Uzawa, Review of Eco nomic Studies, Vol. XXIX, No.1, 1962, pp. 40--47. 48_ "Test of Some Learning Models for Double Contingent Reinforcement," by Patrick Suppes and Madeleine Schlag·Rey, Psychological Reports, February 1962, 10, pp. 259--68. 49_ "Quasi·Concave Programming," by Kenneth J. Arrow and Alain C. Enthoven, Econo· metrica, Vol. 29, No.4, October 1961, pp. 779-800. 50. "The Stability of Dynamic Processes," by Hirofumi Uzawa, Econometrica, Vol. 29, No.4, October 1961, pp. 617-3l. 51. "A Quarterly Econometric Model for the United Kingdom." by Marc Nerlove, The American Economic Review, Vol. LII, No.1, March 1962, pp. 154-76. 52. "Experimental Analysis of a Duopoly Sitnation from the Standpoint of Mathematical Learning Theory," by Patrick Suppes and J. Merrill Carlsmith, International Economic Review, Vol. 3, No.1, January 1962, pp. 60-78. 53. "Experimental Studies of Mathematical Concept Formation in Young Children," by Patrick Suppes and Rose Ginsberg, Science Education, Vol. 46, No.3, April 1962, pp. 230-40. 54. "Optimal Advertising Policy Under Dynamic Conditions," by Marc Nerlove and Kenneth J. Arrow, Economica, Vol. 29, N.S., No. 114, May 1962, pp. 129--42. 55. "Application of a Stimulus Sampling Model to Children's Concept Formation With and Without Overt Correction Responses," by Patrick Suppes and Rose Ginsberg, lour nal of Experimental Psychology, Vol. 63, No.4, 1962, pp_ 330-36. 56. "Optimal Capital Adjustment," by Kenneth J. Arrow, Studies in Applied Probability and Management Science, 1962, pp. 1-17, Stanford University Press. 57. "Models of Data," by Patrick Suppes, Logic, Methodology and Philosophy of Science: Proceedings of the 1960 International Congress, 1962, pp. 252-61, Stanford University Press. 58. "On the Stability of Edgeworth's Barter Process," by Hiro£urni Uzawa, International Economic Review, Vol. 3, No.2, May 1962, pp. 218-32. 59. "Competitive Stability Under Weak Gross Substitutability: Nonlinear Price Adjustment and Adaptive Expectations," by K. J. Arrow and L. Hurwicz, International Economic Review, Vol. 3, No.2, May 1962, pp. 233-55. 60. "Some Current Developments in Models of Learning for a Continuum of Responses," by Patrick Suppes, American Institute 0/ Electrical Engineers 1962 loint Automatic Control Conference, June 1962, pp. 1-9. 61. "Optimal Capacity ScheduIing-I & II," by Arthur F. Veinott, Jr. and Harvey M. Wagner, Operations Research, Vol. 10, No.4, July-August, 1962, pp. 518--46. 62. "A Note on the Theory of Economic Growth," by T. Okamoto and Ken-Ichi Inada, The Quarterly lournal of Economics, Vol. LXXVI, August 1962, pp. 503-07. 63. "Demand Analysis: Mathematical Appendix," by H. Uzawa, The American Economic Review, Vol. LII, No.4, September 1962, pp. 797-80l. 64. "Analysis of Social Conformity in Terms of Generalized Conditioning Models," P. Suppes and Madeleine Schlag·Rey, Mathematical Methods in Small Group Processes, 1962, pp. 334-61, Stanford University Press. 65. "The Economic Implications of Learning by Doing," K. J. Arrow, Review of Economic Studies, Vol. 29 ,No.3, June 1962, pp. 155-73. 66. "Group and Individual Behavior in Free-Recall Verbal Learning," Irving Lorge and Her bert Solomon, Mathematical Methods in Small Group Processes, 1962, pp. 221-31, Stan ford University Press. 67. "A Theorem on Non-Tatonnement Stability," Frank Hahn and Takashi Negishi, Econo metrica, Vol. 30, No.3, July 1962, pp. 463-69. 68. "The Stability of a Competitive Economy: A Survey Article," Takashi Negishi, Econo metrica, Vol. 30, No.4, October 1962, pp. 635-69. 69. "A Two-Sector Extension of Swan's Model of Economic Growth: The Case of no Tech nical Change," Mordecai Kurz, International Economic Review, Vol. 4, No.1, January 1963, pp. 68-79. 70. "A Fundamental Property of All-or-None Models, Binomial Distribution in Responses Prior to Conditioning, with Application to Concept Formation in Children," by Patrick Suppes and Rose Ginsberg, Psychological Review, Vol. 70, No.2, 1963, pp. 139-61.