Intersensory and Sensory Integration PERCEPTION AND PERCEPTUAL DEVELOPMENT A Critical Review Series

Series Editors: Herbert L. Pick, Jr. University of Minnesota, Minneapolis, Minnesota and Richard D. Walk George Washington University, Washington, D.G.

Volume 1 Perception and Experience Edited by Richard D. Walk and Herbert L. Pick,Jr.

Volume 2 Intersensory Perception and Sensory Integration Edited by Richard D. Walk and Herbert L. Pick, Jr.

A Continuation Order Plan is available for this series. A continuation order will bring delivery of each new volume immediately upon publication. Volumes are billed only upon actual shipment. For further information please contact the publisher. Intersensory Perception and Sensory Integration

Edited by RICHARD D. WALK George Washington University Washington, D.C.

and HERBERT L. PICK, JR. University of Minnesota Minneapolis, Minnesota

PLENUM PRESS • NEW YORK AND LONDON Library of Congress Cataloging in Publication Data

Main entry under title:

Intersensory perception and sensory integration.

(Perception and perceptual development; v. 2) Includes bibliographical references and index. 1. Perception. 2. Intersensory effects. I. Walk, Richard D. II. Pick, Herbert L. III. Series. [DNLM: 1. Perception. 2. Sensation. WI PE78GM v. 2/WL 705 162) BF311.I59 153.7 80·29204 ISBN 978-1-4615-9199-3 ISBN 978-1-4615-9197-9 (eBook) DOl 10.1007/978-1-4615-9197-9

© 1981 Plenum Press, New York Softcover reprint ofthe hardcover 1st edition 1981 A Division of Plenum Publishing Corporation 227 West 17th Street, New York, N.Y. 10011

Ail rights reserved

No part of this book may be reproduced, stored in a retrieval system, or transmitted, in any form or by any means, electronic, mechanical, photocopying, microfilming, recording, or otherwise, without written permission from the Publisher Contributors

Eugene Abravanel, Department of , George Washington Uni• versity, Washington, D.C.

Emily W. Bushnell, Department of Psychology, Tufts University, Med• ford, Massachusetts George Butterworth, Department of Psychology, University of South• ampton, Southampton, England Malcolm M. Cohen, Naval Air Development Center, Warminster, Penn• sylvania Bryant J. Cratty, Department of Kinesiology, University of California, Los Angeles, California James E. Cutting, Department of Psychology, Cornell University, Ithaca, New York Paul Fraisse, Centre H. Pieron, Universite Rene Descartes, Paris, France B. Hermelin, Medical Research Council (MRC) , Developmental Psy• chology Unit, London, England

Bill Jones, Department of Psychology, Carleton University, Ottawa, Ontario, Canada James R. Lackner, Department of Psychology, Brandeis University, Waltham, Massachusetts Susanna Millar, Department of , University of Oxford, Oxford, England N. O'Connor, Medical Research Center (MRC) , Developmental Psy• chology Unit, London, England v vi Contributors

Herbert L. Pick, Jr., Center for Research in Human Learning, University of Minnesota, Minneapolis, Minnesota Dennis R. Proffitt, Department of Psychology, University of Virginia, Charlottesville, Virginia Jacqueline M. F. Samuel, Department of Psychology, George Washing• ton University, Washington, D.C. Richard D. Walk, Department of Psychology, George Washington Uni• versity, Washington, D.C. Preface

This volume on intersensory perception and sensory integration is the second volume of the series, Perception and Perceptual Development: A Critical Review Series. The topic of the volume is timely, for in recent years, many investigators have noted that information about any natural event is obtained by a perceiver from a variety of sources. Such an observation immediately leads to the question of how this information is synthesized and organized. Of course, the implication that there are several discrete input channels that must be processed has come under immediate attack by researchers such as the Gibsons. They find it extremely artificial to regard natural information as being cut up and requiring cementing. Nevertheless, the possibility that during ontogene• sis, perception involves the integration of separate information has attracted the attention of scholars concerned with both normal and abnormal development. In the case of normal development, a lively controversy has arisen between those who believe perceptual develop• ment goes from integration toward differentiation and those who hold the opposite view. In the case of abnormal psychological development such as learning disabilities, many workers have suggested that percep• tual integration is at fault. In thinking about the issues raised in this volume, we are particularly indebted to our former teachers and colleagues: Eleanor and James Gibson, T. A. Ryan, Robert B. MacLeod, and . We are pleased to acknowledge the secretarial help of Karen Weeks in the preparation of this volume. Preparation of the book was supported in part by a Program Project Grant from the National Institute of Child Health and Human Development (HD-05027) to the Institute of Child Development, by the Center for Research in Human Learning of the University of Minnesota, and by a National Institutes of Health Biomed-

vii viii Preface ical Research Support Grant (2-S07-RR07019-14) to George Washington University.

HERBERT L. PICK, JR. RICHARD D. WALK Introduction

This book is concerned with the interrelation of modalities. How does stimulation of one modality-vision, for example-interact with that of another-audition, for example? The book's primary focus is on the perceptual aspects of intermodal relations in contrast with sensory aspects. Thus none of the contributions addresses such questions as whether stimulation of one sense modality changes the threshold for detection of stimulation in another. Rather, the chapters address such questions as whether a tactual and visual shape are the same, or whether a visual and auditory spatial locus are perceived as the same place. It seems patently obvious that the way to understand perception is to analyze how sensory stimulation is processed by the different sense modalities. In fact, it sometimes appears that there is no other alternative. This pervasive view is perhaps a tribute to the success of 10hanes MOller's doctrine of Specific Nerve Energies. According to this doctrine the different qualities of our sensory experience were based on the nerve energies which the different nerves carried from receptor organ to brain, or on the particular loci of the brain at which these nerves ended. MOller's doctrine was based on the anatomical and physiological evidence of the time and on a philosophical tradition which argued that we could not know anything of the external world, only the state of our nerves. MOller's doctrine led in turn to the structuralist psychology of Wundt and Titchener, in which our mental experience was analyzed in terms of a set of attributes such as quality, intensity, protensity, attensity, etc. Any given percept could be broken down into sensations of vision, audition, taste, etc., of a particular intensity, duration, clarity and size, or combinations of these attributes. Although the extreme mentalism of this approach is now gone, its heritage is still reflected in our emphasis on the processing of stimulation by the specific sense modalities. Con• sider, for example, the typical courses in a psychology department on vision and audition. (There are usually no courses for the other sense

ix x Introduction modalities simply because there is not as big a knowledge base for how these work.) And consider the listing in psychological abstracts of the main topics for audition, vision, and the lower . Our current commonsense view of how to understand perception has not always been so one-track. Although the concept of separate sense modalities can be traced back to Aristotle, he also had the concept of sensus comunis, which referred to a capacity for awareness of properties which were common to the various sense modalities. These resembled the attributes of the structural , and included, for example, magnitude, number, form, unity, and motion. (See Marks, 1978 for a review and analysis of this concept.) Focus on this aspect of perception draws our attention away from the separate sense modalities and permits us to consider the common information that we gain about the world. Although the attributes of Aristotle's sensus comunis are fairly abstract, thinking about these common attributes permits one to take seriously an alternative way of understanding perception. This alternative, following Gibson (1966), involves analyzing the way stimu• lation provides information about the real world as we believe it exists. The implications of such an approach for intermodal aspects of perception are that instead of an analysis of separate sense modalities perception is analyzed in terms of the information which is acquired about important aspects of the world, or for various specific purposes whether through one or several of the traditional sense modalities. This is a very functional approach to perception and is brought out in this book most explicitly in Cohen's chapter on visual-proprioceptive interactions. Both of these approaches to intermodal perception and sensory integration generate their own problems and questions, many of which are the topics of chapters in this volume. The classic sense-modality approach is both anatomically and phenomenologically based. Each sense modality has its own receptor organ and nerve system and creates particular qualities of experience. But this immediately leads to the question of specifying the sense modalities. This is easy for audition, vision, odor, and taste, which have obvious receptor organs. But what of touch, proprioception, and kinesthesis? What are the receptor organs? For touch, the skin? But what of deep pressure and somatic perception? For proprioception, are joints, muscle spindles, and tendon organs all to be considered part of the receptor organ? These are the anatomical questions. Phenomenologically, there also seem to be problems of spec• ification. For kinesthesis we sense ourselves being moved when the vestibular apparatus is stimulated in an appropriate way. But we get the same perception from visual stimulation, and in fact in some conflict situations the visual kinesthesis is stronger (Lee & Lishman, 1975). Butterworth's chapter in the present volume describes some similar experiments with children. Introduction xi

Perhaps these questions of specifying sense modalities can be solved. We also want to know how the information of the different sense modalities gets integrated. The first aspect of this question is whether the integration is built into the organism or whether it develops as a function of experience. This nativism-empiricism issue was one of the earliest to interest philosophers and psychologists, and many of the chapters in this volume address it in one way or another. The next aspect of the question how information gets integrated concerns what, specifically, the integration is like. One might consider three scenarios. In the first, each sense modality has an equal status, and stimulation via any particular modality is encoded in a modality-specific form. When it is to be integrated for use with other modality-specific information it is translated by some sort of correspondence rules; yet the information never really loses its modality-specific identity. In the second scenario, there is a hierarchical order of sense modalities for any particular type of information. Irrespective of the modality through which the information is acquired, it gets translated and recorded in a form which is relevant to a particular modality. For example, Pick (1974) made such an analysis with respect to spatial information. The argument was that spatial information was encoded in a visually relevant form no matter what the input modality. See Freides (974) for an extension of this line of reasoning. In the present volume, Samuel applies this sort of analysis in making predictions about individual differences in intermodal relations. Finally, according to the third scenario, information going through any particular modality is translated into an amodal form and is equally available to relate to new information originating in any modality. If the classic sense-modality approach is accepted, it seems possible, in principle, at least to distinguish experimentally among these different scenarios. The last scenario of amodal encoding may bring us close to the second general functional approach discussed above. The main difference would seem to be that in the functional approach the amodal represen• tation of information is taken directly from original stimulation. There is no intermediate modality encoding at all. The functional approach poses its own problems, partially overlapping with those of the classical modality approach. First we have to decide what kinds of information can be and are abstracted amodally from the stimulation. For example, information can be extracted from the optic array for self-movement. This is normally redundant with classical kinesthetic information about self-movement. It is difficult from this point of view to know how the separate sense modalities contribute to a variable of self-movement. From the classical modality point of view the separate modalities can be isolated or placed in conflict. But from the functional point of view such procedures are artificial and may not result in normal information xii Introduction processing. Secondly, the functional approach does not avoid the ana• tomical issue. The question of by what mechanism stimulation is trans• duced from the environment must be solved sometime. Receptors and receptor organs are important at some level. It is possible that the functional approach will direct the attention of those investigating recep• tor mechanisms to somewhat different or additional questions, but the basic questions are still there. Finally, the nativism-empiricism question still remains. Its form is somewhat different. No longer is it asked whether the integrating of information from the separate sense modalities is built into the organism or whether it is built up from experience. But a new question is posed as to whether the extraction of amodal variables from various sources and combinations of sources of stimulation occurs as a function of maturation or experience. The functional orientation toward intermodal processing of infor• mation emphasizes perception in the service of some purpose. Very often this purpose is action. Action is an important locus for the study of intermodal processes because it is always accompanied by stimulation via a number of the classical modalities. In most ordinary acts we receive visual, proprioceptive, tactual, and kinesthetic feedback stimulation, and of course have knowledge of efferent innervation. How is all of this integrated? A number of chapters in this volume address issues relevant to this question. The reader will find the issues raised in the foregoing analysis implicit in many of the chapters even when they are not made explicit. The book is divided into three sections. In the first section intermodal relations are analyzed with a focus on perceptual development. (Bushnell and Butterworth are concerned primarily with infants, and Jones and Abravanel with older children.) The nativism-empiricism issue is a central one in the chapters in this section, and the reader will note that there is still a lively controversy as to whether intermodal relations are best thought of as going from a primitive unity to differentiated modalities or whether they are better understood as starting from separate and distinct modalities, being integrated as a function of deVelopment. The second section includes chapters on intermodal relations in adults. It is heavily oriented toward intermodal relations in sensorimotor activities but also includes intermodal aspects of rhythm perception (Fraisse). The final section focuses on intermodal relations with respect to special popUlations. The research discussed in these chapters contributes toward a general understanding of intermodal relations and at the same time helps in explaining the behavior of those particular groups: the blind (O'Connor & Hermelin; Millar), deaf and retarded children (O'Connor & Hermelin), and various types of athletes (Samuel). Motor aspects of intermodal relations are also included in this section in Cratty's analysis of the motor-training programs for children with learning difficulties. Introduction xiii

References

Friedes, D. Human information processing and sensory modality: Cross-modal functions, information complexity, memory, and deficit. Psychological Bul/etin, 1974, 81, 284-310. Gibson, J.J. The senses considered as perceptual systems. New York: Houghton Mifflin, 1966. Lee, D.N., & Lishman, J.R. Visual proprioceptive control of stance. Journal of Human Movement Studies, 1975,1, 87-95. Marks, L.E. The unity of the senses: Interrelations among the modalities. New York: Academic Press, 1978. Pick, H.L., Jr. The visual coding of non-visual spatial infonnation. In R.B. MacLeod & H.L. Pick, Jr. (Eds.), Perception: Essays in honor of James J. Gibson. Ithaca: Cornell University Press, 1974. Contents

Introduction...... ix

PART I • Intersensory Perception and Sensory Integration in Children

Introduction ......

1· The Ontogeny of Intermodal Relations: Vision and Touch in Infancy Emily W. Bushnell

1. Introduction...... 5 2. The Neonate...... 7 3. Equivalence of Location...... 11 4. Equivalence of Perceptual Features ...... 18 5. Equivalence of "Actions" ...... 25 6. Differentiation...... 30 7. Summary...... 32 8. References...... 34

2· The Origins of Auditory- and Visual Proprioception in Human Development George Butterworth

1. Introduction: The Genetic Method ...... 37 2. Some Theoretical and Evolutionary Considerations...... 38 3. The Origins of Auditory-Visual Perception...... 41 3.1. Piaget's Theory...... 42 3.2. Oculomotor Responses to Sound in the Neonate. . . .. 43 3.3. The Primitive-Unity Hypothesis ...... 45 3.4. Detection of Intermodal Equivalence...... 49

xv xvi Contents

4. Summary and Conclusions: Auditory-Visual Coordination ...... 54 5. Visual Proprioception...... 55 5.1. Theories of Visual-Motor Coordination...... 55 5.2. Reaching for Virtual Objects ...... 57 5.3. Neonatal Responses to Looming ...... 59 5.4. Visual Proprioception and Posture ...... 60 6. General Conclusions ...... 65 7. References...... 66

3· Integrating the Information from Eyes and Hands: A Developmental Account Eugene Abravanel

1. Introduction...... 71 2. Undifferentiated Perception and the Unity of the Senses ... 74 3. Studies for Intersensory Coordination during Infancy...... 75 3.1. Looming...... 76 3.2. Haptic-Visual Transfer and Matching of Shape by Infants...... 77 3.3. Evidence from Early Imitation...... 79 4. Postinfancy Developmental Research ...... 83 4.1. Processing Shape versus Length ...... 85 5. Perceptual Activity and Exploration ...... 87 5.1. Perceptual Activity When Perceiving Object Shape .. 90 5.2. Perceptual Activity When Perceiving Length...... 94 5.3. Perceptual Activity When Perceiving Complex Visual Displays ...... 97 5.4. Summary ...... 101 6. Conclusions ...... 102 7. References...... 105

4· The Developmental Significance of Cross-Modal Matching Bill Jones

1. Introduction...... 109 2. Birch and Lefford's Account of Cross-Modal Development. III 3. Visual-Tactual Cross-Modal Matching ...... 115 3.1. Methodological Considerations ...... 116 3.2. A Theory of Visual-Tactual Matching ...... 118 3.3. Visual-Tactual Matching of Form ...... 118 3.4. Cross-Modal Matching of Texture ...... 122 Contents xvii

4. Visual and Nonvisual Judgments of Length ...... 124 5. Visual-Auditory Matching ...... 126 6. Conclusions...... 131 7. References...... 132

PART II • Higher-Order Integration Introduction ...... 137 References...... 141

5· Some Aspects of Sensory-Motor Control and Adaptation in Man James R. Lackner

1. Introduction ...... 143 2. Adaptation to Sensory Rearrangement ...... 144 3. Some Limitations on the Nature of Possible Compensatory Changes in Sensory-Motor Control ...... 145 4. Conditions for Adaptation to Sensory Rearrangement ...... 147 5. Factors Influencing the Form of Adaptation...... 153 5.1. Incremental Exposure to Sensory Rearrangement .... 155 5.2. Influence of Exposure Condition on Adaptation ..... 156 5.3. Adaptational Specificity ...... 159 5.4. Intersensory Factors and Adaptation: Constancies of Auditory and Visual Direction ...... 160 5.5. Adaptation to Coriolis and Cross-Coupled Angular Accelerations ...... 162 6. Intersensory Interactions ...... 162 6.1. Z-Axis Recumbent Rotation: Importance of Touch and Pressure Cues ...... 163 6.2. Skeletal Muscle Vibration: Illusory Postural and Visual Motion ...... 166 7. Conclusions ...... 169 8. References ...... 170

6· Visual-Proprioceptive Interactions Malcolm M. Cohen

1. Introduction ...... 175 1.1. Scope of Chapter ...... 175 1.2. Contents of Chapter ...... 175 2. Classical Views of Proprioception ...... 176 2.1. Proprioceptive and Exteroceptive Fields...... 176 xviii Contents

2.2. Anatomical and Physiological Considerations...... 177 2.3. Cooperative Functioning of Proprioceptive and Visual Receptors ...... 188 3. Modern Views of Proprioception ...... 189 3.1. Historical Perspective ...... 189 3.2. Contemporary Views...... 191 4. Mechanisms Subserving Visual Direction and Orientation .. 192 4.1. Background ...... 192 4.2. Inflow Mechanisms ...... 192 4.3. Outflow Mechanisms ...... 193 4.4. Possible Hybrid Mechanisms ...... 195 5. Experimental Modification of Visual-Proprioceptive Relationships...... 196 5.1. Methodological Considerations ...... 1% 5.2. Visual and Proprioceptive Illusions in Altered Gravitational-Inertial Fields ...... 198 6. References...... 211

7· Multisensory Aspects of Rhythm Paul Fraisse

1. Introduction ...... 217 1.1. Rhythm in Antiquity ...... 217 1.2. Main Characteristics of Rhythms ...... 218 2. Rhythms in the Different Sensory Modalities ...... 220 2.1. Rhythm as Perception of the Succession ...... 220 2.2. Rhythmic Groupings ...... 221 2.3. The Spontaneous Tempo ...... 222 2.4. The Ontogenesis of Rhythms ...... 223 3. Auditory and Visual Rhythms ...... 224 3.1. The Differences ...... 224 3.2. Resemblances ...... 226 4. The Sensory-Motor Aspect of Rhythms ...... 227 4.1. Auditory-Motor Relations ...... 227 4.2. Visuomotor Relations ...... 229 4.3. Sensory Cues in Synchronization ...... 230 4.4. The Role of the Nervous Centers ...... 232 4.5. Rhythm and Affectivity ...... 236 5. Rhythmic Cues in Poetry and Music ...... 237 5.1. Music ...... 237 5.2. Poetry ...... 238 5.3. The Human Voice ...... 239 5.4. The Principal Psychological Characteristics ...... 242 Contents xix

6. Rhythms in Space ...... 243 6.1. The Structures ...... 243 6.2. Simultaneity and Succession ...... 244 7. References...... 245

8· Gait Perception as an Example of How We May Perceive Events James E. Cutting and Dennis R. Proffitt

1. Time, Movement, and Their Place in Event Structure ...... 249 2. Gait Perception without Familiarity Cues ...... 252 2.1. A Search for a Biomechanical Invariant ...... 256 2.2. An Invariant Found: A Center of Moment ...... 258 2.3. Gait Synthesis ...... 260 3. Perceiving Centers of Moment in Other Events ...... 261 3.1. Perceiving Centers of Moment in Wheel-Generated Motions ...... 261 3.2. Perceiving Centers of Moment in Aging Faces ...... 262 3.3. Perceiving Centers of Moment in Flowfields ...... 263 3.4. Perceiving Centers of Moment in the Night Sky ..... 263 4. Toward a Grammar for Perceptual Events ...... 264 5. Summary ...... 270 6. References ...... 271

PART III • Sensory Integration in Special Populations

Introduction ...... 275 References...... 279

9· Cross-Modal and Intersensory Perception and the Blind Susanna Millar

1. Introduction...... 28 I 1.1. Implications of Crossmodal Models for Blind Performance ...... 282 1.2. Some Methodological Considerations in Assessing Blind Performance ...... 286 2. Shape Recognition ...... 288 2.1. Three-Dimensional Shapes in Touch and Vision ..... 288 2.2. Touch and Nanling ...... 290 2.3. Names, Shapes, and Physical Features in Tactual Recognition ...... 292 xx Contents

2.4. Tactual Coding and Information from Other Sources. 295 2.5. Shapes, Names, and Familiarity: An Explanation .... 296 3. Spatial Judgments...... 2Cf7 3.1. Movement and Vision in Length Judgments ...... 2Cf7 3.2. Spatial Location and Direction ...... 301 3.3. Sound, Vision, and Movement in Large-Scale Space. 305 4. Conclusions...... 307 5. References ...... 311

10· Coding Strategies of Normal and Handicapped Children N. O'Connor and B. Hermelin

1. Introduction...... 315 2. Modality-Specific Memory Systems and Sensory Integration ...... 317 3. Temporal and Spatial Coordinates ...... 319 4. Modality-Specific Processing ...... 324 5. Representational Coding ...... 329 6. Phonological Coding ...... 335 7. Interpretation...... 338 8. References ...... 340

11· Sensory-Motor and Perceptual-Motor Theories and Practices: An Overview and Evaluation Bryant J. Cratty

1. Introduction...... 345 2. Theoretical Issues and Child Development...... 348 3. Sensory- and Perceptual-Motor Programs and Changes in Movement Capacities...... 349 4. Perceptual-Motor Programs and Visual and Visual- Perceptual Changes ...... 354 5. Sensory Integration and Neurological Organization ...... 358 6. Summary...... 368 7. References...... 369

12· Individual Differences in the Interaction of Vision and Proprioception Jacqueline M. F. Samuel

1. Introduction ...... 375 2. Developmental Changes in Modality Preference ...... 377 Contents xxi

3. Research with the Blind ...... 381 3.1. Form Perception ...... 381 3.2. Spatial Perception ...... 383 4. Prism Adaptation and Visual Capture ...... 385 4.1. Visual Capture ...... 386 5. Rod-and-Frame Experiments ...... 387 5.1. Visual Skills ...... 388 5.2. Body Skills ...... 389 5.3. Individual Differences and Sports ...... 390 6. Artists and Athletes ...... 391 7. Conclusion...... 394 8. References ...... 394

Author Index...... 399

Subject Index...... 409