Mental Rotation - a Case of Embodied Action

Andreas Wohlschläger

Max-Planck-Institute for Psychological Research Unit Cognition and Action Leopoldstraße 24 D-80802 München Germany [email protected]

Abstract movement makes it convenient to call it a mental rotation. A new view of mental rotation in humans is presented. These results demonstrate how analog operations can be Rather than being a perceptual phenomenon, mental rota- performed by our brains and perhaps mental object rotation of objects is supposed to be an imagined action in the tion is performed by similar changes in the activity pat- sense that its only difference to real action is the absence terns of cell assemblies. Notably, these results were ob- of motor output. A series of experiments is reported which tained by measuring cell activities in the motor cortex. shows that the difference in speed between mental and Cognitive psychologists, so far, mainly have tried to manual rotation are negligible and that performing rotational hand movements interferes with mental rotation and explain the analog nature of mental rotation by looking vice versa. It also could be shown that the preparation of for the relation between mental rotation and the percep- rotational hand movements is already sufficient to influ- tion of rotary motion. Shepard & Judd (1976) addressed ence mental rotation. The general role of motor processing this question by investigating apparent rotational motion in dynamic visual imagery is discussed, considering the and its relation to mental rotation. However, mental rota- underlying neurophysiology. tion speed was about 50 to 60°/s, whereas the slope of the optimum apparent motion function calculates to 1000°/s. Mental Rotation, Perception and Action Furthermore, Friedman & Harding (1990) showed that mental rotation speed depends on the axis of rotation, Mental rotation occurs when a decision has to be made whereas the apparent motion illusion does not. They con- whether two objects differing in orientation are identical cluded that mental rotation and apparent motion don't or mirror-images of each other. The angular disparity have much in common. In addition mental rotation is between stimuli is varied systematically and response strategic (Just & Carpenter 1985) compared to the largely times (RT) and errors are measured. The typical, most automatic processing of apparent motion perception. intriguing result found in many mental rotation studies is Hence, the consultation of the perception of real motion the almost perfect linear increase of RT with angular as the main explanation for the mental rotation phenome- stimulus disparity. Together with introspective reports non is somewhat unsatisfactory. Mental rotation is obvi- from participants and experimenters, these core findings ously a much higher-level process than the perception of led to the term "mental rotation", since it resembles the real or apparent motion. time course of a physical rotation with constant angular Nevertheless, there have been further, more direct at- velocity. tempts to prove the participation of rotary movement This analog nature of the mental rotation process, most perception in mental rotation. Corballis & McLaren ( impressively demonstrated by Cooper (1976) piqued psy- 1982) could show, that inducing a rotary aftereffect by chologists' interest. The resemblance of mental rotation to means of a rotating textured disk influenced RT of the external physical rotation, however, demands for a mental mental rotation of subsequently presented alphanumeric process mimicking external physical rotation, and it is characters. Compared to the standard experiment, RT still not clear how such an analog process is implemented were increased when the aftereffect was in the opposite in our brains. Recent electrophysiological studies could direction (discordant condition) of the presumed mental measure continuous changes in the activity pattern of cell rotation. However, Jolicoeur & Cavanagh (1992) could assemblies in monkeys performing a visuomotor mental exclude the participation of low-level motion analysis rotation task (Georgopoulos et al. 1989). They found that centers in the mental rotation process. Although finding the neuronal population vector - calculated from the cell slight differences in overall RT level when presenting assemblies' activity pattern - continuously changed its alphanumeric stimuli in different surface media, there was direction prior to the onset of a movement pointing 90° to no effect of surface medium on mental rotation rate. A the left of a target light. The fact, that the neuronal popu- pronounced effect was only produced when rotating the lation vector rotated prior and not in parallel to the characters about a small angle by means of apparent mo tion. Concordant rotation of presented characters acceler- individuals the opportunity to turn the stimuli on the dis- ated RT of mental rotation, whereas rotations opposite play by means of a knob fixed to an axis. (discordant) to the presumed mental rotation direction led Subjects' mean correct RT as well as the direction and to delayed responses, compared to a neutral condition. amount of manual rotation were measured and analyzed. Obviously, figure motion interacts to some degree with Similar to standard mental rotation experiments, in both, mental rotation. Summarizing their results, Jolicoeur & the mental and the manual rotation condition, RT in- Cavanagh (1992) concluded that mental rotation occurs at creased linearly with the amount of angular disparity a relatively high and perhaps abstract level of processing. between stimuli. Speed of manual as well as speed of This agrees with a principal difference between motion mental rotation depended on the axis of rotation. Manual perception and mental rotation. Whereas motion percep- and mental rotation functions matched each other with the tion is a rather automatic process, mental rotation is stra- exception of the oblique B-axis, running upward and away tegic and shares some characteristics with voluntary ac- from the subject. Because the B-axis differed from the tions. Mental rotation can be started and stopped volun- Cartesian axes in two other aspects, its results will be tarily (Cooper 1976) and even its speed is at free choice reported and discussed later on. When comparing both (Cooper & Shepard 1973). The processes engaged in rotation conditions, speeds of manual and mental rotation motion perception might well participate in imagining were not different for any of the three Cartesian axes. The rotating objects, but there should be a higher process same rank order of speeds among these axes was found as steering these dynamic imaginations, because they are in Parsons's (1987) experiment, and it was identical for quasi completely under voluntary control. Considering its the manual and mental rotation condition.. Thus, it can be similarity to voluntary actions, it might be possible that concluded that, at least for Cartesian axes, mental and motor or premotor processes are involved in mental rota- manual rotation are similar processes in a phenomenal tion, a conclusion which Kosslyn (1994) meanwhile sense. draws complementary to his earlier work. More precisely, What about the oblique B-axis? It differed in two as- this means that the processes engaged in rotary object pects from the three Cartesian axes. First, manual rotation manipulation might contribute to mental rotation. In this speed for the oblique B-axis was about twice as fast as view, rotating something mentally is rather an imagined mental rotation speed. Second, the threshold function for action than the perception-like imagination of an object in the shortest direction of rotation was substantially differ- rotation. Let us assume that rotating an object mentally is ent from those of the other axes. For trials with 120° something very similar to rotating it physically (e.g., with through 240° stimulus orientation students rotated in one's hand) in the sense that the same processes initiate either direction with equal frequency. Obviously, indi- and rule the rotation, whether it turns up mentally or it is viduals in the B-axis condition often could not figure out actually performed. This "common-processing" assump- the shortest direction. This is probably due to a general tion implies essentially two things. First, mental rotation inability to conceive the position of the axis and the angle should be commensurate with rotary object manipulation: of shortest path rotation between two orientations of an factors that affect mental rotation should have the same object, when there is no coincidence of the object's princi- effect on actual rotation. Second, both processes should pal axes, the rotation axis, and the viewer axis (Parsons, be functionally connected, i.e., depend on each other. 1995). The results of Experiment 1 show that in mental This could be investigated by having both tasks executed rotation shortest path rotation is not used in the case of the simultaneously, i.e., performing mental rotation while oblique B-axes. Otherwise the participants in the manual making rotational hand movements. rotation condition should have been able to figure out the shortest direction of rotation more accurately. Regarding the correspondence of manual and mental Experiment 1 rotation observed with the Cartesian axes, there is good Experiment 1 was designed to test whether rotary object reason to expect correspondence for oblique axes too, manipulation is commensurate with mental rotation. This when using a device which does not force participants to was tested by a comparison of the mental rotation RT use single axis/shortest path rotations in the manual rota- function with the RT required, if in the same task, the tion condition. However first, alternative explanations for object could be manipulated physically. It was expected, the observed correspondence of mental and manual rota- that the differences between both functions would be of tion must be excluded. negligible amount. Corroboration can be acquired by showing the commensurability for different levels of diffi- Experiment 2 culty. Stimulus conditions which slow down mental rotation should also increase operation times of manual rota- In order to exclude alternative explanations, additionally it tion. The rotation about different axes in space, known to must be shown that both, manual and mental rotation are elicit different speeds of mental rotation (Parsons 1987), structurally connected. If either task falls back on the same was used to realize these various levels of difficulty. In structure, they should interfere when executed simultane- order to make mental and manual rotation conditions as ously. The basic idea of Experiment 2 was to investigate similar as possible, manual rotation was realized by giving the influence of rotational movements of the right hand on the performance of mental rotation. The fact that, as 5. Z-LEFT. Should it be crucial that the movement is shown above, an object is predominantly rotated in the performed with the dominant right hand, the Z-LEFT direction of the shortest angular path was used to investi- condition should fail to elicit an interference effect. The gate potential interference effects. Rotating a knob in a Z-LEFT condition was identical to the Z-RIGHT condi- certain direction (as indicated by an arrow presented prior tion, except that the rotary movement about the Z-axis was executed with the left hand. to each trial) should lead to an inhibition of mental rotation in trials on which the direction of the shortest angle is Following this sequence of conditions, it will be possible in opposition (discordant trials). On the other hand, facili- to determine the necessary features that cause interference tation should occur if mental and manual rotation direc- and thus to specify the level of motor processing at which tion are identical (concordant trials). Rotations about the interference occurs. Having determined the level of motor Z-axis, pointing to depth were used in this experiment. processing, it would be interesting to show the reverse The standard findings of mental rotation experiments interference, i.e. an interference of mental rotation with were replicated in Experiment 2. RT increased with angu- motor performance. This possibility was ignored in Ex- lar disparity. RT-functions clearly showed an interference periment 2. Therefore, in the present experiment, the effect of rotational hand movements on mental rotation. movements exerted on the knob were recorded for further RT for concordant trials were on average about 380 ms analyses. If mental and manual rotation processing fall faster than those for discordant trials. back on the same common structure, an interference should also be observed in this reverse sense. The standard findings of mental rotation experiments Experiment 3 were also replicated in Experiment 3. RT increased with On the one hand, Experiment 3 was conducted to replicate angular disparity. The interference effect found in Ex- the findings of Experiment 2. On the other hand, Experi- periment 2 was replicated. It was restricted to the two ment 3 included four different control conditions, which movement conditions involving a rotation of the hand were designed to identify the level of motor processing at about the Z-axis. Under these conditions RT for concor- which an interference of hand movements with mental dant trials again were on average about 380 ms lower than rotation occurs. In order to follow the logic of the present those for discordant trials. The effect was more pro- experiment consider the following features carried of nounced, when the right hand was used (about 460 ms) Experiment 2: (a) Presentation of an arrow: an arrow than when the rotational movement was executed with the pointing either clockwise or counterclockwise was shown left hand (about 300 ms). No such difference was found in prior to each trial; (b) Execution of a directed movement: any of the other conditions. The bare presentation of an a movement had to be executed in the direction indicated arrow was not sufficient to interfere with mental rotation, by the arrow; (c) The movement was rotary; (d) Movement nor was it the execution of a directed translational move- axis and mental rotation axis were parallel: the rotary ment. Rather, interference occurred specifically with rota- movement was performed about the Z-axis, i.e. an axis tional movements, provided that the axes of mental and that was parallel to the axis used to generate the stimuli; manual rotation coincided in space. A rotational move- (e) The dominant right hand was used. ment about the Y-axis - an axis perpendicular to the Five conditions were designed to find out which of the mental rotation axis - did not lead to an interaction be- above features is a necessary condition for eliciting the tween mental and manual rotation. interference effect. The five conditions can be arranged The same pattern of results was found for the reverse logically in the following way: interference effect. Only under the two movement conditions involving the Z-axis an influence of mental rotation 1. NO-MOVEMENT. In order to show that the bare presentation of an arrow is not sufficient to yield interfer- on simultaneous motor performance was found. Occa- ence with mental rotation, a NO-MOVEMENT condition sional transient reversals of the movement direction oc- was established, in which simply an arrow pointing either curred more often when mental and manual rotation di- left or right was shown prior to each trial. rections were opposite. The movement was disrupted 2. TRANSLATION. In the TRANSLATION condition, about twice as often under discordant than under concor- participants had to move their right hand to the right or the dant conditions. Furthermore, in the Z-RIGHT condition left (according to the arrow direction) in a linear way. the rotational movement was executed slower when di- 3. Y-RIGHT. Should it be necessary that the movement rected opposite to the assumed mental rotation than in the is rotary, then any rotary movement should evoke interfer- concordant case. As above, the interference effect was ence with mental rotation. Rotation about a vertical axis more pronounced in the Z-RIGHT than in the Z-LEFT with the right hand (Y-RIGHT condition) was used to test condition, where it was even absent. this. Note that the mental rotation axis is perpendicular to the rotation axis of the movement in this condition. In summary, the symmetric interference of rotational 4. Z-RIGHT. This is the replication of the experimental hand movements with mental rotation found in Experi- condition of Experiment 2. Should it be necessary for the ment 3 provides strong evidence that mental rotation and interference effect that mental rotation and rotary hand manual rotation fall back on the same common structure. movements are executed about parallel axes, then interfer- The interaction between mental and manual rotation was ence should occur only in this and in the following condi- only found, when axes of rotation coincided in space and tion. it was more pronounced when the dominant right hand ment 4 demonstrated that the interference was not due to was used than with rotations of the left hand. the kinesthetic feedback elicited by the execution of hand movements. Thus, the interaction discovered here seems to work at a Experiment 4 relatively high level of motor processing. This level is One could argue, that the interference observed in Ex- specific with respect to the spatial operation which is periment 2 and 3 is not due to the movement component performed but only weakly involves the selection of the of the action but is caused by kinesthetic perception of the hand to be used. This is in parallel to the interaction be- rotation of one's own limb. If this were the case, then the tween the perception of rotational motion and mental ro- above results would reflect an interference of perception tation, which are suggested to use a "shared representa- from one modality with an - perhaps perception-like - tional space [which] must be relatively abstract" (Jolicoeur imagery process of another modality. In order to exclude & Cavanagh 1992, p. 383). Based on the results of the this alternative explanation, in Experiment 4 subjects had present study, one might suggest such a common to prepare rather than execute a rotatory hand movement. representational medium for mental and manual rotation Again they were shown an arrow prior to each trial, which too. Taken together, this would even imply a common now indicated the direction they had to turn the knob to representation of perception, imagery and action. immediately after they had solved the mental rotation Let us now speculate about the brain structures most task. The three conditions TRANSLATION, Z-RIGHT, likely involved in mental object rotation. Mental rotation and Y-RIGHT from Experiment 3 were used in this ex- of visual material surely involves visual areas. Among the periment. visual pathways, processing of the object's location and Results of Experiment 3 were replicated. RT increased orientation - which follows the dorsal route terminating in with angular disparity. The interference effect was not the posterior parietal cortex - should predominate process- observed when preparing a translational movement or a ing of the objects shape and identity, because orientation is rotation about the vertical Y-axis. However, the interfer- the critical feature in mental rotation tasks. Jeannerod ence effect was clearly observed in the Z-RIGHT condition (1994) extended the concept of dissociable processing of involving a rotation of the hand about the Z-axis. Thus, shape and location originally formulated by Ungerleider & again the interaction between mental and manual rotation Mishkin (1982) to a "semantic" vs. "pragmatic" mode of was only found, when axes of rotation coincided in space. representation, respectively. In his terms, the pragmatic More important, however, it was demonstrated that the mode includes, besides locating an object, also aspects of planning of such a movement is sufficient to interfere with object-oriented behavior like grasping and manipulating, mental rotation, and that its execution - and especially the the latter being relevant in manual object rotation. kinesthetic feedback elicited thereby - is not necessary for Jeannerod's view is supported by the fact that in monkeys, the effect to occur. there are neurons in the posterior parietal cortex (area 7a) showing movement dependent properties: these cells re- spond during active manipulation of visible objects, but General Discussion neither to the presentation of the object alone, nor during In Experiment 1 it was shown that mental and manual manipulation of the object in the dark (Taira et al. 1990, rotation RT functions are identical, provided that any of Mountcastle et al. 1975). These neurons might play an the three Cartesian axes was used. Compared to mental important role in both, dynamic visual imagery and ob- rotation, manual rotation speed was much higher in the ject-oriented action, because area 7a shows extensive re- case of an oblique axis for reasons discussed with Experi- ciprocal connections to the supplementary motor area and ment 1. For now, with the restriction to Cartesian axes the premotor cortex which form together Brodmann's area (and probably to conditions where the axis of rotation 6 (Morel & Bullier, 1990). Recently, using functional MRI coincides with one of the object's axes, see Parsons 1995), Cohen et al. (1996) could show an activation of area 6 in mental and manual rotation can be considered as com- half of their subjects during a mental rotation task. I mensurate. Experiment 2 demonstrated that the execution would like to propose that the supplementary motor area of a rotational hand movement, leads to an interference and the posterior parietal cortex closely work together in with a simultaneously performed mental rotation task. RT translating spatial information into motor commands. were considerably higher when mental and manual rota- This would explain why mental rotation could influence tion went in the same direction than in the opposite, dis- motor performance, as shown in Experiment 3. Consider- cordant case. Experiment 3 clarified which conditions are ing the reciprocity of the connections, one could further necessary to yield such an interaction between mental and speculate that the supplementary motor area in turn might manual rotation. The coincidence of the axes of mental modify visuospatial representations in the posterior parie- and manual rotation turned out to be the decisive prere- tal cortex. This could explain the present findings that quisite for the interaction, which was measurable both, in executing a rotational movement interacts with a visual the RT of mental rotation and in the motor performance. imagery process. Particularly, executing an arbitrary rotational movement Since mental rotation is a serial process, probably con- was not sufficient to cause interference. Finally, Experi- sisting of different subprocesses (like identification, deci sion about the direction of rotation, rotation itself, and Corballis, M. C.; and McLaren, R. 1982. Interaction be- decision finding), it is also of interest to investigate the tween perceived and imagined rotation. Journal of Ex- temporal topographic change of brain activity during the perimental Psychology: Human Perception and Perform- ongoing process. It could be shown that overall activity ance 8:215-224. changes from frontal to centro-parietal sites (Rösler et al. Delius, J. D.; and Hollard, V. D. 1995. Orientation invari- 1993) or from frontal over parietal to left temporal loca- ant pattern recognition by Pigeons (Columba livia) and tions if a verbal response was required (Nikolaev 1994). humans (Homo sapiens). Journal of Comparative Psy- The fronto-parietal shift of overall brain activity during chology 109:278-290. the mental rotation process can also be deducted from the Friedman, A.; and Harding, C. A. 1990. Seeing versus detailed analysis of ERPs by Rösler, Schumacher, & Sojka imagining movement in depth. Canadian Journal of Psy- (1990). The temporal change in overall activity from chology 44:371-383. frontal to parietal locations supports the idea that the Georgopoulos, A. P.; Lurito, J. T.; Petrides, M.; Schwartz, supplementary motor area controls the rotation of a mental A. B.; and Massey, J. T. 1989. Mental rotation of the image. As soon as the mental image sufficiently ap- neuronal population vector. Science 243:234-236. proaches the angular orientation of the comparison Jeannerod, M. 1994. The representing brain: Neural corre- stimulus, manipulation of the image (respectively the real lates of motor intention and imagery. Behavioral and object) should stop and activation of the supplementary Brain Sciences 17:187-202. motor area should fade. In order to make an iden- Jolicoeur, P.; and Cavanagh, P. 1992. Mental rotation, tity/difference judgment, however, the aligned image still physical rotation, and surface media. Journal of Experi- should be maintained for a while in the parietal areas. mental Psychology: Human Perception and Performance The results of the experiments reported here and the 18:371-384. role of motor processing in dynamic visual imagery sup- Just, M. A.; and Carpenter, P. A. 1985. Cognitive coordi- port the notion that mental rotation is a case of embodied nate systems: accounts of mental rotation and individual action. According to Varela, Thompson, & Rosch (1991), differences in spatial ability. Psychological Review this view means that "cognition depends on the kinds of 92:137-172. experience that come from having a body with various Kosslyn, S. M. 1994. Image and brain. Cambridge, MA: sensorimotor capacities, and ... that sensory and motor MIT Press. processes, perception and action, are fundamentally in- Morel, A.; and Bullier, J. 1990. Anatomical segregation of separable in lived cognition. ... the two are not merely two cortical visual pathways in the maquaque monkey. contingently linked in individuals; they have evolved Visual Neuroscience 4:555-578. together" (p. 173). This explains, why the mental rotation Mountcastle, V. B.; Lynch, J. C.; Georgopoulos, A. P.; effect is species-specific. Pigeons do not show a depend- Sakata, H.; and Acuna, C. 1975. Posterior parietal asso- ency of RT on angular stimulus disparity in mental rota- ciation cortex of the monkey: Command functions for tion experiments (Delius & Hollard 1995). Lacking hands, operations within extrapersonal space. Journal of Neuro- they never make the experience continuous object rotation physiology 38:871-908. caused by their own action. Nikolaev, A. R. 1994. [Study of the stages of mental rotation of complicated figures by the intracortical interaction mapping]. Zhurnal Vyssej Nervnoj Deyatelnosti 44:441- Acknowledgments 447. Part of this working note is based on a paper submitted to Parsons, L. M. 1987. Visual discrimintation of abstract the Journal of Experimental Psychology: Human mirror-reflected three-dimensional objects at many orien- Perception and Performance. tations. Perception and Psychophysics 42:49-59. Parsons, L. M. 1995. Inability to reason about an object's orientation using an axis and angle of rotation. Journal of References Experimental Psychology: Human Perception and Per- Cohen, M. S.; Kosslyn, S. M.; Breiter, H. C.; DiGirolamo, formance 21:1259-1277. W. L.; Thompson, W. L.; Anderson, A. K.; Bookheimer, Rösler, F.; Schumacher, G.; and Sojka, B. 1990. What the S. Y.; Rosen, B. R.; and Belliveau, J. W. 1996. Changes brain reveals when it thinks: Event-related potentials in cortical activity during mental rotation. A mapping during mental rotation and mental arithmetic. German study using functional MRI. Brain 119:89-100. Journal of Psychology 14:185-203. Cooper, L. A. 1976. Demonstration of a mental analog of Rösler, F.; Röder, B.; Heil, M.; and Hennighausen, E. an external rotation. Perception and Psychophysics 1993. Topographic differences of slow event-related brain 19:296-302. potentials in blind and sighted adult human subjects dur- Cooper, L. A.; and Shepard, R. N. 1973. Chronometric ing haptic mental rotation. Cognitive Brain Research studies of the rotation of mental images. In Chase, W. G. 1:145-159. ed. Visual information processing. New York: Academic Shepard, R. N.; and Judd, S. A. 1976. Perceptual illusion Press. of rotation of three-dimensional objects. Science 191:952- 954. Taira, M.; Mine, D.; Georgopoulos, A. P.; Murata, A.; and Sakata, H. 1990. Parietal cortex neurons of the monkey related to the visual guidance of hand movements. Experimental Brain Research 83:29-36. Ungerleider, L. G. & Mishkin, M. 1982. Two cortical visual systems. In Ingle, D. J.; Goodale, M. A.; and Mansfield, R. J. W. eds. The analysis of visual behavior. Cambridge, MA: MIT Press. Varela, F. J., Thompson, E., Rosch, E. 1991. The Embod- ied Mind: Cognitive Science and Human Experience. Cambridge, MA: MIT Press.