Roe, A.W. (2019). Columnar connectome: towards a mathematics of brain function. Network Neuroscience. Advance publication. https://doi.org/10.1162/netn_a_00088 1 Columnar connectome: towards a mathematics of brain function
2
1,2 3 Anna Wang Roe
4
1 5 Institute of Interdisciplinary Neuroscience & Technology,
6 Zhejiang University, Hangzhou, China
2 7 Oregon National Primate Research Center, OHSU, Portland, OR, USA
8
9 Keywords: primate, cerebral cortex, functional networks, functional tract tracing, matrix
10 mapping, brain theory, artificial intelligence
11 Correspondence: Anna Wang Roe ([email protected])
12 Pages: 9
13 Words: 3880
14 Figures: 5
15 References: 83
16
17 Acknowledgments
18 This research was conducted at Zhejiang University, and was supported by National Natural
19 Science Foundation Grants 81430010 and 31627802 (A.W.R.), National Hi-Tech Research
20 and Development Program Grant 2015AA020515 (to A.W.R.), and NIH NS093998 (A.W.R).
21 Thanks to Charles Gilbert and Akshay Edathodathil for useful discussions.
22
23 Competing Interests: The author declares she has no competing interests.
24 25 Abstract
26
27 Understanding brain networks is important for many fields, including neuroscience,
28 psychology, medicine, and artificial intelligence. To address this fundamental need, there are
29 multiple ongoing connectome projects in the US, Europe, and Asia producing brain
30 connection maps with resolutions at macro-, meso-, and micro-scales. This viewpoint focuses
31 on the mesoscale connectome (the columnar connectome). Here, I summarize the need for
32 such a connectome, a method for achieving such data rapidly on a largescale, and a proposal
33 about how one might use such data to achieve a mathematics of brain function.
34
35
36
2
37 Understanding brain networks is important for many fields, including neuroscience,
38 psychology, medicine, and artificial intelligence. To address this fundamental need, there are
39 multiple ongoing connectome projects in the US, Europe, and Asia producing brain
40 connection maps with resolutions at macro-scale (e.g. Human Connectome Project) and
41 micro-scales (e.g. Brainsmatics, Gong et al 2016). However, still lacking is a meso-scale
42 connectome. This viewpoint (1) explains the need for a mesoscale connectome in the primate
43 brain, (2) presents a new method for studying mesoscale connectivity, and (3) proposes a
44 mathematical approach to representing the mesoscale connectome.
45
46 The cortical column is a modular processing unit. In humans and nonhuman primates
47 (NHPs), the cerebral cortex occupies a large proportion of the brain volume. This remarkable
48 structure is highly organized. Anatomically, it is a two-dimensional (2D) sheet, roughly two
49 millimeters in thickness, and divided into different cortical areas, each specializing in some
50 aspect of sensory, motor, cognitive, and limbic function. There is a large literature, especially
51 from studies of NHP visual cortex, to support the view that the cerebral cortex is composed
52 of submillimeter modular functional units, termed ‘columns’ (Mountcastle 1997). Columns
53 span the 2-mm thickness of cortex and are characterized by 6 input/output layers (laminae)
54 linked together via inter-laminar circuits (Figure 1). The tens of thousands of neurons within
55 a single column are not functionally identical but share common functional preference such
56 that single stimuli maximally activate the population and produce a coherent columnar
57 response. These coherent responses can be visualized using multiple methods, including
58 electrophysiology (e.g. Hubel and Wiesel 1977, Mountcastle 1997, Katzner et al 2009), 2-
59 deoxyglucose (e.g. Tootell et al 1988), optical imaging (e.g. Blasdel and Salama 1986,
60 Grinvald et al 1986), and high spatial resolution fMRI methods (e.g. Cheng 2012, Nasr et al
61 2016, Li et al 2019). More in depth and scholarly articles about the definition and existence
3
62 of the column are available (e.g Horton and Adams 2005, Rakic 2008, Ts’o et al 2009, da
63 Costa and Martin 2010, Rockland 2010).
64 In non-visual cortical areas, data on columnar organization is more limited (DeFelipe et
65 al 1986, Lund et al 1993, Krizter and Goldman-Rakic 1995, Friedman et al 2004, Gharbawie
66 et al 2014). However, there is accumulating evidence, as well as compelling genetic and
67 developmental (Rakic 1988, Torii 2009, Li et al 2012), and computational reasons (Swindale
68 2004, Schwalger et al 2017, Berkowitz and Sharpee 2018) to believe that columnar
69 organization may be a fundamental feature throughout cortex. Borrowing from Pasko Rakic
70 (2008): “The neurons within a given column are stereotypically interconnected in the vertical
71 dimension, share extrinsic connectivity, and hence act as basic functional units subserving a
72 set of common static and dynamic cortical operations that include not only sensory and motor
73 areas but also association areas subserving the highest cognitive functions.” For the
74 purposes of this viewpoint, the term ‘column’ refers to a unit of information integration and
75 functional specificity.
76
77 Why a columnar connectome is needed. Columns come in different flavors and have very
78 specific connections with other columns. For example (Figure 2), in primary visual cortex (V1,
79 dotted lines divide V1, V2, and V4), different functional columns focus on visual features such
80 as eye specificity (ocular dominance columns, Fig 2B), color (blobs; Fig 1C: dark dots in V1
81 are color ‘blobs’, red dot overlies a blob), and orientation (orientation columns; Fig 1D dark
82 and light domains in V1, yellow dot overlies a horizontal orientation domain). In the second
83 visual area (V2), columns within the thin stripe (green dots) and thick/pale stripe (blue dots)
84 types integrate columnar information from V1, to generate higher order parameters of color
85 (thin stripes: hue), form (thick/pale stripes: cue-independent orientation response), and depth
86 (thick stripes: near to far binocular disparity) (for review, Roe et al 2009). Columns in V4 are
4
87 hypothesized to perform further abstractions such as color constancy (Kusunoki et al 2006),
88 invariance of shape position and size (Rust & DiCarlo 2010, Sharpee et al 2012), and relative
89 (vs absolute) depth (Shiozaki et al 2012, Fang et al 2018) (for review, Roe et al 2012).
90 A key aspect of cortical columns is their highly specific connections with other columns
91 (Figure 2CD, red and yellow arrows). This has been demonstrated from studies using focal
92 injections of tracer targeted to single columns. Such studies have revealed sets of patchy
93 connections, both intra-areal (Figure 1, inset) and inter-areal (Figure 2, arrows) (e.g.
94 Livingstone and Hubel 1984, Sincich and Horton 2005, Shmuel et al 2005, Federer et al 2013).
95 Column-specific connection patterns thus embody a functionally specific (e.g. orientation or
96 color) network. However, to date, due to the demanding nature of these experiments, there
97 are only a small number of such studies. Thus far, there has not been a method that permits
98 systematic largescale study of columnar connectivity. In fact, over 40 years after Hubel and
99 Wiesel’s (1977) description of the organization of functional columns in V1, little is known
100 about the organization of cortical connectivity at the columnar level. I propose that we extend
101 the concept of the hypercolumn (all the machinery required to represent a single point in
102 space) to the connectional hypercolumn (all the connections of that unit of representation).
103
104 The primary limitation of current methods is (1) lack of spatial resolution. Most anatomical
105 mapping methods employ tracer injections 2-5 mm in size. Human connectomes are based
106 on resting state or diffusion methods which typically are mapped at 2-3 mm voxel resolution.
107 These volumes (white rectangle in Figure 2E) encompass multiple columns and therefore
108 reveal connections of a population of multiple functionally distinct columns. Since individual
109 nearby columns can exhibit quite distinct connectivity patterns (e.g. Fig 2CD: color blobs to
110 thin stripes vs orientation columns to pale/thick stripes), connections arising from such
111 averages are inaccurate and misleading (Figure 2E). (2) Slow and expensive. Traditional
5
112 anatomical tract tracing typically requires 2-3 weeks for tracer transport, animal sacrifice to
113 acquire tissue, and time-consuming weeks to map label locations and 3D reconstruction. (3)
114 Not largescale. Anatomical studies are limited to several tracers and therefore the
115 connections of only a handful of nodes can be studied in any single brain. Other methods
116 such as electrophysiological stimulation with fMRI mapping has elegantly revealed networks
117 underlying specific behaviors (e.g. Tolias et al 2005); but electrical methods can suffer from
118 current spread, leading to lack of spatial specificity, as well as inability to map local
119 connections due to signal dropout near the electrode. Optogenetic stimulation with fMRI
120 mapping is a powerful cell type specific approach (e.g. Gerits et al 2012); however, in
121 primates, it takes weeks for viral expression and has thus far been limited by the small
122 number of transfected nodes, making largescale mapping of connections in the primate brain
123 challenging. (4) Correlation based functional connectivity. fMRI BOLD signal correlation
124 (resting state studies) non-invasively probes networks in human and animal brains, but are
125 limited to inference about correlation rather than connectivity. Such limitations also exist with
126 neurophysiological cross correlation studies of spike timing coincidence.
127
128 A new mapping method. To overcome some of these limitations, we have developed a new
129 rapid in vivo mapping technique. This method combines an optical stimulation method,
130 termed pulsed infrared neural stimulation (INS), with high field fMRI (Xu et al 2019a). INS is
131 a method that uses pulsed trains of light (1875nm) to induce heat transients in tissue (Wells
132 et al 2005, Cayce et al 2014, Chernov and Roe 2014a). While the mechanism underlying INS
133 is still under study, the leading theory is that the heat transients lead to membrane
134 capacitance change, which leads to induction of neuronal firing. When INS is delivered with
135 a fine fiber optic (e.g. 200 µm diameter) to the cerebral cortical surface, the light distribution
136 is highly focal (roughly the same diameter as the fiber optic) and roughly ~300 µm in
6
137 penetration depth; this penetration reaches cells in the superficial layers (2/3) and apical
138 dendrites of pyramidal cells in deep layers (5/6). Such neuronal activation, in turn, is
139 propagated to downstream neurons at connected sites. Similar activation can be achieved
140 by inserting the fiber optic into deep brain sites (Xu et al 2019b). The resulting BOLD
141 responses associated with this activation constitutes a map of cortico-cortical connectivity
142 arising from a single site. For example, when INS is applied to area 17 with fine 200 µm optic
143 fibers in a highfield MRI, INS evokes focal activations in areas 18 and 19, the ipsilateral LGN,
144 and contralateral 17/18 (Xu et al 2019a). This activation is robust, intensity-dependent, and
145 safe for long-term use (Chernov et al 2014b). One can also use high resolution mapping to
146 examine BOLD activation in different cortical lamina. This reveals local connections that
147 distinguish between feedforward (layer 4 activation) and feedback (superficial and deep
148 laminae) connections (Xu et al 2019a). Importantly, as no animal sacrifice is required, for
149 large dataset collection, the brain can be probed systematically with fiber optic bundles
150 applied to windows on the brain, across multiple sessions. It is also compatible with other
151 imaging, electrophysiological, and behavioral methods for multimodal dataset correlation
152 (Chernov et al 2014a, Xu et al 2019a).
153
154 Potential impacts of columnar mapping
155
156 Common motifs? The longstanding notion that cortical columns perform common
157 functional transformations would be further bolstered by the presence of columnar
158 connectivity motifs. The existence of motifs would suggest that there are indeed common
159 modes of information distribution and integration. The true value of such motifs is the
160 possibility of identifying a small set of general motifs to characterize cortical function (Figure
161 3). This would provide a basis vectors for constructing biological representation of information.
7
162 One could wonder why it matters what the underlying anatomical motif is. For some
163 computational neuroscientists, it matters not what the brain circuit is, as long as a circuit can
164 achieve the desired functional output. However, it is a fact that our brain contains organized
165 anatomical constructs and these constructs perform some pretty sophisticated functions that
166 so far have been difficult to mimic with other architectures. While the brain may not be the
167 only architecture capable of intelligent behavior, perhaps it is one optimal architecture given
168 existing biological and physical constraints.
169 Below, I present two possible motifs: an intra-areal motif (Motif1, binding) and an inter-
170 areal motif (Motif2, transformation). The basis for these motifs comes from prevous studies
171 of mesoscale anatomical connections.
172
173 Motif1: Intra-areal networks (binding). As shown in Figure 3, many cortical areas contain
174 striking local ‘star-like’ topologies in which a central column is linked to nearby columns of
175 similar functional preference. The columns within these networks are typically 200-300 µms
176 in size (red circles) (Amir et al 1993). A few examples from the literature illustrate the ubiquity
177 of these constructs; areas include visual (a-c), sensorimotor (d-e), parietal (f), and prefrontal
178 (g) areas. In V1, injection of tracer into an orientation column labels nearby columns of similar
179 orientation preference, thereby forming an orientation selective network (Figure 3a top).
180 Color blobs in V1 also form star-like networks (Figure 3a bottom). In V2, color, form, and
181 disparity columns lie within separate functional streams but are linked via inter-stripe
182 connections to form single multi-feature networks (Figure 3b). Similar networks are seen in
183 temporal cortex (Figure 3c). In area 3b of somatosensory cortex, columns subserving digits
184 D1 to D5 are linked in inter-digit tip networks, and are hypothesized to underlie sensory co-
185 activation during power grasp (Figure 3d). Other star-like networks are observed in primary
186 motor (Figure 3e), posterior parietal (Figure 3f), and prefrontal (Figure 3g) areas.
8
187 The central node integrates inputs from a subset of surrounding nodes, thereby performing
188 a ‘binding’ function. In some areas, nodes of similar functionality (e.g. V1, Ts’o et al 1986) are
189 linked, while in others nodes bind different modalities into a coherent multimodal
190 representation (e.g. orientation, color, and disparity via interstripe connections in V2, Levitt et
191 al 1994). Note that opposing networks of complementary function are similarly ‘bound’ via
192 inhibitory relationships. Together, these interdigitated networks underlie push-pull functions
193 (e.g. white vs outlined domains in Figure 3g, Cayce et al 2014, Chernov et al 2018; cf.
194 Sawaguchi 1994, Weliky et al 1995, Sato et al 1996, Toth et al 1996).
195 The specific characteristics of these networks (such as overall size, number of nodes, axis,
196 anisotropy, Figure 3h) may be tailored to the functional/computational purpose of each area
197 and may be influenced by more global factors, such as size of the cortical area or by the
198 number of other linked cortical areas. These possibilities remain to be explored and tested
199 via mesoscale connectome data.
200
201 Motif2. Inter-areal networks (transformation). This motif captures the anatomical
202 connections underlying transformations from one cortical area to another. It is well known
203 that representation becomes increasingly abstract with cortical hierarchy. However, it is
204 unknown whether the connections from one area to the next that mediate functional
205 transformations are in any way systematic or standard. If so, one might be able to systematize
206 the transformations into functional classes, potentially reducing all cortico-cortical projections
207 into a small set of functions. In addition to providing a mathematical way to represent the
208 brain concisely, it would provide insight into the function of areas for which there is as yet little
209 understanding.
210 The idea that there are common functional transformations derive from studies in early
211 sensory cortical areas. Vision: From studies that simultaneously monitor responses of V2 and
9
212 V1 function to single ‘illusory’ stimuli, we have shown that neurons in V2 domains respond to
213 the illusory aspect, while neurons in V1 respond to the ‘real’ aspect (Roe 2009 for review).
214 This ‘real-to-illusory' higher order transformation must be mediated by the anatomical
215 connectivity between V1 and V2. Specifically, we observe establishment of modality-specific
216 higher order properties in different stripes of V2: (1) thin stripes: color representation in V1
217 blobs, which is dominated by red-green/blue-yellow axes, transforms to a multi-color map of
218 hue columns in V2 (Conway 2001, Xiao et al 2003), (2) thin stripes: luminance encoding in
219 V1 transforms to brightness encoding in V2 (Roe et al 2005), (3) thick/pale stripes: encoding
220 of simple contour orientation transforms to higher order cue-invariant orientation
221 representation in V2 (Rasch et al 2012), (4) thick stripes: simple motion direction detection
222 transforms to the detection of coherent motion in V2 (useful for figure-ground segregation,
223 Peterhans and von der Heydt) and to motion contrast defined borders (Hu et al 2018), and
224 (5) thick stripes: segregated representation of left and right eyes in V1 to maps of near-to-far
225 binocular disparity columns in V2 (Chen et al 2008, 2017). Touch: In a similar vein, in
226 somatosensory cortex, integration of tactile pressure domains in area 3b (Friedman et al 2004)
227 are hypothesized to generate motion selectivity domains in area 1 (Pei et al 2010, Wang et
228 al 2013, Roe et al 2017). These modality-specific transformations could be achieved by
229 integrating across multiple unimodal inputs in V1 and in S1.
230 Remarkably, common anatomical motifs underlie these functional computations. Injection
231 of tracer into V2 labels multiple columns in V1 (Figure 4a). These motifs are observed
232 following injections into thin stripes (Figure 4b red), pale stripes (Figure 4b blue), and thick
233 stripes (Figure 4b grays). Similarly, injection of tracer into a single digit location in area 1
234 labels multiple (presumed) columns in area 3a (Figure 4cd). Although it remains to be seen
235 whether such proposed transformations are also found in other sensory, motor, and cognitive
236 systems, identification of such common motifs (along with characteristic integration size,
10
237 number, functional type) would be important for generating an understanding of how the
238 anatomy of brain connections leads to and limits brain function.
239
240 Some thoughts about mathematical formulation
241 There is a very large gap between what is known about cortico-cortical connection
242 patterns and what we need to know to guide concepts about mappings in mathematical terms.
243 Ultimately we should like to know what are the organizational principles underlying cortical
244 connection patterns. The work of investigators such as Nick Obermayer (1993), Geoff
245 Goodhill (2000), Carreira-Perpiñán et al (2005), Swindale (2004, 2007), and others have
246 beautifully demonstrated that the arrangement of multiple maps within an area arises largely
247 through maximizing parameters of continuity and coverage. Similar constraints need to be
248 identified and modelled for cortico-cortical connectivity. Progress on this front will be greatly
249 aided by availability of data on columnar connectivity. There are numerous efforts to
250 characterize connectomes using mathematical topology (cf Sporns 2011). However, greater
251 attention needs to be focused at the columnar scale.
252 Here, I propose one possible view of modelling cortical connections. Although
253 representation of some parameters is continuous (e.g. continuously shifting orientation
254 preference), a ‘binned’ representation is valid from the view of functional organization; that is,
255 continuity gets broken up into columnar representations by the constraint of mapping multiple
256 parameters within a single sheet (Obermayer 1993, Swindale 2004).
257
258 Matrix mapping. If one views the cortical sheet as a 2D array of columns, then
259 connections between cortical areas can be viewed as 2D-to-2D mappings (Figure 5A, red
260 outlined portion). The challenge to characterize connectional motifs may then be expressed
261 as identifying generalized 2D-2D matrix mappings that govern how two cortical areas connect
11
262 (Figure 5A, Fij). For any pair of areas, such mappings would be constrained by anatomical
263 and functional constraints such as continuity (topography) and functional specificity. One
264 additional constraint to be considered might include the number of areas with which a single
265 area directly connects (typically this is a small number). Once an anatomical scaffold is
266 constructed, column-specific feedforward and feedback modulation could be implemented to
267 model circuit dynamics. From such treatment of connectivity, general patterns of mappings
268 (motifs) may emerge. Although the problem may, at first glance appear formidable, in my view,
269 constraints of anatomical architecture dictate that each cortical area maps only a few (e.g. 3-
270 4) key parameters in a continuous and complete fashion. I make an argument below that this
271 matrix representation may help reduce the complexity.
272
273 Computational Feasibility: A few back of the envelope estimates suggest that this
274 mapping problem may be computationally manageable (Figure 5B). These estimates are
275 based on published neuroanatomical studies and inferred order-of-magnitude calculations.
276 We use 200 µm as the dimension of a cortical column in macaque monkey (size of orientation
2 277 domains and blobs in V1); each mm then contains 25 columns. The macaque neocortex has
2 278 an area of about 10,000-15,000mm , roughly a third of which is visual cortex (cf. Van Essen
279 et al 1984, Purves and LaMantia 1993, Sincich et al 2003). This means that there are on the
4 4 280 order of 25x10 columns total and about (a third) 8x10 columns in visual cortex. [Note: In
281 humans, although visual cortex is 3 times as large, due to larger column size (roughly double),
282 there are a comparable number of columns (Horton & Hedley-Whyte 1984, Adams et al
283 2007).] Out of this total 80,000 columns, V1, V2, and V4 occupy roughly 20%, 20%, and 10%
284 of visual cortical area, or 15,000, 15,000, and 7,500 columns, respectively (Weller et al 1983,
285 Sincich et al 2003). [This is likely an underestimate as it assumes square columnar packing.]
286 If all columns in V1 mapped to all columns in V2, the number of single node-to-single node
12
8 287 connections would be on the order of 10 (15,000x15,000) (Figure 5B). From
288 neuroanatomical studies, we estimate that a single node in V1 connects to on the order of 10
289 nodes in V2 (Livingstone and Hubel 1984, Federer et al 2013, Sincich et al 2010). This small
290 number is due to (1) constraints of topography: a single point in visual space takes up roughly
2 291 2 mm of cortical space (Hubel and Wiesel 1977), or 4x25=100 columns (10 ) and (2)
292 constraints of functional specificity (Fig 4), which further reduces this number by a fifth or a
293 tenth the number of nodes from a single topographic point. As an example, if a full 180deg
294 cycle of orientations is represented in one mm distance (Hubel et al 1978) or 5 (200um-sized)
2 295 columns, then within a 1 mm area there may be 5 out of 25 columns that represent a single
296 orientation). This reduces the total number of target nodes from 100 to 10-20 or on the order
8 297 of 10. This potentially reduces a problem that is on the order of 10 to 10. Such a 1:10
298 convergence/divergence could underlie a basic architectural motif of inter-areal connectivity,
299 or an ‘inter-areal connectional hypercolumn’.
300 Such connectional hypercolumns may be replicated at a higher level, albeit with distinct
301 convergence/divergence ratios and functional constraints. For example, from V2 to V4, the
7 302 number of connections decreases (15,000x7500, or 10 ), producing a correspondingly
303 smaller cortical area (V4 is roughly half the size of V2). An area such as MT which is only 5%
304 the size of V1, is likely to have a greater convergence/divergence ratio from V1 to MT; this
305 would be consistent with the large receptive field sizes and comparatively spatially broad
306 integrations for computation of motion direction. Thus, inter-areal connectivity would be
307 specified by parameters such as unique topographical constraints, convergence/divergence
308 ratios, functional selectivities. This set of relationships would be represented as a set of
309 connectional hypercolumns: F1 (x1…xm) … Fn(x1…xm). Solving how multiple connectional
310 hypercolumns mutually constrain the entire set of mappings will lead to a representation of
311 total cortical connectivity in a brain.
13
312
313 Columnar organization constrains the global network. Thus far, topological treatment of
314 brain networks has been based on parameters such as connection number, connectional
315 distance, routing efficiency (e.g. Young 1993, Avena-Koenigsberger et al 2019, Chaudhuri et
316 al 2015). These approaches have provided important advances in our understanding of brain
317 networks and information flow at an areal level (typically with each cortical area treated as a
318 single node). However, what is lacking is the concept of spatial location within each area.
319 That is, the X,Y coordinate in the matrix defines a node’s topographical location and its
320 functionality. Each node’s position is not independent of another nodes position; neighbors
321 constrain neighbors and this impacts the organization of the global network. Studies that
322 address issues of multiscale constraints (e.g. Jeub et al 2018) could incorporate such spatial
323 information to further specify the resulting architecture.
324
325 Conclusion
326 To summarize, I have described a column-based view of the cerebral cortex and
327 presented a new methodology (INS-fMRI) to map column-based networks in vivo. I suggest
328 that acquisition of such largescale connectivity data may demand new ways of representing
329 cortical networks (e.g. 2D-2D matrix transformations). From such data, patterns or motifs of
330 cortical connectivity may emerge, and give rise to basic connectivity units termed
331 ‘connectional hypercolumns’.
332 My goal in this viewpoint is to encourage the connectomics field to capture columnar
333 connectivity. New representations and mathematics need to be developed for
334 multidimensional treatment of nodes, one that incorporates the spatial and functional
335 relationships between neighboring nodes. Such representations may simplify the apparently
336 complex connectional relationships in the global network. While we do not yet have enough
14
337 columnar data to do this on a large scale, one could start by using available data to generate
338 prototype solutions. I list a few questions here to motivate future studies. For a given cortical
339 area, how does the number of directly connected areas affect motif architecture? How do the
340 number of total areas affect the total possible mappings? Is there a general solution to the
341 mappings of smaller brains with fewer cortical areas vs larger brains with many? What
342 aspects of cortical architecture produces and simultaneously constrains our behavioral
343 repertoire? Can one design mappings to generate alternative artificial intelligences?
344 Ultimately, I envision a general connectional theory of brain function, complete with a
345 system of theorems, derivations, and corollaries (Sporns et al 2000, Sporns 2011). Such a
346 rule-based representation will lead to new understandings of brain construction and brain
347 evolution, and will inform our understanding of biological intelligence as well as bio-inspired
348 artificial intelligence.
349
15
350 References
351 Adams DL, Sincich LC, Horton JC (2007) Complete pattern of ocular dominance columns in
352 human primary visual cortex. J Neurosci. 27(39):10391-403.
353 Amir Y, Harel M, Malach R (1993) Cortical hierarchy reflected in the organization of intrinsic
354 connections in macaque monkey visual cortex. J Comp Neurol 334(1):19-46.
355 Avena-Koenigsberger A, Yan X, Kolchinsky A, van den Heuvel M, Hagmann P, Sporns O
356 (2019) A spectrum of routing strategies for brain networks. PLoS Comput Biol.
357 15(3):e1006833.
358 Blasdel GG, Salama G. (1986) Orientation selectivity, preference, and continuity in monkey
359 striate cortex. Nature. 321(6070):579-85.
360 John Berkowitz, Tatyana Sharpee (2019) Decoding neural responses with minimal information
361 loss. doi: https://doi.org/10.1101/273854
362 Carreira-Perpiñán MA, Lister RJ, Goodhill GJ (2005) A computational model for the
363 development of multiple maps in primary visual cortex. Cereb Cortex. 15(8):1222-33.
364 Cayce J, Friedman RM, Jansen D, Mahadevan-Jansen A, Roe AW (2014) Infrared neural
365 stimulation of primary visual cortex in non-human primates. Neuroimage, 84:181-190.
366 Chaudhuri R, Knoblauch K, Gariel MA, Kennedy H, Wang XJ (2015) A Large-Scale Circuit
367 Mechanism for Hierarchical Dynamical Processing in the Primate Cortex.
368 Neuron, 88(2):419-31.
369 Chen G, Lu HD, Roe AW (2008) A map of horizontal disparity in primate V2. Neuron, 58:442-
370 450.
371 Chen G, Lu HD, Tanigawa H, Roe AW (2017) Solving visual correspondence between the two
372 eyes via domain-based population encoding in nonhuman primates. Proc Natl Acad Sci
373 114(49):13024-13029.
16
374 Cheng K. (2012) Revealing human ocular dominance columns using high-resolution
375 functional magnetic resonance imaging. Neuroimage 62(2):1029-34.
376 Chernov M, Roe AW (2014a) Infrared neural stimulation: a new stimulation tool for CNS
377 applications. Neurophotonics 1(1):011011.
378 Chernov M, Roe AW (2014b) Histological assessment of thermal damage in the brain
379 following infrared neural stimulation. Brain Stimulation 7(3): 476–482.
380 Chernov M, Friedman RM, Chen G, Roe AW (2018) Functionally specific optogenetic
381 modulation in primate visual cortex. Proc Natl Acad Sci. 115(41):10505-10510.
382 Conway BR (2001) Spatial structure of cone inputs to color cells in alert macaque primary
383 visual cortex (V-1). J Neurosci. 21(8):2768-83.
384 da Costa NM, Martin KA (2010) Whose Cortical Column Would that Be? Front Neuroanat.
385 4:16.
386 DeFelipe J, Conley M, Jones EG (1986) Long-range focal collateralization of axons arising
387 from corticocortical cells in monkey sensory-motor cortex. J Neurosci. 6(12):3749-66.
388 Fang Y, Chen M, Xu H, Li P, Han C, Hu J, Zhu S, Ma H, Lu HD (2018) An Orientation Map
389 for Disparity-Defined Edges in Area V4. Cereb Cortex. 29(2):666-679 .
390 Federer F, Williams D, Ichida JM, Merlin S, Angelucci A (2013) Two projection streams from
391 Macaque V1 to the pale cytochrome oxidase stripes of V2. J Neuroscience, 33(28):11530
392 –1153.
393 Friedman RM, Chen LM, Roe AW (2004) Modality maps within primate somatosensory cortex.
394 Proc Natl Acad Sci USA 101:12724-12729.
395 Gerits A, Farivar R, Rosen BR, Wald LL, Boyden ES, Vanduffel W (2012) Optogenetically
396 induced behavioral and functional network changes in primates. Curr Biol. 22(18):1722-6.
17
397 Gharbawie O, Friedman RM, Roe AW (2014) Localization of grasp domains in frontal and
398 parietal cortex with intrinsic optical imaging in behaving monkeys. Soc Neuroci Meeting,
399 Washington DC.
400 Gong H, Xu D, Yuan J, Li X, Guo C, Peng J, Li Y, Schwarz LA, Li A, Hu B, Xiong B, Sun
401 Q, Zhang, Liu J, Zhong Q, Xu T, Zeng S, Luo Q. (2016) High-throughput dual-
402 colour precision imaging for brain-wide connectome with cytoarchitectoniclandmarks at
403 the cellular level. Nat Commun. 7:12142.
404 Goodhill G, Cimponeriu A (2000) Analysis of the elastic net model applied to the formation of
405 ocular dominance and orientation columns. Network 11(2):153-68.
406 Grinvald A, Lieke E, Frostig RD, Gilbert CD, Wiesel TN. (1986) Functional architecture of
407 cortex revealed by optical imaging of intrinsic signals. Nature. 324(6095):361-4.
408 Horton JC, Hedley-Whyte ET. (1984) Mapping of cytochrome oxidase patches and ocular
409 dominance columns in human visual cortex. Philos Trans R Soc Lond B Biol Sci.
410 304(1119):255-72.
411 Horton JC, Adams DL (2005) The cortical column: a structure without a function. Philos Trans
412 R Soc Lond B Biol Sci. 360(1456):837-62.
413 Hubel DH, Wiesel TN (1977) Ferrier lecture. Functional architecture of macaque monkey
414 visual cortex. Proc R Soc Lond B Biol Sci. 198(1130):1-59.
415 Jeub LGS, Sporns O, Fortunato S (2018) Multiresolution Consensus Clustering in Networks.
416 Sci Rep. 8(1):3259.
417 Katzner S, Nauhaus I, Benucci A, Bonin V, Ringach DL, Carandini M (2009) Local origin of
418 field potentials in visual cortex. Neuron. 61(1):35-41.
419 Kritzer MF, Goldman-Rakic PS (1995) Intrinsic circuit organization of the major layers and
420 sublayers of the dorsolateral prefrontal cortex in the rhesus monkey. J Comp Neurol
421 359:131–143.
18
422 Kusunoki, M., Moutoussis, K., and Zeki, S. (2006). Effect of background colors on the tuning
423 of color-selective cells in monkey area V4. J. Neurophysiol. 95, 3047–3059.
424 Levitt JB, Yoshioka T, Lund JS (1994) Intrinsic cortical connections in macaque visual area
425 V2: evidence for interaction between different functional streams. J Comp Neurol.
426 342(4):551-70.
427 Li X, Zhu Q, Janssens T, Arsenault JT, Vanduffel W. (2019) In Vivo Identification of Thick,
428 Thin, and Pale Stripes of Macaque Area V2 Using Submillimeter Resolution (f)MRI at 3T.
429 Cereb Cortex. 29(2):544-560.
430 Li Y, Lu H, Cheng PL, Ge S, Xu H, Shi SH, Dan Y (2012) Clonally related
431 visual cortical neurons show similar stimulus feature selectivity. Nature.486(7401):118-21.
432 Livingstone, M. S., & Hubel, D. H. (1984). Anatomy and physiology of a color system in the
433 primate visual cortex. Journal of Neuroscience, 4, 309-356.
434 Lund JS, Yoshioka T, Levitt JB (1993) Comparison of intrinsic connectivity in different areas
435 of macaque monkey cerebral cortex. Cereb Cortex 3(2):148–162
436 Malach R, Amir Y, Harel M, Grinvald A. (1993) Relationship between intrinsic connections
437 and functional architecture revealed by optical imaging and in vivo targeted biocytin
438 injections in primate striate cortex. Proc. Natl. Acad. Sci. U. S. A. 90, 10469-10473.
439 Mountcastle VB (1997) The columnar organization of the neocortex. Brain. 120 (Pt 4):701-
440 22. Review.
441 Nasr S, Polimeni JR, Tootell RB (2016) Interdigitated Color- and Disparity-Selective Columns
442 within Human Visual Cortical Areas V2 and V3. J Neurosci. 36(6):1841-57.
443 Obermayer K, Blasdel GG. Geometry of orientation and ocular dominance columns in monkey
444 striate cortex. J Neurosci. 1993 Oct;13(10):4114-29. 445 Pálfi E, Zalányi L, Ashaber M, Palmer C, Kántor O, Roe AW, Friedman RM, Négyessy L. 446 (2018) Connectivity of neuronal populations within and between areas of primate
19
447 somatosensory cortex. Brain Struct Funct. 223(6):2949-2971.
448 Pei YC, Hsiao SS, Craig JC, Bensmaia SJ (2010) Shape invariant coding of motion direction
449 in somatosensory cortex. PLoS Biol. 8(2):e1000305.
450 Purves D, LaMantia A (1993) Development of blobs in the visual cortex of macaques. J Comp
451 Neurol. 334(2):169-75.
452 Rakic P (1988) Specification of cerebral cortical areas. Science. 241 (4862): 170–6.
453 Rakic P (2008) Confusing cortical columns. Proc Natl Acad Sci U S A. 105(34):12099-100.
454 Rasch MJ, Chen M, Wu S, Lu HD, Roe AW (2012) Quantitative inference of population
455 response properties across eccentricity from motion-induced maps in macaque V1. J
456 Neurophysiol, 109(5):1233-49.
457 Rockland KS (2010) Five points on columns. Front Neuroanat. 4:22.
458 Roe AW, Lu H, Hung CP (2005) Cortical processing of a brightness illusion. Proc Natl Acad
459 Sci USA 102:3869-3874.
460 Roe AW, Lu HD, Chen G (2009) Visual System: Functional architecture of Area V2. The
461 Encyclopedia of Neuroscience (Squire L, ed.), Vol 10, pp. 331-349. Elsevier, Oxford, UK.
462 Roe AW, Chelazzi L, Connor CE, Conway BR, Fujita I, Gallant J, Lu H, Vanduffel W (2012)
463 Towards a unified theory of visual area V4. Neuron, 74(2):12-29.
464 Roe AW, Winberry J, Friedman RM (2017) Study of single and multidigit activation in monkey
465 SI using voltage sensitive dye imaging. Neurophotonics, 4(3):031219.
466 Rust NC, Dicarlo JJ (2010) Selectivity and tolerance ("invariance") both increase as visual
467 information propagates from cortical area V4 to IT. J Neurosci.30(39):12978-95.
468 Sato H, Katsuyama N, Tamura H, Hata Y, Tsumoto T (1996) Mechanisms underlying
469 orientation selectivity of neurons in the primary visual cortex of the macaque. J Physiol
470 494:757–771.
20
471 Sawaguchi T (1994) Modular activation and suppression of neocortical activity in the monkey
472 revealed by optical imaging. NeuroReport 6:185–189.
473 Schwalger T, Deger M, Gerstner W (2017) Towards a theory of cortical columns: From spiking
474 neurons to interacting neural populations of finite size. PLoS Comput Biol. 13(4):e1005507.
475 Sharpee TO, Kouh M, Reynolds JH (2013) Trade-off between curvature tuning and position
476 invariance in visual area V4. Proc Natl Acad Sci U S A. 110(28):11618-23.
477 Shiozaki HM, Tanabe S, Doi T, Fujita I. Neural activity in cortical area V4 underlies fine
478 disparity discrimination. J Neurosci. 32(11):3830-41.
1 479 Shmuel A , Korman M, Sterkin A, Harel M, Ullman S, Malach R, Grinvald A. (2005)
480 Retinotopic axis specificity and selective clustering of feedback projections from V2 to V1
481 in the owl monkey. J Neurosci. 25(8):2117-31.
482 Sincich LC, Blasdel GG (2001) Oriented Axon Projections in Primary Visual Cortex of the
483 Monkey. J Neuroscience, 21(12):4416–4426.
484 Sincich LC, Adams DL, Horton JC (2003) Complete flatmounting of the macaque cerebral
485 cortex. Vis Neurosci. 20(6):663-86.
486 Sincich LC, Horton JC (2005) Input to V2 Thin Stripes Arises from V1 Cytochrome Oxidase
487 Patches. J Neuroscience, 25(44):10087–10093.
488 Sporns O, Tononi G, Edelman GM. (2000) Theoretical neuroanatomy: relating anatomical
489 and functional connectivity in graphs and cortical connection matrices. Cereb Cortex.
490 10(2):127-41.
491 Sporns O (2011) Networks of the Brain. MIT Press, Cambridge.
492 Stepniewska I, Cerkevich CM, Kaas JH (2015) Cortical Connections of the Caudal Portion of
493 Posterior Parietal Cortex in Prosimian Galagos. Cereb Cortex 26(6):2753-77.
494 Swindale NV. (2004) How different feature spaces may be represented in cortical maps.
495 Network. 2004 Nov;15(4):217-42.
21
496 Swindale NV (2007) A model for the thick, thin and pale stripe organization of primate V2.
497 Network. 18(4):327-42.
498 Tanigawa H, Wang Q, Fujita I (2005) Organization of horizontal axons in the inferior temporal
499 cortex and primary visual cortex of the macaque monkey. Cereb Cortex. 15(12):1887-99.
500 Tolias AS, Sultan F, Augath M, Oeltermann A, Tehovnik EJ, Schiller PH, Logothetis NK (2005)
501 Mapping cortical activity elicited with electrical microstimulation using FMRI in the
502 macaque. Neuron. 48(6):901-11.
503 Tootell RB, Silverman MS, Hamilton SL, De Valois RL, Switkes E (1988) Functional anatomy
504 of macaque striate cortex. III. Color. J Neurosci. 8(5):1569-93.
505 Torii, M; Hashimoto-Torii, K; Levitt, P; Rakic, P (2009). "Integration of neuronal clones in the
506 radial cortical columns by EphA and ephrin-A signalling". Nature. 461 (7263): 524–8.
507 Toth LJ, Rao SC, Kim DS, Somers D, Sur M (1996) Subthreshold facilitation and suppression
508 in primary visual cortex revealed by intrinsic signal imaging. Proc Natl Acad Sci USA
509 93:9869–9874.
510 Ts'o DY, Gilbert CD, Wiesel TN (1986) Relationships between horizontal interactions and
511 functional architecture in cat striate cortex as revealed by cross-correlation analysis. J
512 Neurosci. 6(4):1160-70.
513 Ts'o DY, Zarella M, Burkitt G. (2009) Whither the hypercolumn? J Physiol. 587(Pt 12):2791-
514 805.
515 Van Essen DC, Newsome WT, Maunsell JH (1984) The visual field representation in striate
516 cortex of the macaque monkey: asymmetries, anisotropies, and individual variability. Vision
517 Res. 24(5):429-48.
518 Wang Z, Negyessy L, Chen LM, Friedman RM, John Gore, Roe AW (2013) The relationship
519 of anatomical and functional connectivity to resting state connectivity in primate
520 somatosensory cortex. Neuron, 78(6):1116-26.
22
521 Weliky M, Kandler K, Fitzpatrick D, Katz LC (1995) Patterns of excitation and inhibition
522 evoked by horizontal connections in visual cortex share a common relationship to
523 orientation columns. Neuron 15:541–552.
524 Weller RE, Kaas JH 1983) Retinotopic patterns of connections of area 17 with visual areas
525 V-II and MT in macaque monkeys. J Comp Neurol. 220(3):253-79.
526 Wells J, Kao C, Mariappan K, Albea J, Jansen ED, Konrad P, et al. (2005) Optical stimulation
527 of neural tissue in vivo. Opt Lett 30(5):504e6.
528 Xiao Y, Wang Y, Felleman DJ (2003) A spatially organizaed representation of color in
529 macaque cortical area V2. Nature. 421(6922):535-9.
530 Xu AG, Qian MZ, Tian FY, Xu B, Friedman RM, Wang JB, Gao Y, Li P, Song XM, Sun Y,
531 Chernov M, Cayce J, Janson ED, Mahadevan-Jansen A, Zhang XT, Chen G, Roe AW
532 (2018) Focal infrared neural stimulation with high-field functional MRI: a rapid way to map
533 mesoscale brain connectomes. Science Advances, in press.
534 Xu AG, Rui YY, Wang JB, Song XM, Gothard K, Roe AW. (2019) INS-fMRI reveals functional
535 connections of amygdalar subnuclei in the Macaque monkey. SPIE Photonics West. San
536 Francisco, CA.
537 Young MP (1993) The organization of neural systems in the primate cerebral cortex. Proc
538 Biol Sci. 252(133 3):13-8.
539
540
23
541 Figures
542
543 Figure 1. The canonical cortical column and its connections (based on nonhuman primate
544 cortex). A cortical column (*) connects to several other nearby columns (gray ovals
545 represent tops of other columns) via local horizontal connections in layers 2/3 (red
546 arrows). Blue dots: neurons. Black arrow: inputs from thalamic or cortical sites. Green
547 arrows: outputs to subcortical sites (from layers 5/6) or other cortical sites (layers 2/3).
548 Yellow arrows: Vertical circuitry between the 6 laminae. Bottom left inset (adapted from
549 Sincich & Blasdel 2001): Star-like arrangement of connections between columns within
550 a local network (top down view from surface of cortex). Labeled orientation columns are
551 roughly 200-300 um in size (one circled in red) have orientation preference similar to
24
552 the injected column. This local anatomical network embodies the concept of orientation
553 selectivity.
554
555 Figure 2. Columnar organization of connections in visual cortex. (A) Cortical surface and
556 vasculature of visual areas V1, V2, and V4 in macaque monkey brain. (B) When monkey
557 views a color (red/green isoluminant) grating on a monitor, this optical image obtained
558 by a CCD camera reveals activated color columns (V1: dark ‘blobs’; V2: dark ‘thin
559 stripes’ indicated by green dots; V4: domains in ‘color band’ indicated by red oval). (C)
560 When monkey views an achromatic grating, orientation columns are revealed (vertical
561 minus horizontal gratings). (V1: orientation columns; V2: orientation domains in
562 ‘thick/pale’ stripes indicated by blue dots; V4: domains in ‘orientation band’ indicated by
563 yellow oval). Arrows are schematics of connectivity between columns in V1, V2, and V4.
564 Red arrows: connectivity between blobs in V1, thin stripes in V2, and color bands in V4.
565 Yellow arrows: connectivity between orientation domains in V1, thick/pale stripes in V2,
566 and orientation bands in V4. (for review see Roe et al 2012) (D) Yellow column projects
567 to Area X and Red column projects to Area Y. If connections are traced via a large
568 anatomical tracer injection or large voxel (represented by white box), the result will show,
569 incorrectly, that each of the yellow and red columns project to both Area X and Area Y.
570 [Note: for simplicity, feedback connections, e.g. from V4 to V2 and from V2 to V1, are
571 not depicted.]
572
573
25
574
575 Figure 3. Examples of Intra-areal connection motifs. Motif1: Intra-areal circuits serve binding
576 function and display ‘star-like’ pattern (cartoon at top). This is exemplified by anatomical
577 tract tracing. a: V1, top: Sincich & Blasdel 2001, bottom: Livingstone & Hubel 1984. b:
578 V2, Malach et al 1993. c: inferotemporal TE, Tanigawa et al 2005. d: S1, Palfi et al 2018,
579 Wang et al 2013. e: motor M1, Gharbawie et al 2014. f: parietal PPC, Stepniewska et
580 al 2015. g: prefrontal PFC, Sawaguchi 1994. h: Motif parameters. Labeled patches are
581 200μm in size (red circles).
26
582
583 Figure 4. Examples of Inter-areal connection motifs. Motif2: Inter-areal circuits serve to
584 transform representation by integration of inputs from multiple columns. (a) Image of
585 labeled blobs in V1 following injection of tracer in V2 (Sincich and Horton 2005). Red
586 arrows: interblob inputs from V1 converge onto a single pale stripe in V2. Vertical dotted
587 line: V1/V2 border. (b) This motif is observed from (red) V1 blobs to V2 thin stripes,
588 (blue) V1 interblobs to V2 pale/thick stripes, and (grays) from V1 ocular dominance
589 columns to V2 thick stripes. (c,d) Red arrows: Converging digit tip inputs from Area 3b
590 to Area 1 in somatosensory cortex (Wang et al 2013).
27
591
592 Figure 5. 2D-to-2D matrix representation of cortico-cortical mapping. (A) Each cortical area
593 (rectangle) is represented as a 2D array of columns (circles). Red outlined portion:
594 mapping between a single pair of areas. This 2D-2D mapping can be expressed as a
595 matrix transformation (below) with identified constraints such as topography and
596 functional selectivity. The hope is that one can reduce the mappings to a small number
597 of motifs (Fij). (B) Some rough numbers used to suggest that computation is
598 manageable.
28