PROCESSES in BIOLOGICAL VISION: Including

PROCESSES IN BIOLOGICAL VISION: including,

ELECTROCHEMISTRY OF THE NEURON

This material is excerpted from the full β-version of the text. The final printed version will be more concise due to further editing and economical constraints. A Table of Contents and an index are located at the end of this paper.

James T. Fulton Vision Concepts [email protected]

7 Dynamics of Vision 1

The dynamics of the visual process have not been assembled and presented in a cogent manner within the academic vision literature. On the other hand, several authors have presented cogent descriptions applicable to the clinical level. The material assembled by Salmon at Northeastern State University2 in Oklahoma is exemplary (but superficial for the purpose at hand). The dynamics associated with the mechanism of interpreting symbols and character groups, called reading, has not been presented at all. Only the major eye movements related to reading have been studied in significant detail. Even the adaptation characteristic of vision as a function of illumination level has not been presented from a theoretical perspective, and the empirical data has not been analyzed in sufficient detail to provide a coherent understanding of the process. When assembled as a group, the mechanisms and processes associated with forming the chromophores of vision provide a new, interesting and unique perspective on the formation of those chromophores. This Chapter will assemble the pertinent data with respect to a variety of processes where their dynamic aspects are crucial to the visual function. 7.1 Characteristics & Dynamics of Retinoids in the body

The following material is based on extensive empirical investigations that were largely lacking with regard to any contiguous theory of what the goals of the mechanisms involved were from the perspective of the visual modality. The interpretations provided here take advantage of the hypothesis of this work; that the family of retinines, known as the Rhodonines() are the fundamental chromophores of animal vision. As noted in Section 3.6, these chromophores do not require chemical combination with an opsin to form the operational receptor of light. No opsin is found associated with the chromophores of Insecta or Mollusca. Opsin is a structural substrate in Chordata supporting the physical orientation of the chromophore coating the individual opsin discs. 7.1.1 Introduction

This section will develop the dynamics of the retinoids from the perspective of transport through the blood stream and operation within the retina. Sections 4.6.2 & 4.6.3 developed the schematic aspects of the transport mechanisms related to both the transport and operation of the retinoids. A later chapter will develop the temporal characteristics associated with these processes as they relate to perception in vision. The complete absorption, transport and metabolism of the retinoids in the body are well beyond the scope of this work. However, a brief, first level, scenario is required to avoid the “floating model” trap and to interpret the known data concerning the retinoids in vision properly. The goal of this chapter is to present one such overall model and scenario. 7.1.1.1 Overall Baseline

Vitamin A plays a major role in the animal body. The term retinol (the alcohol) and vitamin A are frequently considered synonymous in the clinical and biological literature. However, at a more detailed level, the term Vitamin A should probably be considered a synonym for retinoid that may be present in the alcohol, aldehyde or acid form. This modification allows these three forms to target different elements within the body and/or provides greater selectivity among the elements of the body for acquiring the desired chemical form. In 1994, Blomhoff edited an important volume on Vitamin A in Health & Disease3. On page 2 of this very important work, he cited the 1982 IUPAC–IUB Joint Commission on Biochemical Nomenclature, “Vitamin A is a generic term reserved to designate any compound possessing the biological activity of retinol, whereas the term retinoids includes both naturally occurring forms of vitamin A and the many synthetic analogs of retinol, with or without biological activity .” In this work, the term retinoids will be subdivided into two pertinent groups, the retinenes (non-resonant retinoids) and the retinines (resonant retinoids consisting of two oxygen atoms terminating a conjugated carbon chain). The retinines exhibit a spectral absorption that is non-isotropic and maximum along the axis connecting the two oxygen atoms. This resonant absorption is at a wavelength different from the isotropic absorption of the non resonant retinoids.

1Released: April 30, 2017 2Salmon, T. (2012) Vision Science IIIb: Binocular Vision http://arapaho.nsuok.edu/~salmonto/vs3.html 3Blomhoff, R. ed. (1994) Vitamin A in Health and Disease. NY: Marcel Dekker 2 Processes in Biological Vision

The retinoids are normally transported from the liver to the target location by what are called retinoid binding proteins (RBP). These may be present in a variety of forms, both in the blood stream or individual cell types. The following discussions will only address the RBP’s related to the visual modality. These retinoids are absorbed from the intestine, aided by a series of enzymes that play a crucial role in preparing them for transport to the liver. The Blomhoff text is the most definitive work on this subject known at this time. Figure 1 of that text shows the stick versions of the molecules of interest here in their most recently agreed forms with the two methyl groups sharing a carbon of the β- ionone ring shown pointing upward. In actual fact, the retinoids are not planar molecules and cannot be adequately represented in a 2-dimensional configuration (Section 5.5.8). This configuration is used in the “2nd Order calculations (Section 5.5.8.3.1) of the an-isotropic spectral absorption of light by the retinine, Rhodonine(5) in this work. This absorption occurs at 610 nm under the endothermic conditions associated with most mammals. The numeric, 5, defines the location of the carbon bonding with the oxygen atom of the β-ionone ring in the L–channel chromophore of vision. 7.1.1.1.1 Scenario requirements

Any realistic scenario must be anchored by the following factors: + The organism ingests either carotenes or Vitamin A as a source (the fundamental chromogens) of the chromophores of vision. + The blood stream of the animal is antagonistic to the delicate retinoids. The retinoids are particularly subject to oxidation.

+ Up to four separate types of Rhodonine (the actual chromophores) are deposited in liquid crystalline form on individual substrates of protein material in the Outer Segments of a mosaic of photoreceptor cells in each eye of the animal.

The challenge is to determine at least one scenario in which the retinoids can traverse the available pathways between the small intestine and the retina without being exposed to destructive chemistry. 7.1.1.1.2 Reinterpretation of Data Base

Until now, the mechanisms of the binding proteins related to vision have always been discussed on the assumption that they bound only the retinenes. The situation is more complex. The same generic complex, a retinol binding protein present in the serum (SRBP), and a molecule named transthyretin (TTR) may absorb/adsorb a retinol molecule and transport it to other locations unmodified, for a variety of purposes. It may also modify the retinol as part of the transport and/or delivery step. In the case of vision, there are clear indications that the delivered material is not a retinene but a retinine (with two i’s), a diol version of a retinene. The retinines may have been encountered in the biochemical laboratories studying SRBP and the CRBP/CRABP’s where they have been described as non-canonical forms of retinol.

A definition of non-canonical form may be difficult to find. For the purpose of this discussion, “the basic concept behind the canonical structure is whether two consecutive bonds are appearing in equilibrium or not; C if the bonds appear to be in equilibrium the situation will be called canonical, C if not, the situation will be non-canonical and the process can be well understood by resonance.” Resonance is distinguished from isomerism. An isomer is a molecule with the same chemical formula but with different arrangements of atoms in space. Resonance contributors of a molecule, on the contrary, can only differ by the arrangements of electrons. Therefore the resonance hybrid cannot be represented by a combination of isomers4. Upon recognizing that both the retinenes and the retinines are processed by these binding proteins, readdressing the data base in the literature is necessary. This is necessary to discover how the additional process flexibility introduced by this situation is used and where the resultant processes take place. Much of the information associated with the SRBP + TTR complex has been gather in the context of its use in nutrition. However, the complex is used in many other contexts, including vision but also in transporting material to the testes in male animals as an example.

4http://en.wikipedia.org/wiki/Resonance_%28chemistry%29 Dynamics of Vision 7- 3

Furthermore, the literature is generally unclear whether the particular author is speaking about the holo- or apo- form of the protein. Making this distinction is absolutely necessary if the dynamics of the transport mechanism is to be understood. In the visual case, the post transport apo- protein may not be identical to the pre transport apo- form (see Section 7.1.1.2.3). Most of the stoichiometry of the SRBP + TTR complex has focused in the fact that a complex of two molecules of SRBP and one molecule of TTR has been found easiest to crystallize for purposes of X-ray analysis in order to gain maximum knowledge of the conformation of these materials in complex. However, this is not the criteria for asserting this ratio is involved in the physical transport of retinol. Goodman has provided a cartoon illustrating his view of the options for loading the complex in 1984, Figure 7.1.1-1, published originally in 1979 by Smith & Goodman in an obscure press. Goodman’s caption notes the RBP and T4 combine with TTR at different and largely independent sites, whereas the combining of RBP and TTR designed to upload (and protect) retinol involves a more stabilized situation. It is also useful to collect and reinterpret the data in the literature with respect to the stereochemistry of the retinenes, and presumably the retinines, at different locations in the body and the eye. Although the retinenes are the chromogens, and not the chromophores of vision, some of their characteristics provide an invaluable foundation for the study of the retinines (specifically the Rhodonine family). The retinenes are also critical to the formation of the specific Rhodonines and their method of transport to the RPE. 7.1.1.1.3 The BIG QUESTION–What is the shape of retinol in various environments

Advances in the art of crystallography have brought new attention to the question of what is the precise structural shape of retinol/retinal. More specifically, what are the specific distances between the two oxygen atoms in the individual resonant conjugated retinine, Rhodonines()? Figure 7.1.1-1 Goodman’s cartoon model of alternate loadings of the SRBP-TTR complex. (S)RBP will be used The typical text book stick version of these molecules here to describe the protien when in the bloodstream. Note gave way to the ball and stick version during the third the uploading of thyroxine, T4, is distinctly different from quarter of the 20th Century that relied upon poorly known that of retinol. See text. From Goodman, 1984. bond lengths of chemistry. That era was followed by the computer generated ball and stick version of the last quarter of the 20th Century, that continued to rely upon poorly, but better known chemical bond lengths for more complex well documented molecules. Just prior to the start of the 21st Century, the era of computer reconstituted geometry of molecules based on their x-ray diffraction patterns provided much more definitive dimensions of the centroids of individual atoms in a 3D environment for any molecule that could be crystallized. The early computerized data, typified by the Jmol and Jsmol databases of the Royal Society of Chemistry (RSC) continues into 2016 to suggest Retinol_393012 is a planar molecule. Recent data from the crystallography community confirms that it is not, at least when in the liquid crystalline configuration of biology.

Upon close examination of the Jmol files of the Royal Society of Chemistry (acting as a storage facility and not performing curation on the Jmol data sets), most of their files only present 2D representations of a given molecule and the visualizer used attempts to recreate a 3D representation based on plausible (to the computer) constraints. As a result, this section can only present plausible representations of the chemicals found to be important in olfaction. The RSC indicated to this investigator that if undefined stereo-centers are indicated on their main page for a chemical, their 2D & 3D representation of the molecule are at best approximations. They also indicated that various visualizers will prepare a reasonable representation of the molecule using stored bond lengths (of uncertified or identified precision). Luo has presented a full handbook5 of bond dissociation energies BDE), (a quantity usually believed to correlate with bond lengths) for individual bonds between two atoms as

5Luo, Y–R. (2003) Handbook of bond dissociation energies in organic compounds Boca Raton, FL: CRC Press 4 Processes in Biological Vision

found in large numbers of molecules. Just the BDE’s for the C-H bond of the saturated hydrocarbons covers four pages of significantly different molecules with a range from 95 to 105 kcal/mole. Relying upon any visualizer to estimate the distance between two orbitals in a molecule is totally unacceptable within the research community!!! The Jmol program will have long term positive impact on organic and biological chemistry. However, at this time, it lacks significant curation and fact checking by the RSC. The staff listed on the RSC website in 2015 was surprisingly limited in its chemistry credentials. As a result, virtually anyone is allowed to submit a molecular description to the Jmol library. Not even the sources name is required to be included in the record submitted. There appears to be no peer-review of the submissions. News flash: The Jmol files are no longer available in 3D based on the cancellation of their internet security certification based on the “Cessation of Activity” as of 15 October 2015. It appears these files are being supplanted by the JSmol files curated by the same RSC. However, the JSmol database was taken off the internet for an unspecified period as of 19 Nov 2015 (as was the ability to contact the curator via the website). While the JSmol files examined frequently have more header information than the Jmol files, the information is frequently disguised with a dummy author’s name (Marvin) appearing on large numbers of JSmol files. No citation has been provided to date regarding the bond lengths used in the Jmol and JSmol files the RSC has provided.

The XYZ file format most frequently used with Jmol files is designed to accommodate a number of variants as defined by the Jmol.org6. A major problem arises when incomplete data sets from undefined sources are incorporated into the database without significant curation.

See Sections8.4.1.2.3 & 8.6.1.6.3 (interim xxx) of “The Neuron and Neural System” for a broader discussion of this problem.

- - - -

The recent crystallographic data for retinol shows retinol to be three dimensional with a significant dihedral angle of –58 degrees (Table III in Cowan, 1990) between the (nominally planar) β-ionone ring and the (nominally planar) aliphatic conjugated side-chain. Cowan specifically defines the dihedral angle as the angle between the C5-C6 bond and the C7- C8 bond.

In the case of the holo-SRBP molecule incorporating a retinol molecule, the group attached to C5 is not in the plane of the aliphatic side chain and it is not located along the top of the β-ionone ring as typically drawn in ball and stick form on paper. These differences can have a major impact on the resonant properties of the retinines as developed by Platt in the source book of 1964, “Systematics of Electronic Spectra of Conjugated Molecules.” See Section 5.5.7 for detailed discussions of this early work. Fortunately, Platt’s early work predicted spectra for the retinines that could be and were confirmed in both the laboratory and in commercial products of the photographic film industry. However, there remained considerable argument in the psychophysical community concerning the long wavelength peak associated with Rhodonine(5), the long wavelength photoreceptor of biological vision as discussed in Section 5.5.10.3. The question remains, what is the precise structural configuration of each of the Rhodonines when in a liquid crystalline array deposited on a protein substrate (opsin in the vision of Chordata) or directly on the lemma (microvilli in the vision of Insecta and Mollusca) of a sensory neuron. It has been shown that crystallography can determine the parameters of a molecule when it is in the liquid crystalline form (as of 2016). The technique itself is still in a period of refinement to eliminate certain estimating errors in the mathematical transforms involved. See Section 7.1.2.1 where the papers of Cowan (1990), Monaco (2009) and others are discussed. Currently crystallography involves a considerable danger related to the “Bayesian Trap.” See Rhodes7 (3rd ed., 2006). 7.1.1.2 Terminology

6http://wiki.jmol.org/index.php/File_formats/Formats/XYZ 7Rhodes, G. (2006) Crystallography made Crystal Clear, 3rd Ed. NY: Elsevier Dynamics of Vision 7- 5

In discussing the dynamics of the retinoids in the body, using the terminologies found in pharmacology, nutrition and complex proteins is common. Typically the materials involved are known by their functional characteristics and not their detailed formulas. This is primarily due to the extremely high complexity of many of these molecules. This complexity is apparent in their molecular weight and in their detailed geometry. There are several important sources of background material8,9,10,11,12. However, no source in the literature recognizes the resonant forms of the retinoids proposed here. They do describe, in various levels of detail, the multiple stages in the transport of the retinoids of vision from ingestion by the species to their occurrence in the retina. Chen & Heller13 also identify a “retinoid-like material.” They stress it cannot be a retinoid on grounds that are not supported here, but are compatible with the conventional wisdom of the literature. This work offers an alternate reason they cannot be simple retinoids. The above sources do not address the actual retinoids used as the chromophores of vision. They routinely make the assumption that it is always Retinol in a molecular combination with a putative protein, Opsin--frequently involving a specific stereo-chemical configuration. The result is that their introductory remarks and overall description of the transport of the relevant retinoids of vision must be discounted. After recognizing the putative Rhodonines in liquid crystalline form as the chromophores of vision, the data base provided by the above authors can be reinterpreted. With this reinterpretation, new information and fewer dichotomies appear concerning the vision process. One of the dichotomies that will be explained later is between the work of Bridges and Ganguly. Bridges shows IRBP working in conjunction with 11-cis-retinol, 11-cis-retinal and rhodopsin. Alternately, Ganguly holds that IRBP only binds to the all-transform of the retinoids. Ganguly does not recognize the possibility that there are multiple forms of his retinoids. More recent work in crystallography of retinol also supports its utilization in the all-trans-retinol form.

Focusing on the transport proteins associated with vision has been traditional in the above sources. However, taking a broader view may be advisable. Proteins are usually described in terms of three major classes; transport proteins, enzymatic proteins and structural proteins. This is an inadequate classification for the materials found between the ingesting of Vitamin A and the deposition of the liquid crystalline forms of Rhodonine on the structural proteins, Opsin. Considering a broader range of processes and clarifying definitions is necessary. 7.1.1.2.1 Enzymatic activity

In discussing the enzymatic chemistry of vision, it is important to differentiate between enzymes, substrates and surfaces. It is also necessary to differentiate between accelerants and transporters. Much of the reaction chemistry of vision of interest here appears to involve reactions at, on, or in passing through a surface. Besides the reaction chemistry, there appears to be important transport activity. This activity may place additional requirements on the enzymes used. In addition, many reactions involve materials that accept or give up functional groups. If these materials are present along with an enzyme, their definition becomes awkward. They are labeled either a coenzyme or a co-substrate. Both dehydrogenation and demethylization are important processes in the formation of the chromophores of vision.

Overall, enzymes are large protein molecules and the material they catalyze are much smaller molecules14. Modern works frequently define ribosomes as particulate bodies within cells that act as (or are) enzymes15. In most of the following discussion, a retinoid of molecular weight less than 300 will be considered the substrate. Most of the transport proteins will be considered enzymes. They are not consumed in the transportation process. However, the transport protein associated with moving the retinoid from the liver to the RPE is apparently changed in the process and is

8White, A. Handler, P. & Smith, E. (1973) Principles of Biochemistry, 5th ed. NY: McGraw-Hill pg. 209, 401, 403, 661 9Sporn, M. Roberts, A. & Goodman, DeW. (1984) The Retinoids, vol. II, NY: Academic Press pp 42-85 10Saari, J. (1994) Retinoids in Photosensitive Systems, Chap. 9 of Sporn, M. Roberts, A. & Goodman, D. The Retinoids: Biology, Chemistry, and Medicine, 2nd Edition NY: Raven Press 11Ong, D. (1985) Vitamin A-Binding Proteins. Nutrition Reviews, vol. 43, no. 8, pp. 225-232 12Ganguly, J. (1989) Biochemistry of Vitamin A. Boca Raton, FL: CRC Press 13Chen, C. & Heller, J. (1977) Uptake of retinol and retinoic acid from serum retinol-binding protein by retinal pigment epithelial cells. J. Biol. Chem. vol 252, no 15, pp 5216-5221 14White, A. Handler, P. & Smith, E. (1973) Op. Cit. pg. 236 15White, A. Handler, P. & Smith, E. (1973) Op. Cit. pg. 403 6 Processes in Biological Vision

discarded following this function. It is more appropriately considered a co-substrate. 7.1.1.2.2 Naming enzymatic Proteins

An enzyme is a protein performing as an organic catalyst. There are many overlapping classes of enzymes and the naming conventions used have changed over the years. In early works, the enzyme is named by using the root of the substrate name and adding the suffix -ase. In more recent work the name has brought together the name of the initial substrate, the name of the final reaction material, and the process involved followed by the suffix -ase. Two important examples are retinol-rhodonine-dehydrogenase and retinol-rhodonine-demethylase. Even these labels are inadequate for two separate reasons. First, they do not show that oxygen was added to the substrate at the point of dehydrogenation or demethylization. Second, each of them occurs in two distinct reactions. The solution is to add a parenthetic label to the Rhodonine describing the carbon atom where the oxygen atom was added to the molecule. An alternate and more suitable naming convention would be retinol-rhodonine( )-monooxygenase. This leaves the question of whether a hydrogen or a methyl was removed implicit in the parenthetic term. It also indicates that the oxygen was not obtained from a free oxygen molecule, O2. For Rhodonine(5) or (9), a demethylization would be involved. For Rhodonine(7) or (11), a dehydrogenation would be involved. 7.1.1.2.3 Transport Proteins

A protein associated with the movement of a material, at the molecular level, within the organism is called a transport protein. The movement may be intra-cellular, extra-cellular or across the cell wall boundary. Traditionally, vision research has assumed that a transport protein has not been chemically changed while carrying out its transport function. However, this is no longer true.

Transport proteins are usually named by preceding the hyphenated expression “-binding protein” by the name of the material being transported. Thus, RBP is the traditional abbreviation for a Retinoid-Binding Protein. In this work, a more precise description is needed. Many more specific expressions have been developed to describe binding proteins related to the retinoids. These expressions also require more careful definition.

Although it has not become common in the literature, it is also important to recognize where the binding protein under discussion resides. The literature has defined a set of RBP’s that are found within various cells. These have been labeled with a C for a prefix, i.e., CRBP. Other RBP’s found extra-cellularly in the bloodstream should also be labeled. It is suggested that these be labeled with an S (for serum) as a prefix. Other authors have use P (for plasma) to describe the transport proteins in the blood stream16. For reasons to be developed below, distinguishing certain RBP’s that may exist extra-cellularly in the region of the small intestine and the liver is also useful. These will be described by the prefix L. The L identifies the Lacteal channels of the intestine and liver. However, the L could be considered as indicating a lymphatic channel or mere presence in the liver.

When defining a transport protein, two significant situations occur: when the binding protein is attached to the target material, the holo- situation, and when the binding protein is not attached to the target material, the apo- situation. When speaking of a binding protein, authors generally assume the holo- situation. This is unfortunate. Important changes can occur at the beginning or end of the transport process.

Until now, all of the binding proteins of vision were assumed to be binding to either retinol or retinal. In this work, a third condition must be addressed. The resonant nature of the Rhodonines shows that these materials can exhibit the chemical characteristics of an alcohol or an aldehyde of retinene, depending on the environment. Therefore, it is very difficult to specify whether the given binding protein is specific to either Retinol or Retinal, or the resonant form-- Rhodonine. No current literature demonstrates that any of the binding proteins of vision are specific to either Retinol or Retinal and simultaneously discriminatory with respect to Rhodonine. Because of this fact, the R in the name of a binding protein must be interpreted as standing for a retinoid. As a minimum, the retinoid may be either a retinine (a Rhodonine) or a Retinene. Assigning a name to a putative enzyme is common in biochemistry. It can aid in discussion. However, such names have tended to proliferate in the literature, with or without adequate demonstration of the existence of such an actual substance. Correlating the various names used by different authors can be difficult. The following list defines the names used in this work. Following the list is a table showing some aliases found in the older literature. The list of aliases is

16Chen, C. & Heller, J. (1977) Uptake of retinol and retinoic acid from serum retinol-binding protein by retinal pigment epithelial cells. J. Biol. Chem. vol 252, no. 15, pp-5216-5221 Dynamics of Vision 7- 7

incomplete. RBP--1. As used here, a generic descriptor for a Retinoid Binding Protein. 2. Traditionally, a descriptor limited to a Retinol Binding Protein. This brief descriptor is usually expanded to provide more detailed information concerning a specific situation. CRBP-- 1. As used here, Cellular Retinoid-Binding Protein, an enzyme. Usually, the substrate found inter-cellularly is the resonant form of the retinoid, i. e., Rhodonine. For the cellular case, it is more appropriate for the R to stand for Rhodonine. This material is putatively associated with the transfer of the Rhodonines from the RPE cell wall to the storage locations within the RPE cells. 2. traditionally cellular retinoid-binding protein. CRABP-- Traditionally, an intercellular enzyme associated with retinoic acid as a substrate. It is not normally used in the formation of the chromophores of vision. Found in the inner segments of the photoreceptor cells where it is used for growth and metabolism. CRABPII-- An intercellular enzyme associated with retinoic acid as a substrate. Generally found associated with the skin. It is not normally used in the formation of the chromophores of vision. CRALBP--1. As used here, cellular Retinoid-(aldehyde) binding protein. This enzyme is not modified during its function. In general, the substrate found inter-cellularly contains the resonant form of the retinoid, i. e., Rhodonine. For the cellular case, it is more appropriate for the R to stand for Rhodonine. This material is putatively associated with the enzymatic transfer of the Rhodonines from the storage locations, and the pinocytosis region, within the RPE cells to the RPE cell wall facing the IPM. 2. Traditionally the less specific cellular retinaldehyde-binding protein.

IRBP– 1. As used here, a Rhodonine-binding protein found in the Interphotoreceptor matrix (IPM) of the retina of the eyes of several species of animals. It is probably able to transport all four forms of Rhodonine enzymatically. IRBP only binds to the all-transform of the retinoids. (Ganguly, pg. 155.) 2. Traditionally Interstitial Retinol Binding Protein.

LRBP– Lacteal retinoid binding protein. The protein material encapsulating the retinoid(s) absorbed through the intestinal wall for transport to the liver. The implication is that this material is transported via the lymphatic system.

PRBP– See SRBP

SRBP–A retinoid-binding protein resident in the serum of the blood stream. SRBP appears in three forms; a pre-holo SRBP, a holo-SRBP and a post-holo-SRBP. Both the pre-holo-SRBP and the post-holo-SRBP have been labeled apo-SRBP in some literature. pre-holo SRBP is able to complex with many retinoids, but preferentially with all-trans-retinaldehyde holo-SRBP incorporates a retinoid and appears to be a co-substrate along with an associated protein, TTR. holo-SRBP + TTR is the tranport entity for the retinoids via the bloodstream. post-holo-SRBP is the residue when holo-SRBP is functionally destroyed after delivering its retinoid to the RPE. TTR–The plasma protein transthyretin. Formerly known as prealbumin. The protein also binds to one molecule of thyroxine. Some characteristics of the above binding proteins involved in vision have been tabulated. An important scenario begins to appear. 8 Processes in Biological Vision

TABLE 7.1.1.2-1 PROTEINS THAT BIND RETINOIDS

SPECIFIC PROTEIN ALIAS FUNCTIONS WITH DISTRIBUTION DISTRIBUTION SIZE IN BODY IN EYE Lacteals LRBP CRBP(II) Retinol in digestion In intestine Not reported Bloodstream SRBP CRBP(?) Wide distribution 12kDa pre-holo-SRBP apo-CRBP Retinol to form In circulatory system Not reported Rhodonine holo-SRBP holo-CRBP Rhodonine & TTR In circulation & cells Not reported post-holo-SRBP apo-CRBP Nothing, a residue In circulatory system Not reported TTR In circulatory system Not reported 55kDa Non-retinal areas CRABP retinoic acid (all-trans-) Wide distribution Not reported 15.4kDa CRABPII retinoic acid (all-trans-) Skin Not reported ~16kDa Retina CRBP alcohol only Wide distribution 16.6kDa CRALBP aldehyde only Only in eye Only in RPE 33; 36 IRBP Only in eye Only in IPM 140kDa

Values from many sources. Sizes mainly from Berman, 1991, pgs 320 & 377, Marmor & Wolfensberger pg 137 and Gamble & Blaner, 2000.

The enzyme LRBP transports simple retinoids, generally the retinenes, extracelluarly from the point of ingestion to the liver for storage. The co-substrate SRBP, in conjunction with TTR withdraws retinol from the liver, converts it to one of the four Rhodonines and transports the new material extracelluarly to the RPE interface with the blood stream. The enzyme CRBP accepts the Rhodonines at the RPE cell wall and delivers it to a storage area (pigment globule or granule). When needed, the enzyme CRALBP transports the Rhodonine to the RPE interface with the IPM. At that interface, the enzyme IRBP receives and transports the Rhodonine extracelluarly to its point of deposition on the disks.

Ong has provided a more expansive table. However, he includes some “baggage” that may not be appropriate to vision within the current context. He has defined a specific RBP found only in the intestine and labeled CRBP(II). It is possible that this protein is pre-holo-SRBP in the more recent vernacular presented above since it is described as extracellular and found in the intestine. It is also possible that this protein is more appropriately labeled an LRBP.

Goodman has studied the mechanism employed to transport Retinol between the liver and the target location via the bloodstream. This mechanism involves an RBP that he described as the plasma RBP or PRBP. When dealing only with the circulatory system, many authors have used the root RBP to describe the material they are discussing. The target may be unrelated to vision. Some authors have chosen to use the expression serum RBP or SRBP for a material more specifically defined with respect to vision than that of Goodman. The term SRBP will be used here to describe the particular co-substrate (since the material is modified) that transports the retinoids to the RPE cells. Ong provides some very useful caricatures of the overall metabolic situation from the medical or nutritional perspective. However, they are highly conceptual. They are not supported by detailed discussion or equations. As an example, it is implied that CRBP(II) aids in the transport of Retinol from the intestine to the liver by way of the bloodstream. However, CRBP(II) is not shown in the bloodstream. The only Retinol in the bloodstream is shown enclosed in the “RBP/TTR bottle” (See below). Note the large question mark in the middle of the “target cell.” In his Figure 3, he implies there is no 1:1 relationship between photoreceptor cells and RPE cells. Ong updated his 1985 paper in 199417. The large question mark remains in his caricature. In his discussion, he at times drops the suffix associated with retinol or retinal in favor of the more general retinoid. He may have found this necessary since some binding proteins were less specific

17Ong, D. (1994) Cellular Transport and Metabolism of Vitamin A: Roles of the Cellular Retinoid-Binding Proteins Nutrition Rev vol 52(2), pp S24-S31 Dynamics of Vision 7- 9

than desired in the conventional literature (of retinoid movement in metabolism). He does include a caveat that the retina and the testis are special cases. Note also his intimation that the Muller cells of the retina are positioned quite similarly to the retinochrome cells of the arthropod eye.

Ganguly has summarized many measured parameters relating to the serum RBP’s18. Livrea & Packer have also provided information on the RBP’s and more data on TTR19. 7.1.1.2.4 Specifics related to TTR and its relationship to RBP

Johnson et al. have also provided valuable data related to TTR20, “TTR is a globular, non-glycosylated protein with a molecular mass of 54.98 kDa. With one complexed molecule of retinol-binding protein (RBP; 21 kDa), the total mass is approximately 76 kDa, which is still small enough to diffuse out of the vascular space as readily as albumin (66.3 kDa) or transferrin (79.6 kDa); slightly less than 50% of each of these proteins is normally intravascular as a result.” They cite three very old references. Johnson et al. describe the function of TTR in detail, “The protein migrating anodal to albumin on non-sieving, routine serum electrophoresis at pH ~8.6 was initially noted to bind thyroxin (T4) and was thus given the name thyroxin-binding prealbumin, or TBPA. However, it was subsequently shown to bind triiodothyronine (T3) and holo-retinol-binding protein (RBP with retinol, or vitamin A) as well, and the name was changed to transthy(roxin) retin(ol) to denote its dual transport function. TTR is a tetramer of four identical subunits. Although each of the four monomers has a binding site for RBP, the tetramer binds only one molecule of RBP with high affinity and possibly a second with lower affinity. The binding affinity for apo-RBP (RBP without retinol) is very low, and the loss of retinol (e.g., uptake by tissues) results in the separation and renal excretion of free apo-RBP, accounting for the very short biological half-life of RBP of ~ 3.5 hours. Each TTR monomer also has two binding sites for thyroid hormones, but binding of one molecule of T3 or T4 significantly reduces the affinity of the second site. Binding affinity for T3 is lower than that for T4. The TTR-RBP complex normally transports approximately 20% of circulating thyroid hormones (70% is transported by thyroxin- binding globulin or TBG, the rest by albumin) and 90%–95% of retinol/vitamin A. The complex is more important for retinol transport than for thyroid hormones.” Emphasis added.

When discussing the synthesis of TTR, Johnson et al. note, “Essentially all plasma TTR is synthesized by the hepatic parenchymal cells. . . .” With regard to catabolism, “TTR is catabolized primarily by the liver and by excretory loss via the kidneys and gastrointestinal tract. Its biological half-life is approximately 2.5 days and is not altered by stress or acute inflammation.”

Johnson et al. have also noted, “Serum concentrations of TTR are very low in the fetus and neonate, rise slowly to reach a maximum in the fifth decade of life, and then decline slowly.” A table of values is provided. The changes with age are indeed slow. Genetically, “The gene coding for TTR is located on chromosome 18q (26). There are over 100 known genetic variants, including a few with increased or decreased binding affinities for thyroid hormones but clinical euthyroidism. Many of the genetic variants are associated with deposition of amyloid in tissues, resulting in a group of autosomal dominant hereditary amyloidoses. Plasma concentrations of TTR are essentially normal in these disorders and are not helpful in diagnosis; however, some variants do show altered electrophoretic mobility.” Understanding these multiple gene codes could be an important factor in understanding the onset of AMD within families.

Johnson et al. did not discuss the question of local clearance of TTR or SRBP from the vascularization of the eye as it might relate to AMD. 7.1.1.2.5 The CRBP’s of vision

Most descriptions of cellular retinoid binding protein (CRBP) do not differentiate between the holo- and apo- form. Some form of CRBP is reported to exist in nearly every cell in the body. Saari, writing in Sporn21 has defined LRAT--An enzyme based on lecithin as esterfying all-trans-retinol to all-trans-

18Ganguly, J. (1989) Biochemistry of Vitamin A. Boca Raton, FL: CRC Press pg 66 19Livrea, M. & Packer, L. (1993) Retinoids. NY: Marcel Dekker, inc. pp 92-100 & Chap. 12 20Johnson, A. Merlini, G. Sheldon, J. & Ichihara, K. (2007) Clinical indications for plasma protein assays: transthyretin (prealbumin) in inflammation and malnutrition Clin Chem Lab Med vol 45(3), pp 419–426 21Sporn, M. Roberts, A. & Goodman, D. (1994) The retinoids, 2nd ed. NY: Raven Press pg. 363 10 Processes in Biological Vision

retinyl palmitate (along with some oleate and stearate) within the RPE. This description is considered archaic in this work. The enzyme CRBP (as defined above) performs the same function in this work but the substrate has already been converted to Rhodonine. The Rhodonines are always found in the all-trans- configuration. It is likely, the variously colored globule (or granules) stored in the RPE are made up of these palmitates, oleates and/or stearates. 7.1.1.2.6 Structural Proteins

The only structural protein of concern in this work is Opsin, the substrate produced by the IS of the photoreceptor cell and used as a substrate for the chromophores of vision, the Rhodonines. 7.1.1.3 Properties of the fundamental chromogens

Animals are generally able to ingest and use both carotene from plant sources and Vitamin A from animal sources. The phylogenic and environmental history of the animal determines whether the animal can use a specific form of the carotene and/or vitamin A. Carotene exists in at least four forms22. α−carotene when cleaved at its center in the presence of oxygen leads to one molecule of Vitamin A1 and one molecule of Vitamin A2. β-carotene on the other hand leads to two molecules of Vitamin A1 when similarly cleaved. Vitamin A1 (also known as Retinol1) is used by animals of marine origin. This category includes the mammals living in the sea and on land. Vitamin A2 (also known as Retinol2) is used by freshwater-based animals ( primarily freshwater fish). The vision literature has focused more on β-carotene than on α-carotene because of its higher utility for man. The literature also claims a wider occurrence of β-carotene in nature. Although a family similar to the carotenes, the xanthophylls contain additional oxygen in their ring structure. It is not clear whether they are used by animals to create the chromophores of vision. A xanthophyll with an oxygen replacing the methyl group at position 5 (using Karrer’s notation) would form Rhodonine(5) upon cleavage as above. However, all available data suggest that the chromophores are created in the RPE from retinol feedstock.

Recently, a third form of Vitamin A has been recognized. Vitamin A3 (also known as Retinol3) has been identified among the flies (Diptera of Odonta) of Insecta. See Section 1.2.1.1. The material appears to arise from decaying plant material.

The scenario discussed below will not be detailed to the point of differentiating between Vitamin A1 and A2.

Vitamin A1 is readily available commercially in pure, all-transform. It is provided as a white crystalline material resembling table salt to the naked eye. The crystals are of the rhombic form. Under the microscope, the individual crystal appears yellowish with tinges of blue due to scattering. More details of the specific crystalline structure are available in Section 5.5.4.1. The individual molecules of Vitamin A can be enclosed by a cylinder approximately 5 Angstrom in diameter and 15 Angstrom long23. When discussing vision, the three-dimensional molecule is most easily represented in two dimensions by placing all of the ligands containing single methyl groups, C18, C19 & C20, on the same side of the molecule. This accentuates the structural similarity of the chromophores of vision.

Recently, x-ray crystallography has identified the precise 3D arrangement of Vitamin A. The molecule involves a significant dihedral angle between the β-ionone ring and the aliphatic carbon chain (Section 7.1.2.1). The molecule is not planar. Crystallography is destined to play a major role in understanding the transduction process in the vision, gustation and olfaction modalities.

Recognizing that Vitamin A plays more than one role in the animal body is important. It has a critical role to play in the growth and nutrition of virtually every cell in the body. Simultaneously, it plays an extremely important role in vision. Differentiating the pathways and mechanisms used to distribute Vitamin A in these different roles is important.

7.1.1.4 State of the ART in crystallography versus molecular modeling

Crystallography plays a critical role in determining the precise nature of the chromophores of biological vision, just as it did in its earliest days in the field of receptors for photographic film. In photographic film, the molecules of interest were generally simple organics and did not involve proteins. In biological vision, the same is true of the actual chromophores. However, their transport and manipulation by various proteins results in much more complex moieties.

22Wolken, J. (1975)Photoprocesses, photoreceptors and evolution. NY: Academic Press pp. 34-39 23Wolken, J. (1966) Vision, Biophysics & Biochemistry of the retinal photoreceptors. Springfield, Il: Charles C. Thomas, pg. 30 Dynamics of Vision 7- 11

To understand the precise structure of these moieties, very sophisticated crystallographic techniques at the highest available resolution and precision are required. The Protein Data Bank (PDB) of the Royal Society of Chemistry has provided a rational overview of the field of x-ray and solution NMR crystallography as applicable to the current subject matter24. It develops several key points on first reading: C 80% of the data in the PDB originated from x-ray analysis, 16% from NMR and 2% from theoretical modeling. C protein crystals as used for diffraction studies are highly hydrated ("wet and gelatinous") so structures determined from crystals are not much different from the structures of soluble proteins in aqueous solution. Some molecules have been studied both by crystallography and by solution NMR, and in these cases the agreement has been excellent. This statement is encouraging that the crystallographic results will closely relate to the liquid crystalline state in which most chromophores are actually found.

C X-ray crystal diffraction usually cannot resolve the positions of hydrogen atoms or reliably distinguish nitrogen from oxygen from carbon. This means that the chemical identity of the terminal side-chain atoms is uncertain for Asp, Gln and Thr and is usually inferred from the protein environment of the side chain (i.e. the side chain orientation which forms the most hydrogen bonds or makes the best electrostatic interactions is selected and built by the crystallographer as the most plausible choice). Sometimes there is also uncertainty about whether an atom that is not part of the protein is a bound water oxygen or a metal ion.

These situations are unfortunate for the subject of this Section. The role of oxygen is critical to understanding the transport and manipulation of the retinenes/retinines of this section.

C NMR determines structures of proteins in solution, but is limited to molecules not much greater than 30 kD. NMR is the method of choice for small proteins which are not readily crystallized, and yields the positions of some hydrogen atoms. The results of NMR analysis are an ensemble of alternative models, in contrast to the unique model obtained by crystallography.

The PDB overview provides an excellent discussion of the requirements and shortcomings of the x-ray crystallography method. They quote Rhodes in “Crystallography Made Crystal Clear.” On page 183, Rhodes offers this caveat:

"All crystallographic models are not equal. ... The brightly colored stereo views of a protein model, which are in fact more akin to cartoons than to molecules, endow the model with a concreteness that exceeds the intentions of the thoughtful crystallographer. It is impossible for the crystallographer, with vivid recall of the massive labor that produced the model, to forget its shortcomings. It is all too easy for users of the model to be unaware of them. It is also all too easy for the user to be unaware that, through temperature factors, occupancies, undetected parts of the protein, and unexplained density, crystallography reveals more than a single molecular model shows."

Solution NMR does not appear appropriate for the delineation of the “complexed” molecules described below. 7.1.1.4.1 Critical role of disorder and delocalization in chromophore transport ADD

The results obtained using crystallographic techniques are greatly impacted by the degree of disorder within the asymmetrical unit examined by x-ray irradiation. Similarly, interpreting the results obtained through crystallographic techniques is greatly impacted by the state of delocalization of the electrons of the groups within the asymmetrical unit. Glusker, Lewis & Rossi define these terms25; Disorder– 1. A disturbance to the regular organization of an entity (here a crystal). 2. Lack of regularity. In crystal structures, it implies that there is not exact register of the contents of one unit cell with those from all others. The atoms or molecules in the crystal structure pack randomly (non-periodically) in alternative ways in different unit cells.

Delocalization of electrons– The π bonding in a conjugated system is not consider to consist of localized bonding, but to have the electrons delocalized over the entire system of double bonds.

24http://www.rcsb.org/pdb/static.do?p=general_information/about_pdb/nature_of_3d_structural_data.html 25Glusker, J. Lewis, M. & Rossi, M. (1984) Crystal Structure Analysis for Chemists & Biologists. NY: Wiley- VCH 12 Processes in Biological Vision

Disorder is inherent in the intrinsically liquid crystalline materials of the biological system. It is due to the presence of free water molecules within the cavities of a given protein or more complex structures that are the target of investigation. They lead to a loss in resolution in the electron density maps relied upon to ascertain the molecular structure of an organic molecule. Delocalization of electrons is also inherent among the chromophores of vision (unless the chromophores are separated by differential crystallization or similar techniques) due to their variable length of conjugated chains in a specific crystalline sample. Such delocalization leads to a loss in resolution of both the electron density maps and the difference electron density maps forming the foundation of further analyses to identify the specific molecules and /or chemical groups (including peptides) within a macro-molecule. 7.1.1.4.2 Lack of chromophore planarity plays a critical role in crystallographyADD

The use of molecular overlays, based on simple structural models using only 2D ball and stick models, as an aid to interpreting electron maps in crystallography is limited if the molecule is not planar and the bond lengths are not shown in proportion to their actual length. The “gold standard” of molecular overlays uses a projection from a 3D ball and stick model that matches the projection of the electron density map being reviewed. Otherwise, subtle relationships may be overlooked in the investigators evaluation of his representations.

7.1.1.4.3 Application of molecular modeling and visualization versus crystallography

Currently, the art of modeling molecules and visualization of those models in 3D space is moving forward rapidly. However, the majority of this modeling is being performed ab initio based on the best available set of bond lengths, angles, etc available to the modeler. An unfortunate result of the establishment of multiple databases to hold the results of these activities has been the lack of adequate curation of these databases. The more opportunistic databases have begun to fall by the wayside. One of the databases of wide scope is that of the Royal Society of Chemistry (RSC). However, this database remains uncurated and unsupervised as of 2015. The result is a large number of Jmol and JSmol files in that database that are missing (ostensibly required) header information and lack of any discussion of the sophistication of the rules used in the modeling by a given submitter. See Section 7.1.1.1.3.

For the novice, PDB-101 provides an overview of the capabilities provided by the PDB consortium; http://pdb101.rcsb.org/learn/guide-to-understanding-pdb-data/introduction . Another online introduction is called Bioinformatics for the terrified; http://www.ebi.ac.uk/training/online/course/bioinformatics-terrified

See also Section 7.1.2.4 for a broader discussion of this problem as it relates to the vision modality.

As noted above, a technique advancing at a similar rate is that of x-ray crystallography. While a complex, computer intensive technique, it does appear to converge on the actual conformation of any size molecule to a precision higher than that of molecular modeling.

7.1.1.4.4 Unique 3rd order protein structures described via crystallography

Crystallography has begun to explain the structure of various proteins at a level not generally defined in molecular modeling. These structural features involve unusual hydrogen bonds and disulfide bonds. Glusker et al. have illustrated the presence of hydrogen bonds in the helical structures within proteins (page 486). They note quite succinctly, “The α helix is the most common conformational component of proteins, and a single helix may contain up to 40 amino acids. The polypeptide chain is wrapped in a right-handed spiral manner, held together by intra- chain hydrogen bonds, Figure 7.1.1-2. “This helix has 3.6 residues per turn and a translation per residue of 1.5 Angstrom (3.6 x 1.5 = 5.41 Angstrom for one complete turn, cf., the 5.1 Angstrom α repeat). Pauling predicted that this helical structure would be stable as a result of favorable hydrogen bonding patterns.” The interior diameter of these helices is quite small, typically about 5 Angstrom. No other molecular structure can occupy this space. Dynamics of Vision 7- 13

The disulfide bond is important in forming sheets of peptides within a protein. Wikipedia notes, “A disulfide bond, also called an S-S bond, or disulfide bridge, is a covalent bond derived from two thiol groups. In biochemistry, it is considered a functional group. The terminology R-S-S-R connectivity is commonly used to describe the overall linkages. The most common way of creating this bond is by the oxidation of sulfhydryl groups. . . . Cystine is composed of two cysteines linked by a disulfide bond. Disulfide bonds in proteins are formed between the thiol groups of cysteine residues by the process of oxidative folding. The other sulfur-containing amino acid, methionine, cannot form disulfide bonds. The disulfide bond is about 2.05 Å in length, about 0.5 Å longer than a C–C bond. The disulfide bonds are strong, with a typical bond dissociation energy of 60 kcal/mol (251 kJ mol-1). However, they are about 40% weaker than C–C and C–H bonds,

7.1.2 Transport Scenario for the retinoids

Beginning during the 1970's, the biochemists recognized that the chromophores of vision were not formed within the photoreceptor cells of vision. Clearly, these chromophores were found within the RPE (Section 4.6.2.2.3). They were either created within the RPE or during the transport of retinenes from the liver. Bernstein, Law & Rando summarized this situation in 198726. A series of retinoid binding proteins (RBP) are involved in the formation and transport of both the chromogens and chromophores of vision. In this paradigm shift, the biochemists have not encountered or Figure 7.1.1-2 View of an α helix. The segment shown is explored the technique by which the chromophores are from the protein crambin. From Glusker et al., 1984. transported from within the RPE cells across the IPM and through the outer membrane of the photoreceptor cells. They have made the assumption that the disks were located externally to the photoreceptor cell.

Beginning in the 1990's, the nutrition community came to recognize that all retinoids were not processed the same, whether ingested orally or by injection. In 1998, Newcomer, Jamison & Ong even noted that the family of chemicals labeled Vitamin A in the nutrition literature were processed and participated differently with respect to vision and other largely hormonal activities27. “In addition, the eye has RBP’s which are distinct for that organ: a cellular retinal/RBP (CRalBP) and an interstitial/interphotoreceptor retinoid-binding protein (I–RBP).” Note the distinction between retinal used specifically in the first clause and the term retinoid used in the second clause. This distinction is critical, as developed in this work. It is proposed here that the retinoids of the IRBP take on four resonant forms, the Rhodonines() in the latter reference, related to the spectral absorption of the retina. In the case of

26Bernstein, P. Law, W. & Rando, R. (1987) Isomerisation of all-trans -retinoids to 11-cis retinoids in vivo Proc Nat Acad Sci USA vol 84, pp 1849-1853 27Newcomer, M. Jamison, R. & Ong, D. (1998) Structure and Function of Retinoid-Binding Proteins in Jamison, R & Ong, D. eds. Fat Soluble Vitamins, Vol 30 of the series, Subcellular Biochemistry, Quinn, P. & Kagan, V. eds. Chapter 3 14 Processes in Biological Vision

the CRBP’s within the RPE cells specifically, there may be several different RBP’s involved (Sections 4.2.6, 18.8.3.6.2 & later sections of 7.1.2). Volume 30 of the series, “Sub-cellular Biochemistry was dedicated to the latest papers in fat soluble vitamins. The Newcomer et al. paper provides detailed information concerning the various retinoid binding proteins, including the specific structure and conformation of most of those relevant to vision. It defines three functional classes for which the structural motif characteristics are available, extracellular, intracellular binding proteins and the nuclear receptors. It is explicit as to why retinoic acid is processed differently than the retinoids utilized in vision. It provides less detailed information as to the changes in the RBP’s of vision since it relies upon the chemical theory of the neuron and vision. It does provide useful information concerning the unique character of the extracellular IRBP’s found within the IPM of the retina. These characteristics are associated with the electrolytic theory of the neuron and the resonant properties of the retinines. It does not enumerate or discuss the other extracellular RBP’s, the serum RBP’s or SRBP’s. A new field of research has arisen since 1990 concerning potential processes related to the retinoids that is within the nucleus of a cell, and not just within the soma. Newcomer et al. describe the nuclear receptors proteins as controlling gene transcription within the nucleus of cell. It does address the RAR ( retinoic acid receptor and isoforms α, β & γ) and RXR (retinoid X receptor and isoforms α, β & γ). Here again they differentiate between the retinoic acid receptors and the less defined retinoid X receptors. They indicate the former are related to morphogenesis, spermatogenesis and the maintenance of epithelial tissue, and the latter to vision. Niles has provided additional specific concerning these nuclear receptor functions related to the retinoids28. His Introduction is quite conceptual and based on the conventional wisdom of the time, i.e., he defines retinol alone as vitamin A whereas the medical literature defines vitamin A as including the esters of the retinoids. His text is devoted to the specifics of the RAR and RXR family, but the material related to the RXR family is brief and requires additional research as to the specific retinoids targeted. Niles suggests the RXR family may also target retinoic acid. Niles discusses the locations of the genes involved on specific chromosomes within the cell nucleus.

The precise role of each of the RBP’s varies between investigators in the current literature. Figure 7.1.2-1 attempts to summarize the RBP’s as currently identified and delineated. The first group are RBP’s found within various cells. The second group are concerned with the initial transport of the retinoids from the intestine to the liver (for storage or further processing). The lower group cnsist of RBP’s specifically designed to transport the retinenes from the liver and deliver them in modified form (the retinines) to the RPE cells, in conjunction with the cellular RBP’s found within the RPE cells.

The LRBP are particularly effective in protecting the retinyl esters from oxidative destruction by elements within the vascular system. The SRBP’s are particularly designed to protect the retinoids during vascular transport in conjunction with a second RBP previously named transthyretin, TTR. Iindividual mpolecules of SRBP and TTR form a “bottle and stopper” that is particularly effective in delivering a retinene molecule to the RPE cell interface with the vascular system (Section 7.1.2.2.1). During its delivery, the retinene is converted to one of four resonant retinoids known as retinines, Rhodonine(). Within the RPE cell, the retinines are manipulated to their storage areas by one of the CRBP’s variously labeled, CRBP, CrolBP or CRalBP (I) by various investigators during different time periods. The Newcomer et al. paper provides details of the structures of some of these cellular RBP’s.

28Niles, R. (1998) Control of retinoid nuclear receptor function and expression in Dynamics of Vision 7- 15

Figure 7.1.2-1 The currently identified retinoid binding proteins, RBP’s ADD. A; RBP’s found within various types of cells. B; RBP receptors within the nucleus of selected cells. C; RBP’s associated with various terminal cells or matrices. D; RBP’s used in the transport of the previously identified species on the left. See text.

Schreiber et al. have provided a recent large scale review of vitamin A homeostasis29. Nearly every major section concludes with a statement that more research is needed in the area, and Figure 1 includes a question mark in each of the seven major areas depicted–and many conceptually defined esterase’s. Still, it is a very significant source. The focus is on a wide variety of enzymes, retinyl ester hydrolases, REH, that have been documented relative to the various roles

29Schreiber, R. Taschler, U. Preiss-Landl, K. Wongsiriroj, N. Zimmermann, R. & Lass, A. (2012) Retinyl ester hydrolases and their roles in vitamin A homeostasis A review: Biochimica et Biophysica Acta vol 1821(1), pp 113–123 16 Processes in Biological Vision of the vitamin A1 derivatives. Schreiber et al. continue the differentiation between the retinoic acid BP’s and the retinoid BP’s introduced above.

Summarizing the reviews by Newcomer et al. & Schreiber et al., there are several forms of vitamin A1 of interest in human physiology plus the proto-vitamin, beta carotene. These forms appear to be processed differently within the intestine and possibly within and during transport to the liver. beta-carotenes a proto-vitamin that is cleaved into two retinoids during absorption aty the intestinal wall. retinol the most commonly defined form of the alcohol of the retinene (non resonant) form of retinoid retinal the most commonly defined form of the aldehyde of the retinene (non resonant) form of retinoid retinoic acid the non-resonant form of retinene most commonly associated with non-visual uses of retinene Rhodonine() the resonant (retinine) form of the retinoids found within the IRBP and probably stored within the RPE as esters. Beta carotene is ingested from plant-based foods. It follows a different absorption path and is reported to be less efficiently absorbed than the retinenes. The esters of retinol and retinal are ingested from animal-based foods. Retinoic acid appears to be ingested and transported across the intestinal wall in its native acid form. The differences between the absorption of these species appears to be important in the treatment of night vision and macular dystrophy (Section 18.8.3.6.2 & Sections 18.8.9.3 through 18.8.9.5).

A very superficial survey at the local pharmacy indicated that most of the vitamin A supplements describe their source as fish liver oil (no indication of whether it is from fresh water fish, vitamin A2, or marine fish, vitamin A1. One offering indicated 30% of the vitamin A content was β–carotene. A product from Nature Made indicated it contained retinol palmitate (a retinyl ester). Cod fish (Gadidae) are one marine source of vitamin A1. See Section 1.2.1.1.

Rigtrup et al. have provided considerable detail relevant to the absorption of vitamin A and its precursors30. Their Abstract notes,

“Retinol esterified with long-chain fatty acids is a common dietary source of vitamin A, that is hydrolyzed prior to absorption. An intrinsic brush border membrane retinyl ester hydrolase activity had previously been demonstrated for rat small intestine [Rigtrup, K. M., & Ong, D. E. (1992) Biochemistry 31,2920-29261. This activity has now been purified to apparent homogeneity by a three-column procedure to obtain a protein of apparent molecular weight of 130 000. The purified protein retained the pattern of bile salt stimulation, specificity for the acyl moiety of the retinyl ester, and the K, values previously observed for the activity present in the isolated brush border membrane. This protein also had a potent phospholipase activity, while having little measurable ability to hydrolyze triacylglyceride and cholesteryl ester substrates. The retinyl ester hydrolase enzyme was localized to the distal two-thirds of the small intestine.” Italics added for emphasis.

They go on in their main text,

“We discovered two distinguishable activities. One is of pancreatic origin (possibly cholesterol ester hydrolase) and primarily hydrolyzed esters with fatty acyl chains of less than 10 carbons in length. The other is intrinsic to the brush border membrane and constituted the majority of brush border activity toward long-chain retinyl esters that are typical of those that would be found in the diet.” “The retinyl ester hydrolase activity was stimulated more strongly by unconjugated bile salts, while the phospholipase activity was stimulated more strongly by their taurine-conjugated analogs.” “The enzyme is not simply a nonspecific lipase. For example, pancreatic cholesterol ester hydrolase is also known as carboxyl ester hydrolase (or lipase) because it has a broad specificity, capable of significant hydrolysis of long and short-chain triacylglycerols, phosphatidylcholines, retinyl esters, and nonphysiological esters such

30Rigtrup, K. Kakkad, B. & Ong, D. (1994) Purification and Partial Characterization of a Retinyl Ester Hydrolase from the Brush Border of Rat Small Intestine Mucosa: Probable Identity with Brush Border Phospholipase B Biochem vol 33, pp 2661-2666 Dynamics of Vision 7- 17

as p-nitrophenyl acetate, as well as cholesterol esters.” “Hydrolysis in the proximal third of the small intestine, where the intrinsic enzyme is absent, is most likely carried out by the trihydroxy bile salt-requiring activity of pancreatic origin (Rigtrup & Ong, 1992).” They conclude by suggesting an alternate name for the enzyme they studied most completely. Sikkens et al. have reported on pancreatitis and related diseases leading to fat-soluble vitamin absorption problems in 201331. They identified a specific cohort of 40 patients. “Very few studies have addressed the subject of fat-soluble vitamin deficiencies and low bone density in this patient group.” Their study only isolated a very few vitamin A deficient subjects and they referred to the study by Dutta et al. of 1982.

Dutta et al. provided a large clinical study of pancreatic insufficiency leading to deficiencies in the fat-soluble vitamins32. The study showed the serum vitamin A levels and the degree of night vision sensitivity loss among primarily chronic alcohol pancreatitis sufferers. The following sub-sections strongly support the hypothesis of this work that there are four distinct chromophores of animal vision, the Rhodonines, that they are each a diol and thereby members of the retinine family rather than the simpler retinenes that include retinol. The retinines are delivered to the RPE cells of the retina in complete diol form, and that groups of each specific Rhodonine type are stored in separate globules within the RPE prior to their dispersal to the outer segments of the photoreceptors after exudation into the IPM of the retina. These globules have frequently been described as granules, inplying a granular nature. An alternate description has been as micelles, implying a liquid crystalline spherical state with the hydrophilic portion of each molecule facing outward, and the hydrophobic portion facing inward.

Section 7.1.2.5 will review the literature of retinoid transport for non-visual purposes within the body. 7.1.2.1 Transport of the visual modality retinoids within the body

The goal of this subsection is to understand both the delivery of the Rhodonines to the RPE cells and the IPM of the retina, and the removal from the retina of the unloaded SRBP-TTR complex and the disposal or reuse of its components. The delivery begins with the acquisition and protective storage of one molecule of retinol within a SRBP-TTR complex. The protective aspect is necessary because of the increased delicacy of the retinene as it is converted to a retinine (a diol, Section 5.5.8) during its transport.

There are a variety of caricatures in the literature describing the flow of the retinoids through the system. Most of these caricatures illustrate different concepts in individual areas. Chader presents a particularly artistic caricature from a pharmacological perspective33. However, it may not treat the blood/RPE interface adequately from a chromophore chemistry perspective. Because of this complication, no overall flow diagram will be presented at this time. See Section 7.1.1.4.3. 7.1.2.1.1 Summary premises of retinoid transport within the vascular system

The following premises outline the discussion appearing in the following sections and place it in context with the larger theory; 1. The visual capability of all animals, including humans, employs a fundamentally tetrachromatic retina consisting of chromophores sensitive in the ultraviolet (UV-), short wavelength (S-), medium wavelength (M -) and long wavelength (L-) spectral regions. 2. The chromophores of all animal vision consist of a group of resonance conjugate retinoids, described as members of the retinine family to distinguish them from the retinene family. 3. These chromophores do not incorporate any protein material such as opsin. The opsin of Chordate vision is a substrate upon which the retinines are arranged as contiguous single layer liquid crystals. Opsin is not used, and has not

31Sikkens, E. Cahen, D. Koch, A. et al. (2013) The prevalence of fat-soluble vitamin deficiencies and a decreased bone mass in patients with chronic pancreatitis Pancreatology vol 13, pp 238–242 32Dutta, S. Bustin, M. Russell, R. et al. (1982) Deficiency of fat-soluble vitamins in treated patients with pancreatic insufficiency Ann Inter Med vol 97(4), pp 549–552 33Chader, G. (1984) Vitamin A. In Handbook of Experimental Pharmacology, NY: Springer-Verlag, vol. 69, pp. 367-384 18 Processes in Biological Vision

been identified in the retinas of Insecta or Mollusca. 4. The individual ligand of retinol identified within the opsin molecule of Chordata is present to facilitate the creation of opsin prior to its secretion by the inner segment of the sensory neurons of the retina. This ligand is not employed in the transduction of light within the visual modality. Its absorption cross section relative to the cross section of the individual molecule of opsin is minuscule and is not adequate to support a high sensitivity (quantum counting) retina. 5. For each of the four spectral types of sensory neuron receptors found in the retina of vision of most species, the appropriate chromophore is present as a single layer liquid crystal wherein all of the excited electrons generated in that liquid crystal are shared in a single excited state that delivers energy to the villi or microtubules of the sensory neuron. 6. The energy is received and transduced directly into an electrical signal by an active semiconductor device, similar to a man-made transistor, and defined as an Activa. 7.The discs of opsin secreted by the individual inner segments of a sensory neuron are in direct contact with the interphotoreceptor matrix (IPM). 8. The secreted discs of opsin are coated with a liquid crystalline chromophore stored by and released into the IPM from the retina photoreceptor epithelium (RPE). 9. The coated discs and chromophores have a life expectancy of approximately one week in the human species. These discs are continually destroyed and the chromophoric material recovered upon their arrival at the RPE in order to make room for the newly secreted discs at the inner segment interface. This process avoids a problem with cosmic ray damage to the eye. 10 The recovered chromophore material is stored as separate granules within the RPE cells until the required chromophore material is secreted back into he IPM where it proceeds to coat newly formed opsin substrates. These granules are documented in the literature. 11. Some chromophoric material is lost in the recovery and re-secretion process. Thus there is a need to replenish the reservoir of chromophores within the RPE cells.

12. Retinol, and retinal, are very low electrical band gap molecules that are not compatible with the chemical environment of the vascular system. 13. Retinol (al) are transported from the stomach to the liver protected from the vascular environment. 14. Similarly, retinol (al) is prepared for transport to the RPE cells within the liver by complexing with an apo-retinoid- binding protein (RBP). The apo-RBP will be identified as a pre-holo-RBP in this description. 15. The holo-RBP complex involves a single ester now formed by chemical reaction between the pre-holo-RBP and a retinene structure. The new retinoid structure is now resonant due to its conjugated structure extending between its two oxygen atoms. This structure is a pro-retinine (with a second i) rather than a retinene (with a single i) The oxygen atom of the ester was provided by the RBP. 16. To complete the protection of the now even more sensitive retinine, a TTR is employed to physically isolate the only external atoms of the retinine component of the complex. The TTR stopper remains capping the bottle formed by the RBP while the “overall complex” moves via the vascular system to the vascular surface of the RPE cells (adjacent to Bruch’s membrane. 17. Upon arrival at the RPE interface, the TTR cap is discarded back into the blood stream and the pocket of the holo- RBP is interfaced with a similar pocket on a cellular retinine-binding protein (CRBP). 18. The retinine is released from the holo-RBP complex with the oxygen atom previously forming an ester with the RBP now an integral part of the resonant conjugate retinine. 19. The retinine is drawn into the protected environment of the RPE cell by the CRBP and stored as part of a chromophoric granule (see premise 10). 20. The post-holo-RBP has now lost at least one oxygen atom from its pre-holo-RBP configuration and possibly become structurally unstable due to the large void in its interior. In either case, the post-holo-RBP is no longer viable as a RBP. 21. The post-holo-RBP is released back into the vascular system with the intent it be moved to the kidney and cleared from the body of the animal. 22. Failure of the vascular system to remove the post-holo-RBP and the TTR molecule from the immediate region of Bruch’s membrane can result in the buildup of drusen (and not fuscin) within the choroidal vascularization of the eye near Bruch’s membrane. This can result in the earliest stage(s) of the medically recognized condition known as macular degeneration. 7.1.2.1.2 Ingestion or manufacture of Vitamin A

Wikipedia (ca. 2017) declares, “All-trans-retinol is by definition vitamin A.” This naive declaration appears to be untrue for a variety of reasons. 1. The entire nutrition community considers β-carotene a protovitamin A and discusses it as a legitimate source of vitamin A. It is easily hydrolyzed into all-trans-retinol in the proximal intestine. 2. InterPro notes, “Vitamin A has three active forms (retinal, retinol and retinoic acid) and a storage form (retinyl ester).” This latter description is more appropriate when concerned with vision. To make real progress in understanding the role of vitamin A in homeostasis, and certain diseases associated with vision, Dynamics of Vision 7- 19

β-carotene, retinol, retinal, retinoic acid and the retinyl esters must be considered. In a review, Reboul has provided a remarkably clear interpretation of the situation of these molecules in the context of enterocyte absorption in the intestinal tract34. From his position as a nutritionist, his Abstract is remarkably comprehensive. “Abstract: Vitamin A deficiency is a public health problem in most developing countries, especially in children and pregnant women. It is thus a priority in health policy to improve preformed vitamin A and/or provitamin A carotenoid status in these individuals. A more accurate understanding of the molecular mechanisms of intestinal vitamin A absorption is a key step in this direction. It was long thought that ß-carotene (the main provitamin A carotenoid in human diet), and thus all carotenoids, were absorbed by a passive diffusion process, and that preformed vitamin A (retinol) absorption occurred via an unidentified energy-dependent transporter. The discovery of proteins able to facilitate carotenoid uptake and secretion by the enterocyte during the past decade has challenged established assumptions, and the elucidation of the mechanisms of retinol intestinal absorption is in progress. After an overview of vitamin A and carotenoid fate during gastro-duodenal digestion, our focus will be directed to the putative or identified proteins participating in the intestinal membrane and cellular transport of vitamin A and carotenoids across the enterocyte (i.e., Scavenger Receptors or Cellular Retinol Binding Proteins, among others). Further progress in the identification of the proteins involved in intestinal transport of vitamin A and carotenoids across the enterocyte is of major importance for optimizing their bioavailability.” The opening and closing sentences are particularly important in the context of Public Health in the developing and third Worlds. The primary sources of Vitamin A in these areas are usually described as green vegetables. Most of the Vitamin A is derived from the organic chemical and protovitamin carotene. Carotene gives the fruit of green vegetables their orange color. In the more developed world, the primary source are the preformed sources of retinol, primarily the retinyl esters (with retinol palmitate the most important). Since Vitamin A is stored in the liver of Chordata, the liver of prey is a valuable and concentrated source of Vitamin A for carnivores. Whereas most preformed Vitamin A can be absorbed through intestinal wall, carotene apparently cannot. The protovitamin β-carotene is believed to be attacked in a dioxygenase cleavage in the intestine that creates two molecules of Vitamin A. All of the forms of vitamin A absorbed through the intestinal wall are sensitive to destruction by oxygen and are generally protected by transfer to the liver for storage via the lymph system.

Following storage in the liver, the vitamin is transported throughout the body via the blood system. The portion supporting vision is apparently transported as an alcohol, retinol, in complex with a variety of retinol-binding proteins, RBP’s. These are described in greater detail below.

Reboul notes that for the last forty years, absorption through the intestinal wall has been largely conceptual. Beginning with the turn of the Century, new studies have shed more light on the actual mechanisms involved. He notes a critically important aspect, “Although passive diffusion may occur at pharmacological concentrations of these compounds, a protein-mediated transport is clearly involved at dietary doses.” Reboul then presents an Overview of the conventional wisdom from the perspective of the nutritionist in his Section 2. He notes the measured mean absorption efficiency of β-carotene is relatively low, even though its range is wide, from 3% to 90%. The mean absorption efficiency of the retinyl esters appears to be higher with a range of 75% to 90%. He asserts, “the data obtained in healthy subjects have shown that gastric lipase does not significantly hydrolyze retinyl palmitate. The hydrolysis of esters of vitamin A thus occurs essentially in the duodenum” where other enzymes are available. The pancreatic juice contains two main enzymes that could perform this hydrolysis: cholesterol ester hydrolase (CEH) and pancreatic lipase (LP). The results of in-vivo experiments in mice suggest the in-vivo hydrolysis of retinyl esters is achieved by the LP, together with the pancreatic lipase-related protein 2 (with citation). “It is conceivable that some esters are taken up intact by the intestinal cell and hydrolyzed intracellularly.” He presents a figure 1 in his section 3 that is highly conceptual and will not be pursued here. See instead, the figures in Section 18.8.3.6.2 of this work. Section 3 also includes current research material that has not yet become settled understanding. Part of it involves potentially additional mechanisms involving absorption and/or transport enhancers labeled more briefly as “transporters.” Section 4 suggests that genetic differences may be related to the relatively broad differences in absorption efficiency among individuals. Some 56 subsequent papers have cited Reboul as of 2017. Remond et al. have provided an in-depth discussion (15 pages of citations) of digestion in the elderly35. A paper by McClements et al. has attempted to define a classification system

34Reboul, E. (2013) Absorption of Vitamin A and Carotenoids by the Enterocyte: Focus on Transport Proteins Nutrients vol 5, pp 3563-3581 35Rémond, D. Shahar, D. Gille, D. (2015) Understanding the gastrointestinal tract of the elderly to develop dietary solutions that prevent malnutrition Oncotarget vol 6(16), pp 13858-13898 20 Processes in Biological Vision

for the bioavailability of various ingested pharmaceuticals that may be of interest moving forward36. The article highlights potential strategies for increasing the oral bioavailability of nutraceuticals based on their nutraceutical bioavailability classification scheme ,NuBACS, where the designation, B*A*T*. The symbols are described as bioaccessibility (B*), absorption (A*), and transformation (T*) within the gastrointestinal tract (GIT). Bonrath, et al. have prepared an Encyclopedia entry on the history and industrial preparation of Vitamin A37. Saeed et al. have also attempted to define several aspects of Vitamin A absorption in liver disease38. These papers do not always recognize the variety of molecules found under the generic label, Vitamin A. The following citation provides substantial data on the action of pancreatic lipase on fatty acids, triglycerides. Pancreatic lipase ppt - SlideShare https://www.slideshare.net/fathima1995/biochemistry-pancreatic-lipase-ppt Nov 27, 2013 - PANCREATIC LIPASE DONE BY: BARAKATHU PEER FATHIMA INDIA Pancreatic Lipase Slide 10, from Figure 22.3 of the 2012, 7th Edition of “Biochemistry” shows how this lipase breaks down a triglyceride to a monoglyceride, Figure 7.1.2-2. In the case of retinol palmitate, it may go farther and free the retinol from the palmitate, shown here as R2, along with the remaining acetate group.

Figure 7.1.2-2 Breakdown of triglycerides by pancreatic lipase. It is suggested that this same enzyme might free a palmitate at R2 and the acetate group from the retinol group thereby supporting absorption of the retinol molecule through the wall of the small intestine. See text. From Fathima, 2012.

Drugs.com notes the ability of a commercial pharmaceutical to take this last step,

“The pancreatic enzymes in “Pancrelipase” catalyze the hydrolysis of fats to monoglycerides, glycerol and fatty acids, protein into peptides and amino acids, and starch into dextrins and short chain sugars in the duodenum and proximal small intestine, thereby acting like digestive enzymes physiologically secreted by the pancreas39.” - - - -

36McClements, D. Li, F. & Xiao, H. (2015) The Nutraceutical Bioavailability Classification Scheme: Classifying Nutraceuticals According to Factors Limiting their Oral Bioavailability Ann Rev Food Sci Tech vol 6, pp 299-327 37 Bonrath, W. Bruins, M. Mair, P. et al. (2015) Vitamin A in Kirk-Othmer Encycl Chem Tech NY: John Wiley 38Saeeda, A. Hoekstraa, M. Hoekea, O. et al. (2017) The interrelationship between bile acid and vitamin A homeostasis Biochim Biophysica Acta - Mol Cell Biol Lipids vol 1862(5), pp 496–512 39https://www.drugs.com/pro/pancrelipase.html Dynamics of Vision 7- 21

Figure 7.1.2-3 presents a simple schematic of the general metabolism of Vitamin A in vision. This figure can be compared with the earlier version of Dowling and Wald40. It provides more detail concerning the transport of Vitamin A and a more detailed description of the use of Vitamin A in vision. While being transported to the RPE cell interface, the material contains one ligand that can be described as an aldehyde. However, this ligand is not exposed to attack. It is contained within the transport complex. In passing through the RPE cell wall, it is converted into a Rhodonine where it is stored. During transport from the color granules of the RPE, it is again complexed with a transport protein before deposition onto the Opsin substrate. After deposition, the resulting coated substrate is conceptually described as rhodopsin. However, this is not a compound, only a conglomerate from a physical perspective. Upon excitation by light the Rhodonine chromophore of the conglomerate is excited. Excitation is terminated by the creation of a free electron in the dendrite associated with the disk stack and the chromophore is immediately ready for re-excitation. No requirement exists for the chromophore to be transported back to the RPE for purposes of stereochemical reconfiguration.

Figure 7.1.2-3 The general metabolism of Vitamin A. When removed from storage, Vitamin A travels different paths for vision and for growth and maintenance. That used in vision is transported within the SRBP/TTR complex. The material labeled “rhodopsin” in this figure is best described as an aggregate and not a chemical. Light electronically excites the chromophores but does not isomerize them. Modified from Dowling & Wald, 1960.

Figure 7.1.2-4 illustrates how the retinoids progress through the animal body in order to support the visual function.

40Dowling, J. & Wald, G. (1960) in Vitamins and Hormones, vol 18, pg 537 22 Processes in Biological Vision

Figure 7.1.2-4 The general flow of Retinoids within the animal body and various steps used to recover the chromophores. Top, the interconversion of the chromogens of vision. Middle, the conversion of Vitamin A into the chromophores of vision. Bottom, the alternative approaches to the extraction of the chromophores of vision from retinal extracts. Aggressive extraction causes the chromophores to be destroyed, resulting in residues frequently identified as retinol or retinal using simple chemical tests. Low energy extraction and recrystallization allows the separation of the true chromophores. See text.

The figure also shows how they may be recovered for scientific evaluation. Vitamin A plays many roles in the growth and operation of the animal body. Its importance in vision has been recognized (indirectly) for thousands of years. It was recognized that lack of certain foods in the diet led to night blindness--a lack of photoreceptor sensitivity (Section 4.6.3.3). In more recent times, a lack of vitamin A in the diet has been shown to result in poorly formed outer segments associated with the photoreceptor cells of the retina. The upper portion of the figure illustrates the many forms of the retinoids used in the initial ingestion, storage, and transport of the material. These materials can all be considered chromogens. Only the material shown being transported down to the RPE cells will ultimately become chromophores. The proto chromophores are first transported to the liver by LRBP. They are withdrawn from the liver by the holo-SRBP +TTR complex. After delivery to the RPE cells, the material is transported by the CRBP’s within the RPE. The material Dynamics of Vision 7- 23 appears to be converted from a chromogen to a chromophore as part of the transfer process between the holo-SRBP + TTR complex and the CRBP’s within the RPE cell receptor locations. There appears to be a difference in how the retinoids are transported within the blood stream. For those destined to be used in the formation of the chromophores of vision, they appear to be transported by a specific group of retinol-binding proteins. These retinol-binding proteins appear capable of interacting with unique binding sites found on the surface of RPE cells. The nature of this interaction is critical to the formation of the resonant retinoids, the Rhodonines. Heller41 has studied the transport of the retinoids in considerable detail. In the above paper with Chen, he has shown that there are no receptor sites on the photoreceptor cells for the type of retinol-binding protein used to transport the retinoids to the RPE. His conclusion is that the retinoids used to form the chromophores can only enter the IPM via the RPE cells. The retinoids required for growth and normal metabolism must enter the photoreceptor cells by a different mechanism, probably via a complex with serum albumin. He also found that retinoic acid cannot participate in the interaction with the RPE receptors. This is probably due to two aspects of the molecular structure of retinoic acid. First, the carboxyl group is probably too stable to give up one of its oxygen atoms in the interaction. Second, the presence of two oxygen atoms in the terminal ligand of the acid prevents it taking up another oxygen at one of the critical locations along the polyene chain of the retinoid required to form a Rhodonine. Heller, working in vision, and both DeLuca42 and Chader43 working in pharmacology, have found an unidentified group of “retinol-like” materials in their studies. Chen & Heller stressed the fact that the material found in the RPE cells and in the IPM is retinol-like but not retinol. An exception occurs in the 2nd paragraph of the abstract to the paper where an editor has reduced “retinol-like” to “retinol.” It is proposed that their retinol-like material of page 5220 is Rhodonine. Here again, this work does not support Heller’s rationale (for the material not being retinol) but it does support his conclusion.

Once the material has been transformed into the resonant form of the retinoids in the above interaction, the chromophores become attached to a second group of retinol-binding proteins. More specifically Rhodonine-binding proteins support their transport through the RPE cells and onto their final destination in the disks of the Outer Segments. This subject will be discussed in detail in the next Section. Whereas the Rhodonines are transported by diffusion through the IPM to reach the formative region of the disks, this is not the case for their return. The Rhodonines are transported back to the RPE by the physical motion of the disks as they are replaced, on a regular basis, through growth. This presents a different mechanism than generally assumed in the literature and restated by Nickerson, et. al44. It eliminates the need to return the stereo-isomer of the chromophore to the RPE for regeneration as a functional chromophore after each exposure to light.

The lower section of the figure attempts to illustrate the various avenues to the recovery of the chromophores of vision. The right-hand path is meant to describe the conventional techniques using sodium salts and other chemically aggressive media that normally reduce the Rhodonines back to their chromogens. Alternately, the recovered material is only tested for the presence of a retinoid in aqueous solution. In either case, the experiment fails to isolate the individual chromophores of vision. The left-hand path in the figure is meant to describe an alternate path using less aggressive media. It also recognizes the importance of the liquid crystalline state in the isolation of the individual chromophores. The suggested path is through multiple recrystallizations of the material recovered from the Outer Segments.

- - - --

A major question arises as to the routing of the artery supplying blood to the oculars relative to the blood-brain-barrier (BBB). The BBB is generally associated with the walls of the vascular conduits entering (or leaving) the brain. The goal of the BBB is to isolate the CNS from dangerous chemicals that might exist within the general circulation. The SRBP-TTR-Rhodonine loop has two potential points of origin; the choroid plexus located within the BBB of the CNS or the liver. The liver has traditionally been considered the major source of the SRBP-TTR-Retinol complex that eventually delivers Rhodonine() in one of its four forms to the RPE cells and/or the IPM of the retina.

41Chen, C. & Heller, J. (1977) Uptake of retinol and retinoic acid from serum retinol-binding protein by retinal pigment epithelial cells. J. Biol. Chem.. vol. 252, no. 15, pp. 5216-5221 42DeLuca, L. (1977) the direct involvement of vitamin A in glycosly transfer reactions of mammalian membranes. Vitamins & Hormones. vol. 35, NY: Academic Press. pp. 1-57 43Chader, G. (1984) Vitamin A. Hdbk. Exp. Pharma. Vol. 69 pp. 367-384 44Nickerson, J. Borst, D. Redmond, T. Si, J-S. toffenetti, J. & Chader, G. (1991) The molecular biology of IRBP: application to problems of uneitis, protein chemistry, and evolution. In Molecular Biology of the Retina: basic and clinically relevant studies. NY: Wiley-Liss, Inc. pp. 139-161 24 Processes in Biological Vision

The simplest routing of the retinal artery would be to avoid the BBB completely. - - - - Comments appear occasionally in the literature suggesting that Arthropoda and Mollusca do not use Vitamin A. This may be correct regarding growth and general stasis. However, the measured spectral performance of their visual systems suggests that their chromophores exhibit the same resonant conjugated structure as for Chordata. Although this structure could be obtained through other processes, Vitamin A appears to be the most efficient source for such chromophores. As noted in Section 1.2.1.1, there are multiple forms of Vitamin A 7.1.2.1.3 Visual retinoid transport within the vascular system EDIT

[xxx combine with next section or move into 7.1.2.2 ] Although many older vision-related documents say that Vitamin A (now Retinol) is transported through the bloodstream as a free alcohol45, this is not supported in more recent documents focusing specifically on the retinoids. Many more recent articles in the literature suggest that the retinoids of vision are so sensitive to oxidation that they cannot exist within the circulating bloodstream, and quite probably, within many body cells. Protection of the retinoid is an important responsibility of the RBP’s. It also accounts for the presence of RBP’s peculiar to the Lacteal region, the bloodstream and within the cells of the RPE.

Goodman46 described the actual situation in detail, prior to the putative existence of the resonant retinoids that can emulate the retinenes. He said that Retinol was stored in the liver. When needed by the organism, Retinol is combined with an RBP for protection and then released into the bloodstream. However, the transport of Retinol through the animal bloodstream involves another complication. Although a single binding protein can aid the transport of a single delicate retinoid, it cannot fully protect it from attack. To achieve both transport and protection in a hostile medium, the retinoid requires a more complex transport mechanism. The binding protein involved, which will be labeled SRBP according to the above nomenclature, apparently encloses most of the retinoid but requires another protein, plasma transthyretin (TTR) to protect the retinoid fully. The analogy has been made to a cork and a bottle. The SRBP is the bottle containing the retinoid and TTR is the cork. He noted the fact that a different species of SRBP has been isolated from fish. He did not specify, but it can be assumed, he is speaking of fresh water fish where the retinoids present are related to Vitamin A2. It has also been noted that the holo-SRBP complex is released from the Liver. After the retinoid is delivered to its target location, the apo-SRBP is filtered out of the bloodstream and destroyed. It does not re-circulate. This lack of conservation of a valuable binding protein suggests another option. It is possible that the SRBP released by the liver in holo- form is not identical to the apo-SRBP leaving the target site. This would imply that the retinoid absorbed by the target site may differ from the retinoid released by the liver. As a working hypothesis, it can be assumed that the retinol released by the liver is transformed into Rhodonine within the bottle. This would occur through a change in the actual chemical structure of the “SRBP.” The implication being that the protein found in apo-SRBP within the liver and that found in the bloodstream are different. In this scenario, the apo-form found in the bloodstream could not be reused in a re-circulating mode.

Without being more specific, Gamble & Blaner47 have given the half-life of “RBP-bound retinol” as 12 hours based on Valquist, Peterson & Wibell48. In a well-nourished Western population, the concentration of the combination is given as 2-3μmol/liter based on Soprano & Blaner49. Gamble & Blaner also provide additional information on the molecular weights of the SRBP and TTR and their gross interrelationships with retinol. They also address the human gene producing SRBP. If Retinol requires the protection of an SRBP plus plasma transthyretin to move from the liver to the target location, another question must be addressed. How does retinol get from the small intestine to the liver where it is combined with

45Dowling, J. & Wald, G. (1960) Vitamins & Hormones. Vol. 18 pg. 537 46Goodman, DeWitt. Op. Cit. pg. 42-82 47Gamble, M. & Blaner, W. (2000) Factors affecting blood levels of Vitamin A In Livrea, M. ed. Viatmin A and Retinoids: An Update. Berlin: Birkhauser Verlag pg 1-16 48Vahlquist, A. Peterson, P. & Wibell, L. (1973) Metabolism of the Vitamin A-transporting protein complex Eur J Clin Invest vol. 3, pp 352-362 49Soprano, D. & Blaner, W. (1994) Plasma retinol-binding protein In Sporn, M. Roberts, A. & Goodman, D. eds. The Retinoids. NY: Raven Press pp 257-282 Dynamics of Vision 7- 25

the SRBP? Although not widely discussed, Stacy & Santolucito50 present caricatures and say: “There are two routes of absorption in the intestine: (1) absorption directly into the bloodstream, and (2) absorption into the lacteals, which are the lymphatic vessels of the intestinal mucosa.” Thus, two options are presented. After absorption, retinol could move to the liver through the lacteals, thereby avoiding the bloodstream. Alternately, retinol could combine with another RBP while passing through the intestinal wall. Both approaches could logically employ an LRBP (CRBP(II)) such as that discussed by Ong51. - - - - Monaco (2000) provides good information on how the SRBP + TTR may load the retinol molecule prior to transport.

As asserted by Monaco (2000), “The ligand transported by the complex is exclusively all-trans retinol though the affinity of RBP for other retinoids, most notably retinoic acid is quite similar [32].” As developed below, and in Section 7.1.4.1 [xxx confirm ], this statement may be too strong and only apply to the uptake of all-trans retinol and not apply strictly to the form of the retinol during transport following the uptake. The details of how retinol is transported within the SRBP + TTR complex may still be controversial. However, the general pattern can be described based on the literature. Figure 7.1.2-5 is in color to support the following discussion. There is a black & white version in Monaco (2000) that shows the structure but not the inter-relationships between the moieties. Both figures show the tunnel through the TTR where thyroxine would be found if the complex were transporting thyroxine. In this figure, the total complex is carrying two retinol molecules, one in each SRBP connected by a hydrogen (London) bond associated with the alcohol group at the end of the conjugated aliphatic chain. This hydrogen bond joins the TTR molecule at a glycine peptide in position 83. In the human variant of this complex, Monaco (2009) indicates the retinol is joined to the SRBP by an ester-bond nominally at carbon position 5 of the β- ionone ring of retinol as shown in the left frame of [Figure 7.1.2-4 ]. Monaco did not discuss this statement further and did not indicate why the ester could not occur at other locations along the retinol backbone. As propose here, and developed in Sections 5.5 and specifically 5.5.10.1.2 to explain the spectral performance of the four unique chromophores of the Rhodonine family, the ester can be formed at carbon 7, 9 or 11 of the retinol. This alternative ester location may result in a different peptide of the SRBP being involved in the ester.

[XXX coordinate with figure xxx ]

50Stacy, R. & Santolucito, J. (1966) Modern College Physiology. St. Louis: C. V. Mosby pp. 318-320 51Ong, D. op. cit. pp. 225-232 26 Processes in Biological Vision

Figure 7.1.2-5 Model of the structure of the hexameric complex (RBP)2-TTR containing retinol as determined by x-ray analysis of the orthorhombic crystals. The RBP molecules are shown in red, one of the TTR dimers is in green and yellow, and the other is in blue and turquoise. The retinol molecules are represented as space-filling models with grey carbon atoms and a red oxygen. See text. From Monaco, 2009.

Monaco (2009) provided a clearer model of just the single holo-RBP (RBP4) as prepared by Potterton et al52. and shown here as Figure 7.1.2-6. The figure does not appear in the cited reference; that paper describes a new open source molecular-graphics program in development. The monomer corresponds to the right monomer of the above figure where the hydroxyl group of the retinol is protected from the solvent by the adjacent TTR. No details are shown concerning the potential existence of the retinoid combined with the RBP to form an ester.

Monaco describes this protein as follows, the figure “is a ribbon representation of the molecule showing the bound retinol as a space-filling model. The secondary structure assignments for human RBP4 are for the beta strands the following: strand A, residues 22–30; B, residues 39–47; C, residues 53–62; D, residues 68–78; E, residues 85–92; F, residues 100–109; G, residues 114–123; and H residues 129–138. The alpha helix spans residues 146–158. The open end of the barrel is delimited by four loops joining strands A–B (residues 31–38), C–D (residues 63–67), E–F (residues 93–99), and G–H (residues 124–128). The first three are close to the retinol molecule and amino acids present in them participate in the contacts with TTR, the fourth is far from the ligand and is not involved in contacts with TTR in the complex. The loop connecting strands C and D has been found to be disordered in several X-ray structures which indicates a certain degree of flexibility which might be related to retinol release.

52Potterton, E. McNicholas, S. Krissinel, E. Cowtan, K. & Noble, M. (2002) The CCP4 molecular-graphics project. Acta Crystal D Biol Crystal vol 58, pp 1955–1957. Dynamics of Vision 7- 27

Figure 7.1.2-6 Ribbon representation of the plasma holo-RBP (RBP4) molecule. The bound retinol is represented as a CPK model wih its OH exposed to the solvent on the surface of the protein. Note the oxygen of the hydroxyl group shown at the left extreme of the CPK model. The figure was prepared with the program CCP4 mg. See text. From Potterton et al., 2002.

The old papers of Peterson & Berggard and of Peterson appear important to this work. Peterson & Berggard did demonstrate the ready availability of many oxygen atoms in both aspartic acid and glutamic acid peptides in human RBP53. These oxygen bearing peptides are relatively rare in other proteins; remarkably, they are the dominant fractions in Table IV. “The results of the present study are in agreement with the observations that the retinol-transporting protein is acidic, and has a molecular weight of 21,000 and a sedimentation constant of 2.3 S. In contrast to Kanai et al., who

53Peterson, P. & Berggård, I. (1971) Isolation and Properties of a Human Retinal-transporting Protein J Biol Chem vol 246(1), pp 25-33 28 Processes in Biological Vision isolated two forms of the protein, with and without retinol, we have isolated four molecular forms of urinary RBP.” Kanai did record the very high percentages of aspartic acid (15.8%) and glutamic acid (10.7%) in RBP. These percentages varied somewhat in their processing. Kanai also noted regarding the effectiveness of th protection provided by RBP, “The magnitude of the protective effect is indicated by the fact that retinol (bound to RBP) remains intact in whole plasma kept for several weeks at 4VC, whereas retinol itself is highly unstable and decomposes rapidly in solution in organic solvents when exposed to traces of oxygen or to light.” Peterson & Berggard also observed, “One possible mode of action might be a dissociation of RBP from prealbumin, whereafter vitamin A is emitted from RBP and deposited in a target cell. The free RBP could then be excreted into the urine. This view is supported by the observations that small amounts of RBP occur in free form in serum and that free urinary RBP seems to have a lower binding affinity to prealbumin than RBP which is prepared from the RBP-prealbumin complex.” It may be that the free urinary RBP has lost one or more oxygen atoms and is the residue, apo-RBP, after delivering a retinine to the RPE cells. In a subsequent paper, Peterson noted, “Experiments on serum indicated that the characteristics of serum RBP1 were different from those of urinary RBP.” On page 38, Peterson54 noted, “The result of the chromatography is shown in Fig. 6. The prealbumin-RBP material had apparently dissociated and emerged as two components. Immunological analyses by the single radial immunodiffusion technique showed that only the protein fraction eluted last consisted of RBP.” In figure 8, Peterson showed a distinct separation between the serum RBP-TTR complex (presumably containing retinol), purified serum RBP and urinary RBP using an immunodiffusion analysis. A key assertion on page 42 was, “ The plasma RBP and urinary RBP were identical except for a higher content of aspartic acid in the plasma preparation (see Peterson & Berggard, 1971).” The loss of an oxygen atom from an aspartic acid peptide at one or more location along the RBP molecule is exactly what would be expected according to the theory of this work! Table III indicates the surprisingly high number of aspartic acid and glutamic acid residues in both TTR and RBP. After a brief discussion, Peterson surmised, “A mechanism dissociated RBP from prealbumin close to or in connection with the target cells would thus be required.” This appears to refer to the delivery of the retinol to the RPE cells. His closing discussion appears to support this work. “There were clear differences in the content of aspartic acid between the serum and the urinary RBP.”

The 1984 paper by Newcomer et al. provides more information on the electrical charge of the holo- SRBP55. C “Human RBP has been sequenced and is composed of 182 amino acid residues (Rask et al., 1979).” C “The complex containing all trans-retinol has recently been crystallized by Newcomer et al. (1984b) and we are now able to describe its tertiary structure.” C “Figure 1 shows that RBP from three species (Sundelin et al., 1984) displays a mutation-free area involving the N-terminal residues, the C-terminal base of the ca-helix and the loop region around residue 80. This region is rich in charged residues, e.g., 10 out of the first 20 residues in the sequence are charged, which may explain why RBP and prealbumin dissociate at low ionic strength.” They also note, “It is not known how the transfer of retinol from its hydrophobic environment in RBP to the receptor is accomplished, but after the delivery RBP has virtually lost its affinity for prealbumin (Peterson, 1971). Such apo-RBP, isolated from urine (Peterson, 1971), does not crystallize under the same conditions as the holo-protein.” They also note, “ These data taken together suggest that RBP has different conformations depending on the binding of its ligand. Indeed, removal of the retinol from our model leaves a large, empty volume in the hydrophobic interior of the molecule. This would be most unusual and we must assume that the sheets collapse, probably triggering other conformational changes.” They also note, “The β-ionone ring of the retinol lies deepest in the pocket, with the isoprene tail stretching out almost to the surface of the protein. The molecule is totally buried within the protein (Figure 4), with the area accessible to a water molecule essentially zero. Only the very tip of the retinol is non-zero with the value of 1 Angstrom2 and this could be due to errors in the coordinates. The hydrophobic nature of the retinol binding site is consistent with the observations that RBP may interact with many different retinyl derivatives provided they contain the β-ionone ring and the conjugated double bond system of the isoprene side chain (Hase et al., 1976).” The retinines, a.k.a. Rhodonines(), of this hypothesis are clearly retinyl derivatives. Hase et al56. found “The apo- and holo-forms of RBP were separated by DEAE-Sephadex column chromatography and electrophoresis, and the presence of two types of both forms, that is, two apo-RBP's (Apo I and Apo II) (2) and two holo-RBP's (Holo I and Holo II) (6) was demonstrated.” After some calculations, “Based on this calculation, the preparation contained 86% apo- and 14% holo-forms of total RBP. The ratio of Apo I to Apo II was roughly calculated to be 3 : 4 and that of Holo I to Holo II was 4 : 3.” Their goal was specific, “In the present study, we purified RBP from urine freshly collected from patients with "Itai-Itai" disease, and found that the apoforms were more abundant than the holoforms. We then investigated the binding of apo-RBP with vitamin A derivatives and some related terpenoids, in

54Peterson, P. (1971) Characteristics of a Vitamin A-transporting Protein Complex Occurring in Human Serum J Biol Chem vol 246(1), pp 34-43 55Newcomer, M. Jones, T. . . .Rask, L. & Peterson, P. (1984) The three-dimensional structure of retinol-binding protein EMBO J vol 3(7), pp 1451-1454 56Hase, J. Kobashi, K. Nakai, N. & Onosaka, S. (1976) Binding of Retinol-binding Protein Obtained from Human Urine with Vitamin A Derivatives and Terpenoids J Biochem vol 79(2), pp 361-371 J Biochem (1976) 79 (2): 373-380 Dynamics of Vision 7- 29

order to seek a correlation between the binding affinity and the chemical structure of terpenoids.” During their studies, Hase et al. noted the high relative affinity of β-ionone (the stand-alone molecule) and β-ionylideneacetic acid. They exceeded the affinity of retinol to RBP. The state-of-the-art in the time of Hase et al. was not as advanced as now. As a result, the interpretations of those investigators must be re-examined carefully. However, their factual data is supportive of the hypothesis of this work.

Figure 7.1.2-7 reproduces an alternate view of holo-RBP with the ribbons labeled57. The accompanying text provides many more explicit facts about the configuration of the RBP. Page 92 describes the location of various residues most likely to interact with the retinoid.

Figure 7.1.2-7 Ribbon representation of 3D holo-RBP in Blomhoff. Retinol is shown in the binding pocket. The two loop surrounding the tip of the retinol molecule facing the viewer, appear to interact with the receptor as well as transthyretin. From Sivaprasadarao & Findlay, 1994.

57Sivaprasadarao, A. & Findlay, J. (1994) The retinol–binding protein superfamily in Blomhoff, R. ed. Vitamin A in Health and Disease. NY: Marcel Dekker Chapter 4 30 Processes in Biological Vision

The early work of Rask & Peterson also appears important to the hypothesis of this work even though the target was intestinal cells58. There conclusion was, “It is therefore suggested that there is a receptor for vitamin A on the cell surface which recognizes the protein part of the protein. ligand complex.” They note, “The fate of the newly absorbed retinol in the cells has been studied and will be reported elsewhere. The fate within intestinal cells may differ from that in RPE cells. Rask et al. have reported the complete sequence of human RBP59 as shown in Figure 7.1.2-8. The presence of so many peptides of aspartic acid and glutamic acid should be noted. Each of these peptides contains a carboxyl group unrelated to the atoms involved in chaining the peptides. These sources of oxygen atoms available to participate in a hydrogen bond with the aliphatic portion of retinol insures the possibility of forming an ester at multiple carbon locations of the retinol. Upon expulsion of the retinine from the RBP pocket, the apo-RBP will be different from the holo-RBP.

Figure 7.1.2-8 Peptide sequence of human RBP from Rask. See text. From Rask et al., 1979.

Figure 7.1.2-9, also from Rask et al. (1979) shows the preponderance of carboxylic groups within their RBP. Resolving which of the carboxylic groups are most amenable to esterification with retinol is a remaining challenge. Monaco (2009) has provided additional detail with regard to the monomer.

“The secondary structure assignments for human RBP4 are for the beta strands the following: strand A, residues 22–30; B, residues 39–47; C, residues 53–62; D, residues 68–78; E, residues 85–92; F, residues 100–109; G, residues 114–123; and H residues 129–138. The alpha helix spans residues 146–158. The open end of the barrel is delimited by four loops joining strands A–B (residues 31–38), C–D (residues 63–67), E–F (residues 93–99), and G–H (residues 124–128). The first three are close to the retinol molecule and amino acids present in them participate in the contacts with TTR, the fourth is far from the ligand and is not involved in contacts with TTR in the complex. The loop connecting strands C and D has been found to be disordered in several X-ray structures which indicates a certain degree of flexibility which might be related to retinol release.”

58Rask, L. & Peterson, P. (1976) In Vitro Uptake of Vitamin A from the Retinol-binding Plasma Protein to Mucosal Epithelial Cells from the Monkey’s Small Intestine J Biol Chem vol 251(20), pp 6360-6366 59Rask, L. Anundi, H. & Peterson, P. (1979) The Primary Structure of the Human Retinol-binding Protein FEBS Letters vol 104(1), pp 55-59 Dynamics of Vision 7- 31

Figure 7.1.2-9 Schematic outline of the various fragments and peptides used to establish the amino acid sequence of RBP. See text. From Rask et al., 1979.

Zanotti et al. have provided details on the various angles and planes associated with retinol with sufficient precision to allow detailed discussion of how the retinol might be joined to the apo- RBP in order to support the conversion of the retinene to one of the retinines60. They also examined the conformation of the RBP and TTR most supportive of their forming an RBP + TTR complex. Malpeli et al. have studied the RBP interface with TTR in bird species in order to gin further knowledge of the mechanisms involved61. They data may be helpful but is not critical to the discussions in this section.

Monaco (1995) has indicated that a maximum of two RBP molecules per TTR tetramer are required to support retinol transport. Monaco et al. provide detailed data supporting this maximum value62. They provided more specificity, “The complex circulating in blood is thought to be formed by one tetramer of TTR plus one monomer of RBP (the RBP concentration in normal human plasma is about 2mM and that of TTR about 4.5 mM) but it has also been shown that in vitro it is possible to form the complex with the stoichiometry of two RBPs per TTR tetramer.” Subsequently, Bellovino et al63. have stated explicitly in their abstract, “Retinol binding protein (RBP), the retinol-specific carrier, circulates in blood as a 1:1 complex with the homotetrameric protein transthyretin (TTR). Both RBP and TTR are synthesized and secreted by the hepatocyte. In this work, we have demonstrated, using HepG2 cells as a model system, that the association between the two proteins occurs inside the cell before secretion.” They further note, the secretion is from the endoplasmic reticulum. They also assert the RBP is only secreted in holo-RBP form in the presence of retinol. Bellovino et al. also introduced the concept of a chaperone supporting the formation and secretion of the holo- RBP + TTR complex.

When the retinene/retinine is delivered to the RPE cells of the retina, this work proposes that the oxygen of the ester remains with the retinene to form a resonant conjugate retinine when the hydrogen bond at the TTR is also severed. This conversion may occur in conjunction with the binding proteins within the target RPE cell. If this release of an oxygen atom from the holo-SRBP occurs, the apo-SBRP is now a different molecule (still of great complexity but possibly

60Zanotti, G. Marcello, M. Malpeli, G. et al. (1994) Crystallographic Studies on Complexes between Retinoids and Plasma Retinol-binding Protein J Biol Chem vol 26(47), pp 29613-29620 61Malpeli, G. Folli, C. & Bemi, R. (1996) Retinoid binding to retinol-binding protein and the interference with the interaction with transthyretin Biochim Biophys Acta vol 1294, pp 48-54 62Monaco, H. Mancia, F. Rizzi, M. & Coda, A. (1994) Crystallization of the macromolecular complex transthyretin-retinol-binding protein J Mol Biol vol 244, pp 110-113 63Bellovino, D. Morimoto, T. Tosetti, F. & Gaetani, S. (1996) Retinol Binding Protein and Transthyretin Are Secreted as a Complex Formed in the Endoplasmic Reticulum in HepG2 Human Hepatocarcinoma Cells Exper Cell Res vol 222, pp 77–83 32 Processes in Biological Vision

folded differently and recognizable by the organism as a different molecule). As discussed on conjunction with the drusen of macular degeneration (Section 7.1.4), this now foreign molecule must be removed from the choroidal vascularization of the retina to prevent the accumulation of material within the finer arteriole between the RPE and the choroid. Monaco (2000, pg 67) has addressed these changes by asserting, “The most important is a conformational change on the loop extending from amino acids 34 to 37, in particular, Leu 35 and Phe 36. The space left empty by the removal of the vitamin is filled in both cases by the aromatic ring of Phe 36 and solvent molecules and the movement of the Phe side chain drags the nearby amino acids into positions which are different from those adopted in the holoprotein.” More analysis is needed here since Leu does not contain an oxygen readily available for forming an ester (or hydrogen bond), with retinol. It is possible a nearby peptide provides the required oxygen atom. It is possible that the esterification of the retinol to a retinine occurs in conjunction with the transfer of the retinol to the RPE cell. - - - - [xxx edit into previous material ] The sensitivity of the retinoids to oxidation also provides a clear understanding of the reasons for the two barriers associated with the Inter Photoreceptor Matrix (IPM.) The IPM must be isolated from oxidizing agents to avoid destruction of some of its constituents. The Outer Limiting Membrane (OLM) and Bruch’s Membrane (and/or the “tight junction” shown in Ong) between the cells of the Retinal Pigment Epithelium (RPE) provide an isolated cavity. These barriers around the IPM serve to isolate the delicate retinoids within this space from oxidation by the plasma perfusing the remainder of the retina.

When the Vitamin reaches a location where it is needed, it passes through the cell wall by a chemical exchange in which the transport protein releases the vitamin and the vitamin is transformed into a different compound. In homeostasis, Vitamin A is transformed into Vitamin A Acid as it passes through the cell wall and it is then used in the growth and/or maintenance of a specific cell. In chromogenesis, the retinoid is converted to a resonant form of the retinoid, Rhodonine, in passing through the wall of the RPE cell.

- - - -

Cowan et al. provided a valuable paper in 199064 including new and very detailed material concerning retinoid transport by retinoid-binding proteins (RBP) resulting from x-ray crystallography. They also include valuable data on the cellular retinoid-binding proteins, such as CRBP, CRBPII and CRABP discussed in Section 7.1. They noted, “These proteins are highly homologous to each other but are specific for their ligands. They are members of another super-family of proteins that includes fatty acid binding proteins.”

Some assertions related to the spectrum of retinol cite very old references that did not anticipate the retinines as the actual chromophores of vision. The use of the label isoprene tail is slightly misleading when recognizing the actual chromophores of vision. The β-ionone ring of retinol is generally associated with the original isoprene structure. The schematic figures of Cowen et al. showing 2nd and 3rd order structures are not as clearly labeled as in Newcomer et al. of 1984 although the diagram of figure 6 and Table II contain the same information for the human species. They did not comment on the very high proportion of aspartic and glutamic acid residues in the schematic of the amino acid sequence of human RBP.

Cowen et al. note the ability of the RBP to bind both retinol and retinal. This is important because when in the resonant conjugate form, the distinction between these two labels is actually lost. They note citations in the literature related to RBP binding to various cis- isomers of all trans-retinol and all trans-retinal. However, they note, “Although we have modeled these compound in RBP, we consider the models to be speculative (and of no biological relevance).” Figure 13 of the Cowan et al. paper may show correct distances. However, the reader is cautioned that the representation of retinol in a 2D ball and stick form is very misleading. There is actually a very large dihedral angle between C5-C6 and C7-C8 as shown in Figure 1 of Cowen and more fully documented in Calderone (2003) as discussed below. In 1990, Cowan et al. give this angle as 62 degrees. Either of these large dihedral angles place the oxygen at C5 in rhodonine(5) at a significantly different distance from the oxygen at C15 than previously used in this work to calculate the long wavelength absorption spectrum of the L–channel chromphores of vision (Section 5.5.8.2). This large dihedral angle from crystallography is not incorporated into the Jmol file of the Royal Society of Chemistry database and displayed by ChemSpider as of April 2017.

64Cowan, S. Newcomer, M. & Jones, T. (1990) Crystallographic refinement of human serum retinol binding protein at 2 Angstrom resolution Protein Struct Funct Bioinform vol 8, pp 44-61 Dynamics of Vision 7- 33

Cowen et al. discuss the potential for a recognizable family of proteins based on x-ray crystallography whose function is to solubilize small lipid molecules. They explore a variety of molecules that might belong to such a class but do not appear to include the cellular RBPs of the RPE. They do cite Jones et al. (1988) as providing a comparison of RBP and CRBP families. They note a variety of errors in the reported alignments of many of their candidates drawn from the literature. The closing paragraph of Cowen et al. suggesting that the post holo-RBP may no longer be viable is worthy of careful consideration. Section 5.5.8.3.1 discusses the 2nd order spectral peak of the L-channel chromophore based on x-ray crystallographic measurements. - - -- Zanotti & Berni provided a useful review in 200265. Figure 3 shows the relative size of the SRBP and TTR proteins relative to each other and to retinol. They noted, “The molecular mechanism of ligand binding and release for plasma RBP remains to be clarified.” The remainder of the paragraph following this assertion is useful but largely speculative. They do note the position of the retinene within SRBP leaves the hydroxyl group largely exposed to solvents in the absence of TTR. This group is readily replaced by an aldehyde or an acid group without decreasing this exposure or affecting the binding to the SRBP. Much of the paper is dedicated to cellular retinol binding proteins (CRBP’s). The use cartoons to demonstrate the close familial relationship between all of these retinol binding proteins. The extracellular portion of this family are generally classified as extracellular lipid-binding proteins (eLBPs or lipocalins).

- - - -

Calderone et al. have presented very high resolution data relating to holo-RBP66. The material is quite detailed and appears relevant to this discussion. As they noted in 2003, “the mechanism of retinol binding and release remains to be clarified.” They also noted, “It has been proposed that the release of retinol might require a reversible transition of the protein to a poorly structured state, perhaps a molten globule state, under the influence of mild denaturing conditions, such as a relatively low pH.7–9. The retinol–RBP complex might be subjected to mild denaturing conditions, possibly facilitating the delivery of the ligand from its carrier protein, on or near the plasma membrane of target cells.” The conversion of the retinene, retinol, to one of the Rhodonines could involve such “mild” denaturing. However, the literature suggests the apo-RBP is not viable or reusable and is disposed of by the kidneys. Calderone et al. describe RBP as consisting of 183 amino acid residues. Calderone et al. note the earlier work of Cowan et al. and assert with regard to the retinol, C “The C15–OH bond is nearly perpendicular to the plane of the polyene chain (Figure 1(b)), such that the O atom can participate in two H-bond interactions: one with the amide N atom of Q98 (OH–N, 2.85 D) and the other with water molecule 295 (OH–WAT295, 2.72 D). In turn, the latter solvent molecule is H-bonded to the amide carbonyl O atom of F96 (WAT295–O, 2.71 D) and is kept in place in a tetrahedral arrangement by H-bond interactions with two additional water molecules (WAT381, 2.84 D, and WAT425, 2.61 D) (Figure 1(b)).” Their comments related to the β-ionone ring portion of the retinol are considerably more complex. C “The C5–C6–C7–C8 dihedral angle, defining the orientation of the β-ionone ring with respect to the polyene chain plane, is 54 degrees, a value that compares quite well with those for retinol bound to the other RBPs whose structure is known, as well as for some retinoid compounds in the solid state (see Cowan et al. (1990) for a discussion of torsion angle values). Instead, negative values for the same angle characterize retinol bound to rat CRBP I and II and to zebrafish CRBP (–84 degrees). The C5–C6–C7–C8 dihedral angles for RBP and CRBP-bound retinol molecules are such that the orientation of the β-ionone ring with respect to the plane of the polyene chain is opposite in holo-RBPs compared to holo-CRBPs. Therefore, retinol undergoes a significant conformational change when it is delivered by holo-RBP to plasma membranes and is subsequently bound by CRBPs within the cytoplasm of target cells. This change can be assumed to be induced by the quite different and strong interactions established between the vitamin and residues lining the binding cavities of the two types of carrier proteins.” This work indicates that, in the process of generating the long wavelength chromophore, rhodonine(5), C5 becomes esterified with one of the nearby aspartic acid or glutamic acid residues during the formation of holo-RBP and that the oxygen atom at the root of this ester remains with the retinine when it is released at the RPE lemma. As a result, the molecule delivered to the CRBP is not retinol (a retinene) but a rhodonine (a retinine). Calderone et al. continue, “Therefore, retinol undergoes a significant conformational change when it is delivered by holo-RBP to plasma membranes and is subsequently bound by CRBPs within the cytoplasm of target cells. This change can be assumed to be induced by the quite different and strong interactions established between

65Zanottia, G. & Berni, R. (2002) Retinoids in Mammals: A Crystallographic Perspective Croatica Chemica Acta vol 75(3), pp 835-845 66Calderone, V. Berni, R. & Zanotti, G. (2003) High-resolution Structures of Retinol-binding Protein in Complex with Retinol: pH-induced Protein Structural Changes in the Crystal State J Mol Biol vol 329, pp 841–850 34 Processes in Biological Vision

the vitamin and residues lining the binding cavities of the two types of carrier proteins.” It is proposed that this last quotation can by interpreted to mean; the retinol is no longer a retinene but is transformed to a resonant conjugate retinine when it is released at the RPE lemma. Calderone et al. note the presence of several water molecules within the cavity containing the β−ionone ring of the retinene. One of these could be the result of a condensation reaction forming the proposed ester between the retinol and the RBP. Calderone et al. describe the positions of these water molecules in some detail (page 844). They go on to discuss potential RBP/retinol interactions in the vicinity of the β-ionone ring. “Residues R60, R62, D39 and D68 are involved in a network of interactions occurring at the protein surface, in a region that is possibly critical for retinol binding/release (Figure 2). In fact, as mentioned above, the loop 62–67 has been found to be relatively disordered in the RBP structures. The network of salt bridges/H-bonds they form is quite well preserved at the various pHs, except for pH 2, at which pH value the R62–D68 and D68–R60 interactions become looser.” The conclusions of Calderone et al. appear significant to this discussion. They note their work with the crystalline form of the molecules may record results that differ from those that would be obtained in solution. C “In the crystal structures of holo-RBP that we have determined, the ligand is bound nearly unaltered and the overall structure appears to be preserved at pH as low as 2. In contrast, holo-RBP in solution undergoes denaturation and concomitantly releases retinol at pH below 4–4.5. The latter processes are impeded until the crystal state is preserved by crystal packing forces, and the pH-induced modifications that we observe in the crystal state can be considered to be representative of changes occurring at the initial stages of the acidic denaturation of RBP.” C “The flexible state of RBP that we observe in the crystal state at low pH is likely to represent a key intermediate in the processes of protein denaturation and retinol release.”

Calderone et al. note, “Coordinates have been deposited at the Protein Data Bank as 1KT6 (holo-RBP at pH 9), 1KT7 (holo-RBP at pH 7), 1KT5 (holo-RBP at pH 4), 1KT4 (holo-RBP at pH 3) and 1KT3 (holo-RBP at pH 2) for immediate release.”

- - - - Dynamics of Vision 7- 35

Monaco has presented a summary of the knowledge across many fields related to the holo- RBP + TTR complex in 200967. His figure 8.5, compares the RBP and the TTR sequences of six different species and that of the human using the sequence alignment program CLUSTAL W. In 1984, Monaco indicated the RBP had a molecular weight of 20,600 and consisted of 182 amino acids; this figure from 2009 shows 183 amino acid residues. A black dot identifies in each case a residue involved in the contacts between the RBP and the TTR in the complex. It is reproduce here with additional notation as Figure 7.1.2-10. The results are very interesting for the aspartic acid (D) and glutamic acid (E) residues, those capable of providing oxygen to form an ester between the RBP and the retinol. The numbers below the matrix indicate the number, but not the position of aspartic or glutamic acid residues nearly fully conserved between the species shown. Such a high concentration of these residues (25 out of 180+) within the RBP’s suggests they have a role to play that has not been recognized in the literature.

Figure 7.1.2-10 Primary structure sequence alignment of the RBPs of six vertebrate species. The numbers below the matrix indicates a column containing either aspartic or glutamic acid that is nearly 100% conserved. However, the location labeled 10,24 & 25 are less than 100% conserved due to changes in the chicken. See text. Modified from Monaco, 2009.

It is proposed here that the high number of oxygen orbitals provided by the many aspartic and glutamic acid molecules present in the RBP’s found in the liver can support the transition of the retinenes to retinines at various locations along their conjugated carbon chain (thereby forming each of the desired rhodonines(). Alternately, it is possible that a molecule of retinol is required to be present in the liver in order to seed the creation of an RBP with an oxygen orbital shared between these two moieties at one of the preferred locations. With the association of a TTR cap, the resulting complex containing a retinine can be transported to the RPE and delivered as a specific rhodonine, leaving the post holo- RBP residue non-viable for future retinene acquisition and transport. 7.1.2.2 Transport of the retinoids within the retina

[xxx combine with section 7.1.2.1.2 under a new name and edit ] See Section 4.6 for more of the detailed material on this subject

67Monaco, H. (2009) The Transthyretin–Retinol-Binding Protein Complex In Richardson, S. Cody, V. ed. Recent advances in TTR Evolution, Structure & Functions. NY: Springer available online Chapter 8 36 Processes in Biological Vision

In the case of vision, it appears Vitamin A passes from its protective capsule in the blood stream through the cell wall into the RPE. In the process is converted into a “double ended” molecule exhibiting two separate functional groups containing a polar atom, oxygen. It appears that this double ended characteristic makes it very easy to transport the molecule through a series of environments within the RPE. This is advantageous on the way to its target location as a liquid crystalline coating on a target substrate, opsin, in the disks of the Outer Segment, OS. The double ended structure of the chromophores is actually a resonant structure. This structure can be described as having an oxygen atom double bonded to one end of the molecule and a hydroxyl group singly bonded to the other end. Alternately, it can be described as having one oxygen atom attached to each end of the conjugated carbon chain by 1.5 bonds. In the first case, if a more aggressive molecule attempts to attach itself to the doubly bonded oxygen atom, the oxygen atom will drop one of its bonds to the retinoid. This will cause the oxygen atom on the other end to release the hydrogen atom in the hydroxyl group and to establish a second bond with the retinoid. If a more aggressive molecule attempts to attach itself to one of the oxygen atoms by a single bond, the situation is different. The other oxygen bond will now exhibit a double bond with the conjugated carbon chain in order to maintain correct electrical balance. In this case, there is no need to consider in detail whether a hydrogen atom was captured or released by the Rhodonine molecule. However, the concept of a polar atom connecting to a conjugated backbone by 1.5 bonds implies that hydrogen or other simple molecules are available to share a single charge with the Rhodonine molecule at any time via a hydrogen bond. The general flow of the retinoids from the bloodstream to a coating on the disks of the outer segment is illustrated in Figure 7.1.2-11. This figure is heavily modified from, but uses the style of, a recent figure by Crouch & Ma68. The retinol is shown being locked into the SRBP-TTR bottle during transport through the bloodstream by an esterification with the SRBP. Upon transfer through the cell membrane of the RPE, the material is converted into a retinine as the oxygen of the ester remains with the retinoid. The oxygen is shown associated with C5 in the figure as an example. This is the configuration of the L–channel chromophore of vision. It could be shown alternately at the C7, C9 or C11 position and represent the other chromophores (see Section 5.5.8).

The material is transported and/or stored within the RPE through a second esterification assumed to be related to C15 in the figure (as assumed by Crouch & Ma). This is a logical assumption if only one type of cellular retinoid binding protein is present within the RPE. On the other hand, there may be a variety of such proteins within the RPE. This would suggest that these CRBP’s might be selective for the individual chromophores of Rhodonine. In that case, it is more likely the temporary esterification may occur at the oxygen associated with C5, C7, C9 or C11 on a selective basis. This appears to be the case in practice. If so, it accounts for the variety of CRBP’s found in the RPE. This option is shown below the “OR” line in the figure.

The chromophores of vision pass through the RPE cell membrane into the IPM following a similar procedure as when transferring from the bloodstream to the RPE. Here, the chromophore appears to be esterfied with a single IPM retinoid binding protein (IRBP). This leaves the steric specific portion of each chromophore free to associate with its like kind in a liquid crystal on the surface of the disks. When released by the IRBP, the chromophore becomes part of the liquid crystalline structure that is optimally sensitive to light of a particular visual wavelength. This material can be excited by a photon as shown at the lower right. This excited material can be de-excited by the transfer of energy to the neural system. Following de-excitation, the material is again sensitive to light. This process does not involve any steric change (isomerism) or require the transport of any retinoid back to the RPE through the IPM. The excitation and de-excitation events are instantaneous for all practical purposes.

Eventually (typically seven days in humans), the chromophores on the surface layer of the disks are recovered by phagocytosis by the RPE. The chromophores are then returned to an esterfied state in preparation for storage, or transfer to the IRBP of the IPM for reuse. This sequence can be compared with a similar but more conventional conceptual sequence proposed by Crouch & Ma. Their sequence is not completely conventional since it suggests the chromophores of vision reach the photoreceptor space via the RPE. They present a floating model. It does not address the question of whether their putative chromophore is embedded within a rhodopsin molecule or if it passes through the mitochondria of the photoreceptor cell.

68Crouch, R. & Ma, X-J. (2000) The role of Vitamin A in visual transduction In Livrea, M. ed. Vitamin A and Retinoids: An Update. . . Berlin: Birkhauser Verlag pp 59-72 Dynamics of Vision 7- 37

Figure 7.1.2-11 Proposed flow of chromogens (-phores) between the bloodstream and the disks. Box on the left represents the capsule formed by SRBP & TTR to transport retinol via the bloodstream. R represents a RBP ligand within the RPE. R’ represents the RBP ligand within the IPM. Zigzag lines represent membrane walls. Dashed line on right represents the nominal (non-membrane) edge of the outer segment within the IPM. Barred molecule at lower right indicates the molecule is in a quantum-mechanically excited state. Modeled after a competitive figure in Crouch & Ma, 2000.

7.1.2.2.1 Transfer of the retinines from the SRBP + TTR to the RPE cells

[xxx condense this material. ] It is important to describe the status of research related to the SRBP-TTR-retinene in nutrition prior to discussing the details of SRBP-TTR-Rhodonine transport within the vision modality. At least three investigative teams have made similar comments within their published reports.

C As Noy et al69. noted in 1992, “. . .the exact function that may be served by binding of RBP to specific receptors in the plasma membranes of target cells is unclear.” C As noted by Fex et al70. in 1996, “The origin and role of all-trans and 13-cis retinoic acid in serum is unknown.” CA As noted by Monaco in 200071, “Though the exact mechanism of vitamin delivery is still a matter of debate, evidence for the presence of cell surface receptors has been presented. The kinetics of the process leading to the transfer of retinol to the target cells has shown that the vitamin is transferred from the complex so that there is not a previous dissociation step of the two proteins which instead separate from one another after the loss of the vitamin.” This quotation may not be sufficiently granular, separation may occur in conjunction with retinine transfer.

As shown by their titles, these papers were not concerned with the use of retinol in the visual modality, and their observations do not apply to the purpose of supporting vision. However, they contain a great deal of detailed information concerning the SRBP-TTR-retinoid complex. The numbers shown in brackets lead to specific citations in each paper supporting the above quotation. Where appropriate, these citations will be addressed below. The nutrition literature focused on SRBP-TTR-retinene transport is quite extensive, the operational aspects of this literature are not relevant to the transport of SRBP-TTR-Rhodonine(). However, much technical information in this literature elucidates the detailed mechanisms involved in SRBP-TTR-Rhodonine transport.

69Noy, N. Slosberg, E. & Scarlata, S. (1992) Interactions of retinol with binding proteins: studies with retinol binding protein and with transthyretin Biochemistry vol 31, pp 11118–24 70Fex, G. Larsson, K. & Nilsson-Ehle, I. (1996) Serum concentrations of all-trans and 13-cis retinoic acid and retinol are closely correlated J Nutritional Biochem vol 7(3), pp 162-165 71Monaco, H. (2000) The transthyretin-retinol-binding protein complex. Biochim Biophys Acta vol 1482, pp 65–72 38 Processes in Biological Vision

Describing SRBP-TTR-Rhodonine transport is complicated because most of the literature relates to the role of the simpler SRBP-TTR-thyroxine loop or the SRBP-TTR-retinene loop and their role in nutrition. The SRBP-TTR-retinol loop was originally investigated as the SRBP-TTR-thyroxine loop. In many of these cases, the use of the term loop may not be justified; the transport phenomenon may not involve returning to the starting point, or reuse of some of the transport materials. This section and its subsections will provide a logical progression from these positions to a much better understanding of the role of the SRBP-TTR complex within the vision modality. The method of transporting what is initially a retinene, but that is converted into a retinine (a diol) before, or in connection with, the transfer of the Rhodonines (specific retinines) to the plasma membrane of the target RPE cells will be developed. - - - - [xxx ] Monaco (2000) indicated that when at the RPE lemma, the TTR moiety was removed exposing the hydroxyl group at position 15 of the retinol while the rest of the molecule remained enclosed within the SRBP moiety. Naylor & Newcomer have provided very detailed data at the peptide level regarding the SRBP/TTR interface. They also note, “The retinol is encapsulated by the β-barrel in the binding cavity in a hand-in-glove-like fit with the ring end of the retinol innermost. Only the hydroxyl of the retinol is solvent accessible72. 7.1.2.2.2 Clearance of the residues of SRBP & TTR from the RPE cell area

This work is more interested in the clearance of any residues of the holo-SRBP + TTR complex after delivery of the retinine to the RPE cells because of the potential buildup of such residues in the form of drusen within the immediate area of the RPE-choroid space and causing macular degeneration (Section 7.1.2.3, the next section).

Sivaprasadarao & Findley73 addressed the transport potential of SRBP and CRBP in detail but not in the vocabulary of this work and not with respect to the visual modality (in 1988). “As the interface between plasma and the cytoplasm, plasma membranes are a critical part of the transport system for retinol, but the mechanism(s) in vivo by which the vitamin is transferred from extracellular RBP to the intracellular binding proteins has not yet been rigorously established.” They did note the apparent difference between their holo-SRBP and the post-holo-SRBP forms, “The results therefore suggest the presence of at least two forms of RBP, with distinct receptor-binding kinetics, the slow-dissociating component corresponding to high-affinity binding.” In their discussion, they further note, “However, these observations provide experimental support for the proposition (Rask & Peterson, 1976) that RBP undergoes a marked loss in affinity for the receptor upon delivery of its bound retinol to the target cell.” 7.1.2.3 Potential buildup of drusen resulting in macular degeneration

[xxx see Dave Goggin email of 6 May 2016. Send draft with cover letter to Dr Duncan ] This sub-section is in direct support of Section 18.8.9.3.1 discussing the medical state-of-the-art with regard to macular degeneration.

The movement of retinene between the blood stream and the RPE in the space between Bruche’s membrane and the RPE involves very low energy chemical reactions. If these reactions are interfered with, it is possible to generate extraneous material, generally described as drusen that cannot be removed by the bodies normal processes. Drusen (singular, "druse") are tiny yellow or white accumulations of extracellular material that build up between Bruch's membrane and the retinal pigment epithelium of the eye. Drusen are made up of lipids, a fatty protein (this description from the American Academy of Ophthalmology appears to lack precision). Clinically, drusen is observable by the ophthalmologist, frequently consisting of particles larger than 25 microns in diameter. There is a large literature on drusen examined from the immunological perspective. Crabb et al have provided a report on their investigations into drusen based on a presumed protein base for these

72Naylor, H. & Newcomer, M. (1999) The Structure of Human Retinol-Binding Protein (RBP) with Its Carrier Protein Transthyretin Reveals an Interaction with the Carboxy Terminus of RBP Biochem vol 38, pp 2647-2653 73Sivaprasadarao, A. & Findley, J. (1988) The interaction of retinol-binding protein with its plasma-membrane receptor Dynamics of Vision 7- 39 materials74. Their discussion offers two important statements. “Drusen are a hallmark risk factor for developing age- related macular degeneration (AMD), yet little is known about their molecular composition or mechanism of formation. The progression of AMD might be slowed or halted if the formation of drusen could be modulated.” “Waste products from the RPE and blood components from the choriocapillaris provide a ready source of extracellular material for oxidative modification and drusen formation. Over time, oxidative modifications and subsequent immune-mediated events could cause the expansion of drusen on Bruch’s membrane. Accordingly, we hypothesize that oxidative protein modifications are causally involved in drusen formation.” Oxidative transformation as outlined int the previous and next subsection of this work are at the very foundation of the chemical reactions at sites on the RPE lemma facing Bruch’s membrane. It is difficult to relate the various proteins identified by abstract labels in Crabb et al. with the more functional labeling used in this work and the work of other investigators. A paper by Umeda et al. provides more recent data on drusen and presents a figure 2, reproduce in an expanded form as Figure 7.1.2-13, more compatible with the operations described earlier in this section75. They noted as of 2005, “At present there is no fundamental cure for AMD, although some success in attenuating choroidal neovascularization has been obtained with surgical excision or photodynamic therapy.” “Although the most prominent lesion of AMD involves the RPE and Bruch’s membrane, it is degeneration, dysfunction, and death of photoreceptors and its consequences that account for the vision loss. Very little is known about the pathophysiology of this disease process.” “Drusen in both late and early onset monkeys showed immunoreactivities for apolipoprotein E, amyloid P component, complement component C5, the terminal C5b-9 complement complex, vitronectin, and membrane cofactor protein. LC-MS/MS analyses identified 60 proteins as constituents of drusen, including a number of common components in drusen of human age-related macular degeneration (AMD), such as annexins, crystallins, immunoglobulins, and complement components.”

There are two separate papers on the Internet by these authors with the same title and credentials. However, the figures are different. The paper incorporating this figure 2 is http://www.fasebj.org/content/19/12/1683.full.pdf The other paper is http://www.fasebj.org/content/early/2005/09/30/fj.04-3525fje.full.pdf . It contains other useful figures.

There is little text accompanying this figure. “A possible pathological pathway whereby autoimmunity against annexin II could contribute to drusen formation is: 1) anti-annexin II immunoglobulins bind to the basal plasma membrane of the RPE; 2) the inactive C1 serum protein interacts with the Fc portion of the immunoglobulin; 3) this leads to formation of the C5b9 membrane attack complex; 4) causing damage to the RPE cells followed by shedding of the cell membranes in the sub-RPE space. Immune complex formation might continue in the resultant drusen cores leading to further development of drusen.”

The original figure also lacks definition so important in discussing failure modes of a complex physiological structure, such as the relationship between Bruch’s membrane and the lemma of the RPE cells. The location and role of verhoeff’s membrane in protecting the inter-photoreceptor matrix (IPM) from infiltration by chemically active substances is not recognized or discussed.

The normal condition in this figure shows two ellipses (on the right) at the vascular/RPE interface representing an unknown process under way. The ellipses are shown in each frame of the figure. This process was not discussed in the text. It is proposed here that this process involves the delivery of retinenes/retinines to the RPE interface by the SRBP/TTR “bottles” described above. Subsequently the bottle components are released and returned to the kidneys and/or liver for recycling and the capture of additional retinene stored in the liver. “Furthermore, the codistribution of IgG and terminal complement complexes in drusen suggests an immune response directed against retinal antigens and immune complex formation. This hypothesis is supported by the presence of putative anti-retinalautoantibodies in the sera of patients with AMD. Anti-retinalautoantibodies have previously been reported in a number of retinal diseases, including retinitis pigmentosa, paraneoplastic retinopathies, and retinal vasculitis.” - - - - -

74Crabb, J. Miyagi, M. Gu, X. et al. (2002) Drusen proteome analysis: An approach to the etiology of age-related macular degeneration PNAS vol 99(23), pp 14682-14687 doi/10.1073/pnas.222551899 75Umeda, S. Suzuki,, M. Okamoto, H. et al. (2005) Molecular composition of drusen and possible involvement of anti-retinal autoimmunity in two different forms of macular degeneration in cynomolgus monkey (Macaca fascicularis) FASEB J. express article 10.1096/fj.04-3525fje 40 Processes in Biological Vision

Figure 7.1.2-13 Conceptual schematic of potential disease conditions (drusin buildup) associated with the vascular/RPE interface forming the Rhodonines. Left column; Routine operation of the vascular/RPE interface leading to potential accumulation of debris. Right two columns; autoimmune activity at the vascular/RPE interface proposed by Umeda et al. The failure of the routine operation of the vascular/RPE interface may lead to the accumulation of debris between the RPE cells and Bruch’s membrane. See text. Expanded from Umeda et al., 2005.

This work suggests an alternate source of macular degeneration, based on the first figure developed in Section 4.5. Prior to proceeding, the reader is asked to review the terminology of Section 7.1.1.2.3. The items of primary interest are labeled SRBP (a retinoid-binding protein resident in the serum of the blood stream) and TTR (The plasma protein transthyretin previously known as prealbumin). As discussed above, SRBP is believed to form a “bottle” capable of holding one molecule of a retinene. TTR is believed to associate with the SRBP to form a “cork” for the bottle. When loaded, the bottle is used to transport a single molecule of a retinene from the liver to the RPE of the retina. The retinene-SRBP complex is labeled holo-SRBP in this work. Following arrival and deposition Dynamics of Vision 7- 41 at the RPE, it is believed the two proteins constitute “debris” that must be returned via the bloodstream to either the kidneys or liver. The post-holo-SRBP is no longer biologically viable. The failure of complete debris removal after release of the protein “bottle” parts following delivery of the retinenes to the RPE cells may mediate macular degeneration. Frame A of the left column of the figure shows a single RPE cell with its phagocytosis mechanism shown along the top edge that is used to digest old outer discs from the outer segment of the animal photoreceptor neurons. New chromophores are exuded into the IPM from this same surface. The reclaimed and new proto-photoreceptor material (chromophore pigment) is stored within the RPE as individual granules, labeled here U, S, M & L. This depiction is different from that of Umeda et al. where the solid discs are described as the nucleus of individual RPE cells. Note the multiple ellipses along the lower edge of the frame differs from the normal condition, not specifically described, of Umeda et al. There may be separate stenolytic sites along the lemma of the RPE for each chromophore type or only one. Frame B shows the SRBP + TTR & Retinal complex arriving at the RPE cell via the bloodstream from the liver. When complexed, SRBP and a retinene are frequently described as holo-SRBP. As discussed above, there may be four distinct forms of this complex if the retinal is converted to one of the Rhodonines during transport. Alternately, there may be only one form and the retinal is converted to a distinct form of the Rhodonines as part of the transfer from the complex to the RPE at one of four lemma locations. Frame C shows the fragments of the complex as they are released from the reaction sites on the RPE lemma. Absent the retinene, the SRBP is frequently described as apo-SRBP. In this work, the term post-holo-SRBP is used. These fragments must be removed promptly in order to allow additional complexed material to react at the RPE lemma. A constriction or blockage of the vascularization can prevent removal.

Frame D shows the result of failure to remove the debris over an extended period. The debris may accumulate on either side of Bruch’s membrane. With time (aging), the debris can proceed to block the arteriole and/or force the RPE cells away from their location relative to Bruch’s membrane. On a large scale, the forcing of the RPE away from Bruch’s membrane result in a displacement of the outer segments of the photoreceptors away from the focal plane of the stage 1 optical system. This action can cause both perceived optical distortion and defocusing of the image within the foveola. If the debris blocks the arteriole, it can cause necrosis of the RPE cells in a surrounding area. This debris buildup is usually associated with the “dry” type of macular degeneration. Death of the RPE cells can cause additional disruption of the normal RPE/IPM/Photoreceptor operating cycle (Section 4.5) and ultimately destruction of the photoreceptors.

Once overwhelmed, the disruption of the RPE allows the arteriole blood to encroach on the space between Bruch’s membrane and the RPE, penetrate Verhoeff’s membrane (Section 18.8.3.6.2) and eventually enter the inter-photoreceptor matrix (IPM) space (Section 4.5). A more detailed figure supporting this discussion also appears as Figure 18.3.6-19 in Section 18.8.3.6.2 This is described as the “wet” type of AMD. The oxygen of the intruding blood is highly detrimental to the operation of the rhodopsin chromophores coating the outer disks. The resulting failure of the outer disc stack leads to the necrosis of the sensory receptor neurons and associated elements.

Most visual micrographs acquired by ophthalmologists show isolated, typically yellowish, “spots.” The locations of these spots are becoming better understood through the latest optical coherence (sometimes computer-aided) tomography (OCT). See Section 18.8.3.5. A key point is the micrographs show isolated spots, yet the AMD causes total loss of vision over areas larger than that of individual spots, suggesting occlusion of the blood flow. Recent Fourier type OCT is able to identify locations of blood flow interruption (pers. comm. Dr. C. Eifrig MD FACS). A popular trade magazine has provided a very good overview of the rapid introduction of OCT angiography equipment that does not require introduction of dyes into the bloodstream76. It is quite plausible that an immune response is elicited to aid in the removal of the debris blocking the arteriole and maintain normal vision until the system is overwhelmed and the RPE cells are disrupted. (initially resulting in the dry form of age-related macular degeneration, AMD). Nita et al. have recently presented a lengthy review paper with finding that are highly correlated with the above analyses77. However, the terminology is totally different except for the importance of Bruch’s membrane (BrM) and the flow of material into and out of the arteriole channels adjacent to the membrane.

76Schlett, J. (2016) OCT Angiography: Open Eyes BioPhotonics July/Aug pp 21-25 77Nita, M. Strzalka-Mrozik, B. Grzybowski, A. Mazurek, U. & Romaiuk, W. (2014) Age-related macular degeneration and changes in the extracellular matrix Med Sci Monit vol 20, pp 1003-1016 DOI: 10.12659/MSM.889887 42 Processes in Biological Vision

They do succinctly note, “Age-related macular degeneration (AMD) is the leading cause of permanent, irreversible, central blindness (scotoma in the central visual field that makes reading and writing impossible, stereoscopic vision, recognition of colors and details) in patients over the age of 50 years in European and North America countries, and an important role is attributed to disorders in the regulation of the extracellular matrix (ECM).” The difference between that work and this is their focus. The Nita et al. paper deals with the detailed analyses of a wide variety of structural materials found associated with the extracellular matrix ((ECM, (Bruch’s membrane and the choriocapillaris). However, it does not explore the transport oriented materials and the operational aspects of the RPE and Bruch’s membrane relative to the visual modality. They note, C “The 2–4 :m-thick BrM . . .is composed of 2 basement membranes – 1 for the RPE cells and 1 for the endothelium of the choriocapillaris.” C “Between the basement membranes are 2 layers (inner and outer) of structural collagen, composed of collagen type I and type III (fibrillar). Both of the collagen layers embrace the middle layer of the BrM, like a sandwich built mainly of elastin.” Nita et al. also note, “RPE cells control the synthesis of all the structural elements of the BrM, mainly the most abundant proteins – collagen type I, collagen type IV, and laminin (which is not a collagen) – as well as the metalloproteinases and their tissue inhibitors.” However, this is not the principle role of RPE cells and may in fact be highly speculative.

It is not clear if the TTR (transthyretin) is among the molecules described in the Nita et al. paper. The paper is focused on structural materials rather than operational materials involved in the visual modality. They do note, “Deposition of these new structures increases the thickness of basal laminar deposits to a high degree and simultaneously worsens the nutrient and oxygen supply of the RPE/photoreceptors, which in turn maintains neovascularization. The basal deposits, with diameters of over 25–30 :m, turn into clinically visible soft drusen, which are conducive to serous [sic] separation of the RPE from the ECM.”

Nita et al. also note, C “MMP-1 degrades collagen I, II, III and the decrease in its activity favors the development of soft drusen.” C “RPE cells, when exposed to blue light (relative AMD risk factor), diminish TIMP-3 production, which consequently causes an increase in the amounts of various collagens and in the development of early dry AMD; C “on the other hand, TIMP-3 demonstrates a protective influence, since it inhibits the development of neovascularization.”

These statements are supportive of the operational role, and potential failure mode of the ECM leading to AMD as described in this work. Their figure 2 is highly complex and probably primarily speculative. However the discussion under the heading, “Histological and Clinical Aspects of Changes in the ECM and Other Mechanisms” appear highly compatible with the analyses of this work. Their figure 3 discusses the disintegration, and presumed removal, of a variety of molecules from the ECM. It is worth noting that the turnover of the transport proteins, SRBP & TTR are at least an order of magnitude more frequent than the replacement of any structural protein explored by Nita et al.

The Umeda et al. and Nita et al. papers are inconsistent as to the degree and specific location of neovascularization related to AMD. In this work, as indicated in the figure above, the situation may involve infiltration of blood plasma into the IPM region via penetration of Verhoeff’s disrupted membrane, with or without growth of vascular tissue. The potential infiltration of blood plasma into the IPM establishes a functional as well as clinical link between a retinal tear and AMD. In both cases aggressive chemicals from either the vitreous humor of the ocular or the vascular matrix adjacent to the choroid can contaminate the IPM and damage the outer segments of the sensory neurons. Dynamics of Vision 7- 43

7.1.2.4 Transport of Rhodonine within the RPE/IPM space of the retina

A similar process has been described in a cartoon by Starzack78 where two different cations compete for two identical binding sites. Figure 7.1.2-14 illustrates the situation for retinol after it has arrived at the RPE and been converted into a resonant retinoid. It can be converted into any one of four retinoids of the Rhodonine family (a). Each of these forms can then participate in the transport process as shown in (b). Referring to (a), Rhodonine (5) is shown with its two polar groups in solid black. [When displayed in two dimensions, the retinoid molecules in this work are shown with the $-ionone ring rotated to place the #5 carbon at the top of the ring. This orientation places the #5 carbon in line with the conjugated chain to stress the familial similarity of the Rhodonines. Each of these groups shares 1.5 bonds with the backbone of the molecule because of its resonant character. For Rhodonine(5), the leftmost polar group is located at the top of the $-ionone ring, at the location of carbon #5, and the rightmost polar group is associated with carbon #15. Similarly, Rhodonine(7) has its leftmost polar group attached at carbon #7, Rhodonine(9) at carbon #9 and Rhodonine(11) at carbon #11. Each of these molecules exhibits a different physical length between its two polar groups. These different physical lengths and the slow wave nature of the resonant structure account for the spectral absorption characteristic of each of these molecules. (b) illustrates how one of the Rhodonines, here Rhodonine(5), can be transported within the retinal structure. Assume it is normally stored in one of the color centers of the RPE in its neutral but resonant form (caricature 1). A transport protein such as CRAIBP binds to the polar atom associated with the #5 carbon (caricature 2)and moves the combined structure by diffusion through the RPE to the RPE/IPM interface. At that interface, a second transport protein, such as IRBP, located in the IPM binds to the polar atom associated with carbon #15 (caricature 3). The first Figure 7.1.2-14 Proposed transport of the retinoids within binding protein, which is still in the RPE, releases from the RPE. (a) The four Rhodonines showing their pairs of the Rhodonine molecule. The result is caricature 4. The binding sites when in the aqueous state. (b) A cartoon combined molecule now moves by diffusion to an end showing the various configurations of Rhodonine(5) and point such as a disk of an outer segment. At that point, other moeities in the course of transferring rhodonine from the second transport protein releases the Rhodonine one location to another in the RPE. molecule and the neutral molecule is now free. It is also in the proper position to assume its place in the liquid crystal on the surface of the protein substrate, stipulated to be opsin. Crouch has presented a flow diagram, using two-dimensional stick models of rhodopsin, as part of a recent review article79. The article assumes the chemical theory of the neural system to describe the movement of the retinoids from the blood serum, through the RPE and on into the IPM (and apparently their return to the RPE at a later time. Although it loosely follows the flow suggested by this work, it does not appear to represent the most recent literature on retinoid transport. This is particularly true with respect to the transport of the chromogens of vision in the bloodstream. Nor does it explain how the putative rhodopsin is transferred from the disks to the RPE, regenerated in the RPE, and transferred

78Starzak, M. (1984) The Physical Chemistry of Membranes. NY: Academic Press Fig. 2.7, Pg. 42 79Crouch, R. Chader, G. Wiggert, B. & Pepperberg, D. (1996) Retinoids and the visual process. Photochem. & Photobiol. vol. 64(4) pp. 613-621 44 Processes in Biological Vision

back to the disks in a timely manner. Figure 7.1.2-15 presents a simpler diagram based on the more recent literature and this work for comparison. It is based on the discussion of Section 7.1.1.4.3 concerning three conditions. (1) is the inability of the retinoids employed in the formation of chromophores to travel to the RPE unprotected in the bloodstream. (2) involves the concept of the chromophores as resonant forms of a retinoid. (3) involves the electronic excitation of the chromophores leading to electronic de-excitation (as opposed to de-excitation via conversion to a stereoisomer). The conclusion is drawn that the retinoids destined to become chromophores are converted to a Rhodonine as part of the transport process through the bloodstream. In this figure, free retinol is stored in the liver before distribution through the vascular system (a). While retinol may be transported through the bloodstream in its native form for some purposes, the material destined for use in the chromophores of vision requires special handling. To pass through the bloodstream and then pass through the cell wall of the RPE cells and eventually reach the disks of the Outer Segment. It is uniquely packaged. This figure shows a putative scheme for accomplishing the steps necessary to transform the retinol into chromophores during this transit. Alternate schemes can be suggested. However, this one provides a method for the chromophores (while complexed and in dilute form) to be selectively deposited on the appropriate spectrally selective disks within the retina. It suggests that retinoid transport and conversion occur simultaneously in a well-orchestrated sequence. Frame (b) shows retinol in its native form, and packaged within four transport “bottles.” For continuity, it also shows an open empty bottle known to depart the area of the RPE and to travel the bloodstream until it reaches the kidneys where it is disposed of. This scheme suggests there are four distinct retinoid binding sites within the SRBP of the blood. These could result from four different specific variants of SRBP that each contained one site or a single SRBP that contained multiple binding sites. Rask, et. al. have sequenced the entire SRBP molecule using a fractionation technique80. Their results speak of three carboxyl groups occurring in human SRBP. They also speak of four methionyl residues (these do not contain carboxyl groups). However, their data shows a plethora of glutamic acid and aspartic acid sites that each contain a complete carboxyl group. This situation suggests that SRBP could make available carboxyl groups at a variety of locations depending on the stereochemistry of the molecule. These locations could support the formation of all four Rhodonines. See Section 4.6.3. Only the determination of the stereographic structure of SRBP can support the determination of the exact mechanism of retinol to Rhodonine conversion in the SRBP bottle.

It is proposed that the location of the reactant carboxyl group of the SRBP (shown by the rectangular nib along the inner top edge of the bottle) varies in a yet to be determined way. It is this site that complexes with retinol. Following this complexing, the content of the bottle is protected by the “stopper” of TTR and can travel the bloodstream free from chemical attack.

Upon reaching the RPE cells of the retina, each bottle becomes attached to an individual cell wall where it can insert the retinoid into the cell. In doing this, the SRBP forfeits the oxygen atom. The SRBP residue is then transported to the kidney for elimination.

Gamble & Blaner have recently noted the fact the stopper TTR has a total weight of 55 kDa and contains four identical subunits81. The presence of four subunits suggests a role for TTR similar to that described above for the subunits of SRBP. In this case, the SRBP acts as a more passive carrier and the stereo-connection between retinal and TTR determines the ultimate form of the rhodonine precursor delivered to the RPE cells. They suggest the larger TTR prevents the SRBP-retinal complex from renal filtration. In this case, the residue of SRBP and a remnant of TTR would be unsuitable for reuse and would be cleared from the system by the kidneys. Gamble & Blaner noted that, “Deletion of the gene for TTR is not lethal, and in fact, TTR deficient mice are phenotypically normal despite circulating RBP- ROH levels which are only approximately 5% of normal.” They did not discuss the potential for impaired vision in these mice. Frame (c) suggests that the retinoids, which now contain two oxygen ligands per molecule are transported within the RPE by a second family of RBP’s. The research in this area has not yet coalesced. There are suggestions that there are two distinct families of CRABPs. One would be used to transport the retinoids to spectrally specific storage locations within the RPE cells. The second would be used to transport the retinoids from storage to the cell interface with the IPM. These options are discussed in TABLE 7.1.1.2-1 above. At the RPE/IPM interface, the retinoids are transferred to a subsequent RBP designated IRBP because of its location within the IPM. By attaching to the retinoids as suggested by frame (d), the unique spectrum determining feature of each type of retinoid is exposed. This allows these retinoids to

80Rask, L. Anundi, H. & Peterson, P. (1979) The primary structure of the human retinol-binding protein. FEBS Letters, vol 104, no. 1, pp 55-58 81Gamble, M. & Blaner, W. (2000) Factors affecting blood levels of vitamin A In Livrea, M. ed. Vitamin A and retinoids: an update. . . . Basel Switzerland: Birkhauser Verlag pp 1-16 Dynamics of Vision 7- 45 be transported to and be assimilated into a spectrally specific liquid crystalline coating being formed on the opsin substrate of each disk. Frame (e) illustrates the chemistry of the individual fully formed chromophores as they are stored in the RPE and as found on the disks of the Outer Segment.

Figure 7.1.2-15 Overall scheme for retinoid transformation for both chromophore formation and operation. All of the molecules shown contain a conjugated carbon chain. Molecules with two black ligands (oxygens) also contain a conjugated carbon chain between the two oxygens, have an unpaired electron and are resonant. Molecules with only one black ligand are not resonant. Each of the chromophores in (e) are easily raised to an excited state by photons sharing their individual resonant wavelength. See text. 46 Processes in Biological Vision

Chen & Heller have noted that their retinol-like material is found in two forms within the RPE. It is found complexed with one of the above rhodonine-binding proteins (mol. wt. >1.5 x 106) and it is also found in a free form of low molecular weight (<1000) material. The four Rhodonines have a molecular weight of either 285 & 299 (two of each). They went to analyze the cytosol of the RPE using radioactive labeled retinoids. They found that about one third of the total material was in the form of a complex with an RBP. The rest was free “retinol-like” material. The free material was found in a spot corresponding to retinol using thin layer chromatography. Because of the slight difference in molecular weight between the chromogens (retinol, mol. wt. 286.4 ) and the chromophores, this would be expected. The question becomes how accurately was the location of the “spot” measured and did the spot show an unusual broadening or even a dual peak in density? Their processing would need to be reconsidered to determine whether it may have degraded the material with a weight less than 1000 from the palmitate to a “free retinoid.” As palmitates, the molecular weight of the retinoids would have been 444 and 448. Farber & Chader have edited a review focused on IRBP82. Liou, Geng & Baehr, writing in the that review, develop the fact that IRBP also exhibits a fourfold feature in its genetic structure and that it is apparently not related to the other RBP’s. They claim it is secreted from the photoreceptor cells based on Hollyfield, et. al. It may originate in the neighboring glial cells. The fourfold gene feature may support the option found in this work that there are four separate IRBPs that preferentially transport the four chromophores of vision across the IPM to the photoreceptor extrusion cup. The molecular weight of IRBP is given as 134,200 in Methods of Enzymology (1990)83. The half-life of IRBP has been measured at ~11 hours in Xenopus84. Loew & Gonzalez-Fernandez have recently characterized the IRBP molecule in detail85.

It is suggested that most of the non-protein material in the IPM consists of fuels to support the electrolytic processes powering the photoreceptor cells of the neural system. 7.1.2.4.1 Nature of the RBP’s in the RPE

Based on this work, there are several RBPs within the RPE that perform individual and specialized functions. These functions all relate to the transport of the chromogen or chromophore material between the interface with the blood stream and the IPM and their temporary storage. This task also involves the recirculation of chromophoric material recovered by phagocytosis of the old disks of the disk stacks.

The precise terminology used for these RBPs has been improving in the literature with each decade. Therefore, weighing the ideas presented in the earlier literature carefully is important. There appear to be two distinct RBPs in the RPE space. One of these, CRBP, is involved in the transfer of the chromophoric material to the pigment globules of the RPE for storage. The other, CRALBP, is used to transfer the same material to the IPM for application. Because of their function, the names are derived from their Cellular Retinoid Binding Protein characteristics. Initially, it was assumed CRBP preferentially bound the alcohol form of the retinoid. Therefore, the letters AL were added to the second protein isolated, based on its proclivity for binding to aldehydes in-vitro. Since these materials are generally involved in the transfer of Rhodonine and not retinol or retinaldehyde, these names are mostly of historical significance. Rhodonine has two separate ligands. One of them is an alcohol and the other an aldehyde.

Pfeffer, et. al.86 have provided molecular weights for these soluble glycoproteins. Their values are 15,000 Daltons for CRBP and 33,000 Daltons for CRALBP. 7.1.2.4.2 Criticality of IRBP based on genetic mutation testing

82Farber, D. & Chader, G. (1988) The molecular biology of the retina. Progress in Clinical and Biological Research, vol. 362, NY: Wiley-Liss 83— (1990) Methods of Enzymology. Vol. 189 pg. 325 84Cunningham, L. Yang, L. & Gonzalez-Fernandez, F. (1999) Interphotoreceptor retinoid-binding protein (IRBP) is rapidly cleared from the Xenopus interphotoreceptor matrix Exper Eye Res vol 68, pp 399-410 85Loew, A. & Gonzalez-Fernandez, F. (2002) Crystal structure of the functional unit of interphotoreceptor retinoid binding protein Structure vol 10, pp 43-49 86Pfeffer, B. Wiggert, B. Lee, L. Zonnenberg, B. Newsome, D. & Chader, G. (1983) The presence of a soluble interphotoreceptor retinol-binding protein (IRBP) in the retinal interphotoreceptor space. J. Cell Physiol., vol. 117, pp. 333-341 Dynamics of Vision 7- 47

Ripps, et. al87. have recently reported significant tests on the visual performance of mice genetically modified to interfere with IRBP utilization in the IPM. The exact expression of the mutation is not clear and several of the tests would not be expected to be significant based on this work. In what they describe as "knock-out" (IRBP--/--) mice, significant changes were recorded in the cross section of the retinas. The outer nuclear layer, containing the nuclei of the photoreceptor cells, was attenuated to less than 50% of its normal thickness and the Outer Segments were shortened and disorganized. Assume for the moment that the IRBP is secreted from the soma or inner segment of the photoreceptors in the outer nuclear layer. Under this assumption, it is not clear whether the absence of IRBP in the IPM was due to failure of the cells to produce the protein or whether it was due to the failure of the cells to secrete the protein into the IPM. The Ripps, et. al. paper assumes that IRBP plays a role in moving 11-cis-retinol from the RPE to the Outer Segments of the photoreceptors and then plays a role in moving all-trans-retinol back to the RPE on a recurring basis. Jones, et. al.88 make an alternate statement that IRBP can transfer the aldehyde form of the retinoid to bleached rod cells restoring their sensitivity to light. However, his suggested kinetics differ from those of a simple transfer between membranes. Okajima, et. al89. have presented a caricature showing IRBP simultaneously moving 11-cis-retinal in both directions and al-trans-retinol in one direction within the IPM. This work does not support any of the above conflicting hypotheses at the detail level. First, it assumes the material transported from the RPE is Rhodonine (which exhibits both alcohol and aldehyde ligands simultaneously). Second, it assumes the chromophoric material is only transported hydraulically from the RPE to the interior of the extrusion cup of the inner segment. Third, the theory does not require the chromophore material to be returned to the RPE for re-isomerization. 7.1.2.4.3 Nature of IRBP in the IPM

Based on this work, IRBP has only one function, to transport chromophores from the RPE interface to the formative location of the disk stack in the IPM. It is not involved in the transport of isomerized chromophore back to the RPE.

Pfeffer, et. al. have studied this material in detail and compared it with some other retinoid-binding-proteins. IRBP is a soluble retinoid-binding-protein of 146,000 Mr, (250,000 Daltons) that can also be described as a glycoprotein, found exclusively in the IPM space (Pfeffer, et. al. speak of this space as the extracellular matrix, ECM). Beside the Outer Segments, the space is also known to contain glycoconjugates and collagens as well as other soluble materials. When using centrifugal techniques on whole retinas, it is found with other RBP’s predominantly, if not exclusively from the RPE. These RBPs include CRBP of 15,000 Daltons and CRALBP of 33,000 Daltons. They considered the fact that CRBP and CRALBP were not found in their IPM wash one of their most significant findings. It strongly suggests that IRBP is the only RBP in the IPM. Liou, et. al., writing in Farber & Chader noted no significant sequence similarity of IRBP with other RBP’s except in two short segments.

IRBP is a large and elongated molecule. It has a nominal length of 240 Angstroms and consists of four replicated units, a tetramere (Loew & Gonzalez-Fernandez, 2002). It has an axial ratio of about 7:1. It is a very large molecule compared to the retnoids with a length of 15 Angstrom and a diameter of 7 Angstrom. Each of the four 77 by 25 Angstrom sections of IRBP exhibit at least two pockets large enough to accomodate stereo-chemical combination with a Rhodonine molecule. It is widely distributed in the IPMs of the vertebrates.90 Additional material concerning IRBP has been collected in Berman91. The recovered IRBP was found to bind to many materials such as cholesterol and tocopherol. However, these materials are not normally found in the IPM space protected by Bruch’s membrane and the Outer Limiting Membrane. Nor are they found in the closely packed RPE cells and the closely packed photoreceptor and glial cells of the retina. It has also been found to bind to a variety of fatty acids. However, this binding may have been a consequence of the extraction technique. There are several statements in the literature suggesting that IRBP is produced through the Golgi apparatus in the

87Ripps, H. et.al. (2000) The rhodopsin cycle is preserved in IRBP “knockout” mice despite abnormalities in retinal structure and function. Vis. Neurosci. vol. 17, pp. 97-105 88Jones, G. Wiggert, B. Crouch, R. Cornwall, M. & Chader, G. (1989) Bovine IRBP and amphibian photoreceptors: retinoid transfer and protection. Invest. Ophthal. Vis. Sci. Suppl. vol. 30, pg. 488 89Okijama, T.-L. Wiggert, B. Chader, G. & Pepperberg, D. (1994) Retinoid porcessing in retinal pigment epithelium of toad (Bufo marinus). J. Biological Chem. vol. 269, no. 35, pp 21983-21989, fig 7 90Inouye, L. Albini, A. Chader, G. Redmond, T. & Nickerson, J. (1989) Exp. Eye Res. vol. 49, pp. 171-180 91Berman, E. (1991) Biochemistry of the Eye. NY: Plenum pp 375-380 48 Processes in Biological Vision

photoreceptor cells and excreted into the IPM. It is not clear whether the alternative possibility, of production in glial cells, was actively considered92. Ripps, et. al. state in their summary, “the results of these studies indicate that the rhodopsin cycle can occur in the absence of IRBP.” This is additional evidence that the actual photoexcitation/de-excitation cycle does not involve the transport of chromophores to the RPE for regeneration. In addition to the short term situation described in the previous paragraph, Ripps, et. al. make the additional statement more applicable to the long term situation: “In the absence of IRBP, lipid depletion could lead ultimately to abnormalities in the turnover rate of disk membranes, biochemical changes in membrane composition, and a reduction in the receptor- mediated ERG.” The statement is attributed to Anderson, et. al93. This position appears compatible with a premise of this work that so-called cones are in fact immature or degenerate rods. 7.1.2.4.4 Proportion of IRBP in the IPM

Several authors have recently referenced a paper by Pfeffer, et. al. They stress the predominance of IRBP in the IPM space. These include a paper by Nickerson, et. al. writing in Farber & Chader as well as the above paper by Ripps, et. al. Two statements are made; + “In the monkey, it accounts for about 70% of the protein in this space.”

+ “In this connection, it is important to recall that IRBP constitutes >70% of the soluble proteins of the IPM.”

Pfeffer actually said something slightly different. They were developing a very gentle technique of removing material from the IPM without causing physical or chemical damage. In their discussion concerning IRBP, they noted that: “A consistent amount (about 30% of the total found in unwashed retina) of this protein does remain with the retina following washing using the methods described here.” Thus, they extracted 70% of the available IRBP and left 30% behind. They said IRBP was the predominant soluble glycoprotein present but made no comment concerning the amount of IRBP as a percentage of the total IPM content or of the amount of total protein present. Their conclusion was that the IRBP was present in the original IPM as an extracellular soluble or loosely bound, peripheral glycoprotein, except for the 30% left behind on washing the IPM. This 30% was thought to be either more tightly bound to the other structures present or merely not extracted by the technique employed. In this work, it is proposed that this material was essentially trapped in the space between the disks of the Outer Segment with some possibly temporarily bound to the liquid crystalline chromophore during the deposition process. 7.1.2.4.5 Sources of IRBP in the IPM

Hollyfield, et. al94. have provided the results of an autoradiographic study focused on IRBP production in the peripheral retina. Their work clearly shows that IRBP is formed in the retinal space occupied by the Inner Segments of photoreceptor cells and various glial cells. Hogan, et. al. have shown the morphology in the area of the inner segments, Mueller cells and the OLM at 24,000x95. Their imagery clearly shows the “fiber baskets” of the Mueller cells are located in the IPM. They could secrete IRBP into the IPM using materials absorbed from the INM without involving the photoreceptor cells in this process.

Although less than conclusive in proving the glial cells were not involved, Hollyfield, et. al. represent that the nucleotide, 3H-fucose, was incorporated into IRBP and the resulting material secreted into the IPM in time scales of less than 30 minutes. They made the assumption that IRBP was formed within the photoreceptor cells. Based on that assumption, they found that after four hours of incubation, all cells in various retinal layers retained a uniform concentration of the nucleotide except the photoreceptor cells (and the ganglion cells). They say that 75% of the material present after 30

92Ripps, H. Peachey, N. XU, X. Nozell, S. Smith, S. & Liou, G. (2000) The rhodopsin cycle is preserved in IRBP :knockout: mice despite abnormalities in retinal structure and function. Visual Neurosci. vol. 17, pp. 97- 105 93Anderson, R. Benolken, R. Dudley, P. Landis, D. & Wheeler, T. (1974) Polyunsaturated fatty acids of photoreceptor membranes. Experimental Eye res. vol 18, pp. 205-213 94Hollyfield, J. Fliesler, S. Bayvorn, M. Jong, S-L. Landeres, R. & Bridges, C. (1985) Synthesis and secretion of interstitial retinol-binding protein by the human retina. Invest. Ophthal. Vis. Sci. vol. 26, pp. 58-67 95Hogan, M. Alvarado, J. & Weddell, J. (1971) Histology of the Human Eye Philadelphia, PA: W. B. Saunders pg 435 Dynamics of Vision 7- 49 minutes was lost from the inner segments of the fully mature photoreceptors (labeled rods in their work) at the end of a four-hour period. The shorter photoreceptor cells (considered immature in this work and labeled cones in their work) only lost 11% from their inner segments in the similar time period. The cells in the inner nuclear layer (apparently nuclei of the photoreceptor cells as a group) actually lost 14%, which was more than the inner segments of the immature cells lost. Their figure 4 would suggest that their so-called cone inner segments were no more functional in the preparation of IRBP than the material of the inner plexiform layer (which does not contain any mitochondria or other manufacturing media). Their investigations continued. They showed that the nucleotide was recovered primarily from IRBP in the IPM wash. They also addressed the consumption of the nucleotide by the ganglion cells in the production of non-IRBP related material. 7.1.2.5 Background: SRBP +TTR complex in non-visual applications

Considerable data from several investigators related to the tetrameric SRBP’s has appeared. Johnson et al96. have provided useful background related to TTR and this work defining the Rhodonines and suggesting the precise operation of these tetrameres during transport from liver to the RPE cells may be described more fully. Both SRBP and TTR are recognized to have an affinity for, at a minimum, the simple retinenes. These affinities might suggest they are capable of contributing an oxygen atom required to convert a retinene into a resonant conjugated molecule, a retinine (spelled with two i’s). The question then becomes, whether this proclivity can place the donated oxygen atom at the appropriate location along the retinine molecules to create the four distinct Rhodonines? Quoting Johnson et al. again in condensed form, transthyretin “was subsequently shown to bind holo-retinol-binding protein (RBP with retinol, or vitamin A) as well, and the name was changed to transthy(roxin) retin(ol) to denote its dual transport function. TTR is a tetramer of four identical subunits. Although each of the four monomers has a binding site for RBP, the tetramer binds only one molecule of RBP with high affinity and possibly a second with lower affinity. The binding affinity for apo-RBP (RBP without retinol) is very low, and the loss of retinol (e.g., uptake by tissues) results in the separation and renal excretion of free apo-RBP, accounting for the very short biological half-life of RBP of ~ 3.5 hours. The TTR-RBP complex normally transports approximately 90%–95% of retinol/vitamin A.” They cite Goodman et al97., Monaco98 and also Noy et al99.

Additional investigations have been reported by Kim et al100., Bienvenu et al101, Fex et al102 Yves et al103 and summarized in a review by Blomhoff & Blomhoff104.

The conceptual material related to vision (specifically rhodopsin and its formation) presented in Blomhoff & Blomhoff (2006) is conventional wisdom summarized from other sources and is not supported here. Many of the process steps described can be replicated in-vitro but they have little relevance to the in-vivo situation. As Fex et al. have noted, “Many tissues are known to be able to transform retinol to retinoic acid in-vitro. The origin and the role of 13-cis

96Johnson, A. Merlini, G. Sheldon, J. & Ichihara, K. (2007) Clinical indications for plasma protein assays: transthyretin (prealbumin) in inflammation and malnutrition Clin Chem Lab Med vol 45(3), pp 419–426 97Goodman, D. Peters, T. Robbins, J. & Schwick, G. (1981) Prealbumin becomes transthyretin J Biol Chem vol 256, pp 12–4 98Monaco, H. (2000) The transthyretin-retinol-binding protein complex. Biochim Biophys Acta vol 1482, pp 65–72 99Noy, N. Slosberg, E. & Scarlata, S. (1992) Interactions of retinol with binding proteins: studies with retinol binding protein and with transthyretin Biochemistry vol 31, pp 11118–24 100Kim, C-I. Leo, M. & Lieber, C. (1992) Retinol Forms Retinoic Acid via Retinal Arch Biochem Biophys vol 294(2), pp. 388-393 101Bienvenu, J. Jeppsson, J. & Ingenbleek, Y. (1996). Transthyretin (prealbumin) and retinol binding protein. In xxx Serum proteins in clinical medicine, vol 1. Foundation for Blood Research pp 1-9 Very limited circulation in 1996. 102Fex, G. Larsson, K. & Nilsson-Ehle, I. (1996). Serum concentrations of all-trans and 13-cis retinoic acid and retinol are closely correlated. J Nutritional Biochem vol 7(3), pp 162-165 103Yves, I. & Jacques, B. (2008). Plasma transthyretin indicates the direction of both nitrogen balance and retinoid status in health and disease. Open Clin Chem J vol 1, 1-12. http://benthamopen.com/contents/pdf/TOCCHEMJ/TOCCHEMJ-1-1.pdf 104Blomhoff, R. & Blomhoff, H. (2006) Overview of Retinoid Metabolism and Function J Neurobiol DOI 10.1002/neu 50 Processes in Biological Vision retinoic acid is obscure, as it does not bind to retinoic acid receptors (RAR:s).” The lack of continuity in the Blomhoff & Blomhoff presentation is indicated by their assertion, “It is clear, however, that an additional and yet unidentified 11-cis retinol dehydrogenase is the major enzyme responsible for the oxidation of 11-cis retinol to 11-cis retinal in the visual cycle.” These materials, identified conceptually in the 1940's, are not the actual photoreceptors used in vision. They offer no detailed schematics of how retinol is modified to generate a family of materials exhibiting the spectral response of the four signaling channels of the photoreceptors of vision, UV–, S–, M –, & L–. The review by Yves et al. focused on nutrition is extremely important in the context of vision. The abstract notes, “The level of TTR production by the liver also works as a limiting factor for the cellular bioavailability of retinol and retinoid derivatives which play major roles in the brain ageing process.” The paper provides very detailed information about the formation and utilization of TTR in the transport of “retinol.” They go on, “The choroid plexus is the sole site of mammalian brain involved in TTR production. Its synthesis rate by the choroid epithelium is estimated 25 to 100 times higher than that of the liver on a weight basis. As a result, TTR is a major component of CSF (CNS in this work), constituting 10 to 25 % of total ventricular proteins conveying up to 80% of intrathecal thyroxine. TTR thus constitutes an hormonal carrier-protein fulfilling important ontogenic and functional properties in mammalian nervous structures, a concept further corroborated by the observation of its increased CSF concentration during the neonatal period. The data imply that choroidal TTR facilitates the uptake of thyroxine from the bloodstream, governing its transport and delivery to brain tissues following a kinetic model developed by Australian workers. In comparison, CSF contains 10 to 100 times lower RBP and retinol concentrations than plasma whilst retinyl esters from dietary origin are virtually absent.” The choroid plexus is a network of blood vessels in each ventricle of the brain. The network is derived from the pia mater and produces the cerebrospinal fluid. It is also a component of the blood-brain-barrier.” These pronouncements add a different perspective related to the SRBP + TTR + retinene complex. They suggest the liver to retina pathway may pass via the blood-brain-barrier. They further note, “strongly suggesting that its retinol ligand is released in free form and readily taken up by membrane or intracellular receptors of neural cells. The dual TTR production, plasma-derived and choroid-secreted, allows complementary stimulation of brain activities.” They go on to note a 1994 citation, “The prominent place occupied by TTR in defining distal retinoid bioavailability has been too long unrecognized despite the warning expressing that ‘overlooking the crucial role of TTR in vitamin A-metabolism results in unachieved or even misleading conclusions’.”

Following this assertion, the paper proceeds down a narrow path that does not recognize the crucial role of the retinines in the visual modality. It begins with, “Retinol is a precursor substrate that must undergo a two-step oxidation procedure to release firstly retinal and thereafter the two active all-trans- and 13-cis-retinoic acids (RAs).” that requires reinterpretation in the context of vision. The transport mechanisms, including their two-step process to generate retinoic acid, presented by Kim et al. and their conclusions are irrelevant to vision, but the data may be quite useful.

The term “two-step” process has long been associated with the loading of the SRBP-TTR complex with a single molecule of retinol converted to a single molecule of retinal in step one, and the conversion of a single molecule of retinal into a single molecule of retinoic acid in step two. This process may be critical to the nutritional process; however, the defined two-step process employed in nutrition is not relevant to the SRBP-TTR-Rhodonine loop. The loading of the SRBP-TTR complex with either all-trans retinol or all-trans retinal is irrelevant since, the objective is to convert either of these molecular forms into the resonant all trans Rhodonine() form through an additional stage of oxidation.

This work does not address any role for retinoic acid in any stereographic form. It does begin to involve other critical retinol binding proteins (RBP), even though they may involve species that are actually retinol derivatives. “The intracellular activities exerted by retinoid compounds are mediated by a large variety of specific receptors among which are cellular-RBP (CRBP), cellular-RA-BP (CRABP), . .” These RBP’s are discussed in other subsection of this chapter. Additional comment of Yves et al. appear relevant after reinterpretation, “Because protein malnutrition is a common finding in as much as 50 % of elderly AD and MID patients, many of them could well suffer permanent hyporetinolemia still accelerating the declining concentration of retinoid molecules observed over the course of normal ageing.” Also, “In murine models, early depletion of retinoids causes deposition of amyloid β-peptides, initiating the formation of Alzheimer plaques. In aged animals, cognitive and memory deficits are associated with down-regulation of the expression of retinoid receptors which may recover their full activities under RA supplementation. Administration of RA similarly restores expression of proteins involved in the control of amyloidogenic pathways. Along the same preventive line is the demonstration that retinol disaggregates preformed amyloid β-fibrils, more effectively than does RA.” Yves et al. conclude in a related but distant perspective, “The last section devoted to brain maturation and functioning in elderly persons paves the way for new diagnostic and therapeutic approaches. The revival of older RCC studies allows to throw deeper insight into more recent findings and to enlarge the scope of current research.” Dynamics of Vision 7- 51

The Fex et al. (1996) paper provides considerable data concerning the concentrations of various retinenes at various locations within the animal physiology. But the dynamics focused on are those of converting retinol to retinal to retinoic acid. This pathway is important in nutrition and general health but not relevant to vision. They do note, “Retinol is secreted from the liveer in a 1:1 (mol:mol) complex with its carrier protein, the retinol-binding protein (RBP a low molecular weight protein that, in the blood is complexed (1:1) to another protein, transthyretin (TTR). One of the effects of this complexation is to prevent the RBP:retinol complex from being lost in the urine. Monaco (2000) is a remarkably concise paper exhibiting very few sentences that can be interpreted in multiple ways. The paper includes many SRBP-TTR complex parameters of importance with some information on how the retinoids are complexed within this two element complex. The paper contains many discussions relevant to nutrition and some relevant to vision. When referring to vision, it is suggested Monaco’s assertion, “The ligand transported by the complex is exclusively all-trans retinol though the affinity of RBP for other retinoids, most notably retinoic acid, is quite similar.” should be modified to assert, “The ligand initially loaded into the complex is exclusively all-trans retinol. . .” The result is more compatible with a potential discharge of a different retinoid by the complex, and the potential change in the precise formula of the apo-SRBP and/or the apo-TTR. In the nutritional context, Monaco also described the SRBP molecule as shaped like a calyx. and noted, “Into this calyx, the retinol molecule binds with the β-ionone ring buried deepest and with the alcohol moiety pointing to the outside on the surface of the molecule. Recall that the presence of the vitamin bound to RBP is correlated to the formation of a more stable complex with TTR.” Monaco notes, “Though it was initially thought that removal of the retinol molecule from the calyx would result in major conformational changes, X-ray crystallographic studies of human and bovine RBP have shown that the transition from holo- to apo-protein involves only very subtle modifications. The most important is a conformational change on the loop extending from amino acids 34 to 37, in particular, Leu 35 and Phe 36. The space left empty by the removal of the vitamin is filled in both cases by the aromatic ring of Phe 36 and solvent molecules and the movement of the Phe side chain drags the nearby amino acids into positions which are different from those adopted in the holoprotein.” This change may be all that is necessary for the apo-SRBP to be subject to attack and removal by immunological elements of the body. Simultaneously, Monaco has discussed TTR and described, “The two dimers of the tetramer are separated by a channel and in contact through symmetry related loops. The channel, about 10 Angstrom in diameter, has been shown to be the ligand-binding site.” There are two implications from these statements in the nutritional context; first, it is the TTR that is the “bottle” and the SRBP is the “cork.” Second, the apo-SRBP in its modified conformation is subject to attack.

Monaco provided a broader scope focused on the participation of SRBP + TTR in the visual modality in a review in 2009105. That paper is addressed in Section 7.1.2.1.3. 7.1.2.6 Important extraneous material related to retinoic acid

Crist, et. al. have recently provided some useful data in spite of a concept of the photoreceptors not supported here106. They studied a material described as CRABPII (cellular retinoic acid binding protein II) but did not describe where it was normally found in the body of the subject. One of their principle citations asserted this protein is only associated with the skin, and not even the lungs107. The retina was not mentioned. They also modified this material in a number of ways to strengthen it ability to act as a host to retinoic acid.

While CRABPII may be useful as an experimental tool, it is not normally associated with either retinal or retinol and does not normally occur within the retina. It is not likely to qualify as a protein mimic of rhodopsin. Crist, et. al. note the low affinity of all-trans-retinal and CRABPII. It appears their experiments would be more productive if they employed either CRBP or CRALBP (Section 7.1.1.2.3) in conjunction with retinol or retinal respectively, and tried to add an additional oxygen to the retinylidene backbone. This would form one of the proposed rhodonine chromophores of vision after release of the rhodonine from the cellular protein. 7.1.3 A precise redefinition of the aspects of the Visual Cycle involving retinoids GOOD/EDIT

105Monaco, H. (2009) The Transthyretin–Retinol-Binding Protein Complex In Richardson, S. & Cody, V. eds. Recent Advances in Transthyretin Evolution, Structure & Biological Functions. Berlin: Springer-Verlag Chapter 8 106Crist, R. Vasileiou, C. Rabago-Smith, M. Geiger, J. & Borhan, B. (2006) Engineering a rhodopsin protein mimic J Am Chem Soc vol 128, pp 4522-4523 107Wang, L. & Li, Y. & Yan, H. (1997) Structure-function relationships of cellular retinoic acid-binding proteins J Biol Chem vol 272(3), pp 1541-1547 52 Processes in Biological Vision

The visual cycle has historically defined the putative circular movement of trans-retinal from within a molecule of rhodopsin in an individual disk of the outer segments to the RPE for reconstitution as cis-retinal and its return to the rhodopsin molecule described above within a cycle time commensurate with adequate availability of the resulting cis- rhodopsin for photon excitation. This concept is fatally flawed for at least three reasons. First, it does not recognize the prodigious quantities of retinal that would need to be moved to the RPE per second to support such a physical transport mechanism. Second, it does not recognize the presence of multiple, stereochemically distinct, chromophores within the photoreceptors. Third, the suction-pipette experiments of Baylor et al. (Section 5.5.10) demonstrate the continued stable operation of a photoreceptor in-vivo while isolated from the RPE. No exchange of retinal species between the RPE and the photoreceptor cell was possible under these conditions. Saari has noted the difficulty of the transport problem in his Friedenwald Lecture for 2000108. “The transcellular migration of the retinoids during bleaching and regeneration is all the more remarkable, considering the anatomy of the journey. The relatively insoluble retinoid must leave the disc membranes, diffuse through a cystolic compartment to reach the plasma membrane of the rod outer segment, traverse the plasma membrane, diffuse across the subretinal space to reach the plasma membrane of the RPE cell, enter into the reactions of the visual cycle in this cell, and make the return journey!” Interestingly, he omits the additional problem of extracting the retinal moiety from within its surrounding opsin molecule and its reinsertion following the above trek.

Gonzalez-Fernandez describes the above trek in more detail109. “All-trans retinaldehyde released from vertebrate rhodopsin is first reduced to all-trans retinol by an outer segment retinol dehydrogenase. The all-trans retinol then does a remarkable thing. It leave the outer segment, crossed the IPM and the traverses the apical RPE cell membrane . . . .” [Underline added] He goes on to describe the reconstitution of the 11-cis-retinol prior to its reverse travel back to becoming a ligand within a previously depleted rhodopsin molecule.

Both authors provide conceptual schematics of their concepts.

Third, the above visual cycle concepts do not recognize the routine replacement of the complete disks of the outer segments at a nominal rate of ten disks per day per outer segment (Section 4.5.1).

The electrolytic theory of vision described in this work eliminates the entire visual cycle as described above. In the electrolytic theory, the visual cycle is subdivided into two components, the transduction-related visual cycle and the homeostasis-related visual cycle.

In the transduction visual cycle, the excited electrons generated by photo-excitation within the liquid crystalline coating of the individual disks of each outer segment only travel to the edge of the disk before being de-excited as part of the transduction process (Section 5.xxx). The interval between the excitation of an electron within the excited state of the liquid crystalline coating and its de-excitation at the disk/neurite interface is measured in milliseconds or less. This relatively short time constant is a parameter in the adaptation process (Section xxx). There is no requirement for the transport of any chemical moiety over any distance or through any membranes as part of the transduction visual cycle.

The homeostasis visual cycle relates to the routine replacement of the chromophores of vision as part of the replacement of the disks in order to maintain functional viability of the outer segments. 7.1.3.1 Gross retinoid transport in vision

Figure 7.1.3-1 shows the flow of retinoids through the biological system in support of vision. The result is more extensive than a similar figure by Chader.and more specific than a similar figure originated by Bok but published widely110. The figure differentiates between the vascularization from the liver to the choroid artery and the venous flow away from the retina to agree with earlier figures in this work. As in Chader, the figure shows two possible methods of retinol transport from the intestine to the liver. The literature is conflicting on whether Vitamin A is carried via the blood system or via the lymphatic system and whether it is carried in association with chylomicra or a lymphatic retinoid-binding protein, LRBP. This work supports the lymphatic/LRBP

108Saari, J. (2000) Biochemistry of visual pigment regeneration Invest Ophthal Vis Sci vol 41(2), pp 337-348 109Gonzalez-Fernandez, F. (2002) Evolution of the visual cycle: the role of retinoid-binding proteins J Endocrin vol 175, pp 75-88 110Bok D. (1990) Processing and transport of retinoids by the retinal pigment epithelium. Eye. vol. 4 ( Pt 2), 326-32. Review. Dynamics of Vision 7- 53

transport path/mechanism. There is general agreement that the retinoids are stored as esters in the liver111. Chader references Goodman (1979)112 who speaks of the retinoids being transported from the liver in the bloodstream via an RBP in conjunction with plasma albumin, PA. Later work, by the same author113, suggests a more complicated method for the transport of the retinoids to be used in chromophore generation. This method involves transport of the retinoid in a “bottle” formed by a specific serum retinoid-binding protein, SRBP, and a stopper of plasma transthyretin, TTR. These different studies had different purposes. It can be assumed that the transport of retinoids in the bloodstream has at least three end purposes, the growth of the organism, the reproduction of the organism, and provision of the chromophores of vision. It is proposed here that each of these objectives may use a specific and different form of transport. The diagram illustrates this by showing three branches to the arterial system. In this depiction, the ()RBP label is generic and applies to the three paths. The parentheses can be filled by specific labels for different types of RBP’s. The more specific label of SRBP + A + TTR applies to the vision-related path. Both the growth and vision transport mechanisms serve the RPE cells via the choroid artery (probably using two different receptor areas on the surface of each cell). By eliminating the reference to the cis- form of retinol and replacing the broken arrow notation by a double ended Rhodonine symbol, the caricature by Bok is very similar to this figure. Whereas Bok does not discuss how the putative retinol is converted back and forth from trans- to cis- within a timescale acceptable to the visual process (seconds or less), no such problem exists with Rhodonine. [xxx edit below here while citing the earlier parts of chapter 7 as well. ] Section 4.6 discussed the options of how the retinoid (probably retinol) could be converted into one of the Rhodonines as part of the transition between the blood stream and the location of the storage of the esters of Rhodonine. This path may or may not require the participation of the ribosome structures within the RPE cells. In either case, the Rhodonines are stored in up to four different color globules within each RPE cell. It is most appropriate to consider these globules as micelles because they do display the color associated with an individual Rhodonine in liquid crystalline form rather than the non-resonant color near 500 nm associated with the material in granular form. It appears they are transported to the storage areas by CRBP and released through esterification. They are removed from storage and joined to CRAIBP through hydrolysis when required to support disk fabrication by the photoreceptor cells. CRAIBP transports the Rhodonine to the RPE cell wall facing the IPM where it is handed off to the IRBP for short-term storage within the IPM. The IRBP is responsible for delivering the Rhodonine to the new disks forming within the extrusion cup of the photoreceptor cells. The Rhodonine is deposited onto the Opsin substrate of the disk as a liquid crystalline film. The film completely surrounds the disk.

111Berman, E. (1991) Biochemistry of the Eye. NY: Elsevier pp 49-53 112Goodman, DeW. (1979) Vitamin A and retinoids: recent advances. Fed. Proc. vol. 38, pp 2501-2503 113Sporn, M. Roberts, A. & Goodman, DeW. (1984) The Retinoids, vol 2 pp 42-85 54 Processes in Biological Vision

Figure 7.1.3-1 Gross caricature of retinoid transport in vision. ER, endoplasmic reticulum. The choroid included here is inside the blood-brain-barrier. The vascular path below the choroid path and that stressed here does not pass through the blood-brain-barrier. See text for discussion. Compare to Chader (1984)

The unique transport mechanism of SRBP + A + TTR suggests that the retinoid retinol may be converted into the Rhodonine as part of its encapsulation in the bottle. The SRBP-TTR combination would be destroyed in this scenario when it releases the new Rhodonine at the RPE interface because it would have lost at least an oxygen atom to the new Rhodonine. This would explain why the “apo-SRBP,” shown here as the post-holo-SRBP, is not reused after delivery of the retinoid/Rhodonine, but is decomposed in the kidney. This apo-RBP would not be a viable form of the original Dynamics of Vision 7- 55

apo-RBP (pre-holo-SRBP) formed in the liver. It may also explain why the retinoid retinoic acid is not employed in the formation of the chromophores. Retinoic acid contains a carboxyl group that already contains two oxygen atoms. Because of the additional stability of this structure compared with the alcohol or aldehyde, the molecule is not compatible with binding to SRBP. It is also not compatible with the addition of another oxygen to its structure at a location required for vision. The Rhodonine material, in the liquid crystalline film, is capable of repeated excitations by light and de-excitations by local quantum-mechanical means for an indefinite length of time. No requirement exists for the chromophoric material to be transported back to the RPE for reactivation. After a period of about 7-10 days in human, each disk has traveled from the extrusion area to the phagocytosis area of the RPE cells. The entire disk is reabsorbed by the RPE cells at this time and the materials are salvaged. Rhodonine that is still viable is transported back to the storage areas within the RPE cells, probably by CRABP or CRALBP. Material degraded from viable Rhodonine, by any means, is treated as debris and returned, along with the Opsin residues, to the vascular system for disposal in the kidneys. This group of mechanisms avoids many question marks in the Bok figure. It also eliminates the need to penetrate the putative Outer Segment membrane of that work. 7.1.3.2 The overall visual cycle related to homeostasis

With the demise of the previous visual cycle, the homeostasis visual cycle can now be redefined as follows. The visual cycle describes the physical renewal of the outer segments of the photoreceptors in order to maintain their functional viability. The gross physical steps include generation of new opsin-based disks by extrusion at the inner segment/outer segment interface and the simultaneous phagocytosis of old disks as they arrive at the outer segment/RPE interface. The gross chemical steps include the liquid crystalline coating of the new opsin-based disks with one of four appropriate chromophores (the resonant retinal derivatives known as the Rhodonines) with material primarily recovered from the on-going phagocytosis process. The four required chromophoric materials are stored as esters within the RPE cells in distinctly separate pigment granules (Section xxx).

The dynamics of the proposed visual cycle are described in expanded form in Figure 7.1.3-2 based on the static configuration shown in Section 4.5.

56 Processes in Biological Vision

Figure 7.1.3-2 Details of the flow of retinoids supporting the outer segments via the RPE ADD in tetrachromats. The UV channel in humans, and other large mammals, is of limited operational importance due to absorption of incident light below 400 nm by the lens. The elements remain functional in all mammals, and totally operational in smaller mammals.

The virgin material retinal is brought to the RPE layer via the blood stream and processed into four unique chromophore precursors (retinyl esters) that are stored in the pigment granules before being transported to the newly formed disks of each spectrally specific outer segment. It also shows how the chromophores from the phagocyzed disks are recovered and stored as retinyl-esters in the pigment granules. When needed, the chromophore precursors are converted into the precise chemical form of the chromophores in the process of transporting them to the virgin disks.

The literature has specifically noted the retinoid binding proteins used to transport the retinoids, SRBP, TTR and IRBP are all tetramers114,115,116, a situation analogous to the tetrameres of Rhodonine. They contain four subunits of a structure that can participate in a stereochemical reaction with the retinoid being transported. Johnson et al. (2007) noted that the subunits were only identical in the first order; the subunits exhibited different levels of participation in various complexing activity.

The current challenge is to define precisely how these RBP’s support the required stereochemical transformations. As noted in Sections 4.5.2 and 7.1.2, the chromophore precursors exist in four separate non-resonant forms before reaching the IRBP’s. Whether four distinct IRBP’s are needed to support the transport of the chromophores and their conversion to their resonant form within the IPM is open to further study. Note there is no role or need for an IRBP to transport chromophore related material from the disks of the OS through the IPM back to the RPE cells. This transport function is provided by the normal disk movement and phagocytosis. Each RPE cell is in contact with 24-44 individual photoreceptor cells and digests on the order of 2000-4000 disks each day! (Berman, pg 399). While both the SRBP’s and the TTR are known to be tetrameres, it is not clear how each support the transport and conversion of retinal into the resonant Rhodonine-esters before, or on, being delivered to the pigment granules (Section

114Gamble, M. & Blaner, W. (2000) Factors affecting blood levels of vitamin A In Livrea, M. ed. Vitamin A and retinoids: an update Basel: Birkhauser Verlag 115Gonzalez-Fernandez, F. (2002) Op. Cit. 116Chader, G. Pepperberg, D. et al. (1998) Retinoids and the retinal pigment epithelium In Marmor, M. & Wolfensberger, T. eds. The Retinal Pigment Epithelium. NY: Oxford Press pg 139 Dynamics of Vision 7- 57

7.1.2). Section 7.1.3.2.1 will develop the subject of transport through the blood stream using the SRBP + TTR “bottle and cork” at a more detailed level. The literature discusses the presence of several largely conceptual RBP’s within the RPE cells, the CRBP’s and the CRALBP’s. While the terminology suggests CRBP combines with retinol and CRALBP combines with the aldehyde, retinal, this is not the case. Saari, 2000) discussed this fact in detail. “First, the protein (CRALBP) has a high-affinity binding site for either 11-cis-retinal (Kd, 10 nM) or 11-cis-retinol (Kd, 60 nM).” They did not describe the ability of this material to bind to a resonant form of these same materials, the Rhodonines. He notes its presence in both RPE and Mueller cells of the retina. CRBP and/or CRALBP probably play roles in interfacing with the external RBP’s at the membrane of the RPE cells and moving the chromophore precursors to and from the pigment granules. However, the specific chromophore precursor processed by these RBP’s are probably not retinol and retinaldehyde as conventionally assumed. How they interact stereochemically with the chromophore precursors is currently unknown. In the following discussion, a slightly different nomenclature will be used for the RBP’s. A single CRBP will be subscripted to detail its specific function. Under the electrolytic theory of vision, the disks of the outer segments have been extruded by the inner segments and are external to the photoreceptor cell. Therefore, the chromophores and/or their precursors need to cross cell membranes only at the RPE/bloodstream and the RPE/IPM interfaces. Figure 7.1.3-3 shows the simpler visual cycle under the electrolytic theory compared to the previous theory as characterized in the “working model” by Saari (figure 4) and elaborated upon by Bok (figure 1-24 in Tso), by Bridges et al. (figure 2-2 in Tso), by Berman117 (figure 7.15), Besch et al118. and by La Cour & Tezel119. Bessant et al. give an alternate view of the retinal (Vitamin A) cycle that differs from these other investigators and uses different nomenclature120. Most of these figures are conceptual in character. None of these investigators address the stereochemical properties or the spectral differentiation of the chromophores processes in the visual cycle. None note the tetrameric character of the various RBP’s.

Two complete disks are shown along with one disk in the process of phagocytosis at the end of its nominal one week life in an outer segment of the human retina. As new disks are extruded, they are uniformly coated with one species of the liquid crystalline Rhodonines. This makes them photosensitive.

When a photon of light impinges on the liquid crystalline layer, an electron is raised from the ground energy band to the excited energy band. This excited electron can travel to the location of one of the neurites connected to the adaptation amplifier Activa of the photoreceptor cell. At that location, the excited electron transfers its energy to the neurite, generates a free electron within the neurite, and returns to the ground state as shown. This completes the transduction- related visual cycle (with no transport of chemical materials).

Virgin disks are coated with Rhodonine delivered to the disks by the RBP known as IRBP based on its presence in the IPM. This spectrally specific Rhodonine is already in its resonant form and can join its spectral specific brethren by stereochemical association. The Rhodonine transported by the IRBP is obtained from CRBPX,2 by a handshake mechanism at the RPE/IPM interface. The CRBPX,2 in turn obtains its Rhodonine from a spectrally specific pigment granule where it is stored in ester form (as palmitates or stearates). The pigment granule obtains most of its store from CRBPX,3. CRBPX,3 obtains its RhodonineX from the phagocytosis mechanism. Some RhodonineX obtained from the phagocytosis mechanism is unusable and is discarded via the bloodstream. This shortfall is compensated by acquisition of new RhodonineX from the bloodstream by CRBPX,1 in a handshake with the RBP known as SRBP. It is not known whether SRBP exists in four distinct forms or not. However, it is clear the Rhodonines are in resonant ester form when delivered to the pigment granules by CRBPX,1. Several investigators have suggested the enzyme LRAT (lecithin:retinol acyltransferase) is required to support the esterification of the retinoids at the pigment granules121.

117Berman, E.(1991) Biochemistry of the Eye. NY: Elsevier pg 390 118Besch, D. Jagle, H. Scholl, H. et al. (2003) Inherited multifocal RPE-diseases: mechanisms for local dysfunction . . . Vision Res vol 43, pp 3095-3108 119La Cour, M. & Tezel, T. (2006) The retinal pigment epithelium In Fishbarg, J. ed. The Biology of the Eye. NY: Elsevier pp 253-272 120Bessant, D. Ali, R. & Bhattacharya, S. (2001) Molecular genetics and prospects for therapy of the inherited retinal dystrophies Cur Opin Gen Devel vol 11, pp 307-316 121Gonzalez-Fernandez, F. ((2002) Beyond carrier proteins J Endocrin vol 175, pp 75-88 58 Processes in Biological Vision Dynamics of Vision 7- 59

Figure 7.1.3-3 A schematic of the homeostatic and transduction visual cycles. The separate transduction-related visual cycle is shown within the dashed box. The weight of the arrows shows the volume of material carried within the main transport loop relative to the limited amount of new material brought from the blood supply. See text. 60 Processes in Biological Vision

Note specifically that the IRBPX only transports chromophores in one direction, from the RPE to the virgin opsin-based disks. The return of the chromophores to the RPE is via the movement of the disks toward the RPE and their ultimate phagocytosis. The overall movement of moieties within the homeostasis visual cycle is shown in a more concise block diagram in Figure 7.1.3-4. The proposed movements for each of the ultimate chromophores is much simpler, and more defined, than that described by Bridges et al. in Tso (figure 2-7). Note the transduction-related visual cycle is shown at the lower right as a distinctly separate mechanism from the homeostasis-related visual cycle. The transduction-related visual cycle does not involve any movement of moieties in the space defined in this figure. The free electron is transferred to the neural system. In this figure, it is clear that a form of retinol or retinal is stored in the liver and is distributed throughout the body by being encapsulated in a “bottle” consisting of the protein SRBP and TTR, transthyretin. At some point during the distribution, at the RPE cell plasma membrane or upon deposition in the pigment granule, the retinal is converted to one of the four resonant forms of retinal, the Rhodonines. They are resonant in the sense that they consist of a conjugated carbon chain terminated by two oxygen ions. In this configuration, it is impossible to delineate either an alcohol or aldehyde form explicitly. The fact the material is now in resonant form is obvious because the material stored in the pigment granules is highly colored whereas retinol and retinal are colorless. The Rhodonines are stored in the pigment granules as esters, which retain the resonant form of the Rhodonine moiety.

The resonant material stored in the pigment granules exhibit spectral peaks associated with the four Figure 7.1.3-4 Block diagram of proposed homeostasis visual cycle in the Rhodonines (Section 5.5.9). chordate eye. The transduction visual cycle is shown at lower right. The However, the width of their absorption subscript X can represent the UV, S, M or L form of rhodonine or the spectra is believed to be narrow associated carrier. See text. indicating they are not in the liquid crystalline state. Because the resonant form of the Rhodonines cannot be described as either alcohols or aldehydes, the designations historically used for the RBP’s present in the RPE cells, CRBP and CRALBP, are not useful. It is more useful to designate the RBP’s by their function until a more precise method of naming is developed. Thus the CRBP’s are subscripted first to indicate their association with a specific Rhodonine and second to indicate their role; 1 = movement to bulk storage, 2 = movement to the IPM interface, 3 = recovery during phagocytosis and return to bulk storage. As shown at upper right, degraded Rhodonine may be transported to the bloodstream by another RBP of yet unspecified form. Berman has provided an in-depth discussion of the phagocytosis and remnant elimination process122. The weight of the arrows is indicative of the volume of chromophoric material being transported. The system is a closed loop with minor input of new material to replace degraded material. A possible cause of the degradation is cosmic ray damage in long life animals. Such regeneration and disposal of damaged material is only known to occur in the eyes

122Berman, E. (1991) Biochemistry of the Eye. NY: Elsevier pp 399-406 Dynamics of Vision 7- 61

of Chordata. Gonzalez-Renandez attempted to define the visual cycle to include the invertebrates in 2002. H he did not provide substantive data supporting a visual cycle in these species. He did include an interesting observation. “To date, IRBP has not been identified in any invertebrate retina. Furthermore, we have not been able to identify the IRBP gene in the Drosophila database which is now virtually complete.” The lack of any IRBP in the vertebrate retina appears to doom his proposition. As noted earlier, the IRBP’s are only used to transport Rhodonines to the disks of the outer segment. There is no requirement that the Rhodonines be returned to the RPE by chemical transport. Return transport is provided by the movement of the disks and their subsequent phagocytosis. It is not known whether separate IRBP’s are required for each member of the Rhodonine family or whether a single IRBP can transport all varieties of Rhodonine with their selective deposition, on spectrally specific disks, based on their own stereographic arrangement.. 7.2 Dynamics of radiation-chemistry and the photoreceptor cell

To interpret the radiation-chemistry of vision correctly, understanding the operation of the photoreceptor cells of the retina in detail is necessary. The photochemical aspects of the Outer Segment of these cells have been discussed in Chapter 5. The electrophysiology of these cells will be discussed in Section 10.8.7. The fundamental circuit diagrams of the photoreceptor cells developed in that section are reproduced here as Figure 7.2.1-1 for convenience. Frame A describes the morphology of each cell and emphasizes the extracellular nature of the disk stack of the Outer Segment. Only the dendritic structure of the photoreceptor cell extends into the region of the Outer Segment. The zigzag line surrounding the cell in frame B and C represent the electrolytic environment surrounding the cell. Note this environment is divided into two regions, the IPM and the INM. This physical and electrical division plays a significant role in the clinical evaluation of the retina. While frame B shows the electrical circuits of the photoreceptor cell in their correct topology, these circuits are usually rearranged in electronics, as shown in frame C, to emphasis an important feature. The two Activas are arranged in a differential pair with a common emitter impedance (2). This circuit exhibits several special properties discussed in Section 10.8.7. The leftmost Activa employs a unique structural arrangement that introduces a highly nonlinear output current as a function of the input excitation. This circuit is the adaptation amplifier of the visual process and is discussed in Section 12.5.3.

The detailed structure of the disk stack has been presented in Section 4.3.2 through 4.4.2. 62 Processes in Biological Vision

Figure 7.2.1-1 The morphology and electrophysiology of the photoreceptor cell from Section 10.8.5.3. See text for details. Dynamics of Vision 7- 63

Recently, Clark et al. provided a simple differential equation for the dynamic properties of the photoreceptors123. They described their approach, “We introduced a new phenomenological model that captures the response and adaptation properties of cone photoreceptors. The DA model is expressed as a first-order differential equation in time (Eq. (2)) and relies upon a single non-linearity.” Their Table 1 is a useful bibliography. Their model results in a response that is the difference between two exponentials. It can fit a wide variety of idealized qualitative data. They describe their extensive “fitting” operations in the caption to Table 2. However, their model does not incorporate any absolute delay term, any temperature term or the more general adaptation mechanism associated with the sensory neuron as opposed to the transduction mechanism alone (which they attribute to the “photoreceptors” and this work attributes to the disk stack of the above figure alone). Their model does not address the totally different dynamics of the long wavelength transduction mechanism. Their range of adaptation does not approach the multiple orders of magnitude actually encountered in vision. 7.2.1 Radiation Chemistry Photosensitive chemical compounds, upon radiation by photons of a few electron-volt energy, have many reactive and non-reactive options. The initial process involves absorption of the energy and the transfer of the molecules of the material to an excited state. One of the following events must then occur: 1. The energy can be re-radiated within a very short time, 10-9 seconds, leaving the material unchanged. The resulting phenomenon is called fluorescence. 2. The energy can be stored for an appreciable period of time, 10-5 seconds or longer (even minutes or hours), before re-radiating a photon of somewhat lower energy and again leaving the material unchanged. The resulting phenomenon is called phosphorescence. 3. The molecule can de-excite by “internal conversion.” The result is a thermal loss of energy 4. The energy can be transferred to another structure thereby returning the original molecule to its ground state. 5. The molecule can use the absorbed energy to rearrange itself. The process is known as isomerization. 6. The molecule can dissociate into its component ligands or atoms.

The first four options are nondestructive and essentially conservative, although some energy may be lost thermally in the process. The last two are destructive of the original molecular structure. Options 5 and 6 are the only processes that actually use the energy absorbed from the photon to do work. They require additional energy to return the constituent(s) to their original configuration. Option 4 was little known during the early work in vision and it was not adequately considered. Option 4 is quite conservative in energy and requires no physical rearrangement of the material. It has become a well-understood process in recent years and can be used in a continuous process without requiring any additional energy source. In fact, it is the mechanism used in nearly all gas and liquid lasers. It is also the primary mechanism that makes possible color photography based on the underlying silver halide process.

Lacking knowledge of option 4, option 5 was promoted by Hubbard, and subsequently Wald and others, as the most likely process used in vision. Validation of this isomerization hypothesis as a fundamental mechanism in vision is notably lacking after 50 years. In vision research, option 5 has always been a stumbling block because of two questions. Where does the energy come from to restore the isomerized material to its original condition? The amount of energy required can be significant in the confined metabolic system of the eye. How is the rearrangement associated with option 5 reversed in a timely manner?

The dynamics of option 2 and 3 are quite similar and both have relevance to the vision process. This will be seen as the model develops. Option 6 will not be explored in this work. Clearly, the visual process does not consume a significant amount of chromophore material. 7.2.2 Excitation of a Liquid Crystal

The photon excitation of a liquid crystal has been discussed in Chapter 5. It is a complex process involving anisotropic absorption in exchange for a greatly increased absorption cross section. The important point to note here is that the

123Clark, D. Benichou, R. Meister, M. & Azeredo da Silveira, R. (2013) Dynamical Adaptation in Photoreceptors PLOS: Comp Biol •DOI: 10.1371/journal.pcbi.1003289

64 Processes in Biological Vision process is quantum-mechanical in nature and can be described as photoelectric in character as opposed to photoconductive or photothermal. As a result, the rules of quantum mechanics apply to this absorption as do the rules of quantum mechanics. 7.2.3 De-excitation of a Liquid Crystal

As discussed briefly in Section 7.2.1, the relevant de-excitation process in vision involves the transfer of energy from the excited liquid crystalline chromophore to another material through a quantum mechanical process. This process is completely conservative with respect to the physical structure of the chromophores. As with excitation of the chromophore, this transfer is also quantum-mechanical in nature and subject to the statistics of quantum mechanics.

Zollinger has addressed the transfer of energy from an excited dye to an oxidizing or reducing agent124. However, he did not address the potential for transferring energy between two crystalline states. In vision, the hydronium liquid crystal of the dendrites can receive energy from the chromophoric crystals. 7.2.4 Dynamics of excitation

The dynamics of creating an exciton within the energy bands of a liquid crystalline material upon the absorption of a photon involves two separate mechanisms. The actual absorption of the photon results in a near instantaneous creation of an exciton. However, the probability of absorption depends on the availability of unexcited n-electrons. Whereas absorption under “dark adapted conditions” can occur within a few hundred femto-seconds, the observation of the signal at the Activa within the inner segment occurs much later. The time of absorption followed by transport of the charge within the liquid crystal and de-excitation at the dendrite is on the order of 10's to 100's of microseconds. 7.2.4.1 The dynamics of photon absorption

Wang has employed differential bleaching to determine the excitation time of the visual photoreceptor material at 200 femto-seconds when using 500 nm excitation light125. He used a variable wavelength probe to measure the change in the absorption coefficient. His data shows nearly the same excitation time for all wavelengths of light from 570 to 630 nm. There is some indication that the time for the 620 and 630 nm light may be completed in about 100 femto-seconds. Following this excitation, the absorption coefficient remains constant for more than three picoseconds. This data is compatible with the excitation time expected. The open-ended value, “more than three picoseconds,” leaves the question of de-excitation time undefined.

Although the above experiments did not quantify the absorption coefficient, De Grip, et. al. have given a value of 0.67 for the quantum yield126. This value, although not defined in detail, is quite compatible with the value obtained by calculation based on the signal-to-noise properties of light. See Chapter 17. 7.2.4.2 The dynamics of excitation/de-excitation (small signal case)

In most areas of Physical Chemistry, the description of the excitation process is very simple; there are only one or two non-bonding electrons present and once they are excited by radiation of an appropriate wavelength, the molecule is transparent to additional radiation at that wavelength. The decay time of the excited electron(s) is easily observed. There is little interest in illustrating the process in any greater detail. In the case of chromophores in the liquid crystalline state however, the number of shared non-bonding or n-electrons can be quite large. In the region of 109 n-electrons can be found in the OS of a photoreceptor if they are all shared via the spaceframe created by the structural protein, Opsin. Further, the decay process can have different time constants depending on the de-excitation involved. In this situation, the dynamics of photoexcitation and de-excitation are not trivial. The process is probabilistic and will be labeled as “withdrawal with delayed replacement,” a variant of the extreme cases called “withdrawal with replacement” and ‘withdrawal without replacement” generally described in Probability Theory. In this process, a time delay is encountered before replacement. This time delay can be attributed to any of a number of causes. The principle cause is related to the excited state the electrons go into. An additional factor is whether the de-excitation is controlled entirely by the internal structure of the crystal or by an external process. If the electrons are excited into a singlet state, de-excitation is generally rapid and described as fluorescence. If they go into a triplet state, the natural process of de-excitation is

124Zollinger, H. (1991) Color chemistry. NY: VCH pg. 303 125Wang, Q, Schoenlein RW, Peteanu LA, Mathies RA, Shank CV. (1994) Vibrationally coherent photochemistry in the femtosecond primary event of vision. Science. vol. 266, pp. 422-424 126Degrip, W. (1999) In Rhodopsin and phototransduction. NY: John Wiley & Sons, pg. 104 Dynamics of Vision 7- 65

usually one of phosphorescence. However, in some situations, the crystal may be in intimate contact with a substrate or other structure that can cause de-excitation before the normal phosphorescence. In the case of chromophores having two identical (possibly similar) auxochromes symmetrically located at the extremes of a conjugated chain, phosphorescence requires a forbidden transition and it is not observed in these molecules. These molecules may remain excited for an extended period until thermal or other external processes accomplish the de- excitation. This is the situation normally observed in vision research. 1.) Lewis & Perreault127 as late as 1982 indicate the chromophores of vision do not exhibit significant fluorescence or phosphorescence and many investigators report the chromophores, once “bleached” remain excited for long periods. The periods are frequently described in terms of hours. Most frequently, the preparations are left unattended in the laboratory overnight. 2.) If the above forbidden transition, is only present when identical auxochromes are involved, this would be strong evidence for the symmetry of the visual chromophores. It would essentially eliminate the possibility that the chromophores of vision are described in terms of the Amidic system, i. e. contain a nitrogen atom. This situation would in turn prevent the chromophores from participating in a Schiff-base bond to the substrate as proposed by Collins & Morton in 1950. Morton & Pitt further rationalized the Schiff-base behavior proposed by Collins & Morton in 1955 using terms such as “a fortuitous artifact.” See Dartnall128 writing in Davson for a discussion of this area. The energy band structure of a complex organic molecule such as the Rhodonines in the liquid crystal state is very complex. There are a large number of unpaired electrons subject to excitation, and an equal number of empty excited states. Following excitation, these electrons will remain in one of the excited state for a period of time until they are used in some additional process or they return to their original ground state. In either case, explaining the dynamics of the process using a first order differential equation with constant coefficients is possible.

Figure 7.2.4-1 shows the basic flow diagram, an appropriate electronic equivalent circuit and the basic equations involved. The rate of generation of excited states is proportional to the input radiant flux times the absorption cross section times the number of available electrons in the n-electron band of the energy band diagram. The rate of decay of electrons from the excited state is proportional to the number of excited electrons in the π* band. An auxiliary equation is that the total number of excited and unexcited electrons is fixed and finite. This fact is emphasized in the equivalent circuit by showing a capacitor in the lead connected to ground--the excitation process cannot draw on an infinite supply of electrons. This would only be the case if the circuit were connected directly to a ground.

127Lewis, A. & Perreault, G. (1982) Emission spectroscopy of rhodopsin and bacteriorhodopsin. In Methods in Enzymology, vol. 88, Biomembranes, part I 128Davson, H. (1962) The eye. New York: Academic Press, vol. 2, pg 450-454 66 Processes in Biological Vision

Figure 7.2.4-1 Basic flow diagram, equivalent electronic circuit and applicable equations.

If a complete solution to this problem is desired, another factor must be considered. It involves the transport of excited electrons from their excitation point to their de-excitation point on the π* surface of the liquid crystal forming the surface of the disk. The concept is simple, however, the mathematics involves Bessel Functions. When the first excited electron is created, it has very little tendency to move toward the edge of the crystal. Once two or more excited electrons are formed, they will repel each other and move to the periphery of the disk, where the dendritic structures are. As ever more excited electrons are formed, the fields generated will cause more rapid movement of the excited electrons to the periphery. Thus the time to reach the edge is an inverse function of the field strength that is itself a function of the absorbed flux rate. This problem is solved in transmission line theory. It represents a pure delay term in the overall solution of the photoexcitation/de-excitation problem. Here the delay is inversely proportional to the absorbed flux as indicated in much of the test data (but not accounted for in the accompanying analyses). Appendix A addresses the details of this phenomenon. For the immediate purpose, defining a short term transit delay is adequate, Tt, that is a function of the incident photon flux. This delay can be considered the quotient of the average distance traveled by the excitons (between their point of creation and their point of disappearance) divided by the short term average transit velocity of those excitons. However, it is also highly temperature dependent and its full expression must include a temperature term. Many equivalent electrical circuits can be employed to model this process. They differ in what electrical term is related to what term in the original system. For instance, the radiance, F, is taken as the number of photons per unit area per unit time. When multiplied by the cross sectional area of an OS and a time interval, the resulting flux has units of photons. This quantity can be related to either a charge or a current in a circuit using a current generator. Alternately, it can be related to a voltage (or the derivative of voltage) in a circuit using a voltage generator. The form is optional. Here: + the total incident flux is in photons and can be equated to individual electrons, q, in the equivalent circuit.

+ F•σ is taken as a flux rate, photons per second, where s is the effective absorption cross section of the liquid crystal chromophore. The total flux rate due to absorption by a single Outer Segment can be related to the current, dq/dt in the equivalent circuit. Dynamics of Vision 7- 67

+ Tt is the transit delay between the excitation of an exciton and its return to the ground state. Based on the above conditions and definitions, the excitation problem can be put into the form of a standard differential equation. The solution of this equation depends on the form of the forcing function, qf. Appendix A contains solutions to this standard form for a variety of forcing functions. That of most interest here is for a forcing function defined as an impulse. This corresponds to the short flash of light generally used to measure the intrinsic response of photoreceptor cells in the laboratory. Flashes of 50 msec or less qualify as an impulse up through the photopic region of vision. 7.2.4.2.1 Excitation/de-excitation with transport delay, the P/D Equation

Introduction of transport delay complicates the form of the complete P/D Equation considerably. However, it does describe the process of photoexcitation followed by de-excitation in detail. This is particularly true after the effect of temperature is included. The complete derivation of the P/D Equation is provided in Appendix A. The P/D Equation, describing the current injected into the neural system by a single disk in response to an impulse stimulus, in final form is:

Eq. 7.2.4-1

under the condition that σ •F•τ not equal 1.00

Note carefully that the first exponential term contains the imaginary operator, j. This is the delay term in the overall expression. Each exponential term includes a temperature sensitive component, KT. T represents the temperature of the chromophores in degrees Celsius and the eight is indicative of the narrow biological range of this variable. C•T is equal to 0.002 seconds and the term, kd, is a scaling factor carried as a matter of convenience. This equation is shown in Figure 7.2.4-2 with typical values for the variables. 68 Processes in Biological Vision

Figure 7.2.4-2 Theoretical responses to an impulse as predicted by the photoexcitation/de-excitation equation. The latency is shown explicitly, by the departure from the baseline, as a function of the peak flux density, F, in 2 photons/micron -sec. For other temperatures, the time scale can be multiplied by the appropriate value of KT. The value of σ is appropriate for perpendicular illumination, or a stack of individual disks. The Hodgkin Solution (σ•F•τ = 1.000) occurs at F = 12.

Note that the P/D Equation is a first order equation that departs from the baseline abruptly following a delay (latency) given by the imaginary term in the equation. Note also that the value of the equivalent resistor, r or rn in Figure 7.2.4-1, must be reasonable if a reasonable value for the time constant is to be obtained.

Each waveform exhibits a different absolute delay (latency), a different peak amplitude, and a different slope associated with its leading edge. The simultaneous changing of all three of these parameters is the primary cause of difficulty for previous investigators attempting to find an empirical solution to the P/D Equation.

Cattell is credited with first observing the delay as a function of stimulus intensity in 1886129. Green has presented the paper in modern form as part of his “Classics in the History of Psychology130.”

Equation 7.2.4-1 is formidable. However, the delay term is well behaved and can be omitted while looking at the remainder of the equation. The most important feature is the fact that the amplitude response involves the difference between two exponentials. As shown by its derivative, such an equation exhibits a continually varying slope that is not easily described by a single exponential function.

129Cattell, J. M.(1886) The influence of the intensity of the stimulus on the length of the reaction time, Brain vol 8(4), pp 512–515 Alternately, Mind vol 11(41), pp 220-242 via JSTOR 130Green, C. (undated) Classics in the History of Psychology http://psychclassics.yorku.ca/Cattell/Time/ Dynamics of Vision 7- 69

Eq. 7.2.4-2

This response as a function of time, t, remains a function of multiple variables. These variables include the incident flux, F, the absorption cross section, σ, the time constant, τ, of the de-excitation process and a constant, KT, related to the temperature. The appearance of several of the parameters at multiple locations in the equation accounts for many of the problems of interpreting the laboratory data using a far simpler equation. This equation also includes a singularity for the product of σ, F & τ equal to 1.000. Under this condition, the solution of the differential equation takes the form shown in Equation 7.2.4.3 and discussed below.

By taking the Laplace transform of the general solution shown above, it is seen that the function contains two poles in the frequency domain. One pole is the result of a conventional time constant, defined as τ. The second pole is more unique. It is defined by the product of σ times F. The fact that the second pole, σ•F, is a function of the flux is what has caused problems for many earlier investigators. They have attempted to introduce a fixed filter in the transmission channel to provide a fixed pole equal to this variable pole.

7.2.4.2.2 The Hodgkin Solution to the P/D Equation for σ •F•τ = 1.00

For the values shown in Figure 7.2.4-2, the singularity at σ•F•τ = 1.00 occurs for a photon flux of 12 photons/μ2. shown in the above figure, the product of the flux, the absorption cross section and the time constant is 1.875. At the discontinuity, the mathematical form of the P/D Equation changes significantly. The complete form at the singularity is shown in Equation 7.2.4-3.

Eq. 7.2.4-3

where τ is the same time constant of the de- excitation process as found in the complete equation and KT remains the thermal coefficient at a given temperature. Again, the imaginary term is well behaved and can be omitted during the following discussion.

Although based on a more complicated process than the Poisson Distribution of statistics, the P/D Equation at σ •F•τ = 1.00 is identical to the equation of the Poisson Distribution for the trial value, ν = 2. Hodgkin apparently recognized the similarity in the waveforms of the experimental data of Fuortes & Hodgkin to the 70 Processes in Biological Vision

Poisson Distribution. He attempted to correlate the two in 1964 with little success131. Without knowledge of the delay term, he had a problem with the trial value when attempting to correlate the Distribution with the measured data. He arbitrarily assigned ν the value of 10 on page 252 and found a range of 5<ν<14 in Table 1. He intuitively associated the value of ν = 10 with an impedance-isolated ten-stage filter in the visual signal chain. This parameter appears as both an exponent and in a factorial term in the complete Poisson Distribution. It does not relate to any variable in the underlying process of vision. The best that can be said is that the P/D Equation and the Poisson Distribution share a common special case. The Poisson Distribution cannot be used to describe the P/D process of vision, except in the special case of σ•F•τ = 1.00.

7.2.4.2.3 Other attempts to obtain a P/D Equation

Later, Lamb writing in Baylor, Hodgkin & Lamb132, approached the photoexcitation/de-excitation problem from a different statistical direction and derived an “independent activation equation.” This equation was based on the assumption that the initial detection process was inherently linear and related to a photoconductor. The equation remained similar to the Poisson equation but avoided the factorial in the denominator. Being a statistical derivation, the problem of the factor ν remained (Lamb labeled it the number of “reactions” in quotation marks and assigned it the value of six). However, by varying the exponent representing the number of observations, and the other parameters, the equation could be made to fit nearly any individual recorded response. As noted in figure 4 of the article, the equation did not track a series of responses as a function of incident flux unless the variable ν was adjusted arbitrarily for each response. The best fits involved varying the putative time constant of individual “reactions” in a set of six or seven “reactions.” The formulation originating in the above Baylor, et. al. paper does not correlate well with a set of responses representing the photoexcitation/de-excitation mechanism of vision. Nor does the formulation correlate with any combination of that mechanism and any subsequent mechanism. In the final section of the above paper, and in subsequent papers, Lamb has concentrated on using a Michaelis (logistic) equation to describe only the rising phase of the waveforms related to the P/D mechanism.

As recently as 1984, Baylor, Nunn & Schnapf have recognized the limitations of these earlier efforts to employ multistage filters133. They said on page 581 while discussing the falling phase of the measured waveforms: “This feature, evident in the response of figure 3 [ Figure 7.2.4-3], and even more prominent in responses of other cells, could not be fitted by any adjustment of the time constants in the multistage filter models.” The dashed line shows the excellent fit provided by the P/D Equation in this region. This figure is discussed in greater detail in Appendix A.

131Fuortes, M. & Hodgkin, A. (1964) Changes in time scale and sensitivity in the ommatidia of Limulus. J. Physiol. vol 172, pp 239-263 132Baylor, D. Hodgkin, A. & Lamb, T. (1974) the electrical response of turtle cones to flashes and steps of light. J. Physiol. vol. 242, pp 685-727 133Baylor, D. Nunn, B. & Schnapf, J. (1984) The photocurrent, noise and spectral sensitivity of rods of the monkey Macaca fascicularis. J. Physiol. vol. 357, pp 575-607 Dynamics of Vision 7- 71

Figure 7.2.4-3 Response of in-vitro macaque outer segment at 36 Celsius. Data was collected using a 0-50 Hz filter. 500 nm light stimulus was applied perpendicular to the long axis of the outer segment. Each point represents average of 264 flashes of light. Note the 0.1 pA (peak to peak) correlated noise at ~20 Hz. Light was introduced perpendicular to the long axis of the outer segment. Flashes were nominally 0.58 photons/micron2 at 500 nm and 11 ms duration. The integrated exposure was 53 photons/sec-micron2. Thin line is impulse response of six identical RC filter stages in series adjusted to give a peak response at 200 ms following excitation. Heavy dashed line gives predicted responses based on the P/D Equation for σ•F•t - 0.53 & T = 36 Celsius. Data points from Baylor, et. al., 1984.

7.2.4.3 The dynamics of excitation/de-excitation (large signal case)

The small signal analysis leading to the P/D Equation assumed the absorption cross section of the chromophoric material remained constant. This is the condition encountered for low level stimulation under full dark adaptation. No bleaching of the chromophoric material is observed. For higher stimulation levels or significant background levels, the constant absorption cross section is not maintained. There is not a sufficient number of unpaired and therefore excitable ground state electrons associated with the chromophore pool. Therefore, the absorption cross section is reduced in proportion to the total pool minus those electrons that are still in the excited state at a given time.

Figure 7.2.4-4 shows the quantum efficiency as a function of the current through the emitter (dendritic structure) of the sensory neuron under prescribed conditions. The conditions are:

C The graph applies to steady-state conditions and the large signal case. C The figure does not apply to the L–sensory channel due to the fact that a 2-exciton phenomenon is employed in this channel.

C A wavelength of 532 nm is used as a reference point for the center of the M –channel receptor. A correction has been made for the number of photons compared to the 555 nm of the SI standard.

C A 2 micron diameter outer segment is assumed for each PC. C The irradiance applied to a single outer segment is axially oriented and collimated following the lens action associated with the inner segment of the PC.

C The protein based material of each opsin-based disk is essentially transparent at the spectral wavelengths applicable to a given PC. 72 Processes in Biological Vision

C The conversion of the irradiance described in terms of the candela is expressable in photons per watt using the 1979 International Standard (Section 17.1.3) even though the wavelength of 555 nm chosen is not related to the actual maximum spectral sensitivity of the human eye (the wavelength is relatable to the peak perceived sensitivity).

C Any attenuation related to the stage 0 optical system is ignored (typically about 10 percent, Section 2.4.2). The resultant calculations can only be considered an order of magnitude estimate because of the status of the standards etc. Under these assumptions, the human eye is only operating at 10–8 or less of peak quantum efficiency under photopic conditions. Similarly, Crawford carried out his scotopic experiments at a few percent peak quantum efficiency. Finally, peak quantum efficiency is reached at about 10–6to 10–7 candela/meter2, suggesting that the human eye is able to detect individual photons as reported sporadically in the vision literature (although it is not always clear the statistical ramifications of that claim are always elucidated adequately). The ramifications include the slowing of exciton velocity within the excited state associated with one liquid crystalline layer of one disk (Section 17.2.7.3). Section 16.4.1.2.2 reports on the data of Baylor, Nunn & Schnapf in 1984. They established a maximum collector current of 34 pA for their single isolated sensory neuron when stimulated using transverse irradiation at 500 nm. Section 17.6.1 provides the behavioral (operational) aspects related to the dark adaptation and light adaptation characteristics of the human eye. That section also introduces the “exposine,” an expression foreign to the previous vision literature. It describes the response of a 3rd order dynamic system to a short pulse of stimulation. It is the product of an exponential function and a sine function. Dynamics of Vision 7- 73

Figure 7.2.4-4 The idealized quantum efficiency of a photoreceptor cell as a function of irradiance and with the current capability of the 1st Activa of the PC (the adaptation amplifier). Brief references in the literature suggest 30–50 pA is the nominal current capability of the adaptation amplifiers. The figure only applies directly to the M –channel. It can be used for the UV– & S–channels by adjusting the horizontal scales. See text. The L–channel follows a different set of loci.

The figure illustrates two major findings; how large the dynamic range of vision is and how it is provided by the dynamics of the chromophores of vision. Under photopic conditions, the outer segments are highly bleached, leaving only about one part in 108 of the individual chromophore molecules of vision in a sensitive state. Even at the high end of the scotopic region, only about one part in 103 of the available chromophore molecules are sensitive to light. Only under the lowest light conditions does the retina absorb all of the light striking it, appear totally black due to this total absorption and not exhibit any bleaching. 74 Processes in Biological Vision

The total number of chromophore molecules per outer segment is approximately 4A1010 (Sec. 4.3.5.3.5) - - - The candela is now defined in terms of monochromatic light at 555 nm (or 540x1012 Hz) and whose radiant intensity in that direction is 1/683 Watt (4.092"1017 photons) per steradian. At a wavelength of 532 nm, the number of photons per steradian would be reduced proportionally, to 3.922@1017 photons/ per steradian. For UV– and S–channel performance, the horizontal scales should be adjusted to recognize the lower number of higher energy photons per watt at either 342 or 437 nm. The conversion between radiant intensity in cd/steradian and irradiance in cd/m2 is not usually addressed in the vision modality literature. The assumption is that the source is sufficiently far from the pupil that the energy arriving at the pupil is collimated and its radiance can be described in terms of watts per steradian. The radiance is usually reported as measured using a calibrated meter and using a white card (with a reflectance assumed to be 100% rather than the more likely 90-95%) at the location of the pupil. In human vision, the transition between the photopic and mesotopic regions is usually taken as 3–5 cd/m2 measured at the pupil of the eye.. The transition between mesotopic and scotopic is usually taken as 10–3 cd/m2 (Section 17.2.6). Crawford used an irradiance of 3 x 10–5 cd/m2 for his scotopic measurements (Section 17.2.6.1.1).

- - - -

The reduction of the quantum efficiency of each PC with irradiation is a direct result of the quantum-physics of the chromophores discussed in Section 5.4. The excitation of the chromophoric layers of the outer segment of a PC are the result of irradiance through the pupil of the eye. The excitation of the chromophoric layers remains essentially constant over time in the absence of de-excitation in the transfer of acoustic energy from the chromophoric layers to the 1st amplifier of the associated Activa (the adaptation amplifier). In the large signal case, the result of these activities follows the equation, irradiant flux (photons in) x quantum efficiency = emitter (dendritic) current (electrons out) nQaperture of the outer segment. For purposes of this discussion, the total irradiant flux for a 2 micron diameter outer segment will be taken as the irradiance in cd/m2 at 532 nm (3.99@1017 photons/sec) reduced by a factor of π@1012, or 1.27@105 photons/sec. [xxx make consistent with the figure above ] At very low irradiances, the quantum efficiency of the individual outer segment remains near 100% as the emitter current is able to de-excite all of the excitons in real time. This condition can be described as the scotopic irradiance region.

As the irradiance level rises, the limit for emitter current (nQmax) is reached. Without complete de-excitation, the chromophores remain partially excited. At the molecular level, the molecules that remain excited are transparent to the incident radiation, thereby lowering the effective quantum efficiency of the outer segment. This condition remains true throughout the mesotopic and photopic regions. At hypertopic irradiation levels multiple mechanisms limit the performance of the stage 1 sensory neurons.

It is suggested that the product of the average quantum efficiency times the incident flux level (σCF) remains nearly constant over a considerable range based on a nearly constant average emitter current. - - - - As calculated in Section 5.4, the absorption per layer of chromophore in the fully configured outer segment to radiation applied axially is about 0.0004. While this number may appear low, there are two layers per disk and about xxx disks per outer segment. Appendix L proposes the standard PC outer segment incorporates 2000 disks, each with two coated surfaces of chromophoric material. This suggests that essentially all of the irradiance applied to the aperture surface of the PC outer segment is absorbed under the most severe scotopic conditions; the gross quantum efficiency would be 100%. - - - - The significant change in the quantum efficiency of the PC with light level is the key to understanding of the great dynamic range of the visual modality with respect to incident irradiation. This change is not directly measurable at this time. Two parametric phenomena are commonly associated with the reduction in quantum efficiency of the PC’s, the Dynamics of Vision 7- 75

bleaching of the retina (observable by the ophthalmologist) and the processes of light and dark adaptation (observable by the biophysicist). - - - - The relationship between the Candela and power in Watts is a tenuous one, although recent activity within the biophysics community has given it a more solid foundation (Section 17.1.3). The principle problem is it is defined at only 2042 Kelvin (the freezing point of platinum under a high pressure) as of 1979. At that time, the Candela was transformed from a broad spectrum source at 2042 Kelvin to a narrow spectrum source at 555 nm (540 x 1012 Hertz–the presumed peak in the human photopic visibility function, Vλ). The blackbody radiation spectrum at 2042 Kelvin is notoriously insufficient in the blue region. whose radiant intensity in that direction is 1/683 Watt (4.092"1017 photons) per steradian.

See US National Institute of Science & Technology definition.134 The definition of the ampere is considerably simpler. one ampere is approximately equivalent to 6.2415093×10 18 elementary charges moving past a boundary in one second.

7.2.5 The dynamics of transduction to a current

The mechanism employed to obtain a determinate current in the neural system of an animal in response to excitation of the chromophores by light has been defined in Chapters 4 and 11 of this work. The mechanism is somewhat more complex than shown in the above figure. The liquid crystalline structure of each disk is in physical contact with at least one microtubule (that is actually a dendrite like structure) of the photoreceptor cell. This allows the energy of the excited state of the chromophore to be transferred to a suitable structure in the dendrite. This latter structure is the hydronium liquid crystal forming the base of the Activa within this structure. Figure 7.2.5-1 illustrates the overall situation. In part (a), the energy band diagrams are shown for both the liquid crystal of the OS and the hydronium crystal. The ellipse is meant to illustrate the coupling in which the de-excitation of an electron in the liquid crystal results in the excitation of an electron into the conduction band of hydronium crystal. This electron is swept out of the hydronium crystal by transistor action and becomes the initial free electron current of the neural system. For this process to work, the energy associated with the exciton(s) must exceed the energy necessary to elevate an electron to the conduction band of the hydronium crystal. Here one quantum of energy, Ed, in the chromophore is exchanged for one free electron in the hydronium at the expense of the energy, En. In (b), an equivalent circuit is shown consisting of two parts. The left side is the equivalent circuit used earlier to describe the photon excitation & de-excitation process in the chromophoric liquid crystal. The circuit shows two resistors, corresponding to the de-excitation of excited electrons by thermal means (also known as “internal conversion” in some texts) and by transfer means. rtrans is smaller than rtherm and therefore dominates in this circuit. This equivalent circuit is still a model of an energy state process. The quanta, q, do not relate directly to free electrons in an electrical circuit until they reach the conduction band, represented by the capacitor, cn.

Part (b) of the figure shows the extension of the earlier circuit to recognize the creation of this new current. Here, rtrans. represents an equivalent conductive path for the excitons as they become de-excited. The conductive path, rtherm represents any other loss path within the liquid crystal. Normally, this term is negligible when the Outer Segment of a cell is in contact with the microtubules of the cell. The subscript n is used to indicate an electrolytic element of the neural system. The values Cn and rn are only shown as place holders for a more complex circuit to be introduced later. The purpose of this figure is to show how each exciton of the P/D process generates a single, potentially free, electron in the conduction band of the first neuron of the visual system.

134http://physics.nist.gov/cuu/Units/candela.html 76 Processes in Biological Vision

Figure 7.2.5-1 The circuit diagram of the combined P/D and transduction process. The capacitance and resistance on the right are only shown for illustration, the processes remains entirely quantum mechanical in this figure.

The transfer of energy between a liquid crystal and a nerve can be likened to a physical impact. Similar situations are discussed in terms of phonons with properties similar to acoustic phenomena. How the energy is transferred between the crystal and the nerve is not important here. However, properly modeling the process is quite important. In the neuron portion of this figure, the conduction band is quite wide; therefore, an electron can have a wide range of energies and still transition from the valence band to the conduction band. The principal criterion is that it has an energy greater than En. The band gap, En, is approximately 2.0 electron volts in a photoreceptor neuron at mammalian body temperature. Therefore, it would be expected that an excited electron in the liquid crystal would require this minimum Dynamics of Vision 7- 77

energy to exceed the threshold of the neuron. In fact, two or more lower energy quanta in the liquid crystal can combine their energy in order to achieve the minimum energy threshold of the neuron--the same effect observed in silver halide photography. In this quantum mechanical situation, the energy of two quanta acts as one and the resulting transfer characteristic exhibits a “square law” characteristic as will be shown. In this model, the quantum generator, creates one quantum in the neuron for each n quanta in the liquid crystal having the necessary minimum energy, En. n = 1 in all photoreceptor channels except those sensitive to the long wavelength spectrum. For the channel sensitive to the long wavelength spectrum, n=2. n=3 and n=4 do occur in photography, but apparently not in vision. The value of n is directly relatable to the function, “gamma,” in both photography and vision. Gamma is the exponent in the transfer function of the spectral channel. 7.2.5.1 Details of the two-exciton process

The development of the details of this transduction process must allow for or account for the difference found between the process in the long wave chromophoric signal path and the shorter wavelength signal paths. As will be discussed in the next section, the long wavelength channel of animal vision exhibits a “square law” response that is the root of the difference between the photopic and scotopic spectral responses in vision. The “two-exciton ” process employed in vision is different from the “two-photon” process employed in the pumping of flourescent dyes135. The two-photon process in dye pumping involves the addition of two quanta of energy during the process of excitation. Conversely, in vision, the two-exciton process involves the summation of two quanta of energy during the de-excitation of excitons in the chromophoric material.

For dyes, there must be at least two energy levels within the energy state diagram of the dye that are approximately equally spaced in energy. The first photon excites an n-electron into the first excited state. The second photon excites this excited electron, in the same molecule, into a still higher energy state. The resultant single excited electron then de-excites to the ground level, releasing the total amount of energy associated with that single molecule. This amount of energy is frequently associated with an ultraviolet wavelength photon.

In vision, and photography, the first photon excites an individual n-electron into the π* excited state. The second photon excites an additional individual n-electron associated with a separate molecule into the π* excited state. However, these two molecules are associated in the same liquid crystalline structure. Their energies are associated with a common cloud of excited electrons. De-excitation in vision involves transfer of the combined energy of these two molecules simultaneously to a separate third material. There is no requirement for the individual molecules of the chromophoric material to exhibit a sufficiently high energy state to excite the third material. The subject of exciton clusters, both bi- excitons and higher order exciplexes, are discussed in Gutmann, Keyzer & Lyons136. The nonlinearity in the L-channel of vision due to the requirement to use the two-excition mechanism has been amply demonstrated in the spatial frequency down-conversion experiments of MacLeod, Williams and Makous137. Their experiments were limited to the L-channel due to the use of a 632.8 nm laser. Such a conversion requires a nonlinear process within the first stages of the photodetection process. A repeat of these experiments using the 441.6 nm laser of Metha & Lennie would show whether such a nonlinearity was present in the S-channel of vision. A repeat with a light source in the 532 nm region would provide similar data for the M-channel. If a nonlinearity of similar magnitude is found in these channels, this work will require revision. Burton has also provided background on the non-linearities in the initial stages of the visual system138. The dual-oxygen forms of the Rhodonines are known for their unique stability after excitation. Under biological temperature conditions, they do not de-excite through fluorescence or thermal means. This feature is believed to be due to the special properties of the triplet states of Oxygen. 7.2.6 Analysis of the excitation equation

135Williams, R. Piston, D. & Webb, W. (1994) Two-photon molecular excitation provides intrinsic 3- dimensional resolution for laser-based microscopy and microphotochemistry. FASEB Journal, no. 8, pg. 804 (also Laser Focus World, Dec. 1997, pg. 77-82) 136Gutmann, F. Keyzer, H. & Lyons, L. (1983) Organic Semiconductors: Part B Malabar, FL: R. E. Krieger Publishing Co. pp. 61-87 137MacLeod, D. Williams, D. & Makous, W. (1992) A visual nonlinearity fed by single cones. Vision Res. vol. 32, no. 2, pp 347-363 138Burton, G. (1973) Evidence for non-linear response processes in the human visual system . . . Vision Res. vol. 13, pp 1211-1225 78 Processes in Biological Vision

The excitation equation describes the combination of the P/D process with the process of transduction, of the quantum mechanical signal of the OS, into the electronic signal of the nervous system. The result describes the electrical currents from the dendrites of the photoreceptor cells in response to photon excitation, measured within the IPM, without relying on any hydraulic flow of molecules through various putative gates suggested by some analysts. The equation is entirely determinant. There is no need to invoke any statistical processes (other than those previously discussed relating to the transport of excitons within the excited state of the chromophores). This section will only discuss the short wavelength spectral region to avoid the complexity of the two-exciton process described above. Fortunately some of the best experimental data available was acquired using transverse illumination of individual Outer Segments. Transverse excitation of Outer Segments insures that only the intrinsic isotropic retinoid absorption spectrum will be active. In all but the long wavelength spectral channel of vision, the process of transduction from an exciton in the chromophore to a free electron in the conduction band of hydronium base material of the first Activa is a linear process. The efficiency of the process is very nearly equal to 100%. Because of this fact, this section will be primarily concerned with the P/D Equation. 7.2.6.1 Parametric analysis of the P/D equation

The P/D process involves six primary variables when discussed in terms of its impulse response, + the flux density, F, applied to the Outer Segment of the photoreceptor, + the absorption cross section, σ, of that Outer Segment, + the time constant, τ, associated with the time between excitation and de-excitation at a given temperature, + the temperature coefficient, KT, (or in the expanded form, k and the denominator of the temperature argument taken here as equal to 8), + the slowly changing secondary flux parameter, Fd, and + the temperature.

Each of these parametric values can be determined independently in the laboratory. Both the flux and the temperature are controllable variables. The other four are all experimental variables. Knowing the theoretical form of the P/D Equation, each of the variables can be evaluated by systematically varying the experimental conditions.

7.2.6.1.1 Effect of temperature and incident flux on the delay in the response

The temperature coefficient, KT, and the flux parameter, Fd, can be determined unambiguously by repeating a single experiment, on a single in-vivo photoreceptor, to measure the delay between the impulse and the beginning of the response waveform. Although such experiments have been performed in the past, the experiment needs to be performed again because of the inconsistent methods of defining the time delay in those experiments. For precise measurement, the delay must be measured to the start of the response waveform, not at the 10% amplitude point and not at the peak of the response.

The preferred graphical format is to plot the delay as a function of incident flux with the temperature as a parameter. Figure 7.2.6-1 shows a sample of the available data. The data was collected by many investigators over more than 50 years. The data was collected using a variety of animals, light sources, delay criteria and methods of temperature measurement. An alternate plot format would be to use an Arrhenius Plot with the flux as a parameter. Such a plot might aid in the precise determination of the temperature coefficient. Baylor, et. al. have provided such a plot for the turtle. However, considerable scatter exists in their data (+/-13%) and they did not recognize the parametric nature of the flux in their graph. The mechanism underlying this data is not a chemical reaction. The primary delay mechanism is related to the transit velocity of the excitons within the chromophores of vision. Attempts to calculate an activation energy from such a plot, based on an assumption of chemical kinetics, should be avoided. This is particularly so because the curves move as a function of flux level on this plot, causing what would be called the pre-exponential factor to be a variable with light level. See Appendix A. Dynamics of Vision 7- 79

Figure 7.2.6-1 Collage of delay data versus flux level and temperature. The dashed lines are drawn for a delay given by the incident flux raised to the one-sixth power. The data points deviate from these dashed lines because of the inconsistency in defining the flux used and in characterizing the delay interval. See Appendix A for details.

7.2.6.1.2 Determination of the effective absorption cross section of a photoreceptor

If the derivative of the complete P/D Equation is taken and evaluated at the time the response departs from the baseline, the slope of the function is found to be σ•F•KT. With F known and KT determined from the previous paragraph, the value of the absorption coefficient, σ, is easily determined. This is an important variable in vision and should be determined under a variety of conditions for a variety of animals to surface any performance variables related to the structure of their photoreceptors. The importance of specifying the orientation of the radiation used in the experiments must be stressed since different results can be expected as a function of wavelength for transverse versus axial illumination. Transverse illumination will generally give an absorption coefficient that is the same for all photoreceptors whatever their functional spectrum. The measurement should be made at the nominal peak of the isotropic absorption spectrum near 500 nm. Only axial illumination will provide an absorption cross section applicable to the functional absorption spectrum of the chromophores of vision. The axial measurements should be made at the nominal absorption peak of the functional chromophore under test, either at 342, 437, 532 or 625 nm.

7.2.6.1.3 Determination of the effective time constant

The determination of the time constant of the P/D Equation is more difficult than in the above case. However, a unique condition is available that simplifies the task. For the condition where σ•F•τ = 1.000, the complete P/D Equation simplifies to the special case of Equation 7.2.4-4 xxx. If the measured data is examined carefully, the flux level most closely matching the Hodgkin Equation can be determined. The slope of the response at the point where the function 80 Processes in Biological Vision

leaves the baseline defines the desired time constant. Under this condition, the slope of the response is equal to KT/τ and KT is known from above. For this special case, both the time constant and the absorption coefficient are determined since σ•F•τ = 1.000. In the P/D Equation, the primary and secondary processes, using the terminology of radiation chemistry, occur within very short time intervals (less than 10–12 seconds) relative to the overall time constant. The dominant factor in the time constant is the travel time of the excitons between their point of excitation and their point of de-excitation. 7.2.6.1.4 Effect of the experimental configuration on the time constant

The time constant discussed above represents the time constant of an intact and functioning photoreceptor cell as represented by [Figure 7.2.5-1]. For other situations, the measured time constant is likely to be much longer and be more properly represented by [Figure 7.2.4-1]. The difference is in the value of the effective resistance shunting the effective capacitance in this figure. This resistance can be considered formed by the value of four separate resistances in parallel. These resistances would represent the four different paths an exciton might take to de-excitation. They can be defined as, and there effective value given as below:

+ Rflour ~ infinity: A resistance related to the fluorescence of the chromophores when in the liquid crystalline state. This impedance would be expected to be very high for the Rhodonine chromophores due to their excitons being in the triplet energy state. A transition from this state to the singlet ground state is not allowed in the Rhodonines.

+ Rphos ~ infinity: A resistance related to the phosphorescence of the chromophores when in the liquid crystalline state. This term would be expected to be very high for the Rhodonine chromophores. A phosphorescent transition is not allowed by the molecular symmetry of the Rhodonines.

+ Rtrans ~ finite. This is the lowest value term and dominates in the intact and functional photoreceptor cell. It is related to the contact of the dendrites of the neural system to the disks of the OS. This circuit element is usually destroyed during extraction experiments in the laboratory, either by breaking the OS from the IS or by chemical attack on the chromophores and conversion from the liquid crystalline state.

+ Rtherm > Rtrans This term related to thermal decay of the excitons is the predominant term after the destruction of the Rtrans term in the laboratory. The resulting relaxation time constant of the chromophoric material is usually measured in hours.

The observation of phosphorescence in the laboratory is usually an indication that the chromophores have been chemically attacked and are no longer symmetrical with respect to oxygen at the molecular level. 7.2.6.2 Comparison with the literature

Although data related to the response of the basic transduction process could not be found in the literature, some very relevant data is available related to the electrical output from the outer segment as a “total” entity. The data is excellent for a variety of animals: Investigator Year Animal Probe Illumination Fuortes ‘59 Limulus Tomita ‘65 Carp Bortoff & Norton ‘67 Necturus Toyoda ‘67 Scallop Toyoda ‘69 Necturus & Gekko Penn & Hagins ‘69 Rat Baylor et. al ‘73-79 Turtle intracell flash Baylor et. al ‘79 Toad contact flash Torre et. al. ‘86 Salamander contact flash Breton et. al. ‘94 Human ERG flash These were primarily exploratory studies, with the exception of Baylor et. al. in 1974. They attempted to find an Dynamics of Vision 7- 81 appropriate equation without solving the photoexcitation/de-excitation problem. Lamb continued this curve fitting approach and presented his analysis in 1996139. Several investigators used simple electrical circuits to illustrate relationships but all of the models were found to diverge from the data--always in significant respects. No model adequately accounted for the fall in the trailing edge of the response or accounted for the initial time delay that is clearly a function of the applied radiation level (or impulse level, F•T). Baylor & Hogdkin140 made the greatest effort toward a theoretical solution. They made the assumption that the detection process was related to a photoconductor in the wall of the photoreceptor cell membrane. They examined many equations. However, the tone of the discussions was reminiscent of the days before the discovery of Planck’s Distribution Law of Radiation for the black-body. They could get the rise time right but the decay characteristic eluded them. At one point, their model had 16 degrees of freedom. They finally made the interesting comment that “Our tentative conclusion is that any model with n=6-7 [i. e. 6-7 degrees of freedom] can be fitted to the results provided that we allow sufficient dispersion of time constants.” This team also introduced the “suction pipette recording” technique that provides very well defined data without the frequent corruption by adjacent signals. The basic problem in the work since 1970, besides failure to analyze the photoexcitation/de-excitation problem, has been the concentration on a simple equation that does not support a discontinuity in the response upon the application of light and does not allow a falling response after a peak is attained. The focus has been on the so-called Michaelis (or Logistic) Equation. This simple algebraic equation was so named at the turn of the last century and does not relate well to the complex responses recorded for the OS.

Interestingly, the form of the equation for the photoexcitation/de-excitation problem is very similar in form to the Planck Distribution Law. 7.2.6.2.1 Detailed comparisons

In 1979, Baylor, Lamb and Yau141 performed a series of very sophisticated tests that are most informative and very disciplined. They operated entirely under infrared illumination except for the test light exciting the photoreceptor under test (and a subtle comment about during “initial pithing and enucleation”). They used a suction technique to draw a single photoreceptor outer segment (OS) into the entrance to a pipette while it was still attached to the IS and the rest of the retina. They were able to operate in the current mode and to place the OS/IS junction at the entrance to the pipette. Thus, they were able to measure the current passing from the OS to the IS under optical stimulation by pulsed light, long pulse light and with or without background light. They confirmed most of their earlier work but at a much greater degree of precision and control. Their figures 3a, 10, 12 & 13 are highly recommended and will be discussed in this work. Because these curves exhibit considerable saturation, which is related to processes following the actual transduction, the responses they obtained are complex. The details of the circuits leading to the response they obtained will be developed in PART C, Chapter 12. For convenience, Figure 7.2.1-1 reproduces [Figure 10.8.7-3] from PART B. It will be referred to in the following discussion.

Baylor, Lamb & Yau prepared a special micropipette designed to act as a Faraday Cage. This pipette was slipped over a single Outer Segment assembly. The assembly included both the disks and the associated dendrites (microtubules). The current emanating from Figure 7.2.6-2 Fundamental current paths in a photoreceptor cell. See text and Section 11.1.5 for details.

139Lamb, T. (1996) Gain and kinetics of activation in the G-protein cascade of phototransduction. Proc. Natl. Acad. Sci. USA 93, pp. 566-570 140Baylor, D. & Hodgkin, A. (1974) Changes in time scale and sensitivity in turtle photoreceptors. J. Physiol. vol. 242 pp. 729-758 141Baylor, D. Lamb, T. & Yau, K. (1979) The membrane current of single rod outer segments J. Physiol. vol 288, pp. 589-611 82 Processes in Biological Vision

the OS assembly was collected as a function of the position of the lip of the cage. The resultant current represented that from terminal (a) of the photoreceptor cell in the figure until the lip of the cage reached the OS/IS interface. The cage then began to collect the net charge from terminal (a) minus that from terminal (b). The net current they collected under high illumination was shown in their text-figure 6. The net current under variable illumination and before the current from terminal (b) became relevant was presented in their text-figure 3(a). Figure 7.2.6-3 compares the theoretical equation developed in this work (Section 7.2.4 and Appendix A) with two of the curves from the above text-figure 3(a). The solid curves are for a toad in darkness following a 20-msec flash. The curves represent only the response of the outer segment after filtering by a 6-pole filter with a cutoff at 20 Hz. The temperature was not specified. The dotted curves are from the P/D Equation using the parameters listed in Table 7.2.6-1. Dynamics of Vision 7- 83

TABLE 7.2.6-1 Measured by | Theoretical parameters from this work Baylor, et. al. | σ F τ σ•F•τ θ scale photons/μm2 | unit areas quanta/ msec. msec. factor unit area*time upper set 0.7 | 0.028 7600 8.8 1.873 0.3 20 lower set 0.35 | 0.03 3800 5.4 0.677 0.3 20 The delay term was set arbitrarily in the theoretical curves to agree with the measured data. Note the first order break with the quiescent level exhibited by the theoretical curves. The outward current is taken as positive.

The currents are those measured at terminal (a) of the circuit diagram. The illustrated low photon flux curves show negligible distortion due to the operation of the adaptation amplifier. The theoretical waveforms do not include any multistage or variable parameter filter as usually required by other proposed explanation for these waveforms.

Figure 3a of Baylor, et. al. is shown in Figure 7.2.6- 4(left) along with a smoothed set of selected curves from the same figure in 4(right). These curves represent the current at terminal (a) of the above figure under a wider range of conditions. The measured curves, up to 15 pA, in this figure track the theoretical curves of [Figure 7.2.6-1(d)] precisely.

Note: Smith, lacking a detailed model, attributed the current recorded by Baylor, et. al. at terminal (a) to the current at terminal (d) for pedagogical purposes. He Figure 7.2.6-3 A comparison of the theoretical and measure apparently did this under the assumption that the OS currents. See text for specific parameters. impedance (4) was linear and that no current flowed into the orthodromic circuit142. This was a bad assumption. It does not show the amplitude compression associated with the actual impedance of the output Activa circuit of the photoreceptor cell.

Figure 7.2.6-4(b) is annotated to explain the nature of the curves in (a). First, the horizontal line represents the hard saturation of the equivalent input Activa at roughly 22 pA. It represents the maximum current capability of the distributed Activa incorporated into the microtubules of the OS. Since no noise associated with the photon flux appears in this saturation current, the noise shown in the figure can be attributed to the test set. Since the random noise appears to be uniform, whatever the photon flux elsewhere in the figure, it appears the test set is the predominant noise source in the measured data. Second, the slopes of the rising waveforms and their point of departure from the horizontal axis (their delays) are both seen to be functions of the excitation flux. This is in accordance with the P/D equation. Finally, the falling waveforms are seen to all be exponential in character (outside the saturation area) and to exhibit a similar time constant. The actual time constants are a combination of the time constant of the P/D process, shown in TABLE 7.2.6-1 and the time constant of the input Activa in its role as the adaptation amplifier of vision, See Section 12.5.3.

142Smith, C. (1989) Elements of Molecular Neurobiology. NY: John Wiley 84 Processes in Biological Vision

Figure 7.2.6-4 CR The dynamic characteristic of the current collected from the OS. Left frame from Baylor, et. al. 1979. Right frame smoothed and annotated to show operation of the adaptation circuit of the photoreceptor cell. Note the hard limit at 22 pA due to saturation in the Activa. Note also the variable delay before the rising portion of the waveform leaves the horizontal axis and the variation in the slope of the rising waveform, both a function of the flux level. Outside of the saturation area, the falling waveforms exhibit a constant time constant.

The Baylor school, and subsequently Lamb working with others, has proposed a variety of equations to describe the P/D process. These equations have not described the P/D process completely without introducing putative (and generally variable) circuit elements that are not found in the visual system. Baylor, et. al. used simple, as opposed to complex, algebra. As a result, their solution does not include a term for the time delay. Furthermore if Baylor’s equation (4) is replaced by the P/D equation, a redrawn Text-fig. 11 in that paper will result which is a much better representation of the data. The situation near the bottom of this figure requires special attention. The irradiance is so high that the complete P/D equation may be required. Further, the duration of the light pulse is no longer considerably shorter than the response time. A finite pulse width model (as opposed to the impulse model) may be required in the forcing function. Appendix A provides further details in this area. More recently, Lamb has only modeled the rising waveforms of the P/D equation. These models are not based on the actual biological processes and circuits.

Although, Baylor did not characterize the time delay, td between the applied impulse and the beginning of the response, Lamb did143. Lamb shows the delay is clearly a function of the applied impulse (and probably the radiation level in the wide pulse case) although he continued to use an equation with a fixed delay. He merely noted the different values of td. In (A), the response clearly saturates at high signal levels. In (B), saturation is not as evident. Note the significant differences in time scale. Notice that the Michaelis equation has great difficulty tracking an exponential function in (A) and fails completely to account for the downturn in the waveforms of (B), which that author truncated to avoid emphasizing the difficulty. It should also be mentioned here that the data collected by Torre was based on a current from the rod; whereas, the work of Breton measured a voltage (with a return lead attached to the sclera/cornea). It should also be mentioned that the dynamic radiant level used by Breton may have avoided the saturation region, i. e. if a significant background level was present during or before the impulse, the impulse may have equaled a small fraction of the average light level. Lamb’s experiments were based on “Ganzfeld (full-field) stimulation (with very brief white flashes, . . . )” Using this technique would essentially avoid the saturation region of the adaptation amplifier completely. The data collected in the Baylor and the Lamb work is excellent, however the proper interpretation of that data clearly requires the P/D equation and the more complete model of the photoreceptor cell presented in this work.

143Lamb, T. (1996), Op. Cit. Dynamics of Vision 7- 85 7.2.7 Noise measurements of Baylor et al.

Baylor, Lamb & Yau144 have provided data on the noise performance of single sensory neurons in-vitro from the toad. However, they did not provide a schematic of the neurons they were investigating. Their values and assertions appear subject to alternate interpretations. A key factor suggesting this position is in their assertion, “The procedure was to first measure the mean number of quantal electrical events induced by dim 520 nm flashes. The light intensity was then increased and the transmitted flux was measured with the quantum photometer at 520 and 580 nm both before and 30-60 see after the end of a bright 60 see bleaching light.” No calibration relating to the dark adaptation occurring after the bright flash was provided and the bright flash itself was also not adequately specified. Their 520 nm stimulation was polarized and presented perpendicular to the axis of the outer segment. See Section 7.2.4 and Section 16.4.1. [xxx expand if time allows ] 7.3 Dynamics of the physiological optics of vision

The material to be reviewed in this section and sections 7.4 and 7.5 becomes ever more species specific as it proceeds. While the functions associated with the awareness mode of vision are shared among a variety of higher chordates, and particularly the higher primates, this does not hold for the analytical mode. The capabilities associated with the analytical mode of vision vary significantly, even among the higher primates145. This makes confirmation of the performance defined herein more difficult because it becomes difficult to employ surrogates. When looking at the phylogeny of the higher primates, an orderly increase in performance, peaking in the human, can be seen. Sections 1.2.1.5.3 through 1.2.1.5.5 attempt to outline these differences within Anthropoidea. Martinez-Conde & Macknik have recently surveyed the available data for a broader range of species146. Their paper, and conclusions, are important. They suggest, “ocular tremor is possibly ubiquitous to all vertebrates.” Conversely, they say, “Fixational eye movements, and microsaccades (minisaccades) in particular, appear to be most important in foveate vs afoveate species.” They support the position, “Animals lacking fixational eye movements, such as afoveate frogs and toads, may be incapable of seeing stationary objects.” They may mean tremor here, since frogs do exhibit some large fixational eye movements (which may involve movement of what might be called the eye socket). The lack of tremor leads to long term inability to see stationary scenes.

There are significant differences in the brains of the great apes and man, compared to the brains of the lesser apes and monkeys. The difference is primarily in the midbrain. The midbrain is also the least studied and most difficult to study. Chapter 15 will show that the thalamus, and particularly the perigeniculate nucleus and pulvinar of the midbrain, are far more developed in man than in any other species, family, super-family, etc. They are key elements in the ability of man to read and analyze fine spatial detail in object space. Only a few of the great apes, the Gorilla, Gorilla gorilla, and the Orangutan, Pong pygmaeus, can approach the human in these areas. When studying reading and the analysis of fine detail, the lesser apes and monkeys are not homologous with humans. Even the chimpanzee, Pan troglodytes, appears inadequate in these areas. Within the context of this work, it appears the hierarchy based on visual performance are Humans, chimpanzees, orangutans, gorillas and then monkeys (Section 1.2.1.5). Significant differences in performance can be defined between these species.

The primary visual cortex and the lateral geniculate nuclei play negligible roles in the higher level visual functions associated with the analytical mode, such as stereopsis and reading. Without understanding the purpose and operation of the ocular system, understanding the operation of the visual system as a whole is impossible. The ocular system is the only part of the visual system involving closed loop servomechanisms. It is also the only part of the visual system that employs external feedback. 7.3.1 Background 7.3.1.1 The literature of the physiological optics

144Baylor, D. Lamb, T. & Yau, K-w (1979) Responses of Retinal Rods to Single Photons J Physiol vol 288, pp 613-634 145Stone, J. (1983) Parallel Processing in the Visual System. NY: Plenum Press pp 190-192 146Martinez-Conde, S. & Macknik, S. (2008) Fixational eye movements across vertebrates: Comparative dynamics, physiology, and perception J Vision vol 8(14), paper 28, pp 1–16 http://journalofvision.org/8/14/28/ 86 Processes in Biological Vision

The dynamics of the ocular system and its control by the brain has been an underappreciated aspect of vision. While Davson devoted a volume of his work to this subject, it is quite old147. The modern era of work in this field began with Rashbass and Rashbass & Westheimer in 1961148. That material is written in simple language that can also be interpreted by someone versed in control theory. The serious reader should not overlook these papers. Their conclusions are still pertinent, although they can be stated more succinctly using the language of modern control theory and an adequate model. They provide many definitions of terms used in the field to this day. These, and other, definitions will be collected in Section 7.3.2. A compendium was published in 1978 that summarizes the state of the art up to that time149. Much of the modern work in the field bears the stamp of Professor Emeritus L. Stark. His brief 1977 paper begins with a heuristic view of the field150. The paper shows the very simple instrumentation used in that day. Many teams working in the various fields addressed in this section have arisen via the tutelage of Professor Stark. 1979 saw the appearance of a compendium by Records151. In 1983, another compendium appeared152. While covering the subject area extensively, it is remarkable that all of its Chapters remain conceptually based. Keller provided an extensive and in-depth review of the physiology and gross morphology of the brainstem as it relates to eye movements153. It very clearly defines the state of knowledge at that time regarding the level of detail knowledge available concerning each neural engine. Leigh & Zee provided an excellent compendium of eye movement data in 1999154. Hung has offered a short monograph155. It is designed to provide basic concepts understandable by both engineers and non-engineers. Recently, a new compendium of work in this area has appeared that summarizes much of the recent literature156. It will be discussed in Section 7.3.9. On the surface, this volume appears comprehensive. However, from a broader perspective, it is largely limited to the optometric aspects of physiological optics.

Reviewing the literature up to today shows the field remains in an exploratory phase. This is due to one major limiting problem, the lack of a viable model of the operation of the visual system. Such a model is particularly important to understand the fundamental signals used in the pointing (version), convergence (vergence) and focus (accommodation) functions. The aperture control system appears to operate largely independently of the other systems. However, the literature of the lens, aperture and vergence subsystems are so intertwined that they are frequently spoken of as a triad. 7.3.1.2 Overview of the dynamics of the physiological optics

The physiological optical system has generally been studied from two perspectives, pointing or version, and all other optimization procedures. These other subsystems include vergence, accommodation (focus) and aperture control. These have been outlined by Tyler157. Tyler discussed these subsystems individually and from a static perspective. In practice, it is necessary to consider these perspectives “en semble” and under dynamic conditions. This approach leads to the elimination of some of the first order concepts previously proposed that are based on less dynamic conditions.

These subsystems all rely on the same sensory channels and many neurological elements of the midbrain. As a group, these channels and elements are known as the Precision Optical System, POS. Portions of the POS have previously (and inappropriately) been labeled the auxiliary optical system, AOS. Except for the focus and aperture control subsystems these subsystems use the same oculomotor plant.

The empirical data show clearly that the operation of the pointing system and the triad are interrelated (Section 7.4.9). However, the conceptual models of how they are intertwined leave much to be desired (see Semmlow & Hung158).

147Davson, H. (1962) The Eye, Vol. 3 NY: Academic Press pp. 4-151 148Rashbass, C. (1961) The relationship between saccadic and smooth tracking eye movements (and two other titles) J Physiol vol. 159, pp 326-364 149Held, R. Leibowitz, H. & Teuber, H-L. eds (1978) Perception NY: Springer-Verlag 150Krishnan, V. & Stark, L. (1977) A heuristic model for the human vergence eye movement system IEEE Trans Biomed Eng vol. BME-24, no. 1, pp 44-49 151Records, R. (1979) Physiology of the Human Eye and Visual System NY: Harper & Row 152Schor, C. & Ciuffreda, K. eds (1983) Vergence Eye Movements: Basic and Clinical Aspects London: Butterworths 153Keller, E. (1991) The brainstem In Carpenter, R. ed. Vision and Visual Dysfunction. Boca Raton, Fl: CRC Press Chapter 9 154Leigh, R. & Zee, D. (1999) The Neurology of Eye Movements, 3rd ed. NY: Oxford University Press 155Hung, G. (2001) Models of Oculomotor Control. London: World Scientific 156Hung, G. & Ciuffreda, K. eds. (2002) Models of the Visual System. NY: Kluwer Academic/Plenum Press 157Tyler, C. (1975) Stereoscopic tilt and size aftereffects Perception vol. 4, pp 187-192 158Semmlow, J. & Hung, G. (1983) The near response: Theories of control In Schor, C. & Ciuffreda, K. eds Vergence Eye Movements: Basic and Clinical Aspects London: Butterworths Chap 6, pg 185 Dynamics of Vision 7- 87

Specifically, they do not develop the source of the signals input to the servo loops. The way they are interrelated can best be understood by considering the morphological and neurological elements of the Precision Optical System, POS, as a physical layer (using computer-speak). The accommodation, vergence and pointing servosystems can then be considered operating layers overlaid on this physical layer. The physical plant (using control theory speak) of the physiological optics is found in the physical layer. This plant supports all of the above subsystems. Hung notes this commonality of the physical layer with respect to the “plant.”159 However, the plant receives its instructions from the neurological portion of the physical layer. Conceptually, this portion is defined as the controller. Anatomically, it is known as the precision optical system, POS. This controller has been labeled the auxiliary optical system in the literature. This POS includes the perigeniculate nucleus, PGN, the pulvinar, and a major portion of the superior colliculus. The PGN is frequently labeled the pretectum in lower animals. The POS also includes a group of morphologically isolated neural complexes generally known as the oculomotor nuclei. Both the PGN and the pulvinar are elements of the thalamus. It will be shown that the controller is considerably more complex than shown in the previous literature. This is evident in the recordings of eye movements in humans. The physical layer exhibits two operating modes related to deriving signal information from targets in object space and two operating modes associated with the motions of the eyes. It also provides a blanking signal that the TRN uses to suppress signals to the other elements of the thalamus and the occipital lobe during large saccades. One other major factor should not to be overlooked when modeling the physiological optical systems. The accumulation of signals used to provide spatial coordinates of a target in object space varies with position within the retina. This accumulation, and low-level processing, of information by morphologically identifiable means has been given the name computational anatomy.

The physical layer incorporates an enhanced Type I servomechanism. Outside the foveola, it operates as a simple Type 0 servomechanism due to the limited resolution of the lens group at off-axis locations. It operates to bring the target to the line of fixation using only angle location data derived from the illuminated photoreceptors and computational anatomy within the retina. The precision of this tracking loop is better than 0.1 degrees (six arc minutes). Within the foveola, the physical layer operates as a Type I servomechanism. It operates in a velocity tracking mode in response to tremor signals generated within the POS. This velocity tracking offers a precision of better than 6 arcsecond. See Section 7.3.3.2.

The controller exhibits two distinct operating modes related to physical tracking, accepts initial conditions from other portions of the brain, and performs a considerable amount of computation. The latter rely upon Boolean algebra. This computational input to the servomechanisms of the physiological optics has not been considered previously in the literature. A hallmark of this computational capability is the fact that the overall physical layer, exhibits autonomous control over both saccadic and smooth motions.

The intertwining of the pointing, convergence and focus functions is well recognized but not well understood. There is considerable discussion regarding precedence regarding the triad. A variety of conceptual models have appeared that suggest significantly different operating modes within the triad. The exploratory nature of the previous work has led to considerable difficulty with the nomenclature. A particular problem has been the claim that the system is nonlinear without a clear definition of the term. This pervasive problem will be addressed in Section 7.3.2. The challenge of describing the operation of these subsystems based on the exploratory state of the field was highlighted recently by Patel, et. al. in their introductory remarks concerning the human visual system (HVS)160. “Various studies have shown that the HVS is nonlinear and adaptive (Sethi, 1986161). Therefore, analytical tools from linear time-invariant system theory could not be directly applied to study the dynamics of the HVS.” This work will show that the system is definitely nonlinear in the use of logarithmic and transcendental processing (the latter due primarily to mechanical geometries that introduce trigonometric factors). However, it will also be shown that the functions are not nonlinear in the differential equation sense. Neither are they adaptive in the conventional control theory sense. They can be considered adaptive in the sense that information from long-term learning (and stored in memory) is introduced into the servomechanism loops through computational support. The primary reason they have been called adaptive is the same one that has held up the photodetection community since the time of Fuortes & Hodgkin162.

159Hung, G. (2001) Models of Oculomotor Control. London: World Scientific, pg 112 160Patel, S. Ogmen, H. Whilte, J. & Jiang, B. (1997) Neural network model of short-term horizontal disparity vergence dynamics Vision Res vol. 37, no. 10, pp 1383-1399 161Sethi, B. (1986) Vergence adaptation: A review Docum Opthalmol vol. 63, pp 247-263 162Fuortes, M. & Hodgkin, A. (1964) Changes in time scale and sensitivity in the ommatidia of Limulus. J Physiol vol. 172, pp 239-263 88 Processes in Biological Vision

The protocol used by the vergence community and Fuortes & Hodgkins can be simply stated. If you cannot find the precise mathematical solution associated with a phenomenon, adopt a family of similar mathematical functions, or a serial expansion, and define the phenomenon as adaptive. It must be labeled adaptive because it switches between the various members of the mathematical family during its operation. The community studying the absorption spectra of chromophores has adopted a similar protocol. This variant says, if you cannot find the precise mathematical solution associated with a phenomenon, adopt a serial expansion of related form and then empirically determine the coefficients of the chosen expansion. If later work produces different responses, merely change the coefficients in the expansion. This work will show that the conclusion drawn by Sethi, even recognizing that few investigations based on control theory models had been performed, is inappropriate. With the correct functional and mathematical models, the system is linear, in the differential equation sense, and time-invariant in the control theory sense. Authors in the field continue to look to Donders (1864), Hering (1868) and Maddox (1893) for guidance regarding the operation of the pointing system and the triad. This is unfortunate. First, the frequent use of the cyclopean (equivalent unitary eye), based on the Theory of Binocular Eye Movement of Hering, in both pedagogical and research activities, has contributed to the simple use of linear relationships. This has masked the trigonometric relationships critical to understanding the operation of the pointing and convergence systems. Second, the reliance on the Theory of equal innervation of Hering is also self-limiting. It postulates that the two eyes move as though there was only one. Ono has provided an analysis of these first order hypotheses, which he cautions against describing as laws163. The need for a third set of eye muscles to compensate for the non-orthogonal arrangement of the first two pairs of muscles is a good example of an exception to the simple rule.

Owens and Leibowitz have addressed problems with the Maddox baseline164. They make particular note that “later research has also led to fundamental revisions of Maddox’s model of the vergence system. Most notably, the old unidirectional model of vergence control has been rejected in favor of an opponent-process model.” This marked the beginning of the introduction of control theory to vergence research. This modification also marked the recognition of the tonic nature of the signals controlling the eyes (even though they are encoded and transmitted in pulse code over the stage 3 projection neurons within the POS).

The reliance on angle-measuring to explain the functions of vergence and stereopsis165, as opposed to temporal measurements (based on ocular tremor), has left the field related to human vision in a stagnant condition for the last quarter of a century. Section 7.3.9.1.1 will analyze one analogy to vergence in vision that appears to fail the test of rationality. The overt dismissal of ocular tremor in the literature, as a significant factor in the operation of the auto-focus, vergence and analytical modes of vision has contributed greatly to this situation. This dismissal of tremor as a desirable phenomenon has even led to its description as a disease, physiological nystagmus166.

While the lower animals may rely upon angle-measurement to achieve vergence and possibly auto-focus, it is quite clear the higher chordates rely upon a higher performance method to achieve their high visual acuity. This method is found in the higher primates and probably in many of the predator birds. These animals achieve their high visual acuity in the foveal region by making angular velocity and position measurements based on ocular tremor. This mode achieves very high temporal precision within the vergence servomechanism and contributes to the capabilities of the analytical mode (such as reading). It is likely that the awareness mode of vision defined below relies upon angle-measurement in the periphery of the human visual system and on angular-velocity-measurement in the foveola. In this sense, vergence in the human visual system is a two-stage mechanism.

163Ono, H. (1983) The combination of version and vergence In Schor, C. & Ciuffreda, K. eds (1983) Vergence Eye Movements: Basic and Clinical Aspects London: Butterworths Chap 11 164Owens, D. & Leibowitz, H. (1983) Perceptual and motor consequences of tonic vergence In Schor, C. & Ciuffreda, K. eds (1983) Vergence Eye Movements: Basic and Clinical Aspects London: Butterworths Chap 3, pg 29 165Julesz, B. (1978) Global stereopsis: Cooperative phenomena in stereoscopic depth perception In Held, R. Leibowitz, H. & Teuber, H-L eds Perception NY: Springer-Verlag, Chap 7 166Moon, P. & Spencer, D. (1943) The specification of foveal adaptation J Opt Soc Am vol 33(8), pp 444-456 Dynamics of Vision 7- 89

This work moves far beyond a heuristic approach. Sufficient data is now available to provide a much more detailed model of the opto-neuro-mechanical operation of the visual system. As in many other parts of the vision literature, much of the reported data is not accompanied by an adequate report on, or control of, the associated variables affecting the data. Typically, Krishnan & Stark did not discuss control of the brightness or contrast of their scene generating display. It will be shown that both parameters directly affect the performance of the opto-neuro-mechanical system of vision. This limitation in the database forms a major limitation on how far the model of this work can be extended. 7.3.1.3 Subsystems of physiological optics

The complete schematic drawing of the visual system shows the ocular system includes five separate subsystems; + The lens control subsystem + The aperture control subsystem

+ The 1st shutter, the eyelid on humans + The 2nd shutter, the nictating lens in many animals + The pointing subsystem (both pointing and vergence)

Each of these subsystems plays a crucial role in the operation of the visual system. The importance of a given subsystem is directly related to the position of the animal on the phylogenic tree. Simple and compound eyes do not use many of these subsystems. They use the musculature of the body segments to point the entire head instead of just the eyes. The complex eyes of Mollusca are not designed to use a separate pointing system. However, the squid has evolved a pointing system adequate for its purposes. Many chordates use all these subsystems. The human is one of the few chordates that does not use the 2nd shutter, although it is claimed to be vestigial in man.

Besides the obvious importance of the lens and aperture control subsystem in high performance visual systems, the 1st shutter and the pointing system play required roles in the proper operation of the complex eyes of the chordates. The role of the 1st shutter in chordates is an important one. Besides preventing damage to the eye, in the absence of the 2nd shutter, it is nearly--although not totally-- opaque. It is used primarily as an adjunct to the data processing system of the brain. While the brain is commanding the pointing subsystem, it also commands the 1st shutter closed. This prevents the brain from formatting and transmitting data to the brain that is not useful. In this role, the 1st shutter is an integral part of the short term memory clearing function.

The pointing system performs more functions than are normally discussed in the literature. One of the most important is normally not discussed at all. The pointing system is obviously used to establish a line of sight as desired by the brain in response to its cognitive activity. This pointing usually involves an initial “large saccade” to establish a line of fixation to an object of interest. Once the coarse line of sight is established, the eye may make a series of smaller saccades to explore a complex scene by imaging individual scene elements onto the foveola. The eye is observed to pause intermittently for a short period usually described as a gaze. During a gaze, the system actually operates in a finer scanning mode, involving much smaller saccades, associated with recognition of the detailed characteristics of the portion of the scene included in the gaze projected onto the fovea. This last scanning operation is by far the most important in chordates. The obvious gaze is frequently subdivided into subgazes that may not be clinically observable. One function of the pointing system is to transform the chordate eye into an imaging system. The purpose is to enhance the capability of the eye to sense danger when the animal is stationary. This is accomplished by the rapid vibration of the instantaneous line of sight about the nominal line of sight. This function is performed by the unique muscles attached to the ocular globe and is described by the label ocular tremor (or just tremor in this work). Without this capability, the eye would only be sensitive to rapid changes of illumination level on individual photoreceptor cells. It could not perceive stationary threats. An understanding of the operation of the pointing and shutter system leads to an understanding of the spatial encoding by the retina and subsequent decoding by the cortex. This spatial coding provides spectacularly effective transfer of information and the excellent spatial discrimination capability to be performed using only the limited number of neural fibers found in the optical nerve bundle. This capability relies on two separate techniques; a short term memory storage capability, similar to the full frame memory storage used in modern digital television transmission, and an encoding system of the n-ary type. The specifics related to this subject will be found in Section 14.8. 7.3.1.4 Block diagrams of the physiological optical system 90 Processes in Biological Vision

Figure 7.3.1-1 repeats the top level block diagram of the visual system. It is used as a starting point for discussion throughout this work. The details associated with specific parts of this figure will be found in the relevant chapters of this work. For purposes of this chapter, a simpler figure can be extracted. It focuses on the physiological optics and the signal processing support for these subsystems, the Precision Optical System, POS. The POS was formerly known by the ambiguous name of the Auxiliary Optical System. This name illustrated a lack of understanding of its role.

Figure 7.3.1-1 Top level block diagram of the visual system of Chordata, particularly of man Dynamics of Vision 7- 91

Figure 7.3.1-2 is the same figure as used elsewhere in this work. It focuses on the servomechanisms of the physiological optics but is drawn to highlight the vertical motion of the eyes. The vertical performance of the two eyes is virtually identical except under pathological conditions. An alternate representation focused on the operation of the visual system in the horizontal plane will be presented below. It is more useful when discussing vergence and stereopsis. However, it is necessarily more elaborate.

Figure 7.3.1-2 Simplified top level schematic of Chordata focused on vertical oculomotor functions.

The association of the various terminal nuclei and the pairs of ocular muscles is unclear in the morphological literature. The arrangement shown is based on Pansky, et. al167. Borostyankoi-Baldauf & Herczeg have recently claimed the arrangement is different from that shown. They assert the LTN (or lateral terminal nucleus) is associated with the vertical motion of the eyes, which implies the LTN is connected to the superior and inferior rectus muscles168. They assert the DTN (or dorsal terminal nucleus) is associated with the horizontal motion via the medial and lateral rectus muscles. Separating the lateral terminal nucleus from the lateral rectus muscles appears questionable although the designations may relate to different frameworks. They did not discuss the role of the MTN.

167Pansky, B. allen, D. & Budd, G. (1988) Reveiw of Neuroscience, 2nd ed. NY: Macmillan, pp. 136-155 168Borostyankoi-Baldauf, Z. & Herczeg, L. (2002) Parcellation of the human pretectal complex: a chemoarchitectonic reappraisal Neurosci vol 110(3), pp 527-540 92 Processes in Biological Vision

Figure 7.3.1-3 illustrates the dual nature of the pointing system when viewed from the perspective of the horizontal plane. For small motions of the eyes, less than one degree, the system operates as a closed loop servomechanism incorporating the perigeniculate nucleus as its major functional unit. This unit extracts the servo loop error signals and uses them to drive the oculomotor neurons. An alternate source of signals driving the oculomotor neurons is the perturbation generator shown at lower right. Both sources are related to the analytical mode of operation. Two additional signal sources drive the oculomotor plant. Instructions from the LGN (as part of the alarm mode) and the higher cognitive centers (as part of the volition mode) drive the superior colliculus. The lookup table within the superior colliculus is responsible for converting these high level instructions into more elaborate command sets for driving the oculomotor neurons. The switching between these signal sources is under the supervisory control of the thalamic reticular nucleus (not shown).

Figure 7.3.1-3 The dual nature of the pointing system seen from above. See text for details related to the nomenclature.

Note the nearly complete independence of the left and right neurological and physiological plants. Theoretically, the eyes can operate independently in the horizontal plane (as they do for the salamander). However, the signal command generation structure for most chordates is less flexible. The eyes are coordinated. This coordination takes two principal forms. When the eyes rotate in the same direction, the motion is called versional. When the eyes rotate in opposite direction, the motion is called verginal. Failure of the eyes to operate in a coordinated manner related to these two motion types is a pathological condition (Chapter 18). The use of the cyclopean concept (a single equivalent eye) when addressing the exterior geometry of the visual process will not be addressed in this research oriented work. The concept is convenient in lower level pedagogical and clinical settings. However, it obscures the facts required to understand the complete operation of the visual system. By eliminating the cyclopean stimulus category, the terms local versus global can be used in non-stimulus related contexts without confusion. Howard & Rogers provide caricatures of the “cyclopean” visual geometry. They define the reference point as the midpoint of the arc between the two eyes (drawn with the Vieth-Muller circle passing through the centers Dynamics of Vision 7- 93 of rotation of the eyes169. While resting on some logic, this is an unconventional representation as shown in Section 7.4.1. More sophisticated analyses of this concept discuss whether the cyclopean eye can be defined relative to a fixed point. It appears that different experiments suggest the location of the putative eye may vary. The details related to the versional and verginal motions of the eyes will be addressed in separate sections below.

169Howard, I. & Rogers, B. (2002) Seeing in Depth, vol 2, Depth Perception Toronto, Canada: I Porteous, pp 8-11 & 67-100 94 Processes in Biological Vision

Figure 7.3.1-4 also appears in several chapters of this work (Section 17.1.4). It describes the signal processing used to create the various signaling channels of the visual system and begins to define the appropriate parts of the brain involved in processing the information delivered over those channels. Of particular interest in this Section is the operation of the direct channels associated with the individual photoreceptors of the foveola. Each of these photoreceptors connects to a direct and exclusive neural channel to the perigeniculate nucleus, PGN, of the thalamus located in the midbrain. It will be confirmed that the operation of the servomechanisms associated with the physiological optical subsystems does not depend on inputs from (or even the existence of ) the visual cortex. In fact, the routine operation of these servomechanisms does not require the existence of the cerebral hemispheres. In discussing eye movement tracking, Rashbass put it slightly differently170. “It is not clear whether the cortex is necessarily involved in these movements.” Pettigrew has recently come to a similar conclusion171. The cerebral hemispheres do play a role in the operation of these servomechanisms via the volition mode of visual system operation.

Figure 7.3.1-4 The luminance, chrominance and appearance channels of the eye of normal and aphakic humans. The UV channel is partially blocked in normal humans. However, the O-chrominance channel is functional. See Section 17.1.4 for details of the nomenclature.

170Rashbass, C. (1961) Op. Cit. pg 327 171Pettigrew, J. (2001) Searching for the switch: Neural bases for perceptual rivalry alternations Brain Mind vol. 2, pp 85-118 Dynamics of Vision 7- 95 7.3.2 Terminology 7.3.2.1 Classification of the rotational motion of the eyes

The term ballistic occurs frequently in the oculomotor literature. It usually occurs without definition in a non-technical context with technical overtones. Technically, ballistics is the study of projectiles and particularly the flight path of projectiles. The first order flight path of a short range projectile is characterized by its parabolic trajectory. Friction due to the air only enters into the problem in the second order. Mathematically, a ballistic curve (describing the position of an object) is a parabola formed by the action of a single constant force (gravity) operating continually on a single mass (the projectile) given an initial velocity by an impulse ( the firing mechanism). A motion describable by a parabola can be created in many ways. However, these motions are not ballistic in the mathematical or physical sense. In vision, the motion of the eyes is controlled by opposing pairs of muscles operating on a mass partially immersed in a liquid medium. The muscles can be represented by a source of tension and an elastic element. While the resulting motion (velocity) of the eyes can be described as parabolic in the casual use of the word, it is not truly ballistic. This is clearly demonstrated by the second derivative of the position of the eyes (the first derivative of the velocity of the eyes). It is not a constant with respect to time. Other authors have attempted to describe the motion of the eyes as one of constant velocity following an initial impulse and terminating in a nominally equal and opposite second impulse. Here again the derivative of the velocity is a variable as a function of time. Thus, the analogy to a bang-bang type of motion is not a precise one. Tole & Young have graphically compared the idealized bang-bang approach with the actual biological saccade172. Recognizing that the motion of each eye is complex, and generally the result of a pair of muscles working in a push-pull mode in each of two (nominally) orthogonal planes, is best. Because the two pairs of muscles are not physically orthogonal to each other, a third pair of muscles is used to compensate for this physical inadequacy in the architecture of the system.

Discerning a pattern in the literature to the description of the various movements made by the eyes is difficult. The terms used clinically show little correlation to those used in research. Alpern has provided a review through 1970173 and also a briefer summary of the research literature174. Dell’Osso & Daroff have provided a more recent text and a summary table of both eye movements and latencies prior to movement175. While recognizing the requirement for tremor to avoid blindness within a few seconds, they do not explore the functional significance of tremor. They provide a variety of conceptual block diagrams of the oculomotor systems. Unfortunately, they stress the role of the cerebrum (labeled the cortex-- as opposed to the midbrain in many of their figures) excessively. Simultaneously, they note the overwhelming evidence that the oculomotor, vergence, accommodation and light control mechanisms are substantially if not completely autonomous (page 328). The cerebrum plays no role in most of the autonomous oculomotor functions of the eyes. It only participates in the volition related aspects of some of these mechanisms. They also include several external neuron feedback paths (a so-called efference copy) that has not been found necessary in this work. Such a path would be the only external neuron feedback path known in the visual system.

Dell’Osso & Daroff do discuss the dual mode of the command generator of the oculomotor subsystem. They describe conjugate eye motions as controlled by the version command generator and disjugate movements associated with convergence as controlled by the vergence command generator (page 328). This work treats the command signals differently. It treats them as entirely formed within the superior colliculus with individual oculomotor neural commands containing both conjugate and disjugate components. The superior colliculus is relied upon to obtain the necessary vergence control signals from the LGN (coarse) and PGN (fine). These are the sources of autonomous (“reflexive” in the words of Dell’Oso & Daroff) steering commands while the volition (“Voluntary) steering commands are sent from area 7a of the cerebral cortex. The superior colliculus also requires reference information from the vestibular system. See Section 15.2.4, 15.2.5 & 18.8.5.2.2. The figures in those sections differ fundamentally from those of Dell’Osso & Daroff.

172Fisher, D. Monty, R. & Senders, J. (1980) ; Eye movements : cognition and visual perception. Hillside, N.J. : Lawrence Erlbaum Associates pg. 251 173Alpern, M. (1970) Muscular mechanisms. In The Eye, 2nd ed. vol. 3. Davson, H ed. NY: Academic Press, pp 1-252 174Alpern, M. (1973) Eye movements. In Handbook of Sensory Physiology, Vol VII, No. 4, Jameson, D. & Hurvich, L. ed. NY: Springer-Verlag, pp 304-326 175Dell’Osso, L. & Daroff, R. (1999) Nystagmus and saccadic intrusions and oscillations. In Glaser, J. ed. Neuro-ophthalmology, 3rd ed. Chapter 9, pg 340 96 Processes in Biological Vision

The start of the 21st Century has seen a resurgence in interest in ocular motions with Volume 154 of “Progress in Brain Research” devoted to such motions, their purpose and consequences. Papers in that volume by Martinez-Conde176, the editor of the volume, and by Engbert177 are of particular relevance. The nomenclature in these papers differs from the following because their community has not yet resolved the purpose of tremor. Because of that lack, they describe minor saccades in the two to 60 minutes of arc as microsaccades that are described as minisaccades below. Volume 8, number 14 of the The Journal of Vision was devoted to eye motions. The articles generally follow the above descriptions using microsaccades to refer to motion amplitudes approaching one degree. Most of the articles also ignore tremor as irrelevant to visual performance. Collewijn & Kowler follow this approach but provide a good review of the literature as a database178. They cite the recent work of Steinman in saccades in general179. They also cite the work of Spauschus et al180. in “microtremor,” the tremor of this work. They concluded, “Spectral peaks were observed at low (up to 25 Hz) and high (60-90 Hz) frequencies. A multivariate analysis based on partial coherence analysis was used to correct for head movement. After correction, the signals were found to be coherent between the eyes over both low- and high-frequency ranges, irrespective of task, convergence or fixation. It is concluded that the frequency content of ocular drift and microtremor reflects the patterning of low-level drives to the extra-ocular muscle motor units. Figure 7.3.2-1 is offered to present a common ground for discussion. The diagonal expresses the general relationship between the amplitude of a saccade and the frequency associated with it presented by Finlay in 1971181. Specifying the bandwidth of a saccade is difficult. Whatever technique that is used will generate a sloping line as shown but it may be moved horizontally. Two data points are shown from Robinson and three from Westheimer. The Westheimer values were from different times and different methods of calculation. The classification of saccades is based on their size compared with the diameters of the fovea and the foveola as observed in object space. Large saccades are generally larger than the diameter of the fovea and may be much larger. Small saccades are generally smaller than the diameter of the fovea but larger than the diameter of the foveola. Minisaccades, or flicks, are generally observable motions typically smaller than the diameter of the foveola. The finest saccades are not normally observable without special equipment. They are used to support the detailed analysis of fine detail such as letters in text. They are generally less than 100 arc seconds in amplitude and exhibit fundamental frequency components above the fusion frequency of the visual system. Their range is typically greater than 40 and less than 150 Hertz.

To help organize later discussions, saccades greater than the diameter of the foveola will be considered major saccades. This term combines the large and small Figure 7.3.2-1 A conceptual framework for discussing saccades which are readily observable clinically. saccade amplitudes and temporal frequencies. Solid circle Saccades smaller than this diameter will be grouped as data points are from Robinson, 1964. Crosses are from minor saccades. This term includes the minisaccades and Westheimer in 1954 and 1958. Open squares are equivalent the microsaccades which are difficult to observe frequencies obtained from data of Becker, 1991. Solid square is a tremor point computed in Section 7.3.7.

176Martinez-Conde, S. (2006) Fixational eye movements in normal and pathological vision Prog Brain Res vol 154, pp 151-175 177Engbert, R. (2006) Microsaccades: a microcosm for research on oculomotor control, attention, and visual perception Prog Brain Res vol 154, pp 177-192 178Collewijn, H. & Kowler, E. (2008) The significance of microsaccades for vision and oculomotor control J Vision vol 8(14) http://journalofvision.org/8/14/20/ 179Steinman, R. Haddad, G. Skavenski, A. & Wyman, D. (1973). Miniature eye movement. Science, vol 181, pp 810–819. 180Spauschus, A. Marsden, J. Halliday, D. M. Rosenberg, J. & Brown, P. (1999). The origin of ocular microtremor in man. Exp Brain Res vol 126, pp 556–562. 181Findlay, J. (1971) Frequency analysis of human involuntary eye movement. Kybernetik. vol. 8 pp 207-14. Dynamics of Vision 7- 97

clinically. Engbert & Merganthaler have recently published using the term microsaccades to represent the minisaccades range shown here182. For most major saccades, the velocity profile as a function of time closely approaches a parabola. Noting that the largest saccades are characterized by velocity profiles that are no longer parabolic is useful. This deviation would suggest a form of saturation or loss of linearity in the system (See Section 7.3.4.1.3). Making the assumption that the parabolic response can be considered half of a sine wave, an equivalent frequency for the oculomotor motion as a function of angle achieved can be calculated. These rough estimates are shown as open squares in the figure, based on Becker (1991). These values fall along a “main sequence” using the nomenclature of Rolfs, Kliegl & Engbert183 based on Zuber, Stark & Cook of 1965. The Zuber, Stark & Cook data only extended down to 2 minutes of arc in amplitude. The early data of Westheimer is called into question by the other data and the concept of a main sequence. Figure 7.3.2-3 presents a parallel framework expanded from Zuber, Stark & Cook184. Their data is shown in the upper right quadrant and exhibits a saturation described more fully in the data of Becker in Section 7.3.4. The straight line is drawn to represent a nominal “main sequence” for major and minisaccades (including flicks).. These motions are typically autonomic but are under voluntary control when the individuals attention is concentrated on them. The microsaccades are not under voluntary control. They generally occur as brief sequences of binocular motions where the vertical and horizontal components are generally in quadrature with respect to time (except when analyzing diagonal scene elements). These motions generally exhibit a fundamental frequency in the region of 30 Hertz. These brief motions are difficult to analyze using Fourier techniques since they only last for a few cycles when examining complex scene elements. The use of windowed, or Fast, Fourier Techniques generally associate harmonics with these waveforms that are a function of the width of the temporal window used in the analysis. The + mark at 50 degrees/sec and 18 arc seconds amplitude represents a microsaccades sequence at 30 Hertz as calculated in Section 7.3.7.

Figure 7.3.2-2 A conceptual framework for saccade angular 7.3.2.1.1 Classification of eye movement rates and amplitudes based on earlier empirical work. The syndromes or complexes upper right quadrant is from a 1965 report using laboratory equipment of limited sensitivity. The diagonal shows the Clinicians are generally not interested in individual nominal performance for a linear critically-damped impulse- saccades and are rarely instrumented to observe minor driven oculomotor system elements of the tonal type. The saccades. They use a series of global definitions related area to the left shows the potential area of oculomotor to saccadic responses that involve large parts of the performance associated with the twitch type oculomotor overall visual system. The following list is modified elements also operating as a linear critically-damped from Leigh & Zee185. impulse-driven oculomotor subsystem. See text.

182Engbert, R. & Mergenthaler, K. (2006) Microsaccades are triggered by low retinal image slip PNAS vol 103, pp 7192-7197 183Rolfs, M. Kliegl, R. & Engbert, R. (2008) Toward a model of microsaccade generation: the case of microsaccadic inhibition J Vision vol 8(11), pp 1-23, http://journalofvision.org/8/11/5/, doi:10.1167/8.11.5. 184Zuber, B. Stark, L. & Cook, G. (1965) Microsaccades and the velocity-amplitude relationship for saccadic eye movements Science vol 150, pp 1459-1460 185Leigh, R. & Zee, D. (1999) The Neurology of Eye Movements, 3rd ed. NY: Oxford University Press, pg 4 98 Processes in Biological Vision

Clinical definitions of eye movement syndromes

Class of Eye Movement Main Function Vestibular Holds images of the world steady on the retina during brief head rotations Visual fixation Holds the image of a stationary object on the fovea Optokinetic Holds images of the world steady on the retina during sustained head rotation Smooth pursuit Holds the image of a small moving target on the fovea; or holds the image of a small near target on the retina during linear self-motion. In the presence of optokinetic responses, aids gaze stabilization during sustained head rotation. Nystagmus quick phases Reset the eyes during prolonged rotation and direct gaze toward the oncoming visual scene. Saccades Bring images of objects of interest onto the foveola. Vergence Moves the eyes simultaneously in opposite directions so that images of a single object are placed and held on both fovea.

7.3.2.1.2 Classification of clinically observed eye movements–saccades

Several authors have categorized the motions of the eyes. The following table is modified from Leigh & Zee186. Another useful set of classifications is discussed in Newell187. However, the terms are not directly associated with the neural system. Classification of clinically observed Saccades

Classification Definition

Volition saccades Elective saccades made as part of purposeful behavior Predictive, anticipatory Saccades generated in anticipation of or in search of the appearance of a target at a particular location. Memory-guided Saccades generated to a location in which a target has been previously present. Antisaccades Saccades generated n the opposite direction to the sudden appearance of a target (after being instructed to do so) To command Saccades generated in response to a cue

Reflexive saccades Saccades generated to novel stimuli (visual, auditory or tactile) that unexpectedly occur within the environment. Express saccades Very short latency saccades that can be elicited when the novel stimulus s presented after the fixation stimulus has disappeared (gap stimulus) Spontaneous saccades Seemingly random saccades that occur when the subject is not required to perform any particular behavioral task. Quick phases Quick phases of nystagmus generated during vestibular or optokinetic stimulation or as automatic resetting movements in the presence of spontaneous drift of the eyes.

Note: many of the above terms require a prearranged external stimulus or participation by the subject in a training program as part of the evaluation.

186 Leigh, R. & Zee, D. (1999) The Neurology of Eye Movements, 3rd ed. NY: Oxford University Press, pg 91 187Newell, F. (1986) Ophthalmology, 6th ed. St. Louis, MO: C. V. Mosby, pg 105 Dynamics of Vision 7- 99

Tusa has provided a similar list to the above188. He uses “remembered target saccade” as a synonym for memory guided volition saccade. It could also be labeled a proximal saccade through its association with the term proximal vergence. 7.3.2.1.3 Classification of clinically unobserved eye movements–flicks and tremor

Many critically important fundamental eye movements are not normally observable clinically. These are the minor saccades consisting of flicks, minisaccades and microsaccades. Ogle has labeled these motions physiological nystagmus189. Flicks tend to be faster and larger than the minisaccades that are occasionally noted by a clinician. However, smaller minisaccades and microsaccades are generally not observable without specialized instrumentation. Distinguishing among the variety of uses of the above terms (particularly) in the clinical literature is important. Ashe, et. al. describe “microsaccadic flutter” as in the range of 0.1-0.5° in amplitude at a frequency of 15-30 Hz. They describe ocular micro tremor as a “near sinusoidal oscillation” about 0.008° in amplitude at a nominal frequency of 100 Hz190. The term “near sinusoidal oscillation” obscures the more complex nature of this phenomenon. They also compare their term microsaccadic flutter with the term microflutter used by Carlow and by Sharp & Fletcher (both in 1986) for this relatively gross amplitude phenomenon. Shakhnovich reviews the amplitude and frequency range of phenomena described as tremor, drift and saccades by various investigators191. Describing any motion exceeding a minute of arc using the prefix micro does not appear appropriate. 7.3.2.2 Classification of the temporal characteristics of the motion of the eyes

The temporal frequency characteristics of the eyes have not been carefully documented in a comprehensive manner. A few investigators have recognized and explored the high frequency motions of the eyes (frequencies greater than 20 Hz). However, most investigators have not achieved sufficient sensitivity in their test equipment to encounter such high frequencies. As a result, they have generally assumed the temporal frequency of the motions of the eye are limited to less than 20 Hz. This is unfortunate for they have failed to recognize one of the most important operational characteristics of the eyes. The eyes rely upon these higher frequencies to perform two critical procedures. The high frequency tremor is used to change their operating mode from simply that of a change detector to that of a quasi-imager. What is more important, the high frequency tremor is used to scan the fine detail in the image projected onto the foveola. This scanning is a critical part of the detailed analyses involved in recognizing shapes and characters, such as those found in writing.

7.3.2.3 Defining the operating modes within the physiological optics subsystem 7.3.2.3.1 Framework for discussions of pointing and the triad

Defining the operation of the two eyes begins with the realization that the two eyes are physically and neurologically independent regarding both their plant and their oculomotor nuclei. Functionally, they operate independently. It is only through the neurological signal processing and command generation functions within the POS that the eyes are coordinated. Pathological nystagmus is a clear example of the failure within the signaling system between the plant and the computational centers of the POS. The symptoms are frequently that of a high performance servomechanism in which the feedback path has been broken at a point of high impedance. The result is large angular excursions in response to poorly defined “noise” inputs.

When operating as a pair, rotation of the eyes in the same direction is described as conjunctive. Rotation of the eyes in opposite directions is described as disjunctive. First order eye motions are usually algebraic summations of both disjunctive and conjunctive motions. A large and specialized vocabulary has arisen to describe the spatial orientation of the eyes compared with the desired spatial condition. This vocabulary attempts to describe the location of a target in object space relative to the axes of the two eyes. A sub vocabulary has arisen in the clinic attempting to describe the location of a target with respect to one equivalent eye (the cyclopean eye). This vocabulary is needed to support the description of the mechanisms and phenomena encountered within the signal processing of the visual system.

188Tusa, R. (1988) Cortical control of eye movement Chapter 14 In Kennard, C. & Rose, F. Physiological Aspects of Clinical Neuro-Ophthalmology Chicago, Il: Year Book Medical Publishers pg 220 189Ogle, K. (1950) Researches in Binocular Vision. London: W. B. Saunders, pg 42 190Ashe, J. Hain, T. Zee, D. & Schatz, N. (1991) Microsaccadic flutter. Brain vol. 114, pp. 461-472 191Shakhnovich , A. (1977) Op. Cit. pg. 23-24 100 Processes in Biological Vision

The visual system employs two distinctly separate servomechanisms to control the eyes. The outer or coarse servomechanism is designed to rotate the line of fixation of the eyes to the location in object space of an object of interest. This system operates as two distinct servomechanisms. One controls the vertical orientation of the eyes. The second system controls the horizontal orientation of the eyes. The inner or fine servomechanism is designed to support the analysis of the object presented to the foveola after its image has arrived at the point of fixation. Both the horizontal and vertical orientation of the eyes are controlled simultaneously (and coherently) within this servomechanism. Because the two eyes are in a horizontal line compared with the typical environment, and with respect to the motion available to the eyes, their vertical spatial orientation is geometrically simpler than their horizontal orientation. In a first approximation, the two eyes rotate in the same direction (in conjugation) in the vertical plane. However, because of the separation of the two eyes relative to the vertical plane separating them, the motion of the eyes in the horizontal plane is much more complicated. While their gross motion may be considered conjugate, a significant disjunctive motion (the eyes moving in opposite direction) is required to cause both eyes to converge on an object at a specific distance from the head. The conjunctive motion of the eyes is generally described using the term version. The disjunctive motion is generally described using the term vergence. The goal of all eye movements is to bring objects of specific interest to the point of fixation of both eyes so their spatial characteristics can be analyzed (and their nature identified). To do this effectively requires the eyes to both focus at the desired target distance. It also requires that both eyes achieve the required version and vergence angles to allow fusion of the images provided by the two eyes within the visual system. A mechanism called stereopsis is associated with the midbrain. This mechanism leads to the phenomenon of fusion (the merging of the two images with respect to lateral spatial position or eccentricity). This mechanism also leads to the perception of depth in a perceived image. While the visual system is capable of depth perception for images not at the point of fixation, this capability decreases rapidly in quality with distance from the point of fixation. The quality decreases so rapidly outside the foveola (a 1.2 degree diameter area of the fovea surrounding the point of fixation) it is considered qualitative in the remainder of the retina. Within the region of the foveola, the depth perception is considered quantitative. Within this region, the precision of the depth perception is so good, it is frequently labeled verdicial (or linear).

Under unusual conditions, the images presented to the two eyes may not merge properly within the signal processing engines of the midbrain. Under this condition, the visual system perceives two conflicting images. The resulting situation is described as diplopia. The two images may appear to compete for attention in what is described as image rivalry. The temporal characteristics of this rivalry are poorly understood at present. It appears the rivalry is related to the motions of the eyes usually used to analyze a larger scene than can be imaged onto the foveola at once. In this situation, the POS causes the eyes to perform a saltatory scanning of the entire scene. The time scales of rivalry and saltatory scanning are similar.

When evaluating the pointing performance of the eyes, defining two different null conditions is common. The most common is to speak of the angle formed between the lines of fixation of the eyes when they are resting in the dark (the dark vergence or tonic vergence). The eyes tend to converge at a point of fixation at least one meter from the line between the principal points of the eyes (points slightly behind the cornea of the eyes and related to their optical performance). Unfortunately this distance exhibits a large variation among individuals. The other reference is with the eyes pointed to infinity (with their lines of fixation parallel).

When the two eyes are converged on a target in object space, the angle between the two lines of fixation is called the vergence angle. In the clinic, this angle is frequently described as the binocular disparity. When evaluating the performance of the servomechanisms associated with vergence, it is common to describe two different vergence angles. The first is the theoretical vergence angle associated with the target (the target vergence). The second is the actual vergence angle achieved by the two eyes (the eye vergence). The difference between these two angles is called the vergence disparity angle or the vergence disparity error. This angle is actually a measure distance. It is the difference in distance between the point of mutual fixation and the location of the actual target. Retinal disparity is another important term. It is used to describe the disparity of an object in the field of view of the foveola while the eyes are optimally aligned on another target. This is the angle used by the stereopsis mechanism to determine the relative lateral position and the relative depth of individual targets in the fused image of the scene. Another important term is proximal vergence. This is the vergence adopted by the eyes initially when they are commanded to acquire a target image. This value is normally stored in the superior colliculus (memory) and is based largely on experience. A rarely encountered term is cyclovergence. Since the oculomotor muscles are attached to the ocular globes in a slightly non-orthogonal arrangement, introduction of a small rotary correction is necessary for targets close to the eyes. This Dynamics of Vision 7- 101 component is known as cyclovergence. It is influenced by the oblique oculomotor muscles. When viewing object at infinity, the cyclovergence angle is considered zero. Many other terms related to vergence and disparity have appeared in the literature. Schor & Ciuffreda review many of these in Chapters 7 & 8 of their text192. Many experimentalists have used a simple projection screen for introducing vergence disparities into the visual system. They typically place the screen 1-3 meters from the subject. Frequently, rear projection is used to simplify the configuration. Two separate sources are used to project different stimuli to the eyes. The light from the two projected stimuli are isolated so only one image reaches each eye. The use of color selective filters or polarizers is most common. The use of color selective filters frequently leads to difficulties with the calibration of the light levels employed. The net illuminance must be calculated from the combination of the spectral responses of the photometer and the filter. More recently, gate controlled glasses (as used in stereo-cinema presentations) have been used. These can introduce undesirable temporal effects into the information to be processed within the visual system. An alternate test arrangement uses a mirror stereoscope. Figure 7.3.2-3 describes a mirror stereoscope frequently used in disparity vergence studies (compare with Ono variant xxx). As shown, with all fixed mirrors, the system can produce perceived targets at any apparent depth and direction relative to the centerline perpendicular to the plane of the eyes. Note that disparity is created without significant change in accommodation. The system can introduce images from beyond infinity from a vergence perspective, but not with respect to accommodation. The distance to the real scene is virtually constant regardless of the perceived distance. This system introduces no spectral filters or polarizers into the optical path.

Variants of the mirror stereoscope with movable mirrors have been used to study the dynamics of vergence and version under conditions of apparent target motion.

7.3.2.3.2 Defining precedence within the physiological optics subsystem

The literature in this area is descriptive of a common quandary during exploration. “Which came first, the chicken or the egg?” Authors have even debated over accommodation driven vergence versus vergence driven

Figure 7.3.2-3 Mirror stereoscope used in disparity vergence experiments. The perceived range and angle to the target is controlled by the lateral position of the two stimuli on the monitors. See text for discussion. A standard eye spacing is shown.

192Schor, C. & Ciuffreda, K. (1983) Vergence Eye Movements: Basic and Clinical Aspects London: Butterworths 102 Processes in Biological Vision accommodation193. The subject can be addressed more rationally if the description of the physical plant and the POS are considered the physical layer of the combined systems. Accommodation and vergence are only overlay functions performed using this physical layer. The discussion gains even more traction if the major role memory plays in the operation of these functions. They rely heavily upon the information acquired from the awareness channels of vision and the volition channels of vision. This information is used to set the initial conditions applicable to the servomechanism involved. In the volition mode instructions, the initial conditions are already present in the saliency map of the subject. When a human opens her eyes in the morning, the eyes are normally prefocused based on the information stored in the saliency map. It does not matter whether she is looking across the room or at the alarm clock. The saliency map provides the nominal focus and convergence information. Under this scenario, the servomechanisms operate open-loop initially, to carry out the commands containing the initial information. Subsequently, the closed loop performance of these servomechanisms is used to optimize performance. The initial commands contain the proximal vergence and proximal accommodation values provided by the superior colliculus. Hung has noted how the “proximal vergence,” and the similar proximal accommodation, have “frequently been omitted or only vaguely referenced in a qualitative manner such as an ‘injection’ input term.194” His position is that “Recent objective evidence demonstrates that in the absence of visual feedback of target blur and disparity, proximity could drive the accommodative and vergence system substantially.”

The above scenario introduces a different perspective into the discussion of precedence. Clearly, the initial step is to draw upon the salience map or the awareness signals for the initial information supporting pointing and the triad. This data normally brings the system very near nominal performance for any scene viewed. Whether one member of the triad provides fine update information to the others, and in what sequence is largely academic. 7.3.2.4 The Law of Equal Innervation

The control of bilaterally symmetrical functions from a central point suggests that the signals sent to each bilateral mechanism would be symmetrical (of equal size but either the same or opposite sign). Hering is credited with making this observation first (1868). As a first order proposition, it is defendable. However, the geometry of the muscle systems and other mechanisms of vision suggest symmetry is lost when the signals are examined in detail.

Fry has described Hering’s proposition in an expansive form195. As he notes in a footnote, the original proposal applied only to the disjunctive and conjunctive eye movements. However, it is frequently extended to describe the symmetrical operation of the accommodation and aperture control functions as well. 7.3.2.5 Defining the motor unit

When crossing disciplines, interpreting concepts in the literature concisely is frequently difficult. The so-called motor unit of the anatomist is one of these concepts. A careful distinction should be made between a motor unit and the components of the unit. These include both neural and muscular components.

7.3.2.5.1 A functional description of the oculomotor muscle

Many investigators have spoken conceptually of the muscles as sources of electrical signals. This has primarily been when discussing oculomyography. Occasionally, the muscles have been considered to include an electrical battery component discharged during contraction. Particularly when speaking of the twitch fibers of muscles, investigators have also spoken conceptually of the action potentials developed by the fibers. These discussions frequently include statements that the twitch fibers can reproduce the action potentials of the associated neurons. When describing the tension versus time characteristic of those fibers, recognizing that the electrical action potential of a muscle cell is separate and precedes the contractile response of the cell is important.

193Schor, C. & Ciuffreda, K. eds (1983) Vergence Eye Movements: Basic and Clinical Aspects London: Butterworths 194Hung, G. (2001) Op. Cit. pp 49-50 195Fry, G. (1983) Basic concepts underlying graphical analysis In Schor, C. & Ciuffreda, K. Vergence Eye Movements: Basic and Clinical Aspects London: Butterworths pg 404 Dynamics of Vision 7- 103

Ocular muscles are usually categorized as smooth muscles. Individual fibers are found able to exhibit sustained (tonic) and twitch (phasic) responses. Some appear able to produce intermediary responses between these two classes. 7.3.2.5.2 Defining the fatigue factor in the oculomotor response

Several authors have provided excellent data on the fatigue characteristics of the eye muscles. However, their focus has been on the long term fatigue effects (minutes to hours). Differentiating between long-term and short-term effects is important. Individual muscle fibers are not generally able to sustain their contraction over an extended interval. By assimilating a large number of fibers whose responses are not precisely correlated in time, a sustained, or tonic, response is obtained. However, the individual fiber has acted like a leaky integrator and has relaxed after a very short time. Its metabolic system returns it to operational condition in a very short interval and it is then prepared to respond to another action potential from its associated neuron. 7.3.2.6 Glossary

Accommodation–The process of adjusting the power of the physiological optical system to focus on a given element in object space. See also proximal accommodation

Agonist–A contracting muscle that is resisted or counteracted by another muscle, the antagonist. Antagonist–(See agonist)

Attention searchlight– A synonym for the angular beam in object space projected onto the foveola of each eye. Nominally 1.2 degrees in diameter centered on the point of fixation. A concept taken from the ancient Greek wherein light radiated from the eyes.

Binocular disparity– A less precise term than stereoptic disparity. Used widely in the clinic. Generally, the angle between the two lines of fixation when the eyes are fixated on a target. Equal to the target disparity under closed loop conditions. Associated almost totally with stereopsis and the limited field of view associated with the foveola.

Binocular view– the view obtained using both eyes. It is normally merged by the POS if the target is imaged onto the foveola.

Binocular visual direction– The direction of a target in object space relative to the intersection of the vertical and horizontal planes of the subject (see Figure 2.2.1-1). (Schor & Ciuffreda , ‘S & C’ pg 200) Sensory binocular processing– (S & C pg 200)

Command– A neural message executable by the PNS (including the oculomotor subsystem) and generally originating in the superior colliculus and associated structures. Usually using a bit-serial word format and transmitted over a single (or redundant) neuron. See Instruction.

Conjunctive motions– motions where the two eyes rotate in the same direction.

Corresponding points– See Cover points. Cover points– Points in the two retinas that would be overlaid if the two retinas were juxtaposed. Cover region– A region of the foveola in one eye that is within the coherence distance of the spatial correlator of the perigeniculate nucleus with regard to a point in the foveola of the other eye. Crossed Disparate– A descriptor for a scene element located within the Vieth-Muller circle. It has a larger target vergence than the point of fixation. Equivalent to the term convergent when discussing relative disparity. See also uncrossed disparate. Cyclofusion– The mechanism leading to fusion of quasi-parallel lines presented to the eyes dichoptically. Consists of both a physical component (a limited rotation of the eyes) and a neurological component. (220 & 330, S&C)

Cyclopean– Or cyclopian. Used variously according to Tyler & Scott, 1979. 1. (Julesz, 1971) The stereoscopic information first present at a binocular level in the cortex. 104 Processes in Biological Vision

(This work) The stereoscopic information first present within the thalamus of the midbrain. 2. (Hering, 1858) The position in the head from which binocular visual direction is perceived. Cyclovergence– The angular correction required in vergence due to the non-orthogonality of the vertical and lateral ocular muscles. (214, S&C) Dia-stereopsis– A term used in cyclopedean analyses in the clinic. Term is equivalent to diplopia in other environments. Dichotic stimulus– The presentation of the same stimulus to the corresponding points (areas) of the two retinas. Dichoptic– Condition where different stimuli are projected onto corresponding regions of each retina. The differences may relate to spatial, spectral or any other dimension of vision. Diopter– 1. A unit of ophthalmic lens power; one diopter focuses light from infinity at a distance of one2. meter. Basic unit of accommodation and vergence. The reciprocal of the distance from the eyes to the point of interest in meters. See also prism diopter Dioptic stimulus– a single object seen in essentially the same way by the two eyes.

Diplopia– 1. A failure to merge the images from the two eyes when the target is within the normal region of fusion. 2. Similar images falling on non-corresponding retinal points, and therefore projecting to different visual directions; non-fused images; “double” vision.

Disjunctive motions– motions where the two eyes rotate in opposite directions.

Double-duty linkage– An expression recognizing the effect of the common parameter of the correlator of the PGN, the local correlation range, on both the fusion and depth perception phenomenon of vision.

Electromyography (EMG)–A coarse investigative technique used primarily in the clinic, and of limited precision and therefore of questionable value in current research. The technique records the voltages encountered by inserting a probe into the ocular muscles. Similar to probing the S-plane of the retina in that a variety of signals result depending on what section of the muscle is probed. Reviewed from both the clinical and research perspective by Breinin, pgs 27, 36-52 & 134-135.

Essential tremor– A clinical term for postural tremor associated with the skeletal motor system and believed to be caused by a CNS abnormality. Not directly associated with vision or ocular tremor.

False targets– Extraneous images of elements of a scene in object space putatively generated within the signal processing mechanisms of vision and illustrated using a Keplerian projection. Also, described as ghost images in the literature. Largely a spurious concept when the vergence angle associated with the Keplerian projection is held to less than 12 degrees.

Fronto-parallel plane– A geometric construction based on the Gaussian assumption of paraxial optics. A plane drawn through the point of fixation in object space parallel to the line drawn between the nodal points of the eyes. Assumed to match a similar plane drawn through the point of fixation on the retina. The nodal points are not defined under wide field of view conditions. The principle points should be used. The fronto-parallel plane does not project a focused image onto the retina under wide field of view conditions. Fusion– The concept of merging the images acquired by the two eyes within the PGN of the midbrain. Haplopia. Fusional range– The angular range (average disparity in vergence between the scene and the eyes) in which a subject can maintain a fused image acquired using both eyes. Sensory fusion– xxx Ghost images– See false targets. Heterarchy– A term coined by Tyler & Kontsevich to represent the opposite of a hierarchy. An arrangement of computational elements that do not exhibit a hierarchal structure, such as a star network in computers. Hierarchy– A grouping of elements defined in terms of their importance, power, age, etc. Usually pyramidal in form Dynamics of Vision 7- 105

as in a human organization. See also heterarchy. Horopter– Used variously. 1. The locus of points that have zero binocular disparity is known as the horopter (the “horizon of vision”). A term attributed to Aguilonius, 1613. 2. Nonius horopter– named using the Latinized version of Nune, a Portuguese mathematician and instrument maker. First described by Wells in 1792 (pg 57). 3. A device for measuring the disparity in vergence, in multiple planes under specific conditions between the theoretical and actual vergence of the eyes. The most common are designed to measure horizontal disparity. (S & C pp 204-216) HVS– Human visual system Instruction– A neural message not executable as a command by the PNS. Used to direct the actions of the superior colliculus and thalamic reticular nucleus. Typically found in the alarm mode, volition mode and other channels within the CNS. Usually encoded as a bit-parallel word and transmitted over a group of parallel neural paths. See Command. Interp– A message in vectorial form created by the PGN/pulvinar couple (Pretectum) of the POS in response to the interpretation of a symbolic input imaged on the foveola during a single gaze. See also percept. Law of innervation– An archaic (first order) law useful in the absence of a complete understanding of the POS. It is only applicable to the low frequency characteristics of the oculomotor system. It is reviewed in detail in Breinin, where its limitations are described.

Motor unit– A motorneuron plus the muscle fibers that it innervates, is the basic functional unit of skeletomotor systems.

Ocular tremor– See tremor.

Panum’s area– A description of the area in X,Y coordinates in object space at the point of fixation associated with the limits of fusion in normal vision. More precisely, a projection of the maximum effective dimensions of the associative correlator of the perigeniculate nucleus.

Panum’s limit– Used variously in the literature. Generally, the edge of Panum’s area as defined at the associative correlator of the PGN.

Percept– An accumulation of interps, in vector form, that are presented to the higher cognitive centers. It may relate to a message conveyed by a written sentence or clause. Alternately, it may represent a recognized object. See also interp.

Paresis– Partially-paralyzed extraocular muscle.

Phoria– (clinical) A description of the state of deviation of the eye (inward, outward, upward, downward or cyclorotatory in nature) in the fusion free state (typically either with one eye occluded or with prismatic disassociation). A latent strabismus revealed only when the eyes are disassociated (when no fusible stimuli are in view). Esophoria– An inward lateral deviation of the eye in the fusion-free state. Exophoria– An outward lateral deviation of the eye in the fusion-free state. Orthophoria– Lack of deviation of the eye in the fusion-free state. Plant– In control theory, the spatially dynamic portion of a servomechanism as opposed to the control portion. Primitives-- A synonym for features in a scene. Usually used to focus on a specific (but frequently conceptual and open ended) list of features. Prism diopter– A unit of ophthalmic prism power, one prism diopter deviates light from infinity by one cm at 1 meter; 1.745 prism diopters equal 1 degree. Proximal accommodation– initial accommodation assumed based on knowledge of the distance to the target. Nominally the accommodation stored in the saliency map of the subjects environment and available as an initial condition. The existence of this effect has been questioned. (S & C pg 82) Proximal convergence– See proximal vergence 106 Processes in Biological Vision

Proximal vergence– “knowledge of nearness,” frequently described as prior knowledge of nearness. Presumably based on values stored in the saliency map. RDS– Random dot stereogram Recruitment– A coarse term describing the typical number of individual muscle fibers innervated by a given axon. Usually with a value of several hundred for neurons supporting the low frequency operation of the oculomotor muscles and about five to ten for the tremor related (twitch) muscle fibers. Retinal disparity– The geometric angular difference at the eyes between any object in the visual field and the point of fixation. Separable into horizontal (lateral) and vertical components. Reyem’s Loops– The complement to Meyer’s Loops between the LGN and area 17 of the cerebrum. Reyem’s loops are described by the variable axon distance between the various ganglia of the retina and the lamina cribosa. Action potentials travel relatively slowly over these unmyelinated sections of axons. Rivalry– The commonly observed situation (under laboratory conditions) where the visual system will continue to change from one perception of a dichoptic scene to another because of the difference between the two images provided. Saccadic latency– The interval between the change in a test stimulus and the initial movement of the eyes to realign the line of fixation to the new location.

Scene element disparity– Distance, in three dimensional coordinates, between a point in object space and the point of fixation within that object space. Sometimes separated into longitudinal and lateral components. The lateral component is sometimes separated into horizontal and vertical components. See retinal disparity.

Scotoma– A relative or absolute blind area of the visual field (in perceptual space).

Stereopsis– The mechanism within the POS that merges the two images from the foveola of the two eyes. The mechanism results in the phenomena of fusion and depth perception. These phenomena are degraded by vertical disparity. Tyler takes a narrow view and claims stereopsis is observed under and is independent of the conditions of both fusion and diplopia. (S & C pg 200) Ogle differed and defined the following Patent stereopsis– Stereopsis within a range of up to 10 minutes of disparity, roughly aligned with the range of fusion. Qualitative stereopsis– Between 10 and 15 minutes of disparity, where subject still perceives relative depth position but without veridical relationship.

Strabismus– An anomaly of binocular vision in which the visual axis (line of fixation) of one eye fails to intersect the object of interest.

Tremor– The arc-second to arc-minute level motions of the eyes of Chordata and some higher members of Mollusca designed to provide detailed analytical capability to the visual system. Also described as physiological tremor or oculomotor tremor. This tremor in not related to the term “essential tremor” as applied to the postural system.

Uncrossed Disparate– A descriptor for a scene element located outside the Vieth-Muller circle. It has a smaller target vergence than the point of fixation. Equivalent to the term divergent when discussing relative disparity. See also crossed disparate. Vergence– The disjunctive rotation of the eyes to obtain a fused image of an object within the stereoscopic field of view of vision. Target vergence– the angle between the lines joining the center of rotation of each eye with the target stimulus. Eye vergence– the angle between the fixation lines of the two eyes at a given time. Accommodative vergence– vergence angle assumed by the eyes in response to a well-illuminated target in object space. Performance degrades under reduced illumination. (S & C pg 32) Dark vergence– (see Tonic vergence). Morbid vergence– vergence angle following death or under heavy sedation Tonic vergence– vergence angle assumed with the eyes open but in the dark. The quiescent state. Disparity error – the difference between target vergence and eye vergence under operational conditions. Dynamics of Vision 7- 107

In closed-loop operation, the residual error between target and eye vergence. Accommodative vergence– (S & C pg 101 & 114) Proximal vergence– initial vergence assumed based on knowledge of the distance to the target. Nominally the vergence stored in the saliency map of the subjects environment and available as an initial condition. Veridical– Used infrequently to describe the condition where, if the distance of an object at zero disparity is perceived at its true distance, then the change in distance for a given disparity is also correctly perceived. (S&C 238) Version– The conjunctive rotation of the eyes, generally used to cause the line of fixation to pass through the location of an object in the field of view. 7.3.3 The physiology of the pointing subsystem

The pointing subsystem is a very important but little understood part of the visual system. It is a complex servomechanism employing three different sets of muscles per ocular globe and exhibiting several different modes of operation. These modes can be grouped by those serving to rotate the two ocular globes in the same direction and those relating to stereoscopic vision where the globes are caused to rotate in opposite directions. These modes are associated with the term’s version and vergence (along with stereoscopic vision). In this work, these modes will be considered functional overlays on the more fundamental pointing system and will be discussed separately in Section 7.4.

The three primary operating modes of the pointing servomechanism associated with each eye can be discussed from several different perspectives. Concerning their use, the following sequence is appropriate:

+ The static mode, no muscular activity; the eye is in the non-imaging mode (used in many lower chordates).

+ The tremor mode, the overall visual system is in the imaging mode (used routinely in most chordates).

+ The large saccadic mode, a transient mode under both voluntary and autonomous control, depending on the circumstances.

+ The small saccadic mode, a transient mode normally not under autonomous control.

A variety of hybrid operating modes can be described that use a mixture of these individual modes. They may use the individual modes in different sequences and sometimes in combination. They can also be used to different degrees.

Guyton196 provides an excellent discussion of the various motions of the human eye, the neural paths involved and the general characteristics of the individual motions. His figure 60-9, which also appears in Chapter 2 of this work, shows the muscles of the ocular globe and the nerves serving those muscles. His figure 60-10 is quite informative concerning the various neural pathways associated with vision. It illustrates the close proximity of the neurons from the retina terminating in the pretectal nuclei and the neurons emanating from the oculomotor nucleus. His figure 60-11 illustrates most of the motions of the eye as they relate to the motions of the retina in tracking a point source of light. 7.3.3.1 Basic operating scenarios of different species

The operating scenarios available to a given animal are highly dependent on its morphology, (including the shape of the ocular globes) and its ecological niche (which largely determines its vision requirements). The following paragraphs will highlight these differences. Besides the modes listed below, inducing oculomotor-based oscillations into the visual system is possible. These may be pathologically based or artificially induced. The latter have been useful in evaluating the performance parameters of the various closed loop servomechanisms. The former can frequently be traced to a specific break in the signaling path associated with that type of oscillation. 7.3.3.1.1 Static mode

Many animals do not stalk their prey but remain with their head and eyes in an essentially fixed position while they await the presence of food in their field of view. For these animals, the background within their field of view is a distraction. Therefore, they avoid it. The visual system of these animals normally operates in the static mode, although they may have complex eyes. The system may operate in a more dynamic mode on occasion if morphology permits.

196Guyton, A. (1976) Textbook of medical physiology. Phila. PA: W. B. Saunders pp. 818-825 108 Processes in Biological Vision

The typical insect operates in the static mode for purposes of defense. Any significant change in the illumination pattern associated with its field of view initiates flight. The typical stationary fish or frog, operates in the static mode for purposes of finding food. As the food moves through its field of view, its cognitive abilities determine the size, color and trajectory of the potential food. If these characteristics fit a specific template, the animal reacts in a distinctly predatory manner. 7.3.3.1.2 Tremor mode

An animal that needs to search for its food, but is limited in its locomotion capabilities, needs to be able to examine a scene without traversing over every inch of it. The tremor mode provides such a capability. It converts the eye from a change detector to an imaging device. The resulting background scene can be examined cognitively to determine the probable sources of food in the field. It can then react accordingly. This mode requires that the ocular globe be able at least to vibrate about the line of sight. In chordates, this is the principal reason for the morphology of the ocular globe and its mounting arrangement. Only a very simple muscle system is needed to accomplish this objective as the system need only vibrate at an amplitude approximating the angular field of one photoreceptor. It is worth noting the success of the squid in converting a body mounted complex eye to an eye with sufficient freedom to use tremor. The result is a mollusc that can continually image the world around it.

Tremor is extremely difficult to measure in the laboratory because of its small angular magnitude and the necessity of isolating this motion from other cardiovascular, pulmonary or intentional motions of the subject. 7.3.3.1.3 Modified tremor mode

The cat family is frequently noted for its unusual predatory performance and the atypical characteristics of its eyes compared with similarly sized animals. These observed characteristics are principally the result of the operational characteristics of the ocular pointing system. Felines can control the modes of operation of their ocular servomechanism better than most other chordates. This provides it several capabilities desirable in its environmental niche. The cat can sit still for significant periods while watching for the presence of prey in its field of view. By reverting to a static operating mode, the recognition of prey is greatly augmented. In this mode, it can calculate the parameters of any motion in its field of view. By matching these parameters with a variety of templates, it can categorize the object as food, a predatory animal, or the motion of leaves, grass etc. Clearly, its response is dependent on this classification. It will attach, or begin to stalk prey. It will attempt to elude a predatory animal and it will ignore wind or water related motions.

When the cat enters the browsing mode, its eyes are commanded to use the normal tremor mode. However, the cat adopts an intermediate mode when stalking. Under this condition, its computational capabilities associated with the static mode can still be used effectively. As a result, the animal seems to exhibit accentuated spatial capabilities to track its prey. This hybrid mode of operation has attracted the interest of many scientists. They have tried diligently to define a “hard wired” spatial filtering structure either within the signal processing of the retina or in the signal processing of the cortex. It appears this capability is actually a cognitive one in conjunction with the control of the servomechanisms of the pointing system.

7.3.3.2 Fixation motion 7.3.3.2.1 Continuous Tremor

[[ Guyton gives continuous tremor as 30-80 cycles per second, no amplitude ]] The data on continuous tremor is sparse in the literature and contains a large “noise” component since it originates with a variety of investigators. Yarbus197 provides some of the most explicit measurement data; the amplitude of the tremor is 20-40 seconds of arc (1.0-1.5 times the diameter of the photoreceptors in the fovea), the frequency is 70-90 Hertz (much higher than the critical flicker frequency of flicker fusion). For purposes of later calculation, a nominal tremor frequency of 80 Hertz ( 0.0125 period) will be assumed. The amplitude of the tremor is slightly different in the vertical and horizontal plane (one may recall that the muscles are not aligned with the vertical and horizontal planes), resulting

197Yarbus, A. (1967) ibid pp. 113-115 Dynamics of Vision 7- 109

in a the point of fixation traversing an ellipse in the object plane. Guyton also shows that the neurons controlling the muscles driving the ocular globe originate from different oculomotor nuclei. Yarbus also stresses the care required in making measurements of the continuous tremor, to the point of using an inertially stable platform to constrain the subjects head and measuring the extraneous head motion using a small mirror attached to an incisor tooth so that this motion can be subtracted from the measured tremor of the eye. Care must also be taken not to disturb the “tuned” mechanical response of the eye to muscular commands. The eye is essentially a critically damped mechanical structure; if an additional structure is attached to it, such as a contact lens with a mirror attached to it, care must be taken that the frequency or amplitude of the eyes motions are not changed by the test apparatus.

7.3.3.2.2 Slow Drift 7.3.3.2.3 “Flicking movements” 7.3.3.3 Saccadic Motion

See Yee et. Al. In Inv. Ophth. Vis Sci July 85. I have full journal, blue & grey good material See Engbert, R. & Mergenthaler, K. (2006) Microsaccades are triggered by low retinal image slip PNAS vol 103, pp 7192-7197

Liston & Krauzlis have recently provided an analysis of saccadic and pursuit motion that included statistical results from three subjects198. No physiological model, physical description or description of the servo system was provided. 7.3.3.3.1 Small saccadic mode

The small saccadic mode is used in two distinctly different situations. In many animals, the ocular globe is not sufficiently round to allow significant rotation of the globe. However, a small amount of rotation is available. This is the case in a great many chordates. As a result, the animal can rotate the globe a few degrees under either autonomous or voluntary control. This motion reduces the need to rotate the entire head.

In the more important small saccadic mode, the saccades are very small and very fast. They follow a poorly understood motion. However, it is clearly related to cognition since the motions appear to move the line of sight from corner to corner when viewing complex shapes. 7.3.3.3.2 Large saccadic mode

The large saccadic motion is limited to a few families of chordates. The higher primates and the chameleons exhibit this capability since the ocular globe is nearly spherical and large angular changes in line of sight can be achieved. Yee, et. al. have provided velocity information concerning the large vertical saccades199. The paper shows that the saccades represent the motion of the eye operating as an impulse driven inertial body. The peak velocity increases with the distance to be traversed in order to keep the time of traversal as constant as reasonable. The relationship appears to be based on simple inertial mechanics. 7.3.3.3.3 Pursuit Motion

Nystagmus [[Guyton pg 820 ]] [xxx Adler, Bala et. al. is good material200 02 in file] [xxx Rashbass201 1961 [xxx Pola & Wyatt only covers slow (smooth) eye pursuit motion ]

198Liston, D. & Krauzlis, R. (2005) Shared decision signal explains performance and timing of pursuit and saccadic eye movements J Vision vol 5, pp 678-689 199Yee, R. Schiller, V. Lim, V. Baloh, F. Baloh, R & Honrubia, V. (1985) Velocities of vertical saccades with different eye movement recording methods. Inv. Ophthal. Vis. Sci. vol. 26, no. 7, pp. 938-944 200Adler, S. Bala, J. & Krauzlis, R. (2002) Primacy of spatial information in guiding target selection for pursuit and saccades J Vision vol 2, pp 627-644 201Rashbass, C. (1961) The relationship between saccadic and smooth pursuit eye movement J Physiol vol 159, pp 338-362 110 Processes in Biological Vision

[xxx discuss both cues (in Adler, Bala et. al. and distractors] 7.3.3.3.4 Blanking of visual channels during large saccades

To avoid accepting extraneous information into the signal processing system during the positioning movements of the eyes (as opposed to the signal collection movements associated with tremor), it is necessary to suppress the signals generated by the retina during the duration of these movements. As noted earlier, these positioning movements are performed open loop. The xxx subsystem of the POS generates a variety of scanning waveforms that are sent to the oculomotor subsystem under the supervision of the TRN. During the positioning movements, the TRN directs a blanking signal, derived from the signal generator, to the relevant elements of the thalamus, primarily the PGN and LGN. In this mode of operation, the elements of the system prior to the PGN and LGN operate the same regardless of the motions of the eyes. The PGN and LGN use the blanking signal to suppress extraneous inputs during the positioning movements to the pulvinar (foveola related) and area 17 (periphery related). As a result, any signal manipulation engines orthodromic to these bodies are shielded from extraneous signal information. However, the blanking signal creates a different signal level during the blanking interval. This difference in signal level may be measured within later stages of the visual system. This difference in signal level caused by blanking signals has been measured by Sylvester, et. al. using MRI techniques202. Sylvester, et. al. make another important observation in their supplemental material. They describe the LGN as located within a cluster of hot spots associated with the thalamus. This observation suggests that they observed signals related to the PGN and pulvinar but did not interpret them.

The eyelids are normally closed briefly during a large oculomotor positioning. This closure may be in support of the external lubrication of the eye or it may be part of the blanking system. A question remains whether the commands for closing of the eyelids also generate a blanking signal or whether closing of the eyelids during an oculomotor positioning is initiated by the blanking signal.

It should be noted that the open loop suppression of imagery during positioning signals to the oculomotor system is not perfect. In the case of a strobe light on an aircraft flashing during the period of blanking, and a dark adapted eye, the sensing photoreceptor may generate a generator potential that lasts beyond the end of the blanking interval. In this case, the subject will observe a flash of light at the position in the far field calculated by the POS following the rotation of the eyes. The actual source will be at a location prior to, or along the path followed during the, repositioning.

The literature associated with blanking is described in Section 15.3.5.1.

7.3.3.4 Operating modes of the visual system associated with the physiological optics

Major operational differences are found between the servomechanisms associated with the analytical, alarm and volition modes of vision. The major differences are associated with three situations. The first is the significant reduction in off- axis spatial performance of the lens system (Section 2.4). The second factor involves the limitations of the oculomotor plant (Section 7.3.2). Because of these limitations, the correlator formed by the LGN has not evolved to the sophistication found in the correlator of the PGN. It appears the LGN correlator calculates local means for individual objects in the binocular visual space but does not calculate the deviations from the mean. As discussed elsewhere, no indication has been found of any capability of any component of the CNS to perform mathematical manipulations in the spatial frequency domain. Where appropriate, calculations related to periodicity of a stimulus may be made in the temporal domain via the convolution integral. Recognizing the critical role played by memory in the operation of the visual system is difficult. Investigators overwhelmingly consider the visual system as based on the eyes as imaging devices. They are clearly not imaging devices at the fundamental level. They are fundamentally change detectors. The continuous perception of the outside world is a result of the brain maintaining a saliency map of the complete environment exterior to the subject. This capability allows a subject to describe the wall behind him in considerable detail without turning his head (as long as

202Sylvester, R. Haynes, J-D. & Rees, G. (2005) Saccades differentially modulate human LGN and V1 responses in the presence and absence of visual stimulation Current Biology vol. 15, pp 37-41 Dynamics of Vision 7- 111

he was allowed to study the wall in detail previously). The maintenance of a complete saliency map greatly reduces the data processing load on the visual system. The system needs only process current changes to that saliency map. The role of memory is obscure primarily because the laboratory investigators have not yet been able to interpret the signaling code used within the multiple memory and higher cognitive centers of the brain. Thus, the pulvinar, the superior colliculus and cerebellum are some of the least understood areas of the CNS. They are understood more from the flow of signals they are associated with than for the function they actually perform. They are primarily large capacity memory elements. Each appears to contain multiple lookup tables. The element(s) of the CNS hosting the saliency map is yet to be isolated and identified. As noted below and in Section 15.2.2, the saliency map is contained within a general database, similar to that found in a modern computer. It does not contain an image of the outside world. The database is based on spatial position information. It does not contain spatial frequency information. When needed, such information is determined from the spatial-position-oriented database. Because of the above differences, the analytical mode of vision will be closely associated with the spatial domain of the foveola and the operational domain of the PGN/pulvinar couple. The awareness and alarm modes will be associated with the remainder of the retina and the operational domain of the LGN/occipital couple. See Section 15.6.5 for details of these operational configurations. 7.3.3.4.1 The coarse, type 0, (& autonomous) awareness mode

Because of limitation in the spatial resolution of the eye caused by the aberrations of the lens group, the servomechanisms associated with the peripheral (non foveola) retina cannot perceive the motion of the eyes related to tremor. Because of this limitation, the peripheral servomechanisms operate as type 0, or position servos only. The temporal signals within this servoloop do not contain phase information. The signals report the location of changes in brightness in object space on a frame by frame basis (the frame interval is about 33 milliseconds) but do not report the direction of that motion. The precision of the location information is better than 0.1 degrees but far from the few seconds of arc achieved by the analytical mode servomechanism. The direction of the motion is determined by a separate calculation within the CNS. The processing of awareness mode signals is largely autonomous and the subject is largely unaware of the processing. 7.3.3.4.2 The coarse, type 0, (& semi-autonomous) alarm mode

The alarm mode operates in the same manner, and under the same constraints, as the awareness mode, although it is assigned a different functional task. Its task is to detect changes in brightness related to vision as they occur in the awareness mode (covering the overall field of view). The information transmitted to the midbrain is limited primarily to location within object space defined by angles relative to the line of fixation. No information concerning range is provided. Because of the coarseness of the information, these signals are labeled instructions in the above figures. They are converted to more detailed pointing instructions by the superior colliculus. Similar changes are reported to the thalamic reticular nucleus by other sensory systems. The thalamic reticular nucleus filters these reports (and can discount their importance based on training and experience). When deemed important, the TRN instructs the POS (and the rest of the skeletal system when necessary) to reorient the eyes to bring the location of the change into alignment with the point of fixation. This action prepares the system to enter the more precise analytical mode of operation. The control exhibited by the TRN over the alarm mode signals demonstrates the more limited semi-autonomous nature of this channel. 7.3.3.4.3 The fine, type 1, (& autonomous) analytical mode

As new scenes are aligned with the point of fixation, the visual system employs the analytical mode of the visual system to ascertain the detailed properties of the objects in those scenes. The images projected on the foveola exhibit a spatial resolution at least one order of magnitude higher than for typical scenes imaged onto the periphery. This resolution allows the analytical mode to resolve small changes in the image introduced by the physiological tremor of the eyes interacting with contrast edges in the scene. These changes are transmitted to the midbrain synchronously with respect to the signals generating the tremor. By synchronously detecting these signals, the POS can detect both position and direction information with respect to these changes. In this sense, the POS operates as a type 1, or velocity, servomechanism. Using the two eyes in a synchronous mode provides additional information related to the distance to the objects in the scene. This information allows the POS to create a stereo-optical map of the scene through the mechanism labeled stereopsis. 7.3.3.4.4 The (sympathetic and instruction oriented) volition mode 112 Processes in Biological Vision

The higher cognitive centers can access the saliency map of the external environment and command the visual system to point to a previously examined object by issuing the appropriate instructions. These instructions are basically the three-dimensional coordinates of the object maintained in the saliency map. These minimal instructions are passed to the superior colliculus. The superior colliculus interprets these instructions and issues a more complete set of commands to the POS. Besides the necessary angular (version) information, these commands include the additional information required to cause the eyes to focus (accommodation) and to converge (vergence) at the prescribed distance. The volition mode operates essentially open loop. It can only be considered part of a closed loop in the sense that the original data for the saliency map was obtained through earlier operation of the awareness, alarm and analytical modes. This data may have been stored for a very long time before any volition instructions were issued. The TRN exhibits considerable control over the order of implementation of volition and alarm mode instructions and analytical mode commands. 7.3.3.4.5 Accommodation, a fixed reference controlled servomechanism

The accommodation subsystem appears to be fundamentally different from those associated with version and vergence. The performance of the subsystem appears to rely heavily on historical information stored in the memory associated with the superior colliculus. The accommodation values stored there appear to have been optimized over a long period of time and be tied quite closely to the associated vergence values. It appears that the receipt of a pointing command in the form of an alarm or volition instruction generates an accommodation signal obtained from the superior colliculus. Subsequent operation of the POS in the analytical mode may further optimize the accommodation function and provide updated values for storage in the superior colliculus.

Brodkey & Stark appear to have provided the best and most extensive data on the operation of the accommodation system203. However, their nomenclature may be dated. The system they describe in detail appears to be a linear servomechanism within its typical operating range. It does exhibit mild nonlinearity at the extremes of this range (their figure 21). Their discussion defines the limits of their investigation and their assumptions. The last line of their abstract requires modification when one recognizes the sampled but un-clocked nature of the projection system (stage 3) used in the neural system. While the system appears to be continuous at low stimulus frequencies, it is clearly sampled. The driven nature of the sampling mechanism makes it possible to operate at sample intervals of less than 0.01 seconds. This is well below the resolution of the Brodkey & Stark test set. The sampling becomes obvious at higher stimulus frequencies. Fatigue was found to be a significant factor in their protocol.

Brodkey & Stark review several models capable of explaining their data. In each case, they caveat that the model does not contain any memory element. On the other hand, they note the apparent adaptability of the system, a sure sign of a memory function (or some other computational capability). 7.3.3.4.6 The critical role of memory in POS operation

As suggested in the above paragraphs, memory plays a major role in vision. Short term memory is used to detect changes within a scene on a short term basis, typically 33 milliseconds to a second. Longer term memory is used to record the salient features of a scene on a permanent basis. The existence of such a permanent (though up-datable) saliency map, in collaboration with the long term memory of the superior colliculus, is obvious. It allows a subject to return to a dark room after intervals of more than twenty years and reach for the light switch with a precision of less than one inch. Within the precision optical system, POS, short term memory is used extensively to perform the 2-dimensional associative correlation functions required by the stereopsis mechanism. The long term memory capability associated with the pulvinar is then used to compare recent information with previously stored information. This allows the content of a scene to be recognized, labeled and stored in the saliency map. The details of this activity are developed in Section 15.6.6. 7.3.3.5 The pointing system in humans

203 Brodkey, J. & Stark, L. (1967) Accommodative convergence– an adaptive nonlinear control system IEEE Trans Systems Sci Cyber vol. SSC-3, no. 2 pp 121-133 Dynamics of Vision 7- 113 7.3.3.5.1 The general scenario

The pointing system of humans uses all of the above modes of operation except the static mode. Although the static mode is not normally used, it is easily induced pharmacologically. Its effect can also be canceled in the laboratory by several means. It may be shown in future experiments that the human can reduce the significance of the tremor when staring into the distance. The human visual system is very sensitive to small motions in the far field. This feature would be further accentuated (but only marginally) if the tremor were suppressed in favor of trajectory calculation as found in other animals. It is worth noting that the tremor is not known ever to cease in a conscious human. In fact, it is used by medical doctors as one of the vital signs of life. This is typified so frequently in the movies by the doctor lifting an eyelid of the subject to see if he is still alive. The features of the ocular globe are much more prominent when the globe is stationary. The typical operating scenario for the visual system is being impacted by modern technology, which places additional stress on the system. However, the fundamental biological scenario is similar to that of other animals. The system is normally recognized as in the tremor state until something in the far field moves or changes luminosity. The location of this change is n-ary encoded with respect to location within the field of view and transmitted to the brain. The brain interprets the alarm type of signal and commands a large saccade to establish a line of sight between the unknown event and the fovea. Upon establishment of the line of sight, the brain commands a series of small saccades to scan the object and it proceeds to use its cognitive powers to determine the nature of the object. Interspersed among these activities are a series of commands to the shutter to close during major saccades to avoid inappropriate data acquisition and resultant confusion. Many investigators have collected data on the large and small saccades. A much smaller group has studied the tremor.

This work will not explore the large and small saccades. They have been studied extensively and the characteristics of the servomechanisms associated with them have been detailed204. Fuchs, in Bach-y-Rita & Collins, has illustrated the difficulty with semantics in this area. He states that a saccade is the most rapid movement of which the oculomotor system is capable. This is only true for large angle motions. 7.3.3.5.2 Alternate models of the oculomotor portion of the POS

Several electronic analogs of the servomechanisms related to the oculomotor system have been prepared. However, most of these appear to be less flexible than the actual system. Those in the literature are conceptual in nature and do not appear to reflect the contributions of investigators trained in servomechanism theory. Most of these conceptual systems have been defined in terms of continuous, or analog, servomechanism theory205. Young & Stark206 presented a more sophisticated sampled data type analog. However, according to Fuchs, “miniature fixation movements” were considered as disturbances. The Bode Diagram presented in Stark’s later paper207 shows a maximum frequency of only four Hertz, clearly far below the tremor frequency (however, this is the correct value for the low frequency part of the overall oculomotor system). The actual oculomotor system is very sophisticated and incorporates both a sampled data servomechanism and a significant computational capability within its servo loops. A more versatile model of the oculomotor servomechanism, capable of performing in the different modes and to the accuracies found in the human system, is still needed. Such a model will be presented in Section 7.5.4. 7.3.3.5.3 Data related to physiological tremor

Reviewing the data regarding tremor is important. The most important and comprehensive data in this area has been collected by Yarbus208 and Shakhnovich209 in Russia and by Ditchburn210 in England. Additional data has been collected

204Bach-y-rita, P. & Collins, C. (1971) The control of eye movements. NY: Academic Press 205Leigh, R. & Zee, D. (1999) 3rd ed. Op. Cit. pgs. 175, 195 & 206. Also pgs. 150 & 237 in the 2nd ed. 206Young, L. & Stark, L. (1962) A sampled-data model for eye-tracking movements. Quart. Progr. Rep. Res. Lab. Electr. M.I.T. vol. 66, pp. 370-384 207Stark, L. (1971) The control system for versional eye movements. In Bach-y-rita & collins, op. Cit. Ppp. 363-393 208Yarbus, A. (1965-Rus., 1967-Eng.) Eye movements and vision. NY: Plenum Press 209Shakhnovich, A. (1977) The Brain and Regulation of Eye Movement. NY: Plenum Press 210Ditchburn, R. (1973) Eye-movements and visual perception Oxford: Clarendon Press 114 Processes in Biological Vision

by Riggs, et. al211., Krauskopf212, West213, Kelly214, Eizenman, et. al215. and Kalesnykas &Hallett216. Noting that the quality of the instrumentation improves rapidly after the 1950's is important. This was due primarily to the introduction of transistorized equipment. However, significant information is found in many of these papers. With each new generation of equipment, more specific parameters have been quantified about tremor. Zinchenko & Vergiles217 have developed a new equipment using an accelerometer in the eye cap in which they report components up to 200 Hertz as clearly visible. They also show an angular sensitivity to five seconds of arc. They did not provide more information on tremor as their main goal. That goal was to study the relationship between cognition and the small saccades related to image scanning. The use of a reiterative signal processing system, which is now available, to improve the signal-to- noise ratio and then find the spectrum of the angular displacement should lead to more definitive information. Findley provided a broad analysis of tremor and proposed a linear second order system driven by white noise to account for it218. The system had time constants of 0.02 and 0.002 seconds. Putnam et al219. have recently obtained statistical values for tremor using the Rochester adaptive optics ophthalmoscope of 2003. Their precision was much greater than any previous measurements. However, their technique did not readily define the maximum frequency of the highest frequency component of tremor incorporated in their measurements. They describe the standard deviations for the fixation point when using a site at an eccentricity of 1.25 degrees from the fixation point along the horizontal meridian as 1.92 arc min horizontal, 2.86 arc min vertical and 20.7 arc min rotational. They used a photoreceptor diameter of nominally 0.50 arc min (3.2 microns using a focal length of 22.28 mm). At the current time, the best estimates of the parameters of tremor are:

Size of high frequency tremors--20-40 arc seconds in object field, 1-2 photoreceptor diameters in fovea. The standard deviations of the total tremor are on the order of 2 arc min in the horizontal and 3 arc min in the vertical. Frequency spectrum of high frequency tremors--fundamental in the 30-60 Hertz range with harmonics up to 150 Hertz. The test techniques used to measure the frequency of the tremor would suggest the above amplitude is an RMS value.

Yarbus220 says, “the lowest or fundamental frequency is over 40 Hertz. Natural tremor is characterized by a frequency higher than the critical frequency of flicker. Low frequencies during fixation should be classified as drifts.” The premise of this work would suggest that the fundamental, or mean frequency is significantly higher, probably near 90 Hz, and the sidebands extend down to 40 Hz and up to 130 Hz, Ditchburn, et. al221. report on experiments where they introduced an artificial tremor. They found, for tremor amplitudes greater than 18 arc seconds, the fraction of time for which the subject saw the test object increased. This corresponds well with the diameter of a photoreceptor in the fovea. 7.3.3.5.4 Effect of stabilization

Two major discoveries have been made related to the investigation of the tremor associated with the ocular system. The first is the fact that:

The eye becomes blind in a matter of seconds in the absence of tremor, or other change in the object field relative to the line of sight. This change can involve anything that causes a change in the luminosity of a specific pixel in object space.

211Riggs, L. Ratliff, F, Cornsweet, J. & Cornsweet, T. (1953) The disappearance of steadily fixated visual test objects. J. Opt. Soc. Am. vol. 43, no. 6, pp. 495-501 212Krauskopf, J. (1963) Effect of retinal image stabilization on the appearance of hetero-chromatic targets. J. Opt. Soc. Am. vol. 53, no. 6, pp. 741-744 213West, D. (1967) Brightness discrimination with a stabilized retinal image. Vision Res. vol. 7, pp. 949-974 214Kelly, D. (1979) Motion and Vision I & II. J. Opt. Soc. Am. Vol. 69, no. 9, pp.1266-1279 and vol. 69, no. 10, pp. 1340-1349 215Eizenman, M. Hallett, . & Frecher, R. *1985) Power spectra for ocular drift and tremor Vison Res vol. 25, no. 11, pp 1635-1640 216Kalesnykas, R. & Hallett, P. (1996) Fixation conditions, the foveola and saccadic latency. Vision Res. vol. 36, no. 19, pp. 3195-3203 217Zinchenko, V. & Vergiles, N. (1972) Formation of visual images. NY: Consultants Bureau 218Findley, J. (1971) Frequency analysis of human involuntary eye movement. Kybernetic vol 8(6), pp 207-214 219Putnam, N. Hofer, H. Doble, N. Carroll, J. & Chen, L. (2004) The fixational stability of the human eye measured by imaging the cone mosaic J University of Rochester vol 2(2), pp 26-29 220Yarbus, A. op. cit. Pg. 127 221Ditchburn, R. Fender, xx. & Mayne, XX (1959) Dynamics of Vision 7- 115

The second is equally important and profound. The retina need not transmit information concerning a uniformly illuminated region to the brain. Transmitting the shape of the perimeter of that field and the color of that field along the perimeter is quite adequate. The brain will use a “paint” program to uniformly color the area in perception space determined by that perimeter. An accuracy parameter is associated with the term “uniformly.” Unless a sufficient non-uniformity exists to cause a signal in the luminance processing channel, the chromatic perception apparatus will not perceive a change in the color of a region. 7.3.3.5.4.1 Contrast performance of the eye in the static mode

Many of the above authors have reported experiments involving the stabilization of the image presented to the eye. The degree of stabilization has been varied occasionally to gain more information. However, the degree mostly involved the amplitude of the feedback while using an amplifier of more than sufficient bandwidth. Riggs et. al. carried out experiments, using a mirror on a contact lens and large dove prisms. This configuration allowed them to vary the degree of compensation in steps from 0% to 100% and then 200%. At 200%, they obviously overcompensated for the tremor. They were primarily interested in the time to recognize narrow lines. They noted that the lines disappeared with time in proportion to the width of the lines when full compensation for tremor was employed. Under the accentuated condition, the resolution of the eye was actually enhanced.

West found that a stabilized image faded. He also showed the pattern of fading if the total field was larger than the fovea. Generally, but not always, the fading started farthest from the fovea and proceeded inward. This feature suggests a different time constant for different zones of the eye. He also reported data on the effect of color on the fading and the scene contrast. His minus blue filter excited both the L- and M-channel photoreceptors.

Kulikowski pointed out that stabilizing an image to better than 25 to 40 arc seconds is quite difficult. Yarbus went to great lengths to stabilize his images, using a head rest and a bite bar mounted on a seismic block.

Yarbus (pg. 61) also demonstrated that the retina reported a null condition in the absence of motion. This null condition was neither black nor white. This was accomplished by repeating the stabilization experiments and then moving a black object through the field. The subject reported the object as black against a null field. In television terms, the visual system has no black level restoration capability, a common occurrence in cheaper black and white television sets. All color sets were found to require a restoration circuit or the customer would be exposed to bright blank fields between scenes.

The principal finding is that approximately three seconds after initiation of the stabilization system, the observer reports going functionally blind. Without a change in luminosity of an element of the scene, the retina reports a null condition to the subject, neither black nor white. Kelly said it well: “Our results suggest that retinal image motion is the sine qua non of vision.”222 Unfortunately, no curve was found that could be used to determine a more precise time constant for this effect.

7.3.3.5.4.2 Chrominance performance of the eye in the static mode

Several of the above authors have also explored the appearance of various colored shapes within a larger field. Yarbus has explored this area intensely using primitive techniques compared with those available now223. He has considered all of the permutations of a large field surrounding a smaller field in which each field is either stationary or moving relative to the retina. His general conclusion is that a perimeter, no matter how large or small, must be moving relative to the retina to be perceived. The color of the inside of the perimeter will be perceived correctly if the perimeter is moving. This perceived color will be an average of the color perceived along the inside edge of the perimeter itself. If the perimeter is not moving, the perceived color of the area within the perimeter will be that assigned to the area outside of the perimeter but inside a larger perimeter that is moving relative to the retina. It there is no perimeter within the field of view that is moving relative to the retina, the eye will be functionally blind and no color will be perceived

222Kelly, D. op. cit. pg. 1348 223Yarbus, A. (1967) Op. Cit. pp 59-102 116 Processes in Biological Vision

within its field of view. He defines such a field of view as an “empty field.”

7.3.3.5.4.3 Conclusions from experiments by Yarbus

To avoid introducing uncertainty in wording, the following paragraphs are taken verbatim from the English translation of Yarbus224. “We may draw the following conclusions from the results of the experiments described in Chapter II. For optimal working conditions of the human visual system, some degree of constant (interrupted or uninterrupted) movement of the retinal image is essential. If a test field (of any size, color, and luminance) becomes and remains strictly constant and stationary relative to the retina, it will become and remain an empty field within 1-3 sec. Very often, conditions of steady illumination arise on certain parts of the retina during perception. Such conditions arise during the perception of large and uniform surfaces and during small movements of the eyes. If the illumination continues to be constant for more than three seconds, an empty field appears inside this uniform surface (or surfaces). The empty field always takes the color of the surroundings and, in ordinary conditions, is never seen by the human subject. In other words, the visual system extrapolates the apparent color of the edges of the surface to its center. According to electrophysiological findings, I suggest that in man constancy and immobility of the retinal image will banish impulses entering the optic nerve from the eye or will sharply reduce their number. In these circumstances, absence of signals from a certain part of the retina gives the visual system information that this area corresponds to a uniform surface, which does not change and is equal to the color of its edges.

It may be concluded that the visual system ‘identifies’ the empty field arising in artificial conditions with the empty field arising in natural conditions. For this reason, the empty field arising in artificial conditions always appears to the subject as a uniform background (all visible contours disappear inside the field), and the apparent color of the field is always the color of its surroundings.

Two essentially different processes are found in the work of the visual system: the first, a fast process of disappearance of all contours in a stationary test field; the second, a slow process which usually is easily detected by means of afterimages. The fast process may evidently be associated with the appearance of impulses in the optic nerve in response to a change in the intensity of light (the on- and off- effects familiar from electrophysiology). The second, slow process is evidently associated with a change in the state of the retina--with its adaptation.

A definite delay is found in seeing the color of an empty field. This delay enables us, by extrapolating from the edges of a uniform surface, to see this surface unchanged in color when the image of the edges is continuously and saccadically displaced over the retina. The presence of this delay mechanism is essential for us to continue to perceive these edges.”

Similar summaries have been given by Ditchburn225.

Based on this work, it can be assumed that each photoreceptor is a change detector and there is a time constant associated with each photoreceptor cell. This time constant defines the minimum change required for a signal to be generated and passed to the brain. Although not defined precisely in the experiments of Yarbus and others, the mathematical value of the time constant is apparently less than one second at one extreme and less than three seconds in the other extreme. The source of these values will be explored in Chapter 12. Yarbus identified two essentially different processes related to the transient performance of the visual system. His fast process relates to the time constant of the adaptation amplifier discussed above. The slow process is a similar but longer time constant. This time constant is associated with the cessation of illumination. The significance of these two “processes” will be discussed further in Chapter 12. 7.3.3.5.5 Inertial aspects of pointing

224Yarbus, A. op. cit. pp. 100-101 225Ditchburn, R. (1973) Op. Cit. Dynamics of Vision 7- 117

To allow the high angular pointing capability of the human and primate eyes, the optic nerve must be quite long, typically 75 mm. in humans. This long protrusion from the spherical shape of the ocular globe introduces a considerable change in the moment of inertial of the eye and the centroid of that moment. It also requires a considerable volume within the eye socket be available to accommodate the motion of the optic nerve. An optimization has been achieved by sharing this space between the muscles controlling the eye and the optic nerve. When an eye muscle contracts, pulling on one of the ligaments of the globe, it also expands in cross section. This expansion simultaneously pushes the optic nerve in the appropriate direction. Because of this push, the optic nerve can be considered a lever attached to the ocular globe. By adjusting the ratio between the above pull and the push, the effect on the center and magnitude of the moment of inertia caused by the optic nerve can be negated.

7.3.4 Modeling the dynamics of the pointing system

This section and Section 7.4 will provide a broader foundation for understanding both the spatial and temporal dynamics of the oculomotor subsystem and the temporal response characteristics of the overall visual system. Many models relating to the mechanical dynamics of the eyes appear in the literature. Most of them are merely conceptual, suggesting the path to more sophisticated models yet to be defined. Many do not employ sufficient mathematical rigor and some rely upon a putative efference copy loop226. As an example, the equation shown under the upper frame of figure 4-5 of Leigh & Zee is not descriptive of the block diagram presented in that frame. Robinson has provided a carefully constructed model of the mechanical aspects of the POS and compared it with the earlier work of Westheimer227. The Robinson paper includes considerable data on the physical parameters of the human eye and includes a useful summary. However, some values are inconsistent and require careful evaluation. The most rigorous modeling of the dynamics of the eyes has been provided by Stark and his associates228,229. However, neither Robinson nor Stark, et. al. recognized or considered the high temporal frequency components of the eye’s motions above 50 Hz. Both Robinson and Stark, et. al. support the overdamped interpretation of the low frequency mechanical portion of the POS.

Porter, et. al. presented a major review of the structure and function of the extraocular muscles in 1995230. It cautions about the use of certain analyses in the literature dated before 1982 on page 458 and for a similar date on a different subject on page 465. The views of Porter, et. al. appear to be consistent with the recent comments in Leigh & Zee231. Both describe the distinctly different types of muscle fibers found in these muscles. Neither differentiates clearly between the parabolic characteristic of ocular motion and true ballistic motion that it resembles. Porter, et. al. also describe motorneurons of the oculomotor system as firing at up to 600 Hz on page 453 but do not give a citation. This work does not support the conclusion of Porter, et. al. concerning the role of efference copies and proprioception in the operation of the oculomotor system. Although they suggest substantial evidence for the existence of such informational assets exist, no demonstrated need for or evidence of such techniques could be found in the literature by this author. The models developed in this work did not call for these techniques. More than sufficient information was available from the operation of the POS to satisfy the requirements of vision.

Neither does this work support the suggestion by Porter, et. al. on page 465 that Jampel was wrong. Jampel proposed two separate and distinct modes of control system operation based on a segmentation of function within the muscles. They suggest that the oculomotor muscles are essentially uniform and mono-modal in their activity. The following discussion will show that each of these muscles acts as two distinctly different actuators. Although they can be modeled as operating in series or parallel, only additional histology can explain how this is done. Their discussion based on the electromyograph is of limited applicability based on the inability of that technique to achieve simultaneously the sensitivity and noise bandwidth required to detect the tremor signals present. Early in the study of oculometry, a general feeling arose that the ocular muscles were in some way unique in the body. During the middle of the 20th Century, this view was generally denied as unnecessary. Scott & Collins studied the

226Leigh, R. & Zee, D. (1991) The Neurology of eye movements, 2rd ed. NY: Oxford University Press. pg. 150 227Robinson, D. (1964) The mechanics of human saccadic eye movement. J. Physiol. vol. 174, pp. 245-264 228Cook, G. & Stark, L. (1968) The human eye-movement mechanism. Arch Ophthal. vol. 79, April, pp. 428- 436 229Clark, M. & Stark, L. (1975) Time optimal behavior of human saccadic eye movement. IEEE Trans. Auto. Control. pp. 345-348 230Porter, J. Baker, R. Ragusa, R. & Bruecknere, J. (1995) Extraocular muscles. Surv. Ophthal. vol. 39, no. 6, pp. 451-484 231Leigh, R. & Zee, D. (1999) Op. Cit. pg. 327 118 Processes in Biological Vision

subject in 1973 from a clinical perspective232. Their multi-probe EMG data does not include a vertical scale and appears to have employed AC coupling or very poor probe compensation. They did not provide a discussion of their instrumentation but drew two conclusions. One, there was little division of labor between oculomotor muscles. Second, most muscles showed electrical activity with a frequency proportional to eccentricity. However, beginning in the 1980's, the subject was re-evaluated when it was found that the ocular muscles exhibited a dual character233. They were found to contain individual muscle fibers that responded in two significantly different ways. One type exhibited a slow (tonic) response following groups of action potentials. Another group exhibited a very fast (twitch) response to individual action potentials.

The dual nature of the neurons innervating a muscle is well known234. The neuron serving the twitch function is described as an alpha neuron. It is heavily myelinated to achieve maximum signal propagation velocity and connects to extrafusal muscle fibers. The neuron serving the tonic function is labeled a gamma neuron. It is more lightly myelinated and propagates signals at a lower velocity to the intrafusal (spindle) fibers of the muscle. The twitch fibers have been described as reproducing the temporal action potential, of the neuron exciting them, in the spatial motion domain. Only a very small number, five to ten, of twitch fibers were associated with a single nerve fiber whereas hundreds of tonic fibers shared a common neural signal path. Through this organization, an individual neural signal is used to “recruit” a variable number of muscle fibers. The smaller number associated with the twitch fibers assures a higher degree of correlation among their times of response. This feature leads to a shorter response time for these fibers as a group. This shorter response time is supportive of a higher potential frequency response for the process associated with these fibers.

The tonic fibers did not exhibit a phasic response to individual action potentials. They were individually found to only respond to the integrated response of a group of pulses. The larger number of these fibers associated with a single neural path also resulted in a lower degree of temporal correlation between the responses of the fibers. The result was that the tonic fibers exhibited a low pass frequency response and as a group represented a signal integrator.

This work has defined the purpose of the tonic and twitch fibers of the oculomotor muscles and the associated control circuits within the POS. To provide all of the capability required, the POS operates two distinct servo loops within itself. The first is the conventional low frequency loop associated with controlling the line of fixation of the eye(s). The second is a higher frequency loop associated with the precision analyses of scene details while the line of fixation is held nominally constant during a gaze at the scene. Whether the POS operates these two loops in parallel, in series or in time sequence will be discussed below.

Breinin has provided a series of comments concerning laboratory technique that still applies to the current and future investigations235. 7.3.4.1 Developing the model

To understand completely the operation of the servomechanisms of vision requires the application of control system theory. This field generally addresses two major subsystems of a servomechanism, the controller and the plant. The controller (usually electrical) senses some aspect of the mechanism to be controlled and instructs the plant (usually a group of mechanical components) what to do to bring the sensed mechanism into compliance with a desired condition. Control systems are normally divided into distinct types. Type 0– A servomechanism used to sense a position and respond with negligible relative error in its final position. It may respond by pointing at the original position or at any other position, depending on purpose. Type 1– A servomechanism used to sense a velocity and respond by generating an equivalent velocity, which may be accompanied by a finite position error. Type 2– A servomechanism used to sense an acceleration and respond by generating an equivalent acceleration, which

232Scott, A. & Collins, C. (1973) Division of labor in human extraocular muscle. Arch. Ophthal. vol. 90, pp 319- 322 233Leigh, R. & Zee, D. (1999) Op. Cit. pg 327 234Noback, C. (1967) The Human Nervous System. NY: McGraw-Hill pp 93-98 235Breinin, G. (1962) The electrophysiology of extraocular muscle. Toronto, Canada: University of Toronto Press. pp 26-27 Dynamics of Vision 7- 119

may be accompanied by a finite velocity error. These types of servomechanisms can be combined to provide any degree of precision in position, velocity and acceleration tracking required. They can also accept initial conditions that essentially pre-position the output of the servomechanism. The interpretation and modeling of servomechanisms require the use of differential equations. Most of the servomechanism loops of vision are modified forms containing both Type 1 and Type 2 servomechanisms. Such systems are ideal for changing position as a result of a change in position of a stimulus. They are not ideal for tracking (pursuit of ) a moving stimulus. However, this requirement has been met through evolution. Computational and memory functions have been incorporated into the basic analog servomechanisms. As a result, the servomechanisms of vision can frequently be described as Type 1 analog servomechanism with computational enhancement. Many experiments to evaluate the pursuit capability of the system have been documented (Section 7.3.3.3). The visual system employs sampled data techniques in its sensing circuits and servomechanisms. Determining whether a “smooth pursuit” is truly smooth, or whether it is represented by a series of small saccades separated by short analysis intervals, is important. There is the distinct possibility that the series of small saccades involve negligible associated latencies but a finite analysis time between them. Such a series of small saccades might seem continuous within the bandwidth of the low frequency portion of the POS servo-system.

In a 1967 paper, Cook & Stark developed a conceptual model of the mechanical portion of the oculomotor system236. They assembled a group of partial equations describing their proposed system and assigned a set of boundary conditions to these equations. However, they did not obtain a complete and simultaneous solution to these equations. In their 1968 paper, they wrote a series of partial differential equations and asked a computer to determine the response of the plant portion of the system. They did not actually solve for the underlying equation. This methodology had one drawback. It allowed them to ask the computer to find a best-fit set of parameters matching their prescribed initial and final conditions, at least one coarse approximation (to the tension function), and a set of independently measured waveforms. It did not require that they solve the equations of the system in closed form. The best-fit computer generated results employed parameters that were considerably different from those originally specified by Cook & Stark. It was (is) difficult to rationalize the differences between the two sets of parameters without the underlying equations in closed form.

Clark & Stark presented a different model in a short paper in 1975237. Their strategy appears to be to incorporate two time constants in the control portion of the system and to treat the nonlinearity related to the tension source by using a parallel mechanical load. This load was nonlinear in the plant portion of the system. Their figure 1(b)[3] suggests that this strategy did not result in an improved model. Clark & Stark take issue with Cook & Stark over the “order” of the system. They claim to have constructed a sixth order nonlinear homeomorphic physiological model of some imprecisely defined part of the human visual system. This work will continue to rely upon the definition of order in differential equations as defined by the highest order derivative contained in the equations. By arranging the system equations of the mechanical portion of the POS in closed form, they are seen to conform to a second order system with a non constant forcing function. The nature of the forcing function is determined by the controller of the POS system and will be discussed separately in Section 15.2.4.3. 7.3.4.1.1 The Cook & Stark model as a point of departure

Solving the equations of a mechanical plant without considerable experience is difficult. As an example, Figure 7.3.4-1 shows the composite model developed by Cook & Stark. Although the value of Θ is associated with the arrow suggesting it equates to position, the model suffers a lack of clarity. The 1968 Cook & Stark paper discuss the meaning of the variables in this figure. Furthermore, their model does not address the rapid motion of the eyes related to tremor. It is in essence a low frequency model representing only part of a broader capability system.

236Cook, G. & Stark, L. (1967) Derivation of a model for the human eye-position mechanism. Bull. Math. Biophysics. vol. 29, pp. 153-174 237Clark, M. & Stark, L. (1975) Time optimal behavior of human saccadic eye movement. IEEE Trans. Auto. Control vol. 20, pp. 345-348 120 Processes in Biological Vision

Cook & Stark show their interpretation of the applicable data from Breinin and from Wilkie and from Hill in their figures 3a and 3b. The original Breinin data had data points that were remarkably consistent and was for a single muscle. Cook & Stark made the assumption that the data for the oblique muscles of a cat were symmetrical and used the data points for an inferior oblique muscle twice to create a push-pull configuration. The resulting figure was drawn freehand and was apparently intended to be symmetrical. It appears the solid curves were forced to zero at L0/2. It is also becoming clear that the pairs of oculomotor muscles are not truly symmetrical238. Figure 7.3.4-2 shows an alternate presentation of the data in Figures 3 and 6 of Cook & Stark. Different conclusions can be drawn from this figure. Additional scales have been provided to insure consistent notation. The upper half of the graph represents a lateral or superior rectus muscle (although the data was originally from a lateral oblique). This is Figure 7.3.4-1 An initial block diagram of the oculomotor taken initially as the agonistic muscle and is represented plant in one dimension. From Cook & Stark, 1967. by a positive tension on the eyeball. The muscle is shown under three conditions of innervation by the solid lines, 0%, 50% and 100%. The lower half is taken as a medial or inferior rectus muscle. It is initially considered the antagonistic muscle represented by a negative tension.

These muscles are not known to become slack ever. When acting in the antagonistic role, they rely on their elastic properties at zero innervation to maintain a minimum tension239. Cook & Stark noted the finite tension in each muscle at zero innervation and at a nominal zero angle for the line of fixation of the eye. The net tension on the eye at zero innervation is shown by the dashed line. It varies with the length of the muscles. This length is related to the angle of rotation of the line of fixation by the moment arm of the muscle.

Of greater interest is the net tension on the eye as a function of innervation and static position. The dash-dot lines show the net tension between the pair of muscles when one is at 100% innervation and the other is at 0%. Note the net tension goes to zero at ±67 degrees in this caricature. This defines the operational limit of this servomechanism in the absence of any other stops. Note also that the peak in the net tension is not at zero angle for the line of fixation. The system is actually optimized to provide a stronger initial pull for saccades passing Figure 7.3.4-2 A caricature of the static push-pull operation through the zero fixation point. However, it is not of the ocular muscles. This figure is based on Cook & Stark optimized for maximum angular swings. 1967, for the house cat. See text for details. Cook & Stark chose to approximate the maximum tension provided by either muscle as a constant value over its angular operating range, which they defined by the heavy bar at the bottom of the figure. Using the actual value to compute the net tension provides a different operating scenario. In this scenario, the initial net tension at the start of a saccade depends on the line of sight at the start of the rotation. This fact should be considered in any laboratory program investigating saccades. Their use of a constant to approximate the muscular tension probably accounts for the variance

238Cook, G. & Stark, L. (1968) The human eye-movement mechanism Arch Ophthal vol. 79, pp 428-436 239Porter, et. al. (1995) Op. Cit. pg. 454 Dynamics of Vision 7- 121

between their initial parameters and the computer optimized parameters they developed. By adjusting the initial conditions for the net tension characteristics to more realistic values, the net tension characteristics can be approximated by a cosine wave. This approximation simplifies obtaining the closed form solution to the equation for the response of the POS plant.

7.3.4.1.2 The expanded oculomotor plant model

Leigh & Zee have provided some gross models of the oculomotor system that include both eye and head motions240. Only a few detailed models of the oculomotor system have appeared in the literature. None of them have addressed the fine tremor (measured in seconds of arc) present in the normal eyes. By reviewing some of the approximations in Robinson in 1964 and Cook & Stark in 1967-68, a more complete model can be described. It is possible to include both the tonal and twitch servomechanisms in this model. The resulting model is much more complete. An expanded model is presented in Figure 7.3.4-3 that is intended to represent the general case of the entire mechanical portion (the plant) of the oculomotor servomechanical system. Θ =0 is indicative of the direction of the eye under total anesthesia. The defined system is linear and a considerable difference in operating frequency exists between portions of the system. Under these conditions, the system can be separated into two separate systems using the rules of superposition. By eliminating the gray areas (setting Θ3 = Θ3' and Θ2 = Θ2'), a low frequency system quite similar to that of Cook & Stark is obtained that describes the gross saccades (and drift) of vision. By eliminating the two segments beyond the gray segments (setting Θ3' = Θ2' = 0), a system is obtained describing the operation of the total system in the high frequency, fine-motion, regime related to the actual analysis of symbols by the foveola. To interpret the intermediate operating range associated with flicks and drifts associated with the scanning (but not analysis) of textual material may require solving the general model.

Figure 7.3.4-3 An expanded model of the oculomotor subsystem for a single plane of the orthogonal system. The model displays the “push pull” nature of the system. The antagonistic portion is on the left and the agonistic portion is on the right. Shown are the components of the eye ball in the center, the low frequency (tonic) components of the system on each end and the and the high frequency (tremor) components (shaded areas). See text for details.

In this figure, the symbology of Cook & Stark has been maintained but the symbol for some dashpots has been replaced by a sawtooth symbol as a matter of convenience to simplify the artwork. In addition, the symbology for the eye has been modified to more clearly represent the rotary motion involved. Rotation of the eye in the positive Q direction corresponds to rotation of the eye toward the temple. At this initial stage, some labels have been used duplicatively,

240Leigh, R. & Zee, D. (1999) The Neurology of eye movements, 3rd ed. NY: Oxford University Press. pgs. 175, 206, pgs. 150, 237 in 2nd ed. 122 Processes in Biological Vision

partly since the previous works have all assumed the agonistic and antagonistic muscles have been symmetrical. In more detailed discussions, these labels will be subdivided. Although this figure is complex, it can still be represented by a single 2nd order differential equation. The forcing function, due to the various tension sources (shown by the circled arrows) associated with that equation may change with operating mode. If the model is decomposed into high and low frequency models, simpler individual forcing functions may be applied to each simpler model. This figure describes each of the four muscle elements involved in the push-pull operation of the eye in one plane. Each muscle is shown as containing the theoretical elements of inertia, resistive loss, elasticity and a tension generator. Most authors have treated the inertial element of the muscles (shown dotted in the figure) as equal to zero for simplicity. Cook & Stark represented the low frequency, tonal, muscles as a tension source in series with an elastic element, with the combination in parallel with a second elastic element. This approach is followed here but an additional resistive element is also shown in parallel with the source leg. This element will become important later since it introduces a low pass filter into the system. It is believed that the nonlinearity introduced by Clark & Stark based on their interpretation of the data of Hill is better handled by the configuration shown. The pole in this low pass network is believed to be the source of the denominator in their expression for the total tension. All of the relevant elastic elements are shown in their real locations. The elasticity of the ligaments is shown for completeness. They will be considered to have a value of zero in subsequent discussions. This network does not include a net elastic term defined by Cook & Stark as KP. Under total anesthesia, at the so-called surgical plane of anesthesia, the net elastic term is the combination of all of the elastic terms shown with the contractile terms open circuited. The result is mathematically the same but the new model is more precise. More than sufficient data exists in the literature to describe each oculomotor muscle as consisting of two distinct portions, a slow or tonal portion and a fast or twitch portion. This data will be discussed in Section 7.3.2.4.1.

In this model, the tonic portions of each muscle are represented by a single symbol. The individual elements are not physically identifiable. The network formed by each group of T, KCONTRACTILE and B are symbolic of the summation of the tensions supplied by many individual muscle fibers. The result is a tonal response of limited high frequency capability. In the absence of a tension, T, the contractile element is disconnected from the circuit. The muscle is then represented by the parallel combination of B and KELASTIC. This representation is passive in nature and can be considered tonal.

The twitch portion of the muscle is less well characterized at this time. However, it is believed that the contractile elasticity has a nominal value of zero and the dashpot, B, is essentially an open circuit. As a result, the muscle does not exhibit a low pass filter characteristic similar to the tonal portion of the muscle. The muscle can respond to an individual action potential in a characteristic way.

Breinin has described the response of both the tonal and twitch portions of the oculomotor muscle in considerable detail241. However, it must be noted that his test set was inadequate by current standards. It was either AC coupled or he used a very poorly compensated probe. The distortion in the recorded action potentials is considerable. The twitch portion of the muscle exhibits a nominal delay following the stimulus of two msec, a time to peak response of 10 msec, and completion of the response within 40 msec. The first order response suggests that the inertia term was insignificant for the twitch portion of the muscle as tested. The description of the test was not explicit about whether the eye was still attached to the tendon of the muscle under test. Neither the tonic nor twitch data showed any signs of an underdamped second order system. The tonic waveforms did show some ripple due to the temporal spacing of the innervating action pulses.

The resulting network, with all inertial terms removed except the moment of inertia of the eyeball, is easily reduced to a second order equivalent network using Thevinin’s Theorem. Discussion appears in the literature as to the rigidity, and therefore the constancy of the moment of inertia, of the eyeball. The vitreous humor is a gel that generally rotates with the rotation of the eyeball. However, a spring constant and damping factor are associated with this rotation. Similarly, the lens is suspended from the outer shell of the eyeball by a spring and dashpot system242. In detailed analyses, these additional effects need to be included. The original Cook & Stark figure only addressed the tonic operation of the subsystem and only presented the unshaded areas. The shaded areas are introduced to account for the high frequency operation of the system related to the twitch fibers of the muscles. It is this high frequency operation that is responsible for the introduction of the critical tremor

241Breinin, G. (1962) Op. Cit. pp. 118-122 242Deubel, H. & Bridgeman, B. (1995) Fourth Purkinje image signals reveal eye-lens deviations and retinal image distortions during saccades. Vision Res. vol. 35, pp 529-538 & 2897-2902 Dynamics of Vision 7- 123

phenomenon into vision. When defining the electronic analog of a mechanical circuit, two major options are available for simple circuits. One option is to equate mechanical force to electrical current. The second is to equate mechanical force to electrical voltage. In either case, the relationship between time in the two circuits is arbitrary. This arbitrary property allows the design of an electrical analog that operates faster or slower than the mechanical equivalent. Cook & Stark used the force-current analog where the angular position of the eye is represented by the integral of the voltage at a node labeled Θ1. The alternate representation would have equated the angular position of the eye by the charge on a capacitor connected to the node, Θ1. Either approach gives correct results. The force-voltage analog is easier for most people to interpret. This analog provides a DC path between the element representing the position of the eye and the element representing the tension of the muscles. Thus, the overall circuit can be considered a type 0 servomechanism, e.g., one that can maintain a zero position error. The oculomotor circuit is sufficiently complex that it calls for some groups of elements to be transformed from what appears to be one mode to the other by using Thevinen’s Theorem.

The literature exhibits a continuing rivalry between those who consider the low frequency circuit to be underdamped and those who insist it is overdamped. Care should be taken to differentiate the state of damping of the low frequency (tonic) circuitry and the high frequency (phasic or tremor) circuitry. It appears that significant overdamping of the tonic circuitry would contribute greatly to the independent operation of the two distinct circuit profiles. This mode would considerably simplify the mathematics required to describe the circuit and its performance. This mode will be assumed in the following analysis. It will be shown that in this formulation, the two end circuits can be considered as short circuits at high frequencies as far as the twitch circuits are concerned. Alternately, the twitch circuits appear as short circuits to the tonic portion of the overall circuit operating at lower frequencies.

The polarities of the sources in the figure are chosen to conform to the common terminology in the literature when discussing only one eye. A contraction of the agonistic muscle generates an upward conventional current and a positive amount of charge on the capacitance CP representing the mass of the eyeball. This positive amount of charge represents the angle of the eyeball from the quiescent value of the eye under total anesthesia. Following Cook, this positive charge is associated with a deflection of the line of fixation in the temporal direction in object space. This deflection is due to a contraction of the lateral rectus muscle243. Maintaining only one convention in this area is important because, the temporal and nasal oculomotor responses are asymmetrical244. When the orientation of both eyes is being considered simultaneously, the convention becomes more complex. The convention adopted by Breinin should be considered in those situations245.

Cook & Stark noted the earlier work of Hill and of Katz in defining the resistive term associated with the tonic portion of the oculomotor muscle response, represented by GAnt and GAg in the figure. Working in the late 1930's, these investigators defined a “nonlinear” characteristic for these impedances. Their representation was entirely empirical and did not employ any complex notation. They showed that the “damping loss” or resistive impedance of the muscle varied in proportion to the tension of the muscle. Hill showed that the damping loss of the muscle during contraction was also inversely proportional to a sum that included a term proportional to the velocity of the shortening.

Figure 7.3.4-4 extends the previous figure by replacing the simple tension generators by piezoelectric devices more representative of actual muscles and connecting them to their driving neurons to form four motor units. The solid black rectangles represent the electrostenolytic power sources associated with the cytological aspects of each muscle fiber. The mass of the muscles has been eliminated since they are not believed to be significant to the performance of the system. No changes are required in the previous physical model. The neural paths back to the oculomotor controller are discussed in Section 11.6.3.

In this figure, each muscle fiber is treated as a linear piezoelectric transducer. The cytoplasm of the muscle cell normally exhibits a potential of -70 mV relative to its surroundings. This potential is created by the electrostenolytic process on the surface of the cytolemma. When the fiber is excited by its associated neuron, current is injected into the cell at the synapse, the potential of the cytoplasm is reduced and the cell contracts. The contraction along its long axis generates the tension exhibited by the fiber. The contraction is assumed to be a linear function of the potential of the cytoplasm over the operating range of the muscle fiber. The response of either the tonic or twitch portion of each muscle is then the result of the summation in time and space of the contraction of the appropriate ensemble of individual piezoelectric transducers. To insure appropriate contraction with respect to time, the number of fibers recruited by a single neuron is much smaller in twitch portions than in tonal portions of muscle.

243Cook, G. (1967) Derivation of a model for the human eye-positioning mechanism. Bull. Math. Biophysics. vol. 29, pp. 153-174 244Cook, G. & Stark, L. (1968) The human eye-movement mechanism Arch Ophthal vol. 79, pp 428-436 245Breinin, G. (1962) The electrophysiology of extraocular muscle. Toronto, Canada: University of Toronto Press. pg 53 124 Processes in Biological Vision

Under this interpretation of a muscle, the attack time constant of the muscle is determined by the combination of the transducer, the elasticity and the friction associated with the muscle ensemble. The relaxation or deactivation time constant is determined primarily by the ability of the electrostenolytic power supply of the muscle to reestablish the rest potential of the cells’ cytoplasm.

Figure 7.3.4-4 The oculomotor servo plant with driving neurons. All inertial elements have been removed except for the moment of inertia of the eye. Each tension generator has been replaced by a piezoelectric device representing individual portions of the muscles. The black boxes associated with each device represents an electrostenolytic power source attached to the muscle cell. The similar white boxes represent the synapse between the muscle cell and its neuron.

Cook & Stark estimated the activation time constant at one msec and the deactivation time constant at 10 msec at normal chordate body temperatures. Dynamics of Vision 7- 125

7.3.4.1.3 The performance characteristics of the subsystem

Figure 7.3.4-5 presents a graph from Becker246. It is similar to an earlier graph by Baloh, et. al. in 1975. The significant latency between the appearance of the target and the response will be addressed in Section 7.4.7. As addressed in the above discussion, the shape of the velocity responses is parabolic in the first order. However, they are not related to an underlying ballistic phenomenon which calls for a parabolic displacement profile. The displacement profile is linear in the first order. The causal phenomenon is a push-pull arrangement of the oculomotor muscles and the intrinsic properties of these muscles.

Figure 7.3.4-5 The displacement and velocity profile of large-angle human optokinetics. Recorded from the left eye using a magnetic search coil. The angles to the target were 2, 5, 15, 20, 30, 40 & 50°. Note the correction saccades after about 150 ms in several cases. Corrective saccades for 40° and 50° responses have been truncated to save space. Note velocity saturation above 30°. From Becker, 1991.

246Becker, W. (1991) Saccades In Carpenter, R. ed. Eye Movements, vol 8, Vision and Visual Dysfunction. Boca Raton, Fl: CRC Press Chapter 5 126 Processes in Biological Vision

While the response shown in the Becker figure can be associated with a damped ballistic system, it is necessary to supply a holding torque after the ocular has achieved the desired angular deflection. Fuchs & Luschei247 have shown the action potential firing rate required to maintain a given angular position as shown in Figure 7.3.4-6. 7.3.4.2 Expansion of the Top Level Schematic of Vision 7.3.5 Measured open and closed loop performance of the oculomotor system

Robinson has provided extensive mechanical performance measurements for the human oculomotor system248. He performed a series of isotonic, isometric and high inertial load tests. Schiller has provided a series of dynamic measurements for the cat that are very instructive249. They show the “holding” discharge rate as a function of eccentricity angle of the eye for several neurons in the oculomotor nuclei. 7.3.5.1 The low frequency, wide angle(>6.2° diam.), case

This is the region where the saccades are designed to reorient the line of fixation to bring the point of interest into the foveola. Optimally, the saccades should cause the point of interest to overlay the point of fixation within Figure 7.3.4-6 RECOPY Action potential firing rate tens of minutes of arc. The data in the literature generally required to maintain an angular position. Measurements confirms the position that the Oculomotor plant is apply to the abducen motor neurons. From Fuchs & Lushei, overdamped and strongly driven by the controller. It 1970 does not support the earlier claim by Westheimer that the system was underdamped and oscillatory250. This is particularly true when the oculomotor system is recognized as a sampled data servo system. The caricature (no data points) of position versus time on page 81 of Leigh & Zee, 3rd ed. supports this position as do the other two parts of that figure. However, the velocity versus time and acceleration versus time curves have been corrupted by introducing low pass filters into the data reduction scheme. The velocity profile should reach a higher value (~ 390°/sec.) and be much more flat-topped if it related directly to the position plot. The two wings of the acceleration plot should also be larger (peak near 44,000 °/sec/sec) and more distinctly separated with an obvious plateau between them as shown in Tole & Young251. Becker & Fuchs have provided very important data on the wide angle saccades of the eyes252. They explored this area with the eye in the dark and also when viewing illuminated targets. They also provide a good bibliography and briefly discuss the effects of fatigue and medication on the oculomotor system response. Data is provided to demonstrate that the oculomotor system operates slower and at lower velocities of saccades when no input is received from the eyes.

247Fuchs, A. F. & Luschei, E. S. (1970) Firing patterns of abducens neurons of alert monkeys in relationship to horizontal eye movement J Neurophysiol vol 33, pp 382-392 248Robinson, D. (1964) Op. Cit. 249Schiller, P. (1970) The discharge characteristics of single units in the oculomotor and abducens nuclei of the unanesthetized monkey. Exp Brain Res. vol. 10(4), pp 347-62. 250Westheimer, G. (1954) Mechanism of saccadic eye movements. A. M. A. Arch. Ophthal. vol. 52, pp. 710-724 See also; Westheimer, G. (1958) Bull Math Biophys. vol 20, pp. 149-153 251Tole & Young also Sheena & Borah, In Fisher, D. Monty, R. & Senders, J. (1980) ; Eye movements : cognition and visual perception. Hillside, N.J. : Lawrence Erlbaum Associates 252Becker, W. & Fuchs, A. (1969) Further properties of the human saccadic system: . . . Vision Res. vol. 8, pp 1247-1258 Dynamics of Vision 7- 127

They also present data suggesting the larger the subject, the longer the optical signal paths and the slower the responses of the POS. The conclusion can be drawn that volition mode movements operate slower than alarm mode movements. These subjects will be discussed in Section 7.4. Their duration and velocity data is shown in Figure 7.3.5-1. While offering more precision, it is similar to the caricature of Baloh, et. al. Becker & Fuchs point out that most subjects could not make a single saccade of more than 40° without relying on a second corrective saccade. This might explain the bending of the parabola in the Baloh curves at high saccade angles. For larger angles, they found the subject typically made a saccade of 90% of the nominal value and then performed a corrective saccade after a period of error analysis. They considered it noteworthy that the second saccade was always in the direction of the first. They also found that a subject that typically made two saccades in the light also made two in the dark, suggesting action based on programming or earlier training. This suggested to them that such a set of saccades belonged to an operational package used by the POS. They described this “package hypothesis” and further experiments to explore it in detail on their pages 1253-55. Finally, they note the ability of the subject to make a saccade of prescribed angle deteriorated with time in the dark. Becker & Fuchs point out an artifact in EOG recordings not found in competitive methods of oculography. They consistently recorded an 8 ms long pulse waveform representative of a nominal 2° pre- and anti-saccadic motion. They treated this as an artifact that was convenient to use as a point of departure.

Figure 7.3.5-1 Saccadic duration (A) and maximum velocity (B) of human eye movement between well illuminated fixation points (solid lines) and eye movements in the dark (dashed lines) for a representative subject. Each point is the average of at least 10 values with the bars indicating the standard deviation. From Becker & Fuchs, 1969.

It is clear from Becker & Fuchs that experiments involving latencies must describe the size of the individuals involved as this is a variable in the signal projection component of the overall duration of stimulated saccades. Nemire & Bridgeman have provided some values for the precision of the oculomotor system in bringing the line of fixation to a target253. They introduced a target suddenly at 16.5 degrees from the line of fixation in the horizontal plane. They found the response had a mean error of 0.03 degrees but a standard deviation of 3 degrees based on 90 trials. While the mean is excellent, the standard deviation appears excessive. This appears to have been a cumulative error. The subject responded by moving a pointer to signify the location of the target. No eyeball tracker was used. Hebbard has introduced a method of recording wide angle saccades with good precision using an optical levering

253Nemire, K. & Bridgeman, B. (1987) Oculomotor and skeletal motor systems share one map of visual space. Vision Res. vol. 27, no. 3, pp 393-400 128 Processes in Biological Vision technique254. 7.3.5.2 The mid frequency, mid angle (1.2°<Θ<6.2° diam.), case

This is the region of oculomotor operation designed to selectively place areas of interest within the foveola for precision analysis. Figure 7.3.5-2 reproduces one of the figures from Shakhnovich showing the performance of the POS at the level usually associated with the step and repeat type of process such as reading255. It does not address the very fine motion associated with character identification. Only movements from word to word and syllable to syllable were considered. This type of motion is encountered under three conditions, eyes closed and no point of reference, eyes open with no point of reference and eyes open with one or more points of reference upon which to fixate. The major axes vary under these conditions, among subjects, and among pathological subjects, as illustrated in Shakhnovich. To the extent a major axis appears in these figures, a dominant phase relationship can be discerned between the horizontal and vertical saccades. The variation among subjects suggests training plays a role in the operation of the POS at this level.

Although this figure shows all saccades occurring in the upper left quadrant and all drifts occurring in the lower right, this was an unusual characteristic of this subject. Most subjects exhibit double lobed patterns for their saccades. 7.3.5.3 The high frequency, narrow angle (<1.2° diam.), case

This section contains two subsections, the minisaccades associated with scene scanning and the microsaccades associated with detailed image analysis. Engbert & Merganthaler have provided data for the motions associated with scene scanning.

This is the region of oculomotor operation designed to analyze, in detail, individual symbols imaged on the foveola. The area of interest still involves approximately 23,000 photoreceptors. The motions associated with this case have been reviewed in Section 7.3.2.1.3. Many of the finest saccades, true microsaccades, have an angular Figure 7.3.5-2 Spatial orientation of fine movements (<3°) extent of 10-20 seconds of arc. Steinman et al. have of the two eyes. Monocular fixation in subject N.Z. was provided introductory data related to these small carried out by the right eye (solid lines). The amplitude, ρ, motions256. Figure 7.3.5-3, from Ditchburn as is in minutes of angle. The shades areas are drifts and the reproduced in Westheimer, shows the mixed (non- unshaded are saccades. From Shakhnovich, 1977. stochastic) character of these fine microsaccades257. [xxx probably due to Fender in 1956, see below. ]

254Hebbard, F. (1988) Photographing eye movements to obtain both high resolution and large amplitude applied to the experiment of J. Mueller, Am. J. Optom. Physiol. Optics. Vol. 65, no 5, pp 377-382 255Shakhnovich, A. (1977) Op. Cit. pg 57-61. 256Steinman, R. Haddad, G. Skavenski, A. & Wyman, D. (1973) Miniature eye movement Science vol 181(102), pp 810-819 257Ditchburn, R. (1960) Thomas Young Oration. Proc Phys. Soc (London) vol 73, pg 66 also in Westheimer, G. (1963) Optical and motor factors in the formation of the retinal image J Opt Soc Am vol 53(1), pp 86-93 Dynamics of Vision 7- 129

Figure 7.3.5-4, from Shakhnovich, shows the best available high frequency data applicable to tremor and microsaccades258. A high fundamental frequency of near 100 Hz, is clearly seen in the upper traces. Note the distinctly different character of the traces in the upper and lower pair. These are clearly not random noise waveforms. The instantaneous difference between the traces is highly suggestive of the microscanning being Figure 7.3.5-3 RESCAN Record of eye movements during accomplished to analyze the detailed structure of the steady fixation. From Ditchburn, R. (1960). image projected on the foveola.

258Shakhnovich, A. (1977) The Brain and Regulation of Eye Movement. NY: Plenum Press 130 Processes in Biological Vision

Shakhnovich did not provide the amplitude of the waveforms in the above figure but they can be inferred from the data of Yarbus and Figure 7.3.5-5. This figure, from Fender on a slow time axis and with less than ideal bandpass filtering, shows the magnitude and variability of the microsaccades clearly259. The noise floor appears to be less than a few arc seconds. Kowler has provided a discussion of the utility of the fine motions of the eye260. He discusses a series of simple tasks requiring good visual and motor skills, such as Figure 7.3.5-4 Waveforms of tremor resolved into vertical threading a needle and conclude “no useful role for small and horizontal components with the upper two traces saccades in either vision or oculomotor control has been wrapping around to the lower traces. Note the instantaneous discovered.” He did not explore targets more difference between the two waveforms suggestive of a complicated than a simple point or line target. higher level of signal processing. From Shaknovich, 1977. Noting that this work does not support the conclusion of Kowler or the aside in Fisher, et. al. concerning the deleterious effects of microsaccades is important261. It is the position of this work that they are the very key to the performance of the visual system in the higher primates. Their purpose is related to tasks more complex than those explored by Kowler.

The high activity rate associated with the oculomotor system has also been described in electromyographic studies by Scott & Collins as illustrated in Leigh & Zee262. Using probes described as miniature electrode needles, they show activity levels of more than 2000 events per second on a routine basis for the orbital and global portions of oculomotor muscles.

Collewijn has provided additional optokinetic data comparing the eye movements of cat, monkey and human under a variety of conditions263. The material applies to the coarser motions of the eyes.

Figure 7.3.5-5 Bandpass recording of tremor in the human. The noise floor is only a few arc seconds. From Fender, 1956.

259Fender, xxx (1956) xxx 260Kowler, E. (1991) The stability of gaze and its implications for vision In Carpenter, R. ed. Eye Movements, vol 8, Vision and Visual Dysfunction. Boca Raton, Fl: CRC Press Chapter 4 261Fisher, D. Monty, R. & Senders, J. (1980) Eye movements : cognition and visual Hillside, N.J. : Lawrence Erlbaum Associates. pg 229 262Leigh, R. & Zee, D. (1999) Op. Cit. pp 328-330 263Collewijn, H. (1991) The optokinetic contribution In Carpenter, R. ed. Eye Movements, vol 8, Vision and Visual Dysfunction. Boca Raton, Fl: CRC Press Chapter 3 Dynamics of Vision 7- 131

7.3.5.4 The pointing of the eyes under quiescent conditions

Owens & Leibowitz discuss pointing under dark, low stimulus level and quiescent conditions264. Care must be taken in interpreting their material. Their figure 3.2 shows that their measurements of ocular vergence in the dark were not actually in the dark as specified. A laser-generated stimulus was presented against a dark field. Because of the limited database, inferring normative values for the quiescent state of pointing at this time is difficult.

Owens & Leibowitz gave data for a group of 60 subjects Typical range of mean fixation distance in the dark 39-197 cm Typical range of mean fixation distance at morbidity 56-100 cm also under strong anesthesia and deep intoxication Typical range of mean focus in the dark -0.25 to +3.0 diopters They conclude that vergence and accommodation dissociate under low light level conditions. However, it is likely that more work is needed in this area for two reasons. Under truly dark conditions, the eyes can adopt a priori values based on volition. Volition mode activity may also play a strong role in determining the vergence and accommodation values assumed under conditions of limited scene input. They also propose that the dark vergence measures represent the resting or tonus position of the vergence system, unbiased by accommodative or fusional convergence. Other authors say the eyes typically diverge under conditions of morbidity. 7.3.6 Measured Performance of the augmented oculomotor system 7.3.6.1 The interrelationship of saccades, ocular drift and eye position

Several groups have been investigating the relationships between saccades, tremor and drift. Hamstra, et. al. have provided a group of interesting graphs265. They were not directly interested in, nor did they record tremor. Their definition of microsaccades (2-28 min. of arc) corresponds to the definition of minisaccades in this work (See Figure 7.3.2-1). 7.3.6.2 The optokinetic and vestibular-ocular mechanisms

Taube et al. have performed significant electropysiological studies concerning the interaction of the vestibular system with the vergence system266. Their analysis is beginning to show how the POS accepts signals from the vestibular system and uses them in circuitry reminiscent of the typical man-made servo-mechanism resolver. Tweed has provided a model of the complete servo system controlling the eyes, head and torso267. He shows how the laws of Listing and Donder’s are satisfied, or approached, under a variety of conditions.

Optokinetic experiments related to the vestibular mechanism center on the ability of the eyes to track an object moving in object space, or conversely a fixed object in object space when the subject is being rotated in a chair. These experiments attempt to define two different mechanisms. The first is the ability of the Type 0 servomechanism of the oculomotor system to track a constantly moving target. The second is the ability of the eyes to track a fixed point in object space when the subject is rotated. This latter ability is frequently measured in two different contexts. In one, the eyes are open and actively attempt to fixate on an object in space. In the second, the eyes are in the dark, the fixation point is imagined and the rotations of the eyes relative to the body are measured. The latter approach provides information on the performance of the vestibular-oculomotor system independent of external inputs. The former

264Owens, D. & Leibowitz, H. (1983) Perceptual and motor consequences of tonic vergence, Chapter 3 in Schor, C. & Ciuffreda, K. Vergence Eye Movements: Basic and Clinical Aspects. London: Butterworths, pp 35-37 265Hamstra, S. Sinha, T. & Hallett, P. (2001) The joint contributions of saccades and ocular drift to repeated ocular fixations. Vision Res. vol. 41, pp 1709-1721 266Taube, J. & Bassett, J. (2003) Persistent neural activity in head direction cells Cerebral Cortex vol 13(1), pp 1162-1172 267Tweed, D. (1997) Three-dimensional model of the human eye-head saccadic system J Neurophysiol vol 77(3), pp 654-666 132 Processes in Biological Vision

approach introduces another variable into the evaluation process. Much of the clinically oriented exploratory work in this area has been performed by Baloh and his associates268. Lacking a detailed model, they have not differentiated between the role of the extra-foveola retina in the performance achieved. By studying what are described morphologically as afoveate animals, the known directional sensitivity of certain neurons to direction has been examined. However, the role of space and time diversification in generating this directional sensitivity does not appear to have been considered. Those authors used a very simple analog servomechanism block diagram only calling for a velocity input from the vestibular system and relying on an efferent copy of the muscle response to control the performance of the oculomotor system. It also included a symbolic simple neural integrator that they have associated with the pontine reticular formation of the midbrain in an earlier paper269. In that paper, they state that foveate animals consistently have bidirectionally symmetric optomotor responses but they do not quantify or corroborate that statement. They provided an assessment of the pathological asymmetry in the slow phase velocity of the eyes for five patients. Kennard & Rose have also provided substantial material on smooth pursuit270. 7.3.6.3 Oculomotor signals from the controller of the POS system

The signals generated by the POS controller to control the oculomotor plant are very complex and differ for the different operating modes of the system. It appears that the signals controlling the twitch fibers of the muscles are distinct from the signals controlling the tonal fibers. This is suggested by the multiple neural paths from the brain to these muscles. Little data could be found in the literature describing the signals generated to control the tremor implemented by the twitch fibers.

The Superior Colliculus is an area containing at least a million individual neurons and probably more than 1000 individual output signals. Since the individual neurons are much smaller than any man-made probe, isolating a specific neural circuit by current investigative techniques is difficult.

Shakhnovich has discussed the nature of the signals applied to the tonal portion of the oculomotor system271.

Robinson has explored the source of oculomotor signals within the Superior Colliculus of the monkey272. Although, he used stimuli consisting of pulse strings of square wave pulses, it appears he only addressed the signals directed to the tonal portions of the muscles. His figures only relate to responses over intervals longer than 30 msec. The pulses were usually 0.5 msec wide at a rate of 500 pulses/sec. His technique was to probe the Superior Colliculus and observe the resulting saccades.

Tarlov has studied the neural organization of the midbrain of the cat in relation to the oculomotor plant273,274. The technique was to induce lesions and trace the resulting degeneration in the neural paths. A great deal of histological mapping was performed. Some of this mapping showed the high degree of overlap in some functional areas. 7.3.6.3.2 Bandwidth of the neural path

Little data was found defining the temporal bandwidth of the neural signals exciting either the tonic or twitch type oculomotor muscles, but independent of the muscles. McGeer, et. al. have provided data on the bandwidth of a typical projection neuron. The data suggests they only measured the tonic oculomotor drive signals275. Their data indicates a typical half-amplitude-bandwidth of xxx

268Baloh, R. Lyerly, K. Yee, R. & Honrubia, V. (1984) Voluntary control of the human vestibulo-ocular reflex. Acta Otolaryngol. (Stockholm). Vol. 97 pp 1-6 269Baloh, R. Yee, R. & Honrubia, V. (1980) Optokinetic asymmetry in patients with maldeveloped foveas. Brain Res. vol. 186, pp 211-216 270Kennard, C. & Rose, F. (1988) Physiological Aspects of Clinical Neuro-Ophthalmology Boca Raton, FL: Year Book Medical Publishers pg 277 271Shakhnovich, A. (1977) Op. Cit. pp. 92-96 272Robinson, D. (1972) Eye movements evoked by collicular stimulation in the alert monkey. Vision Res. vol. 12, pp 1795-1808 273Tarlov, E. (1970) Organization of vestibulo-oculomotor projections in the cat. Brain Res. vol. 20, pp 159-179 274Tarlov, E. & Tarlov, S. (1971) The representation of extraocular muscles in the oculomotor nuclei. Brain Res. vol 34, pp 37-52 275McGeer, P. Eccles, Sir John. & Mc Geer, E. (1987) Molecular Neurobiology of the Mammalian Brain, 2nd Ed.. NY: Plenum Press pg xxx Dynamics of Vision 7- 133 7.3.6.4 Histological features of the oculomotor system (will move to Chap. 10) 7.3.6.4.1 Histology of the muscles of the plant

Several articles have explored the musculature of the oculomotor system in detail. Fuchs & Binder explored the unique fatigue properties of both the fast-twitch and slow-twitch muscle fibers276. Their regimen was unable to elicit significant fatigue in the oculomotor muscles. Morgan & Proske studied the properties of vertebrate slow muscle in particular277. They struggle with the distinctions between the terms slow, tonic and nontwitch as used in the literature. The statement is made that “true twitch muscle, which is characterized by the ability of the muscle membrane to propagate action potentials and the fact the fiber can contract synchronously in response to each motor nerve impulse.” They also stress the apparent commingling of different fiber types within one muscle. They also provide details on the sizes of muscle fibers and on motor neuron propagation velocity as a function of temperature. Shall & Goldberg provided a description and classification system for the extraocular motor units278. Shall & Goldberg appear to use the term motor unit to refer only to the muscle fibers and not the associated neurons. They have provided a useful discussion on the projection of neural signals along a monosynaptic path from the oculomotor nucleus to a specific abducens nucleus. They have also addressed the subject of recruitment among the muscle types. Their fusion data is significant. It shows that some muscles can respond at more than 230 Hz (although the normally associated neurons may not). Breinin discussed the unusual operation of the oculomotor muscles relative to the waking and sleep states279. 7.3.6.4.2 Histology of the controller circuits

Considerable data is available on the organization of the superior colliculus with respect to oculomotor activity. Much of it predates the fMRI and is based on monkeys. Sparks, et. al. present the size and distribution of movement fields280. Later, Sparks & Nelson provided a set of sensory and motor interconnection maps281. Huerta & Harting have provided organizational maps of the superior colliculus, by layer, and also an extensive discussion of all related interconnection paths282. Their listing includes many references to the pulvinar pathway between the superior colliculus and areas 7 & 8 of the cortex. The associated discussion recognizes the unique importance of this pathway regarding “spatial vision.” No material was found differentiating between those neural paths connecting to twitch versus tonal oculomotor muscles. Shakhnovich has provided a comprehensive discussion of the neural paths between the midbrain and the oculomotor muscles that can be mined effectively283. 7.3.7 Definition and measurement of tremor

Traditionally, measuring the fine natural tremor of the eyes has been very difficult, due to the amplitudes and frequencies involved. Equally important is the phase relationship between the tremor waveforms associated with the two orthogonal axes of the eye. Section 15.2.5.2 discusses the organization of the two-dimensional correlator used in human vision.

7.3.7.1 Potential modes of signal acquisition

The method of signal acquisition at the retina and the architectural design of the correlator found in the perigeniculate nucleus (PGN) are closely tied. No current information could be found describing the precise character of the tremor used to generate the temporal signals at the photoreceptors from the image projected on those receptors. While Ogle

276Fuchs, A. & Binder, M. (1983) Fatigue resistance of human extraocular muscles. J. Neurophysiol. vol. 49, pp. 28-34 277Morgan, D & Proske, U. (1984) Vertebrate slow muscle: ---. Physiol Review vol. 64, pp. 103-111 278Shall, M. & Goldberg, S. (1992) Extraocular motor units: type classification and motoneuron stimulation:---. Brain Res. vol. 587, pp. 291-300 279Breinin, G. (1962) Op. Cit. pp 40-42 280Sparks, D. Holland, R. & Guthrie, B. (1976) Size and distribution of movement fields in the monkey superior colliculus Brain Res. vol. 113, pp. 21-34 281Sparks, D. & Nelson, J. (1987) Sensory and motor maps in the mammalian superior colliculus TINS, vol. 10, pp. 312-317 282Huerta, M. & Harting, J. (1983) Connectional organization of the superior colliculus J. Neurophysiol. vol. 49, pp 28-31 283Shaknovich, A. (1977) Op. Cit. pp. 1-21 134 Processes in Biological Vision recognized the characteristics of the tremor284, the last comprehensive work was by Yarbus and by Ditchburn in the 1960- 70's (Section 7.3.3.5). A critical relationship not reached by them, due at least partly to instrumentation problems, is the phase relationship between the tremor signal applied to the horizontal and vertical motions of the eyes. This phase relationship is critical to the specific design of the circuits of the two-dimensional correlator of the PGN. Because of this, the following material must be limited to a discussion of candidate modes of tremor operation. Only nominal values are available suggesting the tremor has an energy spectrum extending to at least 100 Hz and possibly 150 Hz. The harmonic content of this spectrum would be very beneficial in understanding the analysis mode of visual operation. The physical plant of the oculomotor system is presented in detail in Section 7.3, along with a few comments on techniques for measuring tremor. It is the “twitch” portion of the plant (using a special portion of the oculomotor muscle) that is particularly involved in the analytical mode. The question to be answered is threefold. First, what is the phase relationship between the vertical and horizontal components of the tremor? Equally important, is the phase relationship fixed or a programmable variable? Third, does the plant operate in a linear mode or does it operate in an inertial mode (like the plant used to cause larger saccades)? Figure 7.3.7-1 displays the potential operating modes. The linear scan patterns would suggest the tremor energy spectrum would be a narrow band near the nominal scanning frequency. Only the phase varies between the circular, linear and figure eight Lissajou figures shown. A more efficient scan from the information handling perspective would use a nonlinear drive signal to a linear system that was critically damped by the inertia of the eye. In this approach, the eye rotates at a constant velocity during the emphasized portions of the scan cycle. This approach would lead to maximum linearity in the conversion from spatial position to a temporal waveform.

The square pattern would suggest the presence of a broader energy spectrum for the tremor, as probably encountered in practice. It also suggests separate data frames are collected during each of the four linearized portions of the cycle. The diagonal approach suggests the collection of data during only two intervals associated with the movement of the eyes due to tremor. The Figure 7.3.7-1 Potential scanning modes associated with the crossover approach would suggest data is only collected analytical mode. during the two longer linear intervals. This approach would not be efficient unless data was also collected during the shorter vertical and horizontal intervals. Other feasible scan patterns may exist. These are the simplest.

Burton has discussed the acuity of the human eye to interference fringes and concluded that the eye is most sensitive to horizontally and vertically oriented patterns285. This would support the assumption that the phase relationship between the two sets of oculomotor muscles was 90 degrees. He also confirmed that the acuity of the eye was not related to the photoreceptor mosaic. The mosaic showed no recognizable preference for horizontally or vertically oriented patterns. McKee has discussed the impact of different simple target shapes on stereoacuity286. She concluded that disjointed patterns (either horizontal or vertical lines) exhibited a lower threshold than did patterns containing junctions. While not relating directly to the phase of the tremor, her results are consistent with the assumption that the tremor operates in a square (phase quadrature) manner as drawn in the above figure. This pattern exhibits less performance when exposed to patterns containing junctions rather than separate parallel lines. Let the baseline for discussion be the square, impulse-driven pattern. This pattern would suggest data is collected in two pairs of time intervals. For a 100-Hz fundamental frequency tremor, each complete data collection cycle would be completed in 10 msec. This is less time than generally associated with the interval between minisaccades of the eyes

284Ogle, K. (1950) Op. Cit. pg. 42 285Burton, G. (1973) Evidence for non-linear response processes in the human visual system from measurements on the thresholds of spatial beat frequencies. Vision Res. vol. 13, pp 1211-1225 286McKee, S. (1983) The spatial requirements for fine stereoacuity Vision Res. vol. 23, no. 2, pp 191-198 Dynamics of Vision 7- 135 observed during the study of fine detail by the subject. The process usually involves time intervals of 50-100 msec. Based primarily on fusion frequency data, let it be assumed that the fundamental frequency of the tremor is 30 Hz with overtones extending up to at least 90 Hz. This would suggest the tremor did operate in the inertial mode. Here, the data collected with respect to each side of the square would be acquired during about 8 milliseconds. This frame interval would be sufficient to generate one action potential describing the state of the pixel scanned in one direction relative to the previous frame interval. For examining fine detail, this is all that is required. The visual system is basically looking for edge transitions. To obtain a nominally square movement of the eyes due to tremor, the underlying signals from the oculomotor nuclei would be generated in phase quadrature. Either the nuclei, the twitch muscles, or the combination of both, would linearize the signals to provide a constant velocity movement for roughly 70% of the available time. These waveforms are presented in Section 7.3.4 as a function of time. The caricature in Figure 7.3.7-2 is provided to help visualize the process. In this candidate situation, the individual photoreceptor aperture is made to scan over a distance of 2.5 aperture diameters. Such a value is consistent with the best available data on the amplitude of the tremor (Yarbus, 1957). By examining the output waveform of the scanning photoreceptor, the correlator can detect an edge within the time of interval one and another edge within interval three. No edge will be reported in any of the other intervals. Knowing the phase of the signals from the tremor scan generator, it can say that the first edge is vertical and the second is horizontal. However, the adjacent photoreceptor to the left of the one shown will report a vertical edge in interval two and a horizontal edge in interval three. The cell immediately to the right of the original cell will not report any vertical edge but will report a horizontal edge in both interval eight and interval three. This simple procedure has determined that there is a corner located within the scan pattern of the first cell. The corner has a vertical line extending up from the corner (beyond the scan pattern of the three cells) and a horizontal line extending to the right from the corner (beyond the scan pattern of the right cell).

By completing a more extensive examination of the mapping achieved by a larger group of photoreceptors, the Boolean logic required to be programmed into the correlator can be determined.

The above mode of operation would suggest relatively sophisticated synchronous switching at the input of the correlator in the PGN. Each signal received from the photoreceptors would be in a differential form internally and bear a specific temporal relationship to the signals from adjacent cells. A primary goal of the correlator would be to sum these signals in a way that enhances the ratio of signal data to noise while preserving the geometrical fidelity of the edge determination. Conceptually, this would suggest two possibilities. Either a complex switching of incoming signals from a given photoreceptor sequentially into various time related bins OR the use of additional dimensions in the correlator to represent the foveola map at separate intervals of time.

Look briefly at the switched signal approach first. Each action potential representing a different time interval, and therefore spatial position in the foveola. It would need to be fed into a different time bin at the output of stage 3 and the resulting output would need to be switched to one of four planes of the correlator. Two of the planes would store vertically oriented information (one for the rising scan and one for the descending scan) and two would store horizontally oriented information. The data set would represent edges detected within the image projected onto the foveola. This data would not contain any chromatic information. All of the photoreceptors in Figure 7.3.7-2 Caricature of photoreceptors scanning an the foveola would be treated equally, without regard to edge near the visual acuity limit. Photoreceptor cell is spectral sensitivity. This mode of operation is quite scanning a corner of a right angle in the scene near the compatible with the well-known loss of color vision at acuity limit. The maximum tremor amplitude is nominally very fine resolution. The color information would be 2.5 times the diameter of the cell entrance aperture. 136 Processes in Biological Vision derived only from the much coarser awareness channel. The nominal signal amplitude at each pedicle of a photoreceptor would be the same (in the absence of chromatic adaptation). This is because each of the outer segments of these receptor is of equal size and all of the chromophores exhibit the same absolute sensitivity in response to an equal photon flux light source (nominally 7053 Kelvin for trichromats). Other than the requirement for synchronism, the above switching is quite simple. The requirement for synchronism is not difficult considering the fixed temporal and physical relationship between the tremor generator and the signal recovery circuits. A tremor amplitude equal to the diameter of only one photoreceptor can be accommodated. Larger tremor amplitudes could also be accommodated when called for by other operating requirements and implemented by the POS. The use of additional temporal planes in the multi-dimensional correlator approach replaces the need for the considerable switching circuitry associated with each incoming neural pathway with a simpler mechanism. Instead of switching neural signals individually, they are collected in parallel and then all of the data associated with a given interval of time can be combined with that from adjacent intervals by using a shift-and-sum approach. This is a core approach in the architecture of man-made parallel processing computers. The data in one time plane is combined with that in an adjacent plane by transferring the data of the first plane into the data bin holding the second plane. However, the shift is done along a diagonal that results in a summation in signal amplitude with respect to spatial position regardless of time slice.

This shift-and-sum approach can be employed using various diagonals as test cases. The amplitude of the various outputs obtained can be compared to determine the optimum output amplitude. In similar radar correlators, this step-and- repeat process with respect to the shift-and-sum mechanism can be used to adaptively correct for incorrect estimates of aircraft velocity introduced into the correlator. As a result, one output of the correlator is a better estimate of the aircraft’s velocity than that available from its own instruments. Other outputs provide an enhanced map of the scene of interest. The signal-to-noise ratio of individual line elements in the scene can be improved by about 200:1. In vision, one alternate output of the correlator is a signal indicating the degree of convergence between the imagery provided by the left and right eyes. This signal can be used as a precision vergence signal in the oculomotor control portion of the POS.

Since both of the above approaches involve multiple signal storage planes, it is probably easiest to implement the shift- and-sum approach. This approach requires the fewest discrete switching circuits. The dendritic trees of neurons, acting as temporary signal storage bins, are particularly adapted to the shift-and-sum approach.

This discussion suggests that a candidate relationship between the two twitch modes of oculomotor operation is that of Figure 7.3.7-3.

In this figure, the two orthogonal waveforms are shown idealized to provide two out-of-phase linear movements of the reticle formed by the entrance aperture of the outer segments. The time scale and amplitude are both shown in relative terms. The amplitude appears to have a minimum (for its active portion) approximating the diameter of one photoreceptor within the foveola. Larger amplitudes (particularly multiples of this diameter) appear to be acceptable and the amplitude may be under active control by the POS. The appropriate period appears to be about 1/30 second. The available data suggests the power spectrum of the tremor extends up into the region of 90-150 Hz. The first three harmonics of a sine wave produce a creditable sawtooth waveform of the type shown. This would suggest the fundamental frequency is in the region of 30 Hz. Under this Figure 7.3.7-3 Candidate tremor waveforms for the two assumption, let the period be 32 ms and each portion of orthogonal oculomotor motions. See text. the active scan occupy 8 ms. These numbers would be compatible with an action potential frequency of about 125 Hz in the stage 3 neurons of the optic nerve associated with the foveola. Under this assumption, the action potentials cannot actually be considered to have a frequency. Each interval of 8 ms is represented by either a pulse or a blank space representing the change sensed by the photoreceptor Dynamics of Vision 7- 137

during the previous sampling interval. The result is a pulse-coded stream of action potentials at a clock rate of 125 pulses per second. Under the above assumptions, the average tremor velocity can be approximated using a nominal photoreceptor diameter of two microns. This diameter gives an average tremor velocity of 2.5 cm/sec. If the amplitude is larger than two microns, the velocity may be higher or the scan duration may be extended to equal two sample intervals of the stage 3 action potential generator. Either change would cause a change in architecture of the two-dimensional correlator of Section 15.2.5.2. A small saving in time could be realized if the two waveforms shown above exhibited some rounding of the corners, and the time interval involving rounding was shared by the two quadrature waveforms. The saving is probably marginal if the servomechanism driving the twitch oculomotor responses is highly optimized. No data is available on the subject of twitch responses at the detail required. Harwood & Harris have recently discussed the operation of the non-twitch portion of the servomechanism primarily with respect to the volition mode of operation287. Their investigations appear to be largely exploratory. 7.3.7.2 Methods of measuring tremor

xxx has provided a graphic comparison of the methods of measuring tremor288. It shows the traditional contact lens with mirror was the highest capability approach until the recent introduction of the scanning laser ophthalmoscope married to an adaptive optics system (abbr. AOSLO). The AOSLO can image an edge onto the retina with a positional accuracy of less than one photoreceptor diameter.

The investigator should avoid counting the number of peaks in a recorded temporal waveform of finite duration in order to calculate the mean tremor frequency. This method is far too crude for use when the amplitude of the underlying data exhibits a 1/F frequency spectrum. The tremor is clearly a broad but band-limited pseudo-random mechanism (frequently described as a random-walk mechanism) that can not be described by a mean frequency. A peak counting approach has been used by many past investigators who lacked a clear understanding of the complexity of the problem.

Instrumentation has always been a problem in tremor measurement. Recently, new video cameras have become available that considerably simplify the instrumentation and the introduction of optical coherent tomography (OCT) has quickened the pace of development of fine trackers.

Iscan, Incorporated is now offering full frame video cameras with a frame rate up to 240 frames per second289. Skalar Medical is offering an IR tracker with a Bode plot showing negligible rolloff prior to 100 Hz and a 3dB point at 185 Hz.

Ferguson et al. demonstrated a new tracking OCT in 2005 designed to 3D image the in-vivo retina of the eye290. The reason for the tracking function was because of the poor results with the best high resolution OCT because of the instability of the eye over the image collection interval. Their tracking loop had a closed loop bandwidth of 1 kHz. They claim their system was able to maintain diffraction-limited performance over periods of one minute. Figures 3 & 4 of their paper show the output of their servo tracker with the tracker turned on and turned off. The tremor amplitude driving this waveform is on the order of a few resolution elements.

During the late 1990's the combining of adaptive optics and the scanning laser ophthalmoscope has added a new capability to observe individual photoreceptors of the retina over a significant area at frame rates exceeding 100 Hz. The recent paper by xxx [xxx proc etra paper ] has summarized this capability and provided a useful bibliography. They show it is possible to eliminate various extraneous motions from their data and compute the spectral frequency of the tremor in two orthogonal directions with good fidelity. The fidelity is so good, they had to account for wobble in the human eye due to motions between the lens and its surrounding/supporting sclera during major saccades291. They

287Harwood, M. & Harris, C. (2002) Time-optimality and the spectral overlap of saccadic eye movements. Ann. N.Y. Acad. Sci. vol. 956, pp 414-417 288Xxx (2010) Eye tracking with the adaptive optics scanning laser ophthalmoscope Proc ETRA vol xxx ACM Press 289Iscan, Incorporated, 89 Cambridge Street, Burlington, MA. E-mail: [email protected] 290Ferguson, R. Hammer, D. et al. (2005) Three-dimensional retinal maps with tracking optical coherence tomography (TOCT) SPIE Photonics West, 22-27 Jan 2005. 291Deubel, H. & Bridgeman, B. (1995) Fourth Purkinje image signals reveal eye-lens deviations and retinal image distortions during saccades Vision Res vol. 35, pp 529-538 138 Processes in Biological Vision achieved about one arcsecond (a fraction of a photoreceptor diameter) resolution with their AOSLO. They show the resultant spectra are well represented by a 1/F spectra characteristic of a random walk mechanism as illustrated in the above single trace recordings. They show this spectra is obtained up to at least 700 Hz but includes some artifacts at multiples of 30 Hz because of their scanning protocol. Figure 7.3.7-5. shows their figure 9 representing the mean vertical amplitude spectrum of eye movements during periods free of recognizable saccades. The data was obtained with the subjects fixating on a 16 arc min square target. 16 arc min is equal to the diameter of about 50 photoreceptors at the retina. (Section 19.8.2). Segments 200 ms long were used in the FFT calculation, setting a lower limit for the frequency scale of about 5 Hz.. The trace labeled “grid” is obtained using an “artificial eye.” The details related to this trace were not provided. It exhibits a 1/F0.5 characteristic in their figure 9.. In their horizontal spectrum, figure 8, the raw “grid” trace shows a horizontal component (1/F0) from 5 Hz to 300 Hz followed by a rapid rolloff. This would be the more expected characteristic of their artificial eye described as a “plus lens and a piece of paper mounted rigidly to the AOSLO platform.” The rapid rolloff could them be associated with the diffraction limited performance of their lens. Figure 7.3.7-4 Amplitude spectrum calculated with Matlab’s FFT function from AOSLO video. Segments The other traces relate to four normal subjects. The without saccades were selected from several videos of smoothed versions of these traces all show a near 1/F subjects fixating a 16 arc min square target. The horizontal response with a potential shift in the gain constant in the and vertical sprectra are essentially the same. See text. region of 50 to 100 Hz. It is suggested this change is From xxx, 2009. associated with the change from the tonal muscle to the twitch muscle systems described in Section 7.3.1. This is a different interpretation from the authors suggestion that their data set represents a scale-invariant random walk pattern. As noted by the authors, there is no dominant (sinusoidal) frequency component associated with tremor. This is consistent with the model of this work.

The xxx paper provides a good discussion of their artifact removal protocol.

The Matlab FFT transform apparently does not maintain the cardinal numbers of the individual traces used in the calculation. As a result, a spatial position versus time response has not been recovered from this data. However, Arathorn et al. have recovered such information apparently on the same AOSLO equipment292.

Raghunandan et al. have recently reported AOSLO data using an asymmetrical (3 x 19 arc min) stimulus but focused on the vertical motion of the eye when viewing this horizontal bar293. The paper provides very useful information but the discussion is hindered by the lack of a sophisticated model of ocular dynamics and the neural system. The complexity of the mechanisms they are trying to understand requires an equally complex model. In the introduction, they note the earlier postulates that “the functional significance of such retinal jitter is to overcome the fading of retinal images. . .” using the term fixational jitter when refering to tremor. They did not reference the second postulate that such motion is actually used to extract information from the scene. They proceeded to adopt the position that tremor was a masking noise (“spurious motion due to retinal jitter.”) relative to the performance of the system. They draw a strong conclusion with respect to efference copies of neural commands or proprioceptor signals reporting the motion of the eyes contributing to the servomechanism controlling eye motions.

292Arathorn, D. Yang,Q. Vogel, C. Zhang, Y. et al. (2007) Retinally stabilized cone-targeted stimulus delivery Opt Express vol 15, pp 13731-13744 293Raghunandan, A.Frasier, J. Poonja, S. Roorda, A. &. Stevenson, S. (2008) Psychophysical measurements of referenced and unreferenced motion processing using high-resolution retinal imaging J Vision vol 8(14), pp 1-/11 Dynamics of Vision 7- 139

7.4 Operational overlays on the Precision Optical System

As noted in the introduction to section 7.3, significant differences appear in the performance of the visual systems of each species within the higher primates and the monkeys. These differences make the use of surrogates in the exploratory and precision performance laboratories less than ideal. In many cases, the use of surrogates is completely inappropriate. Historically, the evaluation of the performance of the underlying servomechanisms of the physiological optical system has been performed by different researchers than those interested in the operational performance of the same optical system. Until recently, both of these groups were performing primarily exploratory investigations and their activities did not overlap significantly. The investigations of operational performance were particularly limited in two ways. First, they generally relied upon non-invasive techniques. Second, they assumed that the physiological black boxes they were studying contained only linear analog circuitry. The suggestion that these black boxes might contain memory or complex computational engines was not found in the literature. Leading works summarizing prior investigations of the operational overlays include; Schor & Ciuffreda294, Hung & Ciuffreda295, and Hung296. A problem with these compilations is suggested by the title of Hung & Ciuffreda. The different authors employ a variety of floating models rather than one comprehensive, and consistent, model.

Howard297 and Howard & Rogers298 have completed a massive two-volume work on Seeing in Depth. The first volume is subtitled Basic Mechanisms. However, the term mechanisms is used in its most general. Its use is similar to the way many authors have used the word function. Both speak of functions and mechanisms at the conceptual level. In many chapters, they have used the assumptions of Gaussian optics to explain wide angle optical situations. The wide angle situation requires the use of complete physical optics as shown in Section 2.4 and in Lotmar299. Little discussion is provided relating their psychological discussions to the physiology of the human visual system. The second volume is based entirely on psychological investigations and employs many caricatures that have stretched the applicability of the paraxial optical assumption. The second volume also appears to suffer from a lack of editorial review by the publisher. Several inconsistencies appear between the caricatures presented.

Reading has provided a comprehensive text on Binocular Vision aimed at introductory pedagogy300. It goes into more depth, and provides many more special themes, than in this work.

The question to be addressed in this section was posed by Jones301. “Although all comprehensive theories of fusion recognize that disjunctive eye movements provide the substrate for sensory unification, this realization does not ‘explain’ sensory fusion. It begs the following important question: what is the nature of the sensory process that recognizes retinal disparity and directs the eyes to eliminate it?

Since the answer proposed here does not build on the previous literature, that literature will be discussed after the main discussion (Section 7.4.5).

A key to the understanding of stereopsis and fusion is recognizing the critical role played by physiological tremor in the function of these mechanisms. As noted in Section 7.3.3.5.4, the visual system is blind in the absence of relative motion between the retina and the scene imaged onto the retina. While this fact is well documented, it has largely been ignored in the recent literature. It is this relative motion that is used to convert spatial image information at the retina into temporal information within the neural system. This temporal information can be processed and projected to other anatomical sites within the visual system. Without tremor, no temporal information is created or passed from the retinas

294Schor, C. & Ciuffreda, K. eds. (1983) Vergence Eye Movements: Basic and Clinical Aspects. London: Butterworths 295Hung, G. & Ciuffreda, K. eds. (2002) Models of the Visual System NY: Kluwer Academic/Plenum Publishers 296Hung, G. (2001) Models of oculomotor control, River Edge, NJ : World Scientific 297Howard, I. (2002) Seeing in Depth, vol. 1 Basic Mechanisms Toronto, Canada: I Porteous 298Howard, I. & Rogers, B. (2002) Seeing in Depth, vol 2, Depth Perception Toronto, Canada: I Porteous 299Lotmar, W. (1971) Theoretical eye model with aspherics J Opt Soc Am vol 61, no. 11, pp 1522-1529 300Reading, R. (1983) Binocular Vision. London: Butterworths 301Jones, J. (1983) XXX [ correct paper] Parallel Processing in the Visual System. NY: Plenum Press 140 Processes in Biological Vision

to the CNS. A second key is the availability of a model in sufficient detail to explain how the perigeniculate nucleus of the thalamus performs a two-dimensional correlation function over a spatial range of photoreceptor cell signals constituting the foveola of each eye. A third key to understanding stereopsis and fusion is to accept the fact that a “merged image” is not formed within the central nervous system. There is no need to “eliminate retinal disparity” in the cortical image as frequently suggested in the recent literature. The data extracted by the thalamus and other elements of the CNS is processed in tabular form. The Thalamus calculates both the mean and deviation associated with each pair of edges corresponding to one edge in object space. These tabular values are used to provide the perception of depth. They are incorporated into the saliency map of the visual system and can be recalled by a variety of subsystems (related to the command implementation process) of the system. 7.4.1 Framework for evaluating the operational overlays

Attention to detail is mandatory when considering the complex functional overlays to the pointing system of accommodation, version and vergence. Regarding the physiological optics, the use of the Gaussian Optics assumption is to narrow. Only the incorporation of the broader laws of physical optics allows an understanding of the systems. Physical optics introduces the losses in performance related to field angle, aberrations and other effects (such as the various Stiles-Crawford Effects). These losses must be accounted for when performing the data analysis associated with performance of the visual system.

A similar situation occurs regarding terminology. Attention to detail is necessary. Understanding the above overlays involves many complex relationships. Much of the historical work has been at the conceptual level. This has led to the conceptual, and frequently imprecise, definition of many terms. When later investigators have attempted to use these terms, they have replaced them with their own similarly (but not identically) defined terms. The result has frequently been similar to that found in translating a foreign language. The nuances intended by the original author are lost. The serious reader is encouraged to seek out original articles whenever possible to avoid being misled.

Figure 7.4.1-1 defines the geometry of vision frequently found in the pedagogical literature. It is based on an early concept associated with Vieth & Muller. This concept was in turn based on earlier ideas of Maddox and Fechner. It originated with Aguilonius in 1613. The Vieth-Muller circle is defined by the nodal points of the two eyes and the natural fixation point located along a perpendicular bisecting the line drawn between the two nodal points (on the survace of the sagittal plane). Three major problems with this concept appear at the research level. First, the “natural fixation point” is a variable with a high standard deviation among the population. This makes it awkward to define the diameter of the circle with any Dynamics of Vision 7- 141

Figure 7.4.1-1 Geometry of horizontal disparity. Note the Gaussian optics approximation. For large angles from the line of fixation, the rays do not pass through the lens without bending. The fronto-parallel plane (or tangent plane) is shown for discussion. It does not represent a useful concept. See text. precision. Second, the angles associated with horizontal disparity are usually defined with reference to the lines of fixation leading to the “natural fixation point.” Since the fixation point is poorly defined, the angles associated with the lines of fixation are also poorly defined. Jones discusses this difficulty in Schor & Ciuffreda but proceeds with it for pedagogical purposes302. Finally, the Vieth-Muller circle does not represent actual performance well. The Hering-Hillebrand deviation from this geometric horopter is found in nearly every individual (See Records, pg 649 and the more extensive analysis in Ogle, pp 24-49). The deviation is so pervasive that the Vieth-Muller circle can only be considered a first order approximation of the fundamental performance of the human visual system.

302Jones, R. (1983) Horizontal disparity vergence, Chapter 8 in Schor, C. & Ciuffreda, K. Vergence Eye Movements: Basic and Clinical Aspects. London: Butterworth pp 297-303 142 Processes in Biological Vision

While pedagogical discussion of vergence are frequently couched in terms of two or more points, it is critically important to recognize that stereopsis and fine vergence rely on features rather than points at the locations highlighted. These features are generally larger than a few times the diameter of a single photoreceptor of the retina. It is the scanning of this feature by the tremor of the eye that results in time correlatable signals from multiple photoreceptors in each eye that can be used to provide fine stereopsis throughout the receptive field of the foveola. Jones defined a set of equations based on a proposition of Hering known as the Law of Equal Innervation (Section 7.3.2.5)303.

Jones defines the net disjunctive eye movement (of the point A relative to the physiological resting condition) as αL – αR where these angles are measured from the nominal point of fixation. While useful for pedagogy, it has problems at the research level. Owens & Leibowitz describe these problems in some detail304. Their discussion highlights the tendency of the eyes to diverge under conditions of deep anesthesia and death. The divergence angle is highly variable among subjects. It has been reported to be as high as 71 degrees at birth. Under physiological conditions, the eyes tend to converge. However, the resting condition, leading to the point of fixation, is also quite variable. Owens & Leibowitz give the distance to the point of fixation as 39 to 197 cm. These numbers lead to resting vergence angles of 1 to 7 degrees. With this kind of variation, the measurement of any angle from the resting vergence angle is poor practice. Determining the resting vergence angle for each subject under a variety of conditions, such as fatigue level, becomes necessary.

A more precise definition would be given by (βL + αL) – (αR + βR ) = γL – γR where the values of beta represent the angle to the resting position from the collimated condition.

Similarly, Jones defines the net conjunctive eye movement (of a point A relative to the natural resting condition) as αL + αR. This value goes to zero for any point along the vertical axis through the fixation point. A more precise definition would be given by (βL + αL) + (αR + βR ) = γL + γR where the values of beta and gamma are measured from the collimated condition.

The terms in these equations can be summarized as,

α = vergence to point A with reference to the physiological (resting) condition. β = physiological (resting) condition with reference to the collimated condition. γ = vergence to point A with reference to collimated condition. δ = anatomical (morbid) vergence with reference to collimated condition.

The angle between the two lines of fixation, 2β, is usually described as the target vergence (or sometimes simply as the eye vergence).

Jones uses these equations to support two measurement regimes. He relates the terms disjunctive and conjunctive as descriptors referring to eye position in terms of the signals applied to the pointing system. He then uses the terms vergence and version as referring to the physiological responses of that system.

Another obvious problem is the use of the nodal points of each eye as references. The location and the distance between the nodal points change with rotation of the ocular globes. Thus, the Vieth-Muller circle moves relative to the defined circle of equal convergence. When discussing vision over wide fields of view, it should be recognized that the Gaussian Optics approximation does not hold. The ray passing through the lens at greater angles than a few degrees from the axis cannot be considered straight (Section 2.4.1). It is only because the eyes are optically symmetrical that points such as A can be considered imaged on similar (cover ) points of the two retinas. The preceding and following figures show a fronto-parallel plane through the point of fixation. Under the Gaussian assumption, such a plane would be imaged as a similar, and parallel, plane at the point of focus on the retina. However, the retina is highly curved to satisfy the physical optics of a wide field of view optical system. Thus, the fronto-parallel

303Jones, R. & Kerr, K. (1971) Motor responses to conflicting asymmetrical vergence stimulus information Am. J. Optom. vol. 48, pp 989-1000 304Owens, D. & Leibowitz, H. (1983) Op. Cit. pp 26-46 Dynamics of Vision 7- 143 plane is only imaged on the retina under the Gaussian assumption (that the field of view is less than a few degrees). To achieve this imaging, the appropriate value of accommodation must be established. This condition is only met for the foveola, and possibly a part of the fovea. The fronto-parallel plane cannot be used to discuss wide field of view scenes under the assumption of a fixed condition of accommodation. For a constant state of accommodation, the point of fixation of the eyes rotates about the point of rotation of the eyes at a constant radius. This radius is much larger than that of the horopter. The locus of constant accommodation falls between the horopter and the fronto-parallel plane at all angles. While the horopter defines a circle of constant vergence angle, it does not define a circle of constant accommodation. Blakemore illustrates the problems with trying to use a fronto-parallel plane in his figure 3 reproduced here as Figure 7.4.1-2305. In the caption, he asserts “The scale, showing the distance from the eyes, is also appropriate for the fronto- parallel coordinate.” Unfortunately, the distances from the eyes are in radial coordinates while that for the fronto-parallel plane is in rectilinear coordinates. If two arcs of 43.7 mm radius are drawn through the point of fixation representing the condition of prime focus for each eye, the problem becomes obvious. Both eyes are in optimum focus over only a small region around the point of fixation. Ogle illustrated this in his figure 13, although he took some license in drawing a “Normal” locus306. The resolution of the eyes also falls rapidly at angles greater than a few degrees from the point of fixation. His measured points do not follow the Vieth-Muller circle or his fronto-parallel plane. Nor do the points satisfy the definition for points along the horopter (since their vergence angles are obviously different from those along the horopter). As he states in his text, they are the locations of points that appear single while the eyes are fixated at F. The eyes observe these points under conditions of considerable defocus and reduced resolution.

7.4.1.1 Monocular, binocular and stereoscopic domains of vision

When discussing the operational aspects of pointing, e. g., version, vergence and accommodation, it is extremely important to differentiate carefully between the fields of spatial vision, Figure 7.4.1-3. In pedagogy, these regions are usually only defined for vision in the far-field (infinite distance). The monocular field of vision is that angular region of object space observed instantaneously by one eye. The monocular fields of the two human eyes are not congruent. However, they do contain a common angular region described as the binocular field of vision. This field is normally only described for the region defined when the two eyes are pointing perpendicular to Figure 7.4.1-2 Figure from Blakemore with dashed arcs of a line drawn between them (they are looking straight best focus added. Points show where an object would have ahead). This region has two important features. First, it to be placed to appear single during fixation at F. Modified describes a region that includes a series of equivalent from Blakemore, 1970. areas in the two retinas. These equivalent pairs of areas are sometimes described as covering points (areas) in the literature. The extent of the binocular field of view is constrained by the physical structure of the nose. Second, the binocular field of vision also describes the range over which the line of fixation can be moved while still maintaining a common point of fixation for both eyes in object space. It is this latter feature that is critical to stereopsis. Both foveola must be able simultaneously to observe the same region (within tens of milliseconds) in object space to achieve stereopsis.

305Blakemore, C. (1970) The range and scope of binocular depth discrimination in man J Physiol vol. 211, pp 599-622 306Ogle, K. (1950) Researches in Binocular Vision. London: W. B. Saunders, pg 29 144 Processes in Biological Vision

For scenes closer to the eyes than defined above, equivalent areas can be defined. However, more detail is required in the wording. The total and binocular fields are reduced for off-axis viewing. A great tendency exists to use the terms binocular vision and stereoptic vision interchangeably in scientific literature. However, they are operationally quite different. Monocular vision is used primarily in the awareness mode of vision. It is used to provide directional (versional) signals to the POS. The POS uses them to align the line of sight to significant areas of interest. The required computations occur in both the retinas and the LGN. Binocular vision is used by the LGN to generate coarse vergence signals for use by the POS. Stereo-optic vision (limited to the spatial regions imaged by the two foveola) is used within the PGN to extract a precision vergence signal related to the point of fixation in object space. Stereo-optic vision is also used to calculate differential vergence signals describing the location of other objects imaged by the two foveola relative to the location of the point imaged at the point of fixation. Figure 7.4.1-3 The visual fields of monocular, binocular and stereoptic vision. The stereoptic field is shown in While the narrow field of view mechanism of stereopsis cyclopean form for simplicity. Each eye exhibits a similar can be directed anywhere within the wider field of view field. They are usually convergent at the point of fixation. of binocular vision, these two concepts are quite The fronto-parallel plane is shown for discussion. It does independent and separate. not represent a useful concept. See text.

Interpreting the above definitions carefully leads to an unambiguous description of how the operational overlays to the pointing subsystem work. It also explains how stereo- optic vision is achieved within the analog domain of the PGN without using a large amount of computational power.

When exploring questions concerning fusion and stereopsis, Section 7.4.1.1 will show that only the region of the Vieth- Mueller circle (the horopter) imaged instantaneously by the foveola is involved in precision stereopsis. Under these conditions, the relevant angles associated with retinal disparity are very small, less than 1.2 degrees, and the approximations associated with Gaussian Optics can be used. Outside of this small area, only coarse performance stereopsis can be achieved.

Within the field of view associated with the foveola, the normal eye can perceive a relative disparity of ten arc-seconds or 0.0028 degrees. The best trained observers can achieve two arc-seconds (0.00056 degrees) at a 75% discrimination level. These numbers suggest that differences of 0.001 inches can be discerned at ten inches and differences of two miles can be discerned at the horizon (S&C pg 238). Better documentation is needed to confirm these values. Figure 7.4.1-4 provides a summary of the functional capabilities of the human visual system relative to object space. Dynamics of Vision 7- 145

Figure 7.4.1-4 A summary of the on-axis parameters of vision in object space. The vertical dashed line separates fine depth perception related to the foveola and the analytical mode of vision from the coarse depth perception related to the peripheral retina and the awareness/alarm mode of vision. The latter depends heavily on scale cues and angle-rate information.

When discussing depth perception, two conditions are frequently described. The First is stereoptic depth perception where depth perception is based on a fusion of the images from the two eyes. The second is diplopic depth perception where depth perception is estimated without fusion of the two images. Finding a statement that “Depth is perceived in the region of both fusion and of diplopia” is common in the pedagogical literature307. Section 7.4.5 will discuss the significant differences between these two domains in the character and the mechanism generating the perception of depth. 7.4.1.1.1 Global, local, fine & coarse in discussing functional overlays

The terms global, local, fine and coarse have been used in many ways in the literature of functional visual performance. Most of the usage has been at the conceptual level. Adopting more formal and precise terminology in this area is useful.

Sperling defines two distinct neural binocular fields (NBF)308. He defines these NBF’s as internal three-dimensional representation of the external world. Although not stated specifically, these representations appear to be based on individual planes representing different distances from the subject. He defines “a primary NBF for the fine-detail functions of binocular vision, and a secondary NBF for coarse-detail functions. The interaction of these two systems is crucial for many phenomena of binocular depth perception.” He does not elaborate on the spatial extent of the NBF’s.

Richards & Kaye describe stereopsis as occurring within two domains309. One domain is small and corresponds well with the foveola defined in this work, although it may extend to the diameter of the full fovea. They note that this area requires a high degree of binocular scene similarity and yields fusion. Their second domain does not require as high a degree of similarity in scene elements and generally does not result in fusion. Their criterion was based primarily on disparity values. They provide a graph of relative perceived depth versus stimulus disparity but placing it in context is hard. It can be compared with the graphs of Section 7.4.5.

307Tyler, C. & Scott, A. (1979) Op. Cit. pg 645 308Sperling, G. (1970) Binocular vision: a physical and a neural theory Am J Psychology vol. 84 pp 461-534 309Richards, W. & Kaye, M. (1974) Local versus global stereopsis: two mechanisms Vision Res vol. 14, pp 1345-1347 146 Processes in Biological Vision

Julesz defined fine and coarse in his 1978 paper by painting a scenario310. He said “The stereopsis of narrow bars (or small dots) will be called fine stereopsis, while stereopsis of wide bars (or large dots) will be called coarse stereopsis.” He went on to conceptualize that “fine stereopsis is carried by the high spatial-frequency disparity detectors and coarse stereopsis by the low spatial-frequency disparity detectors.” The two types of disparity detectors were not definitized further. Tyler & Julesz311 suggest that Richards & Kaye312 used local and global to mean the same as the fine and coarse of Julesz. Tyler expanded on the theory of the horopter in 1991313. See Section 7.4.5 where the label precision is used instead of fine. Julesz (1978) went on to define global and local stereopsis much as in this work. Paraphrasing, he said, a global process is needed that evaluates different sets of corresponding element pairings (local stereopsis) and selects one set of pairings from the many pairings in a scene as the dominant element. This work proposes the global stereopsis process calculates a nominal point of fixation for the scene based on a mean of all the element pairings. This mean is then used as a reference for describing each corresponding element pairing (local stereopsis). There seems general agreement in the literature that there is a fine depth perception mechanism associated with the foveola and a coarser mechanism associated with the outer retina. Richards & Kaye referenced several other authors and defined the disparities associated with local stereopsis as occurring within the range of +/– 0.5 degrees. They claim local stereopsis customarily yields fusion and requires a high degree of binocular similarity of the disparate images. They again reference several authors and define global stereopsis as accepting less similar targets in the two eyes. They say these targets are generally not fused but appear diplopic. Their global stereopsis occurs for disparities much larger than +/– 0.5 degrees. They build a two-mechanism model around the differences in performance associated with these two zones. They show that their experiments did not support such a separation into two distinct mechanisms.

Figure 7.4.1-5 shows the plan view from above of the optical system of the right eye. The plan view of the left eye is symmetrical with this image (the optic nerves both exiting on the nasal side of the ocular). The optical rays in this figure are symmetrical about the vertical axis. The optical system of the human eye is highly anamorphic in the language of opticians. Note, the optical rays cannot be represented as straight lines passing through a single nodal point for rays beyond 0.6 degrees from the optical axis. For larger angles, it is appropriate to employ two principle points, one (P1) near the projection of the external rays at 0 and 90 degrees and a second (P2) at the intersection of the internal optical rays near the external surface of the lens. While not important when conceptualizing version, vergence and fusion, this optical diagram shows conclusively that the angles relating to two points on the external horopter do not correspond to two similar points on the retina of each eye. The two images projected onto the two retina can not be overlaid in any realistic sense as is frequently assumed in conceptual discussions (whether flattened or not). The process of fusing the two images occurs entirely in the signal processing (mathematical domain) of stage 4 of the neural system. Even the signal processing related to precision stereopsis must rely on stage 4 signal processing (within the PGN-pulvinar couple) to merge the two images, except within a few photoreceptor diameters of the point of regard of the two images.

310Julesz, B. (1978) Global stereopsis: cooperative phenomena in stereoscopic depth perception, Chapter 7 In Held, et. al. ed. Handbook of Sensory Physiology, Vol VIII, NY: Springer-Verlag 311Tyler, C. & Julesz, B. (1980) On the depth of the cyclopean retina Exp. brain Res. vol. 40, pp 196-202 312Richards, W. & Kaye, M. (1974) Local versus global stereopsis: two mechanisms Vision Res. vol. 14, pp 1345-1347 313Tyler, C. (1991) The Horopter and Binocular Fusion. In Regan D. ed, Vision and Visual Disorders. Vol. 9, Binocular Vision. NY:Macmillanpp. 19-37 http://www.ski.org/CWTyler_lab/CWTyler/TylerPDFs/Tyler_HoropterCh1991.pdf Dynamics of Vision 7- 147

Figure 7.4.1-5 Plan view from above of the right ocular. The optic nerve exits on the nasal side of the ocular. Beyond 0.6 degrees from the central axis, the optical rays are not represented well by straight lines. The center of the foveola is about 5 degrees from the central axis, well beyond the field angle compatible with the simplified optical systems of LeGrand and others. See text. From Lotmar, 1981. 148 Processes in Biological Vision

Tyler & Julesz also define two distinct mechanisms associated with depth perception based on their definition of a cyclopean “retina.” The cyclopean retina of their terminology appears to be a conceptual retina within the cortex per Julesz, 1971. Their first mechanism employs a “binocular cross-correlator.” Their second mechanism is less well defined. They suggest it is associated with four distinct, but relatively abstract, features. Their binocular cross-correlator appears to be quite similar to the parallel processor described within the perigeniculate nucleus of this work (Section 15.6.3). Their figure 4 shows a distinct transition between their two types of depth discrimination. However, the data is quite sparse and a marginally higher value is possible. A conflict appears to exist between their figure and their text. The horizontal scale of the figure is given as area. The break point is described as near 0.5 square degrees. For such a circular area, its diameter is 0.8 degrees. In the text, the area is described by the dimensions of 0.5 by 0.5 degrees. The 0.8 degree number is in better agreement with this work. They continue the confusion in their claim that “the spatial integration area of 0.5 degrees is specific to the depth discrimination system.” They go on, “Consequently, the fine disparity system seems to have a rather restricted spatial integration area, in contrast to the coarse disparity system, for which no spatial integration limit was found up to about 100 times this area.” Based on these, and other ideas in the literature, this work will propose that two distinct mechanisms support depth perception in human vision. The first mechanism is limited to the diameter of the foveola, nominally a 1.2 degree diameter, centered on the point of fixation True stereopsis, and image fusion, is achieved within this area by means of a two-spatial-dimension parallel processor (cross-correlator). This correlator is associated with the analytical mode of vision and is found within the perigeniculate nucleus of the thalamus (Section 15.6.2).

The correlator of the PGN is more than just a single 2-D cross-correlator. It is more properly called a two-spatial- dimension associative processor. It can cross-correlate information within regions smaller than the total extent of the correlator space, as well as correlate over the entire space. The first leads to what will be discussed below as local correlation and the latter will be described as global correlation.

The two-spatial-dimension associative correlator plays a major role in version, vergence and accommodation. It is at the very heart of the precision optical system, POS. Defining the performance of a specific element within this multiple- closed-loop servomechanism, is difficult. Therefore, defining the performance of the correlator alone, with respect to one of its major tasks, is difficult. Similarly, defining its overall performance is extremely complicated. The performance of the individual loops containing the correlator is somewhat easier. Different aspects of the performance of the 2-D associative correlator of the PGN will be developed at several places within this work.

The limiting performance with respect to version precision is better than 0.1 degrees. The limiting performance with respect to vergence precision is better than one second of arc. The limiting performance with respect to accommodation precision is less well defined.

The second mechanism is associated with the peripheral retina and its projection to the LGN of the thalamus (and possibly the further direct and retro projections to the occipital lobe of the cerebral cortex, see Section 15.6.5). The spatial area associated with this mode exceeds ten degrees and probably includes all of the binocular field of view. Section 17.4.4 will show the quality of depth perception is one or more orders of magnitude poorer than for the stereopsis region.

The spatial demarcation between the first and second mechanisms is generally related to Panum’s Limit (although this limit may need further definition as suggested by Richards314). Tyler & Julesz describe a significant difference in limiting noise performance with respect to these two mechanisms. They also describe a different relationship between stimulus size and disparity for these two mechanisms. These considerations lead to the definition of coarse and fine regions of depth perception and both local and global regions of stereopsis as defined in the above figure. The fine region of depth perception relates directly to the mechanism of stereopsis and is associated with the analytical mode of vision. The coarse region of depth perception does not rely upon stereopsis and is associated with the awareness (and alarm) mode of vision. Within the region of stereopsis, the global region relates to the overall extent of the three-dimensional field that can be processed by the two-dimensional correlator of the PGN. This region is limited by the field of view of the foveola. The local region of stereopsis refers to the range of correlation associated with a specific point within the spatial range of the two-dimensional correlator. It is directly associated with the precision of depth perception achievable by the visual system. In the mathematics of stereopsis, these terms have a different connotation. The global value of the stereo-optic function describes the mean distance to the point of fixation of the foveal field of view. This function is the sum of the integrals associated with each of the individual local stereo-optic integrals associated with each element of the foveal image.

314Richards, W. (1971) Independence of Pamum’s near and far limits Am. J. Optom. vol. 48, pp 103-109 Dynamics of Vision 7- 149 The precision of depth perception falls rapidly as the selected point approaches the edge of the stereopsis space due to the limited correlation range available near the edge. This function is shown in [Figure 7.4.5-2]. Beyond the edge of the foveola, the precision of depth perception is quite low and difficult to measure. 7.4.1.1.2 Fusion & rivalry differ in foveola and peripheral vision

The phenomena of fusion and rivalry have been widely studied. However, the studies have been largely psychologically based and have not involved physiology directly. The studies have generally followed the assumption that fusion and rivalry were centered in the occipital lobe of the cerebral cortex. More recently, these studies have begun to focus on fusion and rivalry as phenomena associated with the midbrain315. Pettigrew confesses to a recent epiphany regarding his thoughts on rivalry316. His article is quite extensive and defends his new position that rivalry is not centered on the occipital lobe as he so forcefully argued for more than thirty years. However, he did not describe the physiology of the human visual system, or the portion of the retina to which his discussions relate. His figure 2 provides a histogram describing the rivalry alternation rate in Hz for a few “bipolar patients” (manic depressive disorder) and many normals. Typical values are between 0.25 and 0.8 Hz (4000 to 1,200 msec). The lower value does not differ greatly from the typical quiescent interval between saccades during reading (Chapter 19). The findings of the above investigators agree with the model of this work developed in the following sections and in Chapter 15. In this model, stereoptic fusion and stereoptic rivalry relate to the the analytical channel of vision, the foveola and the perigeniculate nucleus (PGN). Binocular fusion and rivalry occurring outside the foveola relate to the awareness channel, the peripheral retina and the lateral geniculate nuclei (LGN’s). The role of the occipital lobe in fusion and rivalry outside the analytical channel is not addressed in this work.

Of specific importance is the fact that fusion and rivalry are phenomena related to two different fundamental mechanisms associated with the foveola (analytical channel of vision) and the peripheral retina (awareness channel of vision). Designing any laboratory investigation to account for these differences is very important. In the O’Shea & Corballis study referenced above, the stimuli were applied outside the foveola. However, the authors did not express any distinct reason for adopting that test configuration. They review several extant theories regarding binocular rivalry. They also studied two subjects who had their corpus callosa severed for medical reasons. While they expected their results on split brain subjects to point clearly to rivalry within the cerebral cortex, their conclusions were that the cerebral hemispheres did not play a major role in binocular rivalry. They concurred with the suggestions of several other recent researchers that rivalry was a phenomenon of the midbrain. They noted that their conclusions, and those of others, placed their experiments outside the scope of current rivalry theories. Section 15.2.7 shows the presence of a corpus principia that was not severed in the subjects examined. That section also discusses the role of the thalamic reticular nucleus in such rivalry phenomena.

Current methods of studying rivalry can be described as traffic analysis studies. That is the term used by cryptoanalysts who initially study traffic patterns to learn the source, intermediate points and terminal points of radio traffic. In most psychophysical experiments, a stimulus is applied to the retina and a signal is sought in the cortex that relates to that stimulus. No information is collected concerning any other locations between the retina and the occipital lobe where a similar signal might be obtained. Thus, no assurance can be given that the signals were not processed at an intermediate location such as the PGN or LGN. In such cases, the occipital lobe may only act as a receiver of processed information and not as a feature extraction engine.

This work will not discuss binocular or stereoscopic rivalry at length. This is a high-level phenomenon that takes on many guises in the absence of a detailed physiological model. To begin to understand it, the literature must be sorted with respect to foveola versus non foveola participation. It must then be categorized with respect to the type of stimuli used. Since the data obtained is traffic analysis data, determining the latency of the detected signals relative to the time of stimulus is also important. The time constants of the various stages of the visual system must be accounted for. Only then can the results of various investigators be correlated. Laing & Chow have discussed some examples of rivalry on page 49 of their largely conceptual and mathematical paper317. The recent inauguration of fMRI techniques to evaluate rivalry is quite interesting. The experimental protocols reported

315O’Shea, R. & Corballis, P. (2003) Binocular rivalry in split-brain observers Invest Ophthalmol Vis Sci (in press) 316Pettigrew, J. (2001) Searching for the Switch: Neural bases for perceptual rivalry alternations. Brain Mind vol. 2, pp 85-115 317Laing, C. & Chow, C. (2002) A spiking neuron model for binocular rivalry J Comp Neurosci vol. 12, pp 39- 53 150 Processes in Biological Vision specifically omit the foveola from the stimulus pattern318,319. This may be because stimulation of the foveola does not result in any recognizable fMRI pattern at the occipital lobes. Such a conclusion is in accord with the predictions of this work. Rivalry, and the accompanying fusion phenomenon, related to the foveola occurs within the PGN/pulvinar couple, not within the LGN/occipital lobe couple. In the recent fMRI experiments, very complex, and very large dichoptic imagery were used. Checkerboard patterns, with squares one degree on a side, and sinusoidal patterns, at less than one cycle per degree, were used. These suggest the qualitative nature of signal processing in the peripheral retina. Simpler test stimuli, such as that used earlier by Hubel should be adequate. They would also provide data that was easier to interpret. However, the resulting signal to noise ratios may be too low for current fMRI techniques. The fMRI technique remains too slow to provide any latency data concerning the arrival of signals at the occipital lobe following stimulation. VEP techniques appear to remain the only non-invasive techniques able to provide latency information to augment the simple traffic analysis information provided by fMRI. O’Shea is maintaining a current bibliography on binocular rivalry at http://psy.otago.ac.nz/r_oshea/br_bibliography.html . Fusion will be discussed in Section 7.4.5 as it relates to the analytical channel of vison, the PGN and the mechanism of stereopsis. It will not be discussed from the perspective of the awareness channel, the peripheral retina and the LGN’s. 7.4.1.2 General physiology and operating modes associated with binocular vision and stereopsis

This work will only address the framework and first order mechanisms of depth perception. To accomplish the necessary analysis, a detailed model of vision is needed.

The most well recognized forms of depth perception involve:

1. Stereopsis associated with fixed objects imaged on the foveola.

2. Mechanisms associated with fixed objects imaged on the retina beyond the foveola.

3. Mechanisms associated with moving objects imaged on the peripheral retina.

4. Mechanisms not involving any of the above mechanisms; cues such as scale changes between familiar objects.

Much of the scientific literature omits discussion of the third category. However, the angle-rate information associated with such objects plays a large part in determining their perceived relative distance from other objects in the field of view. Blakemore has discussed categories 1, 2 & 4 in some detail and taken particular care to avoid cues of the fourth type320. He introduces some discussion of the neuroanatomy of the visual system. However, he did not recognize the presence of the foveola, and any elements associated with it, when discussing the performance of the visual system. He did incorporate many ideas of Ogle in the discussion.

The most important cues generally involve context, scale and angle-rate information. Clearly, a red spot in a generally uniform green field is a context worth investigating. It might be food. The following discussion will depend heavily on the material of Sections 15.3, 15.6, 15.7 and 17.8, and particularly [Figure 15.6.5-8 ]. Figure 7.4.1-6 is a greatly simplified form of that figure. It is focused primarily on the responses of the system to changes in stimuli in object space. Although eliminated from this figure, the crucial role of time delay along the various paths should not be overlooked in the following discussion. As reviewed in Section 7.3.1, the visual system supports a variety of operating modes. Some are autonomous and some

318Polonsky, A. Blake, R. Braun, J. & Heeger, D. (2000) Neuronal activity in human primary visual cortex correlates with perception during binocular rivalry Nature Neuroscience vol. 3, no. 11, pp 1153-1159 319Tong, F. & Engel, S. (2001) Interocular rivalry revealed in the human cortical blind-spot representation Nature vol. 411, pp 195-201 320Blakemore, C. (1969) Binocular depth discriminaton and the nasotemporal division J Physiol vol. 205, pp 471-497 Dynamics of Vision 7- 151 are sympathetic.

Figure 7.4.1-6 A simplified pointing schematic based on the revised Functional Diagram of human vision, ca 2002, in Section 15.6.4. It shows the TRN in its role as the central control point of vision. It also stresses the paired nature of many functional elements of the visual system. Signals from the retina are separated into three groups. Those related to the foveola pass to the Analytical Channel. Both monocular and binocular signals from the peripheral retina pass to the LGNs. The figure shows information signals passing to the right through the TRN. It also shows instructions passing down through the TRN and control signals passing to the TRN. The heavy lines from the PGN/Pulvinar couple passing through the Superior colliculus and the Oculomotor couple to the eyes represent signals within the Precision Optical System. The diagram is compatible with dual mechanisms of depth perception and shows only abstract instructions being transmitted to the SC/cerebellum for implementation as commands.

In this figure, the chiasm is shown in two places for convenience. The two nerves from the chiasm labeled binocular are meant to represent the combined output from both eyes that represent their common field of view in one field. The two nerves from the chiasm labeled monocular are meant to represent the output of the one eye that has no counterpart from the other eye. In this figure, each occipital lobe consists of two areas, the dorsal area above the calcarine fissure and the ventral area below the calcarine fissure. The resulting signals passing to the parietal cortex via the TRN consist of five distinct commissure representing the four quadrants of the peripheral retina and the one high acuity region of the foveola. The fact that the photoreceptors of the foveola are not processed within the LGN/occipital lobe couple is confirmed by the work of Polonsky, et. al. and Tong & Engel referenced above. The signals from the foveola are not passed to the occipital lobes in high acuity form. To measure responses in the occipital lobes using fMRI techniques, stimuli illuminating the peripheral retina was necessary. 7.4.1.2.1 Major features of the Functional diagram and schematic related to version and vergence 152 Processes in Biological Vision

The figure shows that the visual field is divided into three classes of object space. It also seeks to show the signal processing is organized similarly. The field imaged onto the foveola is very small. The information from this field is processed at high spatial resolution within the analytical channel by the PGN/pulvinar couple. The phenomenon of fusion is a feature of the processing carried out in the PGN/pulvinar couple. The large binocular field in object space is processed within the two portions of the awareness channel. These areas merge the images from the two retinas. However, the merging mechanism is different from the fusion mechanism of the PGN/pulvinar. Finally, the peripheral monocular areas are also delivered to the LGN/occipital couple. These areas are entirely independent of each other. The information from the monocular, binocular and foveola areas are passed to the parietal lobe. The signals from the PGN/pulvinar to the parietal lobe are by far the most complex and are represented by multiple signal lines in parallel. These lines project complex percept vectors associated with each object in the field of view of the foveola. These vectors describe many properties of the individual objects, such as position in three dimensions, orientation, and detailed shape. Whether the saliency map is stored within a specific region of the brain or is distributed within it is unknown at present. This question will probably be answered within a few years using the new imaging techniques now available. A feature not normally found in diagrams of this type is the thalamic reticular nucleus. This thin layer virtually surrounding the thalamus is being recognized as most likely the primary control mechanism of the neural system, at least in higher primates. It plays both a supervisory and control role. The supervisory role involves observing the signals being passed through it. The control role involves switching the paths that signals follow based on instructions from the higher cognitive centers. Further discussion of this role will be found in Section 15.6.

The superior colliculus, and for more complex signals the cerebellum as well, act primarily as coding and decoding centers for the brain. The decoding role is emphasized in this figure. They are primarily involved in receiving brief high level instructions and converting those into more detailed commands that can be distributed to the muscular-skeletal system for implementation. 7.4.1.2.2 Key features of the Precision Optical System

The figure highlights the elements of the POS by darkening the lines on the left side of the drawing connecting them. To achieve maximum performance in a variety of tasks supported by the POS, it does not include any elements of the cerebral cortex. The delays related to the projection of signals to these parts of the CNS are inordinately large.

The POS includes the photoreceptor elements of the foveola connected directly to the PGN/pulvinar couple without significant processing within the neural circuitry (stage 2) of the retina. The PGN/pulvinar provide pointing instructions directly to the superior colliculus for decoding into pointing commands. These pointing commands are delivered directly to the oculomotor nuclei. The nuclei in turn deliver them directly to the appropriate oculomotor muscles. This signal flow provides the minimum possible delay within the POS servoloops. When examining reading, and other highly stylized visual functions, it can be shown that even these minimal delays are a hindrance. To reduce the delay further, a set of default actions is programmed into the repetitive functions related to these tasks (Section 19.8.2).

The PGN contains a unique two-dimensional correlative cross-correlator used in many feature extraction and identification operations. This correlator provides raw signals related to the primitive features of images projected onto the foveola. After passing to the pulvinar, these primitives are interpreted (in real time) into signatures previously associated with recognizable objects through training (experience). The presence of primitives that cannot be recognized causes the POS to enter a default training loop. This loop is much slower than the normal process loop and frequently involves the cerebral cortex in the training exercise. 7.4.1.2.3 Signal paths associated with version and vergence

The figures of Section 7.3.1 show the signal paths and plant accepting version and vergence inputs from the neurological sections and operating modes associated with pointing. The gross instructions intended for purposes of pointing the lines of fixation are generally represented in conjugate form. The pointing system accepts these instructions and generates the appropriate commands in the same conjugate form. These commands cause the eyes to rotate in the same direction. The degree of rotation can be quite large. The instructions intended for purposes of converging the lines of fixation at a given distance from the subject are generally represented in disjunctive form. The pointing system accepts these instructions and generates the appropriate commands in the same disjunctive form. These commands cause the eyes to rotate by small amounts in opposite direction. What is not shown in the figures of Section 7.3.1 is the extensive use of memory in supporting these functions. The previous figure and Section 15.6.5 develop the important role of memory associated with the LGN, the PGN and the Dynamics of Vision 7- 153 superior colliculus. The importance is so great that these individual elements are redefined within individual couples containing a feature extraction element and a memory element. Significant elements of very short, short and long term memory are involved in the operation of the version, vergence and accommodation functional overlays. The saliency map and much of the memory associated with the pulvinar, superior colliculus and cerebellum are obviously long term memories. However, the pulvinar and possibly the superior colliculus also appear to incorporate short term memory elements associated with assembling interps and percepts.

7.4.1.3 Forms, cues and protocols of depth perception

Depth perception has been studied intensely for a very long time. The literature is immense. It includes considerable minutiae collected under less than precise circumstances and often difficult to place in proper context. Howard321 and Howard & Rogers322 have recently provided a massive compendium on the subject of depth perception. However, their use of the word mechanisms is quite different than here. Their mechanisms are largely conceptual and based on non- invasive observations of a very complex “black box.” In this work, the term is used only when describing much more detailed operations based on physiology and electrophysiology. It is difficult to separate the forms of depth perception, the cues found in depth perception and the test protocols needed to describe various phenomena distinctly because of their complexity and intertwined relationships. Several authors have attempted to organize this group of materials. Howard has provided a framework separating cues based on monocular and binocular vision. Based on this work, a better framework contains three classes of depth perception based on the field of view related to monocular, binocular and foveola vision. Noting that this three level framework is hierarchal and not exclusionary is important. Most monocular aspects of depth perception also occur in binocular vision. Similarly those cues found in monocular and binocular vision are not constrained from being present in foveola vision. Noting the great loss in spatial performance with angle from the line of fixation due to the wide angle lens group is important. This is a particular problem in human eyes.

Figure 7.4.1-7 provides an alternate to the figure of Howard (vol. 1, pg 5) using as much of his terminology as practical. However, the term stereopsis is used more restrictively here. Stereopsis is the term used to describe the perception of depth within a scene in object space projected onto both foveola (and only the foveola) simultaneously. This scene is perceived by the two eyes as one. It is said to be fused. The perceived depth of various individual objects in the scene is represented veridically. An object represented veridically is perceived to have a depth relative to a reference surface in the field that is proportional to its difference in vergence relative to the same surface. Objects in the scene imaged onto the remainder of each retina from areas observed binocularly are not observed veridically. In fact, the quality of the perception of depth is typically two orders of magnitude or more, lower than for stereopsis within the foveola. It is so poor it is usually described only as qualitative depth perception. Because of this feature, this work describes the images obtained binocularly as being merged within the neural system rather than being fused. Finally, the monocular fields associated with each eye can evaluate a variety of cues, but only at a qualitative level. This level is even lower than for the binocular field. The loss in performance is due to the further decrease in performance of the optical system at angles greater than fifty degrees from the axis.

The various types of cues are discussed extensively in Howard and Howard & Rogers. See pages 3-5 and section 4.6.5 in Howard and chapters 24-29 in Howard & Rogers. Their presentations are based primarily on psychological testing and analysis. A single framework encompassing all of the terms used to define cues could not be found in this material. As a result, the lower classifications of cues are not mutually exclusive, appear to overlap significantly, and appear arbitrary in their classification. They will not be discussed in depth here.

321Howard, I. (2002) Seeing in Depth: Volume 1, Basic Mechanisms. Toronto, Canada: I. Porteous 322Howard, I. & Rogers, B. (2002) Seeing in Depth: Volume 2, Depth Perception. Toronto, Canada: I. Porteous 154 Processes in Biological Vision

Figure 7.4.1-7 A Re-classification of cues found in depth perception from Howard. Veridical depth perception is at least two orders of magnitude more precise than qualitative depth perception. The subtitles show considerable overlap due to their largely conceptual definitions. See text. Compare with Howard, 2002.

Personal observation is that qualitative depth perception relies as much, or more, on cues than it does on the processing of local vergence values associated with an object in the image. Improvements in the classification of cues could lead to considerable improvement in the protocols used to evaluate the response to these cues. 7.4.1.4 Overview of theories of binocular and stereo-optic vision in the literature

It is difficult to discuss other theories of binocular and stereo-optic vision in the absence of a firm combined physiological and psychophysical basis. As a result, most of the discussion of the literature will be delayed until Section 7.4.5.4.1. Tyler has provided a brief discussion of the various theories of binocular and stereoptic vision found in the literature ca 1983 in Schor & Ciuffreda (S&C)323. While he does discuss binocular (S&C pg 216), stereoptic (S&C pg 244) and “other” (S&C pg 250) theories of depth perception, he does not differentiate clearly between the binocular and stereoptic modes with respect to retinal fields of view. He briefly discusses the previous conceptual theories of binocular vision. Three of these were developed in the 19th Century and one dates from 1935. Tyler noted the exploratory nature of the various theories of stereopsis and fusion (pg 245, first line). He pointed out that “great plausibility” and “explanatory range” are predominant features in the published analyses of stereopsis. These features are shared with the lack of definition between binocular depth perception and stereopsis within the foveola. Tyler makes four points related to his general discussion of stereopsis.

323Tyler, C. (1983) Sensory processing of binocular disparity, Chapter 7 in Schor, C. & Ciuffreda, K. ed. Vergence Ey Movements, London: Butterworths Dynamics of Vision 7- 155 1. Neurophysiological experiments point strongly to the existence of special-purpose mechanism supporting this function. 2. There is substantial psychophysical evidence for the existence of these special-purpose mechanisms. 3. Special-purpose mechanisms are an efficient method of network operation. 4. Current models of non-stereoscopic cortical function also strongly point to the existence of special-purpose mechanisms. The conceptually defined series of special-purpose mechanisms defined by Tyler, in his figure 7.2.2, appear to correspond to the feature extraction engines of this work. Their output is stored in the saliency map developed in Section 15.2.2. The model portrayed in that figure and attributed to Nelson is entirely open loop. Such a model cannot explain the origins of the neural signals found in the visual system. Nor does an open loop analysis lead to the mechanism of feature extraction used in the visual system.

Tyler has recently prepared a new chapter on the various aspects of stereovision aimed at a clinical audience324. 7.4.1.4.1 Recent papers on stereo vision from psychology laboratories

The psychology community appears to have struggled to avoid physical facts, particularly with regard to the physiology of the visual modality in its entirety.

Schwartz provided a very learned but nearly incomprehensible volume in 1994 focusing entirely on the geometry of the stationary eyes (no tremor or microsaccades)325. The entire volume was his interpretation of the thesis of Berkeley, an early 18th Century philosopher of wide scope but very little depth with regard to vision. Relying upon the static geometry of the eyes and the eyes in conjunction with multiple targets at different positions in the external field of view, Berkeley hypothesized that the human could not determine the distance to an object immediately326, “It is, I think, agreed by all that distance, of itself and immediately, cannot be seen. For distance being a Line directed end-wise to the eye, it projects only one point in the fund of the eye, which point remains invariably the same, whether the distance be longer or shorter.” In subsequent assertions he asserts that the system can estimate distance to an object by determining the vergence angle of the two eyes. He did not consider the challenge of determining the distance to multiple objects in object space during the time the vergence angles of the two eyes remains constant. While a rational position for the 18th Century, the introduction of tremor and the computing power of the brain into the discussion provides an entirely different hypothesis that is more viable (Sections 7.4.5 & 7.4.6). No physiologist or neurologist participated in or was cited in the preparation of Schwartz’s book.

Pizlo and colleagues have presented a variety of papers from a purely psychology laboratory perspective. These have generally ignored completely the physiology of the visual modality and the immense importance of memory in human vision.

Pizlo presented a book in 2008 based entirely on the concept of “recovering” 3D imagery from a single 2D image projected onto a single retina327. While possessing two PhD’s in engineering, he notes in the introduction, “I did not include a treatment of the neuroanatomy or neurophysiology of shape perception.” That was followed by “The text concentrates on the discussion of the main concepts; technical material has been reduced to a minimum. This makes it possible to tell the ‘story of shape’ without interruption.” His six goals stated farther down page xii appear difficult to achieve based on the above constraints. The opening paragraph on page xiii seems highly implausible based on the above framework. While Pizlo did cite one paper by McKee (from 1990), he did not cite any of the work of Tyler. He distinguished between recovering information about the 3D image from “reconstructing” the 3D image in perception space. His discussion was entirely geometric and employed “symmetrical” objects. He did not introduce any operational features of the visual modality other than to note the hyperacuity encountered in 3D vision compared to the expected acuity based on the physical size and spacing of the photoreceptors of the retina (and no field lens due to the curvature of the outer retinal surface). His figure 3.13 is incompatible with the hyperacuity he encountered. He does explore the

324Tyler, C. (2004) Binocular vision In Tasman, W. & Jaeger, E. eds Duane’s Foundations of Clinical Ophthalmology Chapter 24 325Schwartz, R. (1994) Vision: Variations on some Berkeleian Themes. Oxford: Blackwell 326Berkeley, George (1709) An Essay Towards a New Theory of Vision http://psychclassics.yorku.ca/Berkeley/vision.htm 327Pizlo, Z. (2008) 3D Shapes. Cambridge, MA: MIT Press 156 Processes in Biological Vision work of Gibson and the work of Marr but primarily in conceptual language. Pizlo did not address the horopter of human vision, any aspect of retinal image motion due to tremor, or the photoreceptors as change detectors. Interestingly, he did not address the high percentage of people who are essentially naive and untrained with respect to stereopsis. His reconstructions relied upon a Simplicity Principle (page 37). “The Simplicity Principle is incorporated in perceptual mechanisms in order to make up for information lost due to (i) the projection from the distal to the proximal stimulus and (ii) the presence of noise in the visual system.” He explains this principle further on page 171, “According to this (Gestalt) principle, the shape most likely to be perceived is the simplest possible interpretation of the ritinal shape produced by a 3D object ‘out there’.” There is no guarantee (or even high probability) such a shape is veridical with the actual distal shape. Even to support the simplicity principle, it was necessary to adopt the “figure-ground organization” of page 27. Figure-ground organization “refers to the fact that closed contours establish special closed regions in the percept that correspond to objects in the visual scene. These regions which were called ‘figures’ are perceived as lying in front of the ‘background.’” A corollary to this figure-ground organization is that the contours always belong to the objects (figures), never to the background.” An additional concept introduced by Pizlo involves “shape constancy” (page 3). “Formally, ‘shape constancy’ refers to the fact that the perception of the shape of a given object remains constant despite changes in the shape of the objects’ retinal image.” Shape constancy in this definition is strangely unassociated with the learning and memory functions of the human neural system that contributes so greatly to the overall perception process. His assertion that “Shape constancy has profound significance because the perceived shape of a given object is veridical (the way it is ‘out there’) despite the fact that its shape on the retina . . .has changed.” is undoubtedly true but the number of hidden variables and mechanistic processes within the overall shape constancy concept is quite large.

Pizlo did address the difference in definitions and standards encompassing the words theory, model and explanation over the years (his section 3.3.1).

Li, Pizlo & Steinman328 presented a paper in 2009 that also focused on the projection of a 3D image onto a single retina and attempted to show how a recovered 3D perception could be obtained through purely geometric computations. They stress the veridicity of the results in their assertion that the subject perceived the object as “out there” in space. Veridicity is commonly found in the psychology literature as a synonym for veracity when applied to a perceived visual image. Their computational model contains no elements of the physiology or electrophysiology of the visual modality. As in the book by Pizlo, their recovery of a veridical perception relied upon a simplicity principle.

Pizlo, Sawanda et al329. presented another paper, labeled a mini review, in 2010 that was primarily a bridge to their 2011 paper, Li, Sawada, et al330. This paper adopted a different thesis, based on the use of both eyes, and described two major experiments. The first employed physical objects. The second employed images prepared on a 3D class flat screen HD television monitor of large size. As a result, the subject actually scanned across a wide field in order to make judgements about relative distances to various points of light or object features. These investigators again noted the hyperacuity of their subjects compared to the expected resolution of a fixed retinal matrix. The importance of this hyperacuity is discussed in the following section based on the findings of Yarbus.

They noted that all of their subjects had “normal vision” and that only one of their subjects was naive about the purpose of the tests. There was no discussion of, or data confirming, what they considered normal vision. To use knowledgable subjects in a series of experiments so dependent on human memory seems unusual. Even the naive individual had a wealth of experience in 3D imaging defined by his age.

Their experiments involved dot separations on the order of three degrees. This value suggests their experiments were outside the range of the normal human horopter for optimum disparity measurements and/or the subjects were allowed to scan a large flat field in order to perceive the dots separately. Their summary and conclusions discuss multiple differences between their results and those of others based on similar experiments. Neither set of experiments were achieved using adequate protocols based on the known physiology of the human visual system. The above works suggest the deductive nature of the current vision work in the psychology community related to stereopsis. While the concepts discussed above can be described as rational they appear largely irrelevant; they lack

328Li, Y. Pizlo, Z. & Steinman, R. (2009) A computational model that recovers the 3D shape of an object from a single 2D retinal representation Vision Res vol 49, pp 979–991 329Pizlo, Z. Li, Y. et al. (2010) New approach to the perception of 3D shape based on veridicality, complexity, symmetry and volume Vision Res vol 50, pp 1–11 330Li, Y. Sawada, T. et al. (2011) A Bayesian model of binocular perception of 3D mirror symmetrical polyhedra J Vision vol 11(4), pp 1–20 Dynamics of Vision 7- 157 any foundation in the physiology of vision available from more inductive analyses. They also fail to make a distinction between stereopsis based; C entirely on relative motion in the peripheral field of view, C entirely on parallax–based vergence calculations and C higher level neural calculation based on time differences derived from tremor. The latter form of stereographic vision is defined as true stereopsis in Section 7.4.5.1. McKee makes specific note of this difference in his 1983 paper discussed in that section. Stereopsis provides the high precision disparity values and disparity range found only within the 1.2 degree diameter field associated with the foveola. Harris & Jenkin edited a book based on a workshop in Toronto in 2011331. The organization of the work is well detailed on page 5. Two summary figures (fig 1-2 & 1-3, are provided with little supporting detail. They are reproduced below. While the work is an excellent source of examples of binocular vision and stereopsis, it is bereft of physiological details of the biological systems of vision. The attendees were primarily psychologists and those working in machine vision. Many geometric drawings are used to explain binocular vision. However, none of the papers reference either Yarbus or Ditchburrn and those investigators do not appear in the extensive index of cited authors. An important indication of the maturity of the field as viewed by Harris & Jenkin is given on page 1, “the problem of perceiving 3D shape and layout is a classic example of an ill–posed and under constrained inverse problem.” After making these assertions, they carefully caveat their individual statements concerning them. Their position is clearly the analog of an inadequately educated audience in front of a magician. In the general case, biological 3D vision performs adequately within the normal environment. If the environment is abnormal; poor lighting, carefully obscured placement of mirrors, etc., the audience may fail to perceive the obvious.

The papers in Harris & Jenkin generally fail to address realistically the physiology that the authors claim to be familiar with. They routinely rely upon the paraxial simplification of the optics of the eye introduced by LeGrand and by Gulstrand (Section 2.4 and particularly Figure 2.4.1-4) without realizing this simplification only applies for scene elements occurring at angles of less than 0.6 degrees from the point of fixation. The simplification depends on the approximation, sin(x) . x. They also fail to recognize the significant spatial non-linearity of the projection of the external field onto the curved retinas of each eye (Section 2.4.2 & Section 15.5.3). Further, they fail to recognize the major spatial transform introduced between the retinas and the striate cortex of the occipital lobe (Section 15.6.5). Most of these papers must be considered inadequate without a more robust description of the physiology they claim they are attempting to emulate. Section xxx describes the net result of the spatial transforms between the exterior field and the striate cortex of the monkey. A similar transformation appears to be intrinsic to virtually all mammals,

Jenkins & Harris (page 1) describe the inverse problem as, “how to build a three–dimensional representation from such two–dimensional patterns of light impinging on our retinas or the cameras of a robot.” Under constrained, the term they employ, is clearly associated with a lack of knowledge concerning all of the variables, and neural mechanisms, associated with the problem. From this conceptualization of the problem, they proceed to a discussion of how difficult the problem is. The text ignores any fine motion of the oculars associated with tremor, thereby implicitly ignoring any fine motion between the image projected on the retinas and the retinas themselves. Figure 7.4.1-8, their figure 1.3, illustrates this implicit assumption. No data points are provided and the area of most significance in human vision, distances on the order of 0.5 meter, has been marginalized. The terms are not defined in this introductory figure in Harris & Jenkin. The maximum just-discriminable depth threshold is recognized to be highly dependent on lighting conditions and the size and surface character of the individual figures in object space. Relative thresholds of less than 0.005 (fraction of mean disparity distance representing the minimum change in disparity recognizable) are seldom reported.

331Harris, L. & Jenkin, M. (2011) Vision in 3D Environments. NY: Cambridge Univ Press 158 Processes in Biological Vision

Figure 7.4.1-8 The just–discriminable depth threshold (detectable difference in depth as a fraction of viewing distance) for information provided by various cues. The area under each curve indicates when that information source is suprathreshold. Some annotation added; see text. From Cutting & Vishton, 1995, as redrawn by Harris & Jenkin, 2012.

Figure 7.4.1-9 has been significantly expanded from their figure 1.2. It now includes material from their psychology school on the right and an alternate description of 3D information extraction on the left based on the physiology of the visual portion of the neural system. This physiology model is addressed extensively in this work. It is important to note this psychology school focuses its attention initially on a single monocular representation of 3D object space. Thus, the disparity between the two retinal images (a “hidden” variable) and disparity calculations using disparity are largely outside of the realm of psychology–based hypotheses of 3D information extraction. Figure 5.11 in Pizlo (2008) diagrams the conventional thinking associated with this school. Only later is the disparity between the two 2D images projected onto the retinas of the eyes during 3D operation, considered: see Chapter 2 in Harris & Jenkin (2011).

There is no physiological evidence for stage 4 signal manipulation (information extraction, such as identifying figures in a figure–ground conceptuallization) occurring within the stage 1 and 2 engines of the retina. Dynamics of Vision 7- 159

Figure 7.4.1-9 An overview of 3D Information Extraction from two schools of thought. There is a large amount of psychology literature addressing 3D information extraction from a 2D image presented monocularly. It does not address the disparity between the two images presented in the binocular situation. The physiology literature is smaller but more recent. It identifies the engines of disparity calculations and oculomotor optimization in detail. See text. Psychology School portion of figure from Harris & Jenkin, 2011. 160 Processes in Biological Vision

Pizlo describes a number of “principles” developed within this school to explain how 3D images are analyzed by a monocular view of the 3D object space (Section 7.xxx). This psychology school has identified a large number of cues within a static 2D image that can be used to infer the characteristics of a 3D scene, as identified by Harris & Jenkin in this figure.. These cues cannot be relied upon however; reliance upon them leads to many illusions332 and misunderstandings that provide professional magicians with an excellent living. These cues are heavily dependent on learning and memory (as suggested by the comments of Pizlo, page 45, relating to the maturation of the visual system in a young child who was maintained in a controlled environment until about the age of 19 months when that level of control became impractical). Another psychology school does address the subject of disparity between the two images presented to two retinas binocularly. However, its analyses largely ignore the physiology of the visual system. The analyses are fundamentally geometric optics. It has also largely restricted itself to analyses based on the narrow-field (less than one degree) representation of vision implicit in its use of the thin lens model of the visual system. This has resulted in a poor understanding of the location of the loci of the theoretical Vieth-Muller circle. A third psychology school suggests it is providing a physiological model, but is actually proposing numerical models based on a linearity assumption and complex correlation integrals outside the realm of neural computation333. Qian & Anderson note regarding their 1994 model, “while the spatial receptive fields of cortical simple cells can be modeled accurately by Gabor functions the temporal responses of the cells are clearly not Gabor like. The integrated model we developed using spatiotemporal Gabor filters is therefore not completely physiologically realistic.” They then present several complex double integrals purported to represent the amplitude of the signal at “simple cells” found in the occipital cortex (without any discussion of the neural paths leading to these simple cells. See Section 7.4.5.1.1.

Incorporating the physiology of the visual modality, and particularly the electro–physiology of the modality, leads to a much different model of 3D vision than that achieved to date based on the various psychology schools. The left portion of the figure describes the computational mechanism leading to cognition of 3D imagery. Upon recognizing that the photoreceptors act as change detectors, and not imaging detectors, the only other pertinent stage 1 feature is the delay as a function of illumination associated with the complete P/D equation (Section 7.2.4). This delay, normally present and equal in both eyes, is not particularly important except potentially in the case of the Pulfrich Effect due to the stimulus intensity reduction in one eye. The stage 2 signal processing is of limited concern, except to recognize that the O, P & Q chrominance difference signals and the R-channel luminance summation signals are created in stage 2. These signals are passed independently to the initial engines of the thalamus (sometimes labeled morphologically the pretectum) where they are processed separately in the lateral geniculate nuclei (LGN)and in the perigeniculate nuclei (PGN) in support of foveal (qualitative) binocular vision and precision (stereoptic) binocular vision respectively. All of the processing between the disparity calculations and the information extraction operations occur within stage 4 signal manipulation.

It is in the LGN that the disparity calculations are most obviously performed. The R–, O–, P– & Q– channel signals from the two left ocular fields and the two right ocular fields are processed pair–wise. The pair of R–channels are aligned with the O–, P– and Q– channels so that any calculations can be readily compared between the layers or transferred to the other layers (Section xxx). It is at this stage that the coarse level mean disparity associated with a 3D object space is initially determined. This mean disparity signal is used to optimize the ocular parameters associated with coarse convergence, coarse accommodation and myosis (aperture control if appropriate). Subsequent to this coarse disparity calculation and coarse oculomotor optimization, the PGN performs a similar calculation of precision mean disparity using the signals from the foveola. These calculations provide a fine or vernier signal to the ocularmotor optimization mechanism. To minimize the analog calculation load, the oculomotor engines employ preset parameters, selected from the quiescent condition, or from a previous (a matter of tens of milliseconds) signal manipulation cycle. Upon optimization of the ocular parameters (in an open loop mode), the calculation of the final binocular disparity parameters and the differential disparity parameters are calculated for each significant element (figure) in the 3D field.(within an accuracy determined by the signal to noise ratio of the individual channels). With these values in hand,

332Adamovic, J.(2004– ) Optical Illusions http://brainden.com/optical-illusions.htm 333Qian, N. & Andersen, R. (1997) A physiological model for motion stereo integration and a unified explanation of the Pulfrich like phenomena Vision Res vol 37(12), pp 1683–1698 Dynamics of Vision 7- 161 the LGN and PGN are able to fuse the various O–, P–, Q– and R– images and attach lateral and depth disparity values (as tags) to the signals describing the elements in the fused images. It is at this point that the visual information can be considered cyclopean (but with both lateral and depth disparity information attached to the amplitude information associated with a figure). The fused and tagged signals are then passed to the stage 4 engines responsible for further information extraction and delivery to the saliency map for access by the stage 5 cognition engines. The signals from the LGN are passed to the area 17, a.k.a. V1, of the occipital lobe (over relatively long and time consuming neural paths) while the signals from the PGN are passed to the adjacent pulvinar.

As confirmed in the conclusions of Cumming & Parker334, the process of binocular fusion has already occurred by the time the cyclopean information reaches the occipital cortex (V1) from the LGN. They continued their analyses of V1 signals without reaching a significantly different conclusion335. Subsequently, they transitioned to random dot pattern analysis and invoked and described in some detail the energy model of Ohzawa et al. (1990)336. They continued to conclud that much more work was required to demonstrate a role for V1 in binocular vision information extraction. They did not provide any schematic of how the signals from the retinas propagated to V1. Cumming & Parker did not arrive at any value for the range of qualitative (fovea–based) depth perception or precision (foveola–based) stereopsis based on their studies. Results in this area will be discussed in greater detail when addressing the work of Allison & Howard in Section 7.4.7.4. Information extraction at this point in stage 4 operations involves much more than just 3D perception. The information still exhibits a degree of scene-optic organization, although in a cyclopean format. Additional information is extracted in both V1 and in the pulvinar relative to the shading, texture, etc. are cues that are transferred to appropriate memory sites for future comparisons with other figures in 3D object space images (as suggested by the arrow pointing to the list of static cues on the right). Note, the term familiarity at the extreme lower right is a synonym for prior learning and memorization and not a cue in itself. It should be noted that the quality of these cues extracted in the pulvinar are considerably better than those extracted in V1. The thalamic reticular nucleus (TRN) compares the signals returned from the pulvinar and the occipital lobe and transfers the best quality information to the saliency map associated with the parietal lobe.

In many cases, the information extracted in the pulvinar is transferred to the saliency map of the parietal lobe before the original information even arrives at the occipital lobe for processing (a well documented oddity not explained by earlier discussions of visual modality operation, Section xxx).

It is proposed that the inductive process of the physiological school, relying upon both the physiological and electro–physiological databases to develop the details of the signaling chain leads to stronger null hypotheses and protocols than the deductive approach of the psychology school based primarily on observation and behavioral studies. As noted, the focus on a monocular presentation, essentially eliminates disparity from any discussion of binocular vision and stereopsis.

The fact that the two images projected on the retinas contain different information, particularly regarding disparity, is readily seen by observing a 3D HDTV presentation without using the required stereo-glasses. The depth disparity information is clearly observed to be represented by differential lateral disparities. Interestingly, chapter 8, by Allison & Howard, introduces the concept of the change in disparity but only as applied to the “cyclopean domain.” They define an interocular velocity difference (IOVD) signal as a change in disparity (CD) related to a single object moving in object space. They then introduced a binocular presentation, Figure 7.4.1-10, of considerable utility to the discussions below. The information was introduced dichoptically, separate sources for each eye using 45 degree semi–silvered mirrors (a Wheatstone stereoscope). There was no discussion of the linearity of the presentations created using a pair of Tektronix 608 oscilloscopes. The nominal diameter of the foveola associated with precision stereopsis and the diameter of the fovea associated with qualitative binocular vision are shown the the dashed

334Cumming, B. & Parker, A. (2000) Local disparity not perceived depth is signaled by binocular Neurons in cortical area V1 of the macaque J Neurosci vol 20(12), pp 4758–4767 335Prince, S. Cumming, B. & Parker, A. (2002) Range and mechanism of encoding of horizontal disparity in macaque V1 J Neurophysiol vol 87, pp 209–221 336Prince, S. Pointon, A. Cumming, B. & Parker, A. (2002) Quantitative analysis of the responses of V1 neurons to horizontal disparity in dynamic random-dot stereograms J Neurophysiol vol 87, pp 191-208 162 Processes in Biological Vision

overlays. Obviously, this protocol was not designed taking the physiological parameters of the retina in mind. Specifically the area of maximum stereopsis was not included in the test protocol except for purposes of maintaining vergence and convergence. The experiments with this presentation are discussed in greater detail in Section 7.4.7.4.

Figure 7.4.1-10 Basic stimulus arrangement of Allison & Howard. The observer fixated the binocularly visible dot in the center of the stimulus and monitored fixation via adjacent nonius lines (the left eye sees one line and the right eye sees the other). In one half of the stimulus (top in this example) was the test display, moving in opposite directions in the two eyes at a given frequency and vlocity. For experiments using matching tasks, the oscillation of the correlated comparison image (bottom in this example) was adjusted by the observer to match the motion in depth of the test stimulus. The dashed circles have been added to signify the size of the foveola (1.2 degree circle) and fovea (8.7 degree circle) in this non rectilinear representation. Modified from Allison & Howard, 2012.

[xxx put words to this display Be sure it matches the caption of figure 8.2 ] The basic protocol is to present a stationary set of random dots to each eye in the upper half of the image whle maintaining fixation on a single point and simultaneously presenting a set of correlated dots to each eye in the lower portion of the field of view. The sets of dots projected to each eye in the lower portion can be moved about using a dichoptic optical system. When correlated, the “opposed to-and-fro motion,” or lateral motions of the images projected into the two eyes are perceived as changes in depth of the merged dot sets.

Many experiments were performed using variants of this display. It is useful to note the expanse of this display. The horizontal width indicates no preference was given to the foveola as the region concentrating on stereopsis. The fact that only the fixation point and the two nonius lines were imaged on the foveola, suggests the experiments are not useful Dynamics of Vision 7- 163 for evaluating precision binocular vision (stereopsis). The individual experiments will be addressed later in Section 7.4.5 on stereopsis.

Recently, Pizlo called for a conference in 2013 to “help move our field forward, because substantial progress in any field is not possible without formal theories.” He called for a “focus on the role of mechanisms (vision algorithms)” in stereopsis. The work of the psychology community would be better served by protocols more firmly based on the physiology of the human visual modality as developed in Sections 7.4.5, 7.4.6 & 7.4.7 below. Recognizing the role of the photoreceptors as edge detectors (Section 15.1.4.6) and the role of the perigeniculate nucleus/pulvinar couple (Section 15.6.3.5.1) in creating the 3D perception within the brain would also be useful. An explanation of hyperacuity is readily available based on these roles.

7.4.1.4.2 The Yarbus Test as a critical hurdle for theories of fusion and stereoptic vision

An important criterion for any description or caricature related to vision is whether it passes the Yarbus Test. Yarbus, Ditchburn and others have shown that the visual system of chordates (demonstrated with humans) is blind in the absence of motion between the retina and the scene in object space (Section 7.3.3.5.4). Such motion is generally provided by the mechanism of tremor, the microradian amplitude angular vibrations produced by the eye muscles.

The requirement for such motion is due to the adaptation amplifier found in each photoreceptor cell. These amplifiers have a zero at zero frequency in their frequency response. That is, they cannot transmit a signal that is not changing with time.

The result of the circuitry within the adaptation amplifiers is that the retina is not an imaging device. It is a change detector. Thus, any light imaged on the retina must be either moving spatially or changing temporally if a signal is to be generated at the output of the individual photoreceptor cells. The Yarbus Test incorporates these requirements.

The theories of fusion and stereoptic vision discussed in the above section do not pass the Yarbus Test and can generally be disregarded for purposes of research. They will be examined in greater detail in Section 7.4.5. 7.4.1.5 Alternate interpretations of the horopter

The following material will be brief. The literature is so convoluted in this area that only a significant effort could produce a useful discussion here. Any serious reader of the following material should first read the history and theory of the horopter of Shipley & Rawlings337. They claim “that investigators in this area have often been unclear as to which aspects of their analyses were immutable physiological, contingent physiological, purely hypothetical and analytical, or empirical.” [emphasis in original] It appears this is just as true today. As an example, the 2002 text by Howard & Rogers, editors, titled Seeing in Depth, contains entire chapters based on the so-called paraxial approximation, or Gaussian optics of the most elementary form338. The authors of those chapters fail to appreciate that the use of this approximation is limited to scenes within 1.2 degrees of the point of fixation. The simple form used also fails to incorporate Snell’s Law of optics. That law accounts for the differences in the optical index of refraction on the two sides of a lens. As a result, most of the figures in those chapters are only useful for pedagogical purposes at the undergraduate level. Much of the earlier material in the psychological literature suffers from these same shortcomings.

Ogle has presented a major discussion of the horopter as a phenomenon and as an instrument339. Shipley & Rawlings go on to focus on another problem. “We see here the very core of the problem. Unlike the other senses, it was felt that vision “exteriorizes: projects sensations outside of the self.” The fundamental geometry of vergence discussions is based on the archaic Keplerian model that assumes that optical rays emanate from the eyes. The Keplerian model completely ignores the laws of physical, and physiological, optics. Schreiber, Tweed & Schor have provided an interesting analysis of the horopter that is entirely mathematical (and relies

337Shipley, T. & Rawlings, S. (1970) The Nonius horopter– I. History and theory and II. An experimental report Vision Res vol. 10, pp 1225-1262 & 1263-199 338Howard, I. & Rogers, B. (2002) Seeing in Depth. Toronto, Canada: I Porteous, Chapters 20, 24 & 25 339Ogle, K. (1950) Op. Cit. pp 14-49 164 Processes in Biological Vision upon an approximation)340. Their field angles appear to be much larger than those associated with the fovea and the region of high quality stereopsis of primary interest in vision. An additional shortcoming in the psychological literature is the complete absence of any material reflecting findings in the physiological literature of the last forty years. Because of the paucity of physiological models in the psychological literature, investigators have adopted the policy of introducing a great variety of less than precise unique definitions relating to vergence, stereopsis and fusion.

7.4.1.5.1 The horopter as a physiological characteristic EXPAND xxx The concept of the horopter is an ensemble of points in space where each point subtends an equal angle at the eyes. This definition includes the Vieth-Muller circle and a vertical line through the point of fixation. The physiological horopter is a largely conceptual, and difficult to portray, characteristic. Most portrayals of the horopter are greatly simplified geometrical caricatures. They ignore the physiological characteristics of the eyes and the requirements of physical optics. The discussion of Tyler & Scott is one of the best for pedagogical purposes341. Howard & Rogers have also provided a readable description of the horopter342. A more theoretical discussion appears in Solomons343. While Tyler & Scott open with a definition of a horopter, their second paragraph discusses a range of different horopters based on details related to the definition. Their opening definition is a geometry “in which corresponding points on the two retinas are defined as being at the same horizontal and vertical (or monocular visual direction) from the center of the fovea of each eye. (The rotation of the eyes must be taken into account, but may be considered identical when the eyes are in the primary, straight-ahead position.)” This definition is based on the Gaussian (or paraxial) assumption in optics. It completely disregards the actual conditions of physical optics (Section 2.4).

Howard & Rogers probably give the clearest definition of the horopter. They first define the point horopter as “the locus of points in space that project images onto corresponding points in the two retinas.” This definition is compatible with the actual optics of the eyes but is incomplete. It should be extended to require the eyes to be fixated on a point in the median plane of the head. They go on to subdivide this horopter into a horizontal (or longitudinal) horopter and a vertical horopter. The horizontal horopter is limited to those points in space lying in the horizontal plane of regard. The vertical horopter is limited to those points in space lying in the median plane of the head.

The caricatures defining the horopter invariably show various principal rays as straight lines between a point in object space and a point on the retina. Such a case is patently wrong, except within less than one degree of the line of fixation. However, the eyes are sufficiently symmetrical and the points of correspondence between the two retinas can still be defined. [As an aside, there is evidence that the retinal arrangement of photoreceptors does not exhibit mirror symmetry as in most of the body of a bilateral animal. It appears the mosaics of photoreceptors in the two retinas can be overlaid at a very fine level of detail.] The caricatures also invariably fail to show the resolution capability of the physiological optical system. At more than one degree from the line of fixation, the resolution capability of each eye declines rapidly. Additionally, the caricatures fail to delimit the portion of the Vieth-Muller circle describing the binocular field of view. Beyond about 51 degrees from the fixation point, the circle is irrelevant. Finally, as Tyler & Scott alluded to in the above quotation, the conventional point horopter only applies to points of fixation along the perpendicular bisector of a line connecting the 1st principal points of the two eyes. As the eyes rotate, new Vieth-Muller circles must be drawn With the above understandings, Figure 7.4.1-11 illustrates the geometry of a simplified longitudinal horopter useful for pedagogical purposes. Being an ensemble of points, it is formally known as the point horopter in the horizontal plane of regard. The figure does not show the rays from the point of fixation passing through the lens to the retina. They are not straight extensions of the rays in object space except within about one degree of the point of fixation.

340Schreiber, K. Tweed, D. & Schor, C. (2006) The extended horopter: quantifying retinal correspondence across changes of 3D eye position J Vision (on-line) vol 6, pp 64-74 341Tyler, C. & Scott, A. (1979) Binocular vision, Chapter 22 in Records, R. ed. Physiology of the Human Eye and Visual System NY: Harper & Row. 342Howard, I. & Rogers, B. (2002) Seeing in Depth, Volume 2, Depth Perception. Toronto Canada: I Porteous, pp 20-26 343Solomons, H. (1976) The three-dimensional space horopter Ophthalmol Opt pp 101-111 Dynamics of Vision 7- 165

Figure 7.4.1-11 A horizontal horopter showing effect of accommodation and rotation of the eyes. The horopter is only defined within the binocular field. Note the significant difference in the surfaces of best focus of the two eyes. Note also the difference in distance to the target, resulting in different magnifications in the images. These problems are aggravated for off-axis positions. Only the regions of the Vieth-Muller circle projected onto the foveola remain in good focus. This is the field of view where stereopsis and fusion occur. The center of the Vieth-Muller circle has moved with the rotation of the eyes.

The caption to a similar figure by Tyler & Scott is instructive,

“For convergence at any distance other than infinity, all points that do not lie on the Vieth-Muller circle or the vertical horopter line project to the retina with either a vertical disparity or both a vertical and horizontal disparity. Dashed lines show geometric horopter for symmetric fixation. Dotted lines are construction lines. Full lines represent relevant light rays. The vertical disparity arises from the differential magnification occurring when the point is closer to one eye than the other, as must occur with all points off the vertical axis. The three-dimensional point horopter is therefore not a surface, but two lines in space.”

Their comments were aimed at a horopter such as shown on the right when they say that the dashed lines only apply to “symmetrical fixation.” Tyler & Scott go into considerably more detail concerning asymmetrical fixation. Several types of empirical measurements designed to confirm the theoretical point horopter have shown consistent deviations. Tyler & Scott provide a good introduction to these problems. 7.4.1.5.2 The horopter as a test instrument

The basic non-invasive instrument for studying the operational performance of the POS and the physiological optical system is the horopter. The general horopter is defined primarily conceptually. Conceptually, it derives from the Vieth- Muller Circle shown in [Figure 7.4.1-1]. This is a circle drawn through the 1st principal point (1st nodal point344) of the two eyes and “the natural fixation point” on an axis perpendicular to, and bisecting the line drawn through the two principal points. The horopter is an instrument used to determine the disparity between scene elements. These disparities describe the location of points in 3-D object space relative to a reference point. Such disparities can be the actual disparities associated with a scene or pseudo-disparities associated with a pseudo-scene created by an optical

344 See Ogle, K. (1950) for a discussion on the value and limitations of nodal points. 166 Processes in Biological Vision instrument. They can also be the disparities determined by the visual system and found by measurement of the angular relationships between the eyes when observing the actual scene (or pseudo-scene). More specific, and limited, horopters are defined sufficiently to be fabricated and used in the clinic and laboratory. Ogle has described a variety of horopters. Most of them are focused on the longitudinal (or horizontal) horopter. These come in two distinct variants. The first is the simple (non reflecting) horopter. The second is the reflecting horopter or haploscope (as defined originally by Hering). In general, the first type is so crude mechanically that it can only be used for qualitative measurements related to the peripheral retina. The second is better suited to measuring veridical disparity associated with stereopsis. However, the instrument must be made with considerable precision if it is to provide reproducible measurements at the arc-second level. Figure 7.4.1-12 shows one of the simplest non reflecting horopters. Its accuracy is limited as shown by the practice of using thin rods as test targets until the precision provided by thin threads is required. Even the size and reflective properties of the threads cause accuracy problems. Because of the limited capability of this type of test set, it is usually used initially to determine an apparent field of constant perceived depth. This parameter is usually labeled, rather inappropriately, the apparent fronto-parallel plane. The surface defined is never a plane. As shown earlier [Figure 7.4.1-2], this surface is not normally planar. It is not planar with respect to focal conditions or depth conditions. The visual system is based on spherical geometry. Ogle attempts to define this plane precisely. However, he resorts to two footnotes to incorporate all of the caveats in the definition. He also notes the necessity of avoiding cues in the scene or test set that would skew the data. He even points out the necessity of randomizing the position of the handles used to adjust the rods used to establish the plane. Following adjustment of all of the rods by the subject, the platen could be raised to mark the position of the rods on graph paper.

Figure 7.4.1-12 A simple horopter test set. Because of the scale involved, the “rods” are frequently replaced by taught threads. A handle is provided for moving each rod. The null position of these handles is randomly arranged in depth to avoid providing cues to the subject. The pins are ink pens that can be brought into contact with the paper to record the position of the rods.

The horopter shown can be modified by introducing a series of apertures immediately in front of the subjects eyes. The description of a modern clinical horopter, used to measure steropsis performance, is difficult to locate in the academic literature. Johnston provided some information345. Earlier equipments were usually based on the Howard- Dolman apparatus of the early 1900's, a simple viewing box able to measure the ability of a subject to establish his horopter relative to the theoretical Vieth-Müller Circle. Both eyes have uninterrupted view of all of the test objects. The angular field of view has never been standardized but tests beyond a 15 degree total field of view are unusual. The Nonius Horopter is more sophisticated. It separates the view of the two eyes, and presents two separate sets of

345Johnston A. (1971) Clinical horopter determination and the mechanism of binocular vision in anomalous correspondence Ophthalmologica vol 163, pp102-119 (DOI: 10.1159/000306329) Dynamics of Vision 7- 167 vertical lines to the subject. He is asked to align the upper set of lines viewed by one eye with the lower set of lines viewed by the other eye. See Shipley & Rawlings cited above.

The literature has not provided a concise list of definitions describing the many potential horopters. Howard & Rogers probably provide the best overview and guide to the literature of simple horopters346. Most of these units can demonstrate the presence of qualitative depth perception under conditions of merging of the two images. This binocular merging is not synonymous with fusion in this work. The modern computer controlled horopter used for screening as part of a general eye examination provides a very narrow field of view, estimated at 1.5 degrees or less. It effectively determines the subject’s depth perception on–axis, i.e. using only the foveola. 7.4.1.5.3 Representations of horopter data

Measurements designed to confirm the theoretical horopter generally do not. Significant systemic deviations appear relative to the fundamental assumption on which the horopter is based. These deviations are frequently labeled the Hering-Hillebrand deviations. Figure 7.4.1-13 shows the frequently reproduced conceptual figure of the empirical horopter dating from at least Ogle (1950). The deviation of the horopter from the Vieth-Muller circle is shown. The fields of view of the foveola have been added for clarity. Describing the Hering-Hillebrand deviations precisely using graphs at this scale is difficult. It is also a bit embarrassing to point out that the versions of the figure by Tyler & Scott in Record (pg 656), by Tyler in Schor & Ciuffreda (pg 222), and in Howard & Rogers (pg 27) are drafted improperly. They all have the Vieth-Muller circle passing through the point of rotation of the eyes and the point of fixation. Such a circle is more commonly known as the circle of equal convergence. The Vieth-Muller circle is defined with respect to the 1st principal point of each eye and the point of fixation. Although it is usually reproduced without attribution, the concept appears to go back to Alhazen in the 11th Century (Howard, pp 50-52). Interest in it was only revived by Hering in the 19th Century. As usually reproduced, it is drawn without scales and any definition of the criteria used to draw it. Tyler, writing in Schor & Ciuffreda, notes that the form of the empirical horopter is not as shown in the figure under many conditions. He demonstrated that it even changed in local areas depending on the nature of the stimulus used.

Tyler discusses the empirical horopter in terms of Panum’s area. He then concluded with, “the traditional concept of Panum’s area as a fixed property of a given retinal region must be abandoned.” This is clearly the case. The situation will be discussed in greater detail below. The description of the empirical horopter depends greatly on the geometry of the stimulus used, the criteria used to define the limit, the light level and whether the stimuli are presented dichotically or dichoptically. Failure to account for these parameters leads to much of the conflict found in the literature concerning Panum’s area.

Howard & Rogers lists fusion range (the range in x,y dimensions defined by Panum’s area), mid-fusional-zone and maximum stereoscopic acuity as criteria for defining the empirical horopter on pages 26-33. Ogle discussed four criteria but only described them in contexts and protocols (pp 29-49). One was the maximum differential stereoscopic sensitivity. A second involved the same primary subjective visual direction (along the z-axis) for two threads in a horopter. He discussed several protocols Figure 7.4.1-13 A typical empirical horopter frequently for meeting this criterion. A third criterion involved the used in pedagogy, with the so-called fronto-parallel plane match of the horopter to the apparent fronto-parallel (X,Y) defined. Frequently called a fusion horopter and plane. The fourth criterion concerned the center of the drawn without scales. The Vieth-Muller circle is drawn region of “binocular single vision.” incorrectly. It should pass through the coreas of each eye. The 1.2 degree fields of view of the foveola have been Ogle (1950) proceeded to derive an additional added.

346Howard, I. & Rogers, B. (2002) Op. Cit. pp 20-40 168 Processes in Biological Vision mathematical framework for evaluating the empirical horopter against these criteria. His derivation employed two parameters that describe the horopter when it is allowed to become a conic section rather than only a circle. The first, H, described the asymmetry of the Vieth-Muller circle. The second, R0, is always small and defines the difference in magnification between the two eyes in the horizontal meridian. The precision in discussing the empirical longitudinal horopter is greatly enhanced using his equation. It introduces a series of geometric horopters of which the Vieth-Muller circle is only one. Figure 7.4.1-14 provides an extended theoretical framework based on a correct Vieth-Muller circle, the equations of Ogle plus several other additions. Ogle’s equations introduced a series of ellipses transitioning from the Vieth-Muller circle to the horizontal axis of the fronto-parallel plane through the point of fixation. With the parameter H = 0.0, the Vieth-Muller circle is obtained. If H = 2a/b, where a is the inter-pupillary distance and b is the distance to the point of fixation from the mid point of the inter-pupillary line, the horizontal axis is obtained. Intermediary values of H equate to intermediate ellipses as shown. One of these ellipses is particularly important because it corresponds to the surface of best focus for an equivalent “cyclopean eye,” or the actual eyes where a is much smaller than b. This ellipse will be important when looking at the data. The impact of the depth of focus of the visual system on horopter measurements suggests the data in the literature may be skewed toward the ellipse of best focus unless precautions are taken.

Little data is available on the precise shape of the focal surface of the human visual system in object space. The variable focal length with field angle of the design could introduce unexpected variations in the shape of that surface. Because the off-axis resolution of the system decreases so rapidly, the subject is largely academic. It will be assumed here that the eye is in optimum focus for all field angles when focused at infinity and that any accommodation changes the focal surface proportionally for all field angles.

The figure also includes the approximate limits (the dashed lines intercepting the ellipses) of the empirical horopter based on the binocular field of view. Finally, a deviation from the true vertical axis is shown. This small deviation of zero to two degrees is due to the Volkmann- Helmholtz Effect. This Effect is due to the relative tilt of 347 the vertical axis of the eyes when they converge . This Figure 7.4.1-14 Theoretical framework for displaying is due to the non-orthogonal motions introduced by the empirical horopter data. The Vieth-Muller circle is drawn oculomotor muscles. When aligned for vision at infinity, correctly and a series of ellipses from Ogle are also shown. the Effect is negligible. At near distances, the Effect can The limits on the field of binocular vision are shown and the cause the empirical horopter to tilt away from the eyes area imaged by the foveola is highlighted. The potential for points above the horizontal by up to two degrees. shift in the vertical axis of the horopter is also shown. See text. Finally, the figure includes the small circular area imaged by the foveola. This small area is the only area involved in stereopsis. By expanding the image, this small area is clearly the only area of the horopter that lies in the fronto- parallel plane. It is also the area of maximum spatial performance of the physiological optics. The data in chapter 4 of Ogle show that the actual horopter of an individual does not correspond to the Vieth-Muller circle or the fronto-parallel plane. His figure 11 would suggest the “typical” empirical horopter at 40 cm corresponds almost exactly with the ellipse of best focus (H = 1/b) rather than the Vieth-Muller circle. The data for several other distances to the point of fixation for this observer are shown in his figure 16. Tabulations of the value of H are also provided for a group of subjects using a variety of test criteria. Tyler & Scott summarized this data but chose not to show the data for six meters range, probably because it would have required more discussion348. The empirical horopter reversed curvature and became hyperbolic at six meters for the subject tested. While such a situation is not addressed in the simple conceptual horopter, the Ogle equation supports it for values of H greater than 2a/b. The data clearly shows that the theoretical Vieth-Muller circle and fronto-parallel plane generally circumscribe, but do

347Tyler, C. & Scott, A. (1979) Binocular vision, Chapter 22 in Records, R. Op. Cit. pp 650-656 348Tyler, C. & Scott, A. (1979) Binocular Vision, Chapter 22 in Records, R. Op. Cit. pg 649 Dynamics of Vision 7- 169 not bound the empirical horopter. Their utility is mostly in introductory pedagogy. Ogle has shown that the empirical horopter can vary significantly with the precise parameters of the subjects eyes. Differences in the phoria of the subjects can lead to significant differences in performance, at least using the Nonius (vertical bars arranged vertically and displaced horizontally) method of measurement (his figures 16, 22 & 23). Ogle provides a definitive answer to the question of the median location of the empirical horopter but he says very little about the diameter of the horopter as suggested by the caricature in [Figure 7.4.1-9]. His figure 27 shows the region of “binocular single vision” as expanding from about 5 mm about the point of fixation at 40 cm to about 7.5 mm at 12 degrees from fixation. However, his table 5 suggests the mean variations of setting defining the empirical horopter varied about seven to one for the two observers measured. This would suggest that the horizontal tube shown in the referenced figure should flare to a much larger diameter near its ends than shown. The question remains, what parameter can the empirical horopter display that will best describe the depth perception performance of the visual system? One answer would be to describe the angular precision obtained by measuring the stereo-acuity of the subject. This would produce a grossly different empirical horopter from the conventional caricature. It would focus specifically on the difference between qualitative depth perception and stereopsis. Figure 7.4.1-15 shows a caricature of an empirical horopter that can illustrate two criteria. When based on the best available data and the criteria of stereo-acuity, the tube drawn along the Z-axis has a length approximately 100 times the diameter of the empirical horopter drawn along the ellipse or the vertical axis. The range is similar when based on the criteria of maximum distance from the point of fixation to the edge of the perceived field of depth perception. The depth of perceived depth perception is much greater within the field of the foveola than outside it. This figure provides a clear representation of the difference between the region of stereopsis and the region of qualitative depth perception. The variation in depth perception outside the foveola using these criteria are so small, compared with that of the foveola, they cannot be shown easily on the same graph.

The above figure provides a framework for displaying a range of empirical horopter data. It also shows the great differences involved in discussing stereopsis and qualitative depth perception. It clearly shows the fronto- parallel plane plays no role in qualitative depth perception for fields beyond the foveola. It also shows, along with [Figure 7.4.1-7], the importance of selecting distances to the point of fixation compatible with the depth of focus of the lens group. 7.4.1.5.4 Parameters important in horopter protocols

The depth of focus of the lens group and thus the f/# of the lens group, and ultimately the light level in the laboratory are major limitations on the quality of horopter data collected. It is important that the light level and color temperature of that light emulate daylight when collecting data of high quality, particularly concerning stereopsis. Figure 7.4.1-15 Caricature of an empirical horopter based on stereoacuity or on maximum to minimum distance of Similarly, it is important that the effect of target disparity perceived depth perception, relative to the point of fixation. angle on the overlapping of the surfaces of best focus of See text. the two eyes be controlled, or at least noted. When collecting high quality stereopsis data at the edges of the foveola fields, the inter-ocular distance should be small compared with the distance to the point of fixation. The phoria of the subject should be determined before making horopter measurements.

Clements has provided data on the criticality of the plane of the horopter versus the horizontal plane of the head349.

349Clement, R. (1985) The geometry of specific horopters Ophthol Physiol Opt. Vol 5(4), pp 397 -401, 170 Processes in Biological Vision

7.4.1.5.5 A more realistic horopter for discussions of stereopsis EMPTY

Figure 7.4.1-16 presents a horopter applicable to discussions of stereopsis. It clearly defines the area of precision stereopsis compared to the much broader extent of initial coarse stereopsis. The Vieth-Muller circle is shown for H = 0.00, see chapter 4 of Ogle for important values of H. The fixation point is in the sagittal plane of the head and the eyes are converged at equal angles. The region of precision stereopsis and optimal fusion is defined by the 1.2 degree visual cone associated with each eye (Figure 7.4.1.4-14). If the eyes are significantly rotated to fixate on a point outside the sagittal plane, the quality of the image focused on the retinas will suffer. from rotation of the oculars about the lines of sight (Figure 7.4.1–10). Some investigators define a tangent board,” typically a vertical panel with a horizontal line aligned with H = 2a/b and a vertical line aligned with the sagittal plane.

Figure 7.4.1-16 The optimal horopter for stereopsis discussions ADD. See text.

Note the only optical rays that are straight lines between the external environment and the retina are those satisfying the simplification, sin x . x. These rays are within 0.6 degrees of the optical axis of the lens. These are not the rays leading to the fixation point or center of the foveola of each retina. The angles, αL and αR are the most useful in discussing the stereoptic field and the phenomenon of fusion. However, they do not correspond to the equivalent angles within the oculars for large angles where sin x … x. As will be developed in Section 7.4.5.xxx, it is not necessary that the object at the point of fixation, or at other locations such as A, do not need to be multidimensional objects, they may be multidimensional features of a multidimensional object of greater extent. The feature can extend beyond a single or few photoreceptors at each retina. However, stereoptic performance will improve if the feature extends over more than a few photoreceptors in dimension (the signal- to-noise associated with the stage 2 through 4 signal processing will be higher. The most important distance in the figure is the distance to the fixation point from the midpoint between the nodal points of the two oculars (the nodal point of the mythical 7.4.1.6 Description of Panum’s area, depth perception and the fusional horopter

7.4.1.6.1 Panum’s area relates to the X & Y axes of object space

Panum’s area, or Panum’s fusional area, is a convenient concept used inconsistently in the literature. This makes it extremely difficult to define the term precisely. Panum reported on his work beginning in 1858. It has been discussed conceptually by many authors. Conceptually, it has long been considered a relatively constant area defined in terms of Dynamics of Vision 7- 171 a small difference in visual angle relative to the point of fixation. Within this area, points of light presented to the two retinas will fuse into a single perceived image. Howard gives a simple illustration of the concept on page 51 and refers to a fuller discussion on pages 272-282. These more complex discussions quickly become generic and revolve around the concepts of fusion and diplopia. They get into the question of whether the area is measured under dichotic or dichoptic conditions, how the area varies with eccentricity from the point of fixation, etc. The subject of hysteresis is also introduced. As two dichoptic points are moved farther apart, fusion is lost at a definable disparity angle. However, the angle at which this occurs is different from that where two widely disparate points fuse as they are brought closer together. The subjects of scene contrast and brightness must be dealt with. In addition, the relevant temporal factors must be considered. The problem is further complicated by the use of the term spatial frequency when the targets invariably consist of only two discrete objects separated by a finite space (interval). In 1979, Tyler & Scott writing in Records said flatly “the traditional concept of Panum’s area as a fixed property or a given retina region must be abandoned. Instead, the fusional extent is strongly dependent on the stimulus used to measure it.” They went on, “Hence, the fusional horopter (frequently) presented (in the literature) must be taken only as an indication of the fusional range in the real world, which will expand and contract according to the objects present in the field and the optical characteristics of the eyes viewing them.” This position appears strongly influenced by Tyler’ papers of 1973 and 1975. These papers introduced the apparently ultimate simple variable to evaluate fusion and Panum’s area simultaneously. They used various permutations of a straight line, a wavy line and dashed versions of straight and wavy lines in their experiments. However, his methodology introduced cues related to both eccentricity and depth. No comment was found saying a goal was to challenge and evaluate the performance of the correlation mechanism of the visual system. Others have used even more complex shapes to study fusion and rivalry. However, these variations usually lead to a more complex explanation or set of rules to explain the phenomenon observed, rather than a simplification of the basic mechanisms underlying the phenomenon.

The first challenge here is to decide whether Panum’s area refers to a two or a three-dimensional phenomenon.

Tyler, writing in 1983 in Schor & Ciuffreda, (pages 220-227) begins by defining the binocular visual direction as applying to the x,y position in the “frontal plane.” He discusses Panum’s area as a two dimensional plane and cautions about the importance of separating the fusion task from the depth perception task. He summarizes by saying, “Instead of a fixed fusional region there is a strong dependence of fusion on the local stimulus characteristics. Panum’s area is a dynamic entity that is continually being adapted to the prevailing features of the stimulus environment.” As noted earlier, He also cautions that “the fusional horopter presented above is not a fixed range around the point horopter and the conventional depictions of figure 7.11 must be taken only as an indication of the fusional range in the real world, which will expand and contract according to the objects present in the field and optical characteristics of the eyes viewing them.”

Howard discusses the area as an ellipse on page 272 to account for the difference in vertical and horizontal performance. He also notes that “one must ensure that subjects are judging fusion rather than apparent depth between the disparate stimuli. This is a severe problem with the forced-choice procedure because subjects tend to rely on apparent depth if that is the only difference they see [emphasis added].”

The term Panum’s limit occurs occasionally in the literature to describe the boundary of Panum’s area. This limit varies as a function of the specific stimulus configuration. Section 7.4.6 will show that Panum’s area is more directly related to the effective dimensions of the correlator of the PGN than the physical dimensions of the foveola.

7.4.1.6.2 Depth perception involves the Z-axis

The expression depth perception is nearly self-defining. It is the perception of different distances to objects in the visual field of view. As discussed in Section 7.4.1.3, a variety of cues can contribute to the perception of depth. Some provide only a qualitative perception while others provide a quite distinctive perception defined as veridical. 7.4.1.6.3 The maximum range of fusion and depth perception

Section 7.4.6.2 develops the relationship between the phenomena of fusion and depth perception and shows they share an underlying mechanism. As a result, Panum’s area, describing the x,y region of fusion in object space, and the equivalent range of depth perception, along the z-axis in object space, are plastic regions. They are both derived from a more fundamental expression of Panum’s limit associated with the associative correlator of the PGN. This Panum’s limit defines the maximum effective spatial range of the correlator when implementing the mechanism of stereopsis. In this correlator space, Panum’s limit refers to a combination of the eccentricity and depth, relative to the nominal point of fixation in object space, that can be processed by the correlator. 172 Processes in Biological Vision

This broader limit in 3-dimensional space can be described as Panum’s volume limit. The generally egg-shaped space enclosed by this limit can be described as Panum’s volume (although he did not study the depth perception aspect). It reduces to Panum’s limit in x,y space for z = 0. The area enclosed by this limit is known as Panum’s area. This limit can also be expressed by the maximum value of z for the condition x = y = 0. This limit describes the maximum range of depth perception relative to x0,y0,z0. For other values of x,y,z, a volume can be defined that is conceptually equivalent to Panum’s area. It defines the combined range of fusion and depth perception achievable by the subject. 7.4.1.6.4 The fusion horopter involves the X,Y & Z or θ, φ, ρ axes

While much of the early literature describes stereopsis in terms of X, Y & Z coordinates, this is not the system used within the stage 4 saliency map. To accommodate, hearing inputs as well as visual inputs, the saliency map of the external environment is almost certainly maintained in a gravity oriented inertial framework using θ, φ, ρ coordinates. The inertial framework in healthy humans is biased so that the φ = –90° axis is parallel to the gravity vector when standing or sitting. [xxx what does it say?] The definitions of Panum’s area cited above relates to the X,Y plane perpendicular to the (Cyclopean) line of sight. When combined with the idea of depth perception providing the third or Z-axis, a different Panum’s volume would be defined. This volume would consist of a torus, as frequently drawn to represent the “fusional horopter.” The torus would consist of a series of laminates or curved planes orthogonal to the line of sight (not the line of fixation) and varying in their vertical size with their distance from the 1st principal point of each eye. Each plane consists of a series of overlapping ellipses representing Panum’s area at each specific eccentricity. The variation in vertical size with depth is controlled by the change in size of Panum’s area with radial distance from the point horopter. The variation in horizontal size with depth also varies with radial distance from the point horopter. However, this dimension is obscured by the merging of adjacent ellipses to form the horizontal extent of a single laminate (plane). This merging is similar to the row of shields on the side of a Norseman’s medieval sailing ship merging to define the outer surface of the ship. As noted by Tyler, the specific size of the fusional horopter is highly dependent on the precise nature of the stimulus used and the characteristics of the subjects eyes.

The online text, Webvision, sponsored by the University of Utah provides additional material related to Panum’s area for the undergraduate350. Webvision as an introductory text does not appear designed to be accurate at the theoretical level.

7.4.2 The version control subsystem: pointing

First order version eye movements are by definition conjunctive. Rashbass and Rashbass & Westheimer have provided a series of papers shedding light on the version control subsystem. See references in Section 7.3.1.1.

The version control subsystem calls upon the underlying pointing system to operate over an extended range of angles and velocities as discussed in Section 7.3.

Delineating between the operation of the version control system for large signals (and those originating outside the foveola) from those smaller signals associated with the foveola (and originating within the precision optical system) is important. The former is generally associated with the awareness and alarm modes of vision. The latter is generally associated with the analytical mode of vision. The analytical mode employs a different type of servomechanism than do the others. In fact, the differences extend to differences in the actual muscles supporting the two servomechanism types. Because of the complex interplay between these servomechanisms (and the role of memory in their operation), describing them individually in-toto is difficult. An introductory description of the operation of the version subsystem can be provided. The alarm mode of vision provides instructions to the TRN relating to the angular location, from the point of fixation, of any changes in the external environment detected by the awareness mode subsystem, the LGN/occipital couple. These changes occur outside the view of the foveola by definition. These instructions are passed to the superior colliculus where they are converted into detailed commands for implementation by the pointing and accommodation subsystems. Similarly, the cognitive portion of the HVS can provide instructions via the volition mode channel to the TRN for implementation. Here again, the signals are passed to the superior colliculus for conversion into implementable commands that can be acted upon by the pointing and accommodation subsystems. The precision optical system can detect and respond

350http://webvision.med.utah.edu/book/part-viii-gabac-receptors/space-perception/ Dynamics of Vision 7- 173 autonomously to changes occurring within the field of view of the foveola. However, these actions may be subject to override by the TRN. 7.4.2.1 The type 0 version control servomechanism (related to awareness)

The version control servomechanism associated with the awareness and alarm modes of vision have the primary responsibility of keeping the organism current on the state of its environment. This is a particularly important role with respect to changes in that environment. The organism maintains a complete saliency map of its environment in memory and does not rely upon the awareness mode for static information. The awareness servomechanism is tailored to detect changes in the environment and report the location of those changes to the thalamic reticular nucleus (TRN) for action. It is the TRN that forwards the coordinates provided by the alarm mode channel (primarily from the LGN) to the oculomotor functions within the POS for implementation. Following that implementation, the awareness and alarm modes re-evaluate the environment. Thus, closed-loop operation is achieved. The fact that the awareness and alarm modes only deliver position information to the TRN during any single cycle of the servomechanism defines it as a type 0 servomechanism. Information related to the spatial velocity of changes is only extracted over the long term. This capability is limited by the time involved. 7.4.2.2 The type 1 version control servomechanism (related to analysis)

The analytical mode of vision employs the much higher spatial resolution of the combined foveola and associative correlator of the PGN. It also relies upon the twitch capability of the oculomotor subsystem and the tremor generator. With these aids, it can compute both location and velocity information within the response time associated with the tonic portion of the oculomotor subsystem and also within the response time of the accommodation subsystem. In this sense, the analytical mode provides the capabilities of a type 1 servomechanism. 7.4.2.3 Smooth pursuit versus salutatory motions EDIT

The awareness and alarm modes of vision are primarily responsible for causing the visual system to redirect the line of fixation to specific locations within object space. They are generally not involved in the pursuit function, the tracking of moving objects. The analytical mode of vision is primarily responsible for tracking a moving object. For an object moving within the field of view of this mode at a low rate, the type 1 servomechanism of that mode is able to provide tracking signals to the POS. With these, the POS and the oculomotor subsystem can provide a pseudo-continuous tracking capability (the two images remain fused within the correlator/pulvinar circuits of the PGN). This mode of operation is called smooth pursuit. For targets moving faster than the type 1 servomechanism can process, the TRN causes the system to go into a salutatory tracking mode. The eyes will be redirected to a new calculated point of fixation (new version, vergence and accommodation values) different by more than the diameter of the foveola from the previous point and a new correlation cycle will be initiated. Rashbass has concluded that the smooth pursuit capability extends up to 10 degrees/second (pg 338). The capability of the analytical mode is usually degraded under salutatory tracking conditions due to the short integration times available.

7.4.2.4 Data related to smooth and salutatory pursuit

A team at the Salk Institute of San Diego has been active recently in the psychophysics of target acquisition, recognition and tracking. Adler, Bala & Krauzlis have provided data built around two experiments351. [xxx add more here ]

Liston & Krauzlis352 have provided good psychophysical data related to salutatory pursuit (based on Rashbass’s criteria of 10 degrees/second). However, their conceptual model, discussed only in words, is not compatible with that of this work. They offered no schematic of the oculomotor system supporting their conclusion that the motor elements are different for the saccadic and pursuit phase of operation. Thus, their proposed operation of the oculomotor system and the conclusions drawn are not supported and should be examined critically by the reader. The frequent use of the words might and could in their discussion is suggestive of the firmness of their findings. However, their discussion and references related to neural activity associated with the reticular formation is very interesting.

351Adler, S. Bala, J. & Krauzlis, R. (2002) Primacy of spatial information in guiding target selection for pursuit and saccades J Vision vol 2(9), pp 627-644 352Liston, D. & Krauzlis, R. (2005) Shared decision signal explains performance and timing of pursuit and saccadic eye movements J Vision vol 5, pp 678-689 174 Processes in Biological Vision

Liston & Krauzlis used two moving windows on a kinescope monitor and monitored the motion of the eye with an infrared tracker to a precision of ~0.1 degree and a 1kHz interrogation rate. Each window was filled with a spatially stationary pseudo-noise pattern. Their test criteria was based on a sensitivity criteria developed by Green & Swets353. Horizontal scene motion was limited to 14.2 degrees/second. The eye was required to jump 2 degrees from a point of fixation and then track the motion of the stationary noise pattern within the moving window. The protocol included operations during two modes of vision. Awareness Mode • a period of fixation on a 0.5 degree diameter cross • a random interval • the appearance of two moving windows, one above (2°) and one below (-2°) the fixation point and spanning the full width of the 45° field of view framed by the monitor. C an analytical interval while the brightest window was determined. Analytical Mode C a vertical saccade of nominally 2° to bring the brightest window over the foveola. C a period of analysis while the character of the pattern applied to the foveola was determined. C a subsequent time while the POS determined and instituted a predictive sequence of pursuit saccades. When exploring the performance of the POS, it is useful to categorize and document the time delays associated with the individual functional elements.

Type Stage Nominal Value Comment C Sensory neuron delay (without adapt.) 1 Varies with avg. light level C Sensory neuron delay ( adaptation) 1 up to 25 ms2 Varies with delta light level C Peripheral signal processing delay 2 Xxx ms in humans C Afferent signal propagation delay 3 Typically xxx ms (human) C POS signal processing delay 4 Optional based on protocol C Cognitive signal processing delay 5 Optional based on protocol C Volition/Command generation delay 6 Optional based on protocol C Efferent propagation delay 3 Typically xxx ms (human) COculomotor plant delay 6

2 Ritter & Gegenfurtner. Probably includes all sensory delay.

The sensory delay varies with light level as shown in Section xxx. It is typically xxx ms under photopic illumination conditions.

DeValois has used 50 ms as the typical cumulative delay recorded at the LGN. As seen from the above table and this discussion. The 50 ms appears to apply to the output side of the LGN. [xxx add additional comments here] 7.4.3 The vergence control subsystem: coarse convergence via the LGN

In this section, noting two situations related to nonlinear mechanisms versus nonlinear analytical techniques will be important. Differentiating between a nonlinear biological mechanism being modeled by a mathematical equation that is also nonlinear; and a piecewise linear model that results in a nonlinear output will be important. The piecewise linear model relies upon linear mathematical equations over individual intervals of a larger interval. It may or may not represent the actual mechanism being modeled. Differentiating between circuits that are actually parallel in the mathematical model of a servomechanism and circuits that are drawn in parallel but are active only individually (under the control of some form of switch) will also be important. As stated in Section 7.3.3, the vergence system is an overlay on the pointing subsystem. While it may only be a single- stage system in most lower animals, who do not exhibit the ability to analyze fine detail, it is a distinctly two-stage

353Green, D. & Swets, J. (1988) Signal Detection Theory and Psychophysics. Los Altos, Ca: Peninsula Dynamics of Vision 7- 175 servomechanism in humans and a few of the higher primates. The precision achieved by the coarse system, employing the LGN, is much lower than the remarkable capability achieved by the precision optical system (the POS) and the PGN, described in Section 7.4.5. The performance of the opto-mechanical systems of the human eye has been studied primarily by four distinct groups. The optometric community has studied the subject using primarily non-invasive techniques. The morphological community has studied the optics primarily as a physical structure. Only recently has either of these two groups studied its parameters associated with the wide angle optical performance of the system. A third group, the medical surgeons have studied the physiological optics from a perspective outside the scope of this review. The fourth group of surgically oriented physiologists have studied (disparity based) coarse stereopsis using electrophysical techniques after initial convergence has been achieved. The optometric community has studied the opto-mechanical systems primarily by evaluating the performance parameters resulting from a stimulus applied to a black box. They have not dissected (in the intellectual sense) the system into a broad description of its internal components and evaluated them individually. The dissection that has occurred has been largely limited to the separation of the system into three components. The first is a physiological optical component(s). The second is a muscle system (frequently described as the plant based on servomechanism terminology). The third is a mathematical description of the overall circuitry required to realize the performance measured. 7.4.3.1 Review of vergence models in the literature

Schor has provided a brief summary of the various states of convergence in the visual system354. Dell’Osso & Daroff have provided information from the clinical perspective, of the vergence system355. The block diagrams are suggestive but not in agreement with the diagrams of this work (See Chapter 15). Their diagrams concentrate on the volition mode of signal processing to the near elimination of the more automatic modes of awareness and analysis concentrated in the mid brain.

Gamlin has recently discussed the vergence system in non-human primates. Although his block diagram contains many question marks, the data appears useful356. Pierrot-Deseilligny, et. al. have discussed similar eye movements in humans due to lesions in the volition mode circuits357. Several block diagrams and time lines are presented. However, the focus appears to be on the volitional High-level commands generated in the cerebrum almost exclusively. Jiang, Hung & Ciuffreda have recently provided a major chapter on vergence and accommodation models and interactions. The characteristics of these models were discussed in Section 7.3.1 above. They represent early floating models that have not been integrated into a system model compatible with the physiology of the visual system. While of great value to an analyst attempting to create a more comprehensive model, they will not be discussed in detail here.

Recently, a large compendium of work associated with the opto-mechanical systems of human vision has appeared358. It was edited by Hung & Ciuffreda. This compendium exhibits the characteristics just enumerated. By exhibiting these characteristics, it simultaneously defines the state of the art in the field. While titled, “Models of the Visual System,” it is largely focused on individual floating models of various parts of the opto-mechanical systems of vision. Little attention is given to the other aspects of the visual system. The volume does provide an excellent summary of the large number of floating models developed by different teams. Many of these teams had short lives while under the tutelage of Professor L. Stark. His name appears in a secondary position on the author line of many important papers in this area of the literature. One of these papers will be adopted as a starting point in developing the pointing subsystem underlying the operation of the focus and vergence subsystems (Section 7.3.4.1).

While the performance of the system is generally recognized to exhibit large signal performance (and therefore generally nonlinear) properties, the system modeling techniques used to date have been largely limited to the small signal regime. Alternatively, they have adopted piecewise linear models to achieve a nonlinear mathematical model of an underlying physiological mechanism (whether that mechanism is linear or nonlinear). This is clear from a review of the papers in Hung & Ciuffreda. The papers have provided a great deal of useful experimental data along with a wide variety of

354Schor, C. (1991) Effects on the resting states of accommodation and convergence. In Grosvenor, T. & Flom, M. Op. Cit. pp 310-317 355Del’Osso, L. & Daroff, R. (1999) Eye movement characteristics and recording technique. In Glaser, J. Neoro-ophthalmology, 3rd ed. NY: Lippincott Williams & Wilkins Chapter 9, pp 327-344 356Gamlin, P. (2002) Neural mechanisms for the control of vergence eye movements. Ann. N.Y. Acad. Sci. vol. 956, pp 261-272 357Pierrot-Deseilligny, CH. Ploner, C. Muri, R. Gaymard, B. & Rivaud-Pechoux, S. (2002) Effects of cortical lesions on saccadic eye movements in humans. Ann. N.Y. Acad. Sci. vol. 956, pp 216-229 358Hung, G. & Ciuffreda, K. (2002) Op. Cit. 176 Processes in Biological Vision floating models. The papers in even this recent volume appear to reflect the limited formal training in servomechanisms of many individual authors. In general, each model attempts to rationalize the performance data collected. While these models are all different, they do provide excellent source material for an analyst with a broader background in servomechanism theory and circuit realization techniques. Be carefully examining the data, and the individual models and equations, it is possible to synthesis the underlying, and more fundamental, equation of the overall process. Once this is done, the mathematical model can be converted to a circuit model using realization techniques compatible with the physiological and neurological limitations of the visual system. This consolidation of the largely conceptual models into a single comprehensive model (rich in features not shared by linear models) will mark a major advance in the state of the art. The papers in Hung & Ciuffreda are generally limited to the small signal regime (in the mathematical context). As a result, the formal structure of the papers is limited to the linear case. It is interesting that at least six different models are provided to describe the vergence servomechanism of the visual system (their Section 9.2). A progression of models is also provided related to the saccadic aspects of the oculomotor system. The papers generally provide estimates of the s-plane characteristics of the underlying servomechanisms required to achieve the measured temporal response characteristics. Only a few papers alluded to the use of root-locus or other sophisticated (but linear) servomechanism analysis techniques. They do introduce a finite, but lumped, time delay into the overall performance model. They also recognize a “dead space” in the response associated with the accommodation system. Some graphic models incorporate a range limiter, shown as a graphic symbol, but do not introduce the same constraint into the equations. Only in a very few cases have the potentially discontinuous nature of the signal processing elements been suggested. None of the included papers has recognized the intrinsically sampled-data characteristics of the system. Thus, no reference to Z- plane, or other higher order servomechanism modeling techniques, were noted. Progress in this field will be slow until the transcendental and exponential properties of the opto-neuro-mechanical systems of physiological optics are introduced into these models along with the time delay and dead-zone characteristics noted above.

Several papers in Hung & Ciuffreda incorporate multiple parallel signaling paths in their models. However, most of these paths are only active sequentially. The switch required to arrange the sequential operation is not always apparent. This notation is introduced to satisfy the nonlinear operation by using a piecewise linear model in the analysis. While easy to draw, confirming this mode of operation within the neurological networks of the brain is difficult. It is likely that recognition of the logarithmic processes used in the neural system (and the transcendental processes in the plant due to geometry) will eliminate the need for this method of analysis in all but the largest saccadic motions.

The various plant models presented in Hung & Ciuffreda vary from early systems with constant gain characteristics to second and third order systems. These models represent attempts to more closely match the performance data of the system based on better test design and instrumentation. Many plant models in Hung & Ciuffreda assign a time constant of about 160 milliseconds to the plant associated with oculomotor response. This value will be used here for the nominal slow time constant of the tonic portion of the oculomotor plant. However, an entirely different time constant must be associated with the twitch component of the oculomotor plant (Section 7.3.4.1.3).

A vast majority of the papers in Hung & Ciuffreda show the physical plant in the forward path of the servomechanism, with conceptual outputs taken from the output of the plant. These are generally labeled vergence or vergence response (VR) and accomodative response (AR). No example was found of any signal being extracted from the models and projected to the higher neural centers of the brain. In Section 10.4 of Hung & Ciuffreda, Pola briefly refers to a potential extra-retinal signal used in the higher neural centers. He then asks a question. “What is the source of this signal?” His only response is to mention a putative efference signal that has appeared occasionally in the literature. Several papers in Hung & Ciuffreda attempt to explain the interaction between the vergence and auto-focus (accommodation) systems (their section 9.3). However, the presentations are largely conceptual and based strictly on floating mathematical models. They typically treat the input signals to these two systems as independent, as if they did not arise from the same photoreceptor cells. They then introduce cross-linking between the two subsystems in an attempt to rationalize the common elements of the output performance seen in the two systems.

As noted in Patel, et. al., these models do not address the source of the neural signals in the real retina359. They say the following. “For example, most control theory models of the HVS [the human horizontal vergence system] use disparity as input and vergence eye-position as output without specifying how disparities are computed from retinal activities and how motoneurons are driven to generate the desired disjunctive movements.” Based on this lack of adequate understanding of the signal generation process, the model presented by Patel, et. al. is more conceptual than other recent

359Patel, S. Ogmen, H. White, J. & Jiang, B. (1997) Neural network model of short-term horizontal disparity vergence dynamics Vision Res vol. 37, no. 10, pp 1383-1399, quote on pg 1384 Dynamics of Vision 7- 177 models reviewed in Hung & Cuiffreda. An intriguing aspect of the literature is its complete lack of association of tremor with the operation of the opto- mechanical (more properly opto-neuro-mechanical) aspects of the visual system. The term tremor does not even appear in the index to Hung & Ciuffreda. Neither does the fact, that the visual system becomes blind within about three seconds in the absence of tremor, appear in their text (See Section 7.3.3.5.4). This lack of appreciation of the critical role of tremor has prevented the optometric research community from associating their measured data with the actual source of the signal processing within the visual system. It has also prevented the psychological community from making significant progress in how animals analyze the imagery in a scene. This work will show the focus and vergence systems (as well as the ability to analyze, including read) are critically dependent on the tremor introduced into the oculomotor system for their performance. The neurophysiology of the visual system appears only peripherally in Hung & Ciuffreda (their Section 10.2.3). This material does not differentiate the operating modes of the visual system. Without such differentiation, providing a formal description of the neurophysiological description of the visual system is difficult. A clearer differentiation of this area is provided in this work (Sections 11.1, & 15.2). This work will provide a broader conceptual foundation for the opto-neuro-mechanical model of the physiological optics. The foundation is compatible with the larger overall model of vision presented in this work. A single broader servomechanism model will be introduced that serves all of the vergence, accommodation, analysis and pointing functions of the visual system. It is based on accepting only a single complex signal from the overall retina, with maximum emphasis in processing place on the portion from the foveola. It will output neurological signals to both higher neural centers and the multiple motor “plants” associated with the ocular. The oculomotor, the accommodation, and the pupil control plants will be shown in separate feedback paths of the common servomechanisms. The delay associated with the systems will be subdivided into the transport delays associated with the stage 3 projection neurons, processing delays associated with the neural system and plant delays associated with the musculature.

Bodis-Wollner has provided a recent discussion of some neural aspects of saccades and attention360. While this material touches on the opto-mechanics of the visual system, it is brief. It will be reviewed further in Chapter 15.

The references listed here and in Hung & Cuiffreda have provided excellent data. However, interpreting that data by the investigators or others is quite difficult. Interpretation is more difficult than the original investigators assumed because they had no appreciation of the actual disparity signal generation mechanisms. Because of this situation, the next two sections will develop the appropriate model of the vergence mechanism before the data of the literature is reviewed. 7.4.3.1.1 Models of the physiology of the vergence control system

Julesz presented a caricature representing depth perception in a discussion labeled “Ambiguities in Stereopsis” in 1978361. It was a reproduction of a similar figure of 1971362. A more complex but similar caricature appears in Tyler & Julesz363. Related caricatures have also appeared in two different articles in Leibovic. One is by Dow (pg 111) and one is by Wildes (pg 341). The latter is attributed to Grimson. These caricatures have all suffered from several problems. First, they are all drawn without any scales and place the eyes farther apart than the objects in the scene. This causes the convergence angles to far exceed those achievable by any human, other than a young child. It also causes principal rays drawn between adjacent objects and the two eyes to appear to intersect. Julesz uses these intersections to define “phantom targets.” He proposes that these phantom targets, which do not appear in more realistic geometries, must be eliminated by undefined neural processing. Second, they invariably employ the Gaussian optical approximation that suggests that all optical rays passing through the lens of the eye travel in straight lines. This is entirely incorrect for angles larger than a few degrees. Third, they are all based on a time invariant geometry. This leads to the assumption that the visual system operates on spatially static information. Because of the above caricatures, the community has limited its discussions of depth perception to such a degree it has not moved beyond the conceptual realm. In recent times, it has discussed the phantom targets of Julesz under the name,

360Bodis_Wollner, I. (2002) Beyond classical retinotopy: striate cortical mechanisms associated with voluntary saccades and attention In Hung, G. & Ciuffreda, K. eds (2002) Models of the Visual System. NY: Kluwer Academic/Plenum Press Chapter 7. 361Julesz, B. (1978) Global stereopsis: cooperative phenomena in stereoscopic depth perception Chapter 7 In Held, R. Leibowitz, H. & Teuber, H-L. Handbook of Sensory Physiology NY:Springer-Verlag 362Julesz, B. (1971) Foundations of cyclopean perception Chicago, Il: Univ. of Chicago Press 363Tyler, C. & Julesz, B. (1980) On the depth of the cyclopean retina Exp. Brain Res. vol. 40, pp 196-202 178 Processes in Biological Vision the “false target problem.” Anderson & Nakayama have recently written “The severity of the false target problem has been the primary theoretical constraint that has shaped virtually all extant models of stereopsis.” They then asked the question, “Is the false target problem a false problem?364” They conclude it is not a real problem based on their assumptions and analysis. Specifically they say, “we believe that the fundamental mistake that has been made in the statement of the matching-noise problem (an element of the false target problem) is the confusion of a method of stimulus construction with a theory of visual perception.” The conceptual method of stimulus generation adopted by the community has long been a basic problem. The next paragraph will show the dynamics of the eyes relative to any fixed scene results in a totally different interpretation of the stereopsis mechanism. It shows the false target problem is totally spurious. Figure 7.4.3-1 provides a caricature using more realistic geometries than those found in the literature. It contains three views. The left view represents an overhead view of a four-lane bowling alley with the subject playing in lane C. Each lane is sixty feet long and 42 inches wide. The pins are 4.75 inches in diameter and arranged in an equilateral triangle with the closest pin to the subject spotted 34.73 inches from the farthest edge of the lane. The size of the pins causes considerable overlap in their images. This aids the subject greatly in perceiving their relative position in depth. To allow for gutter lanes, the lanes are on seventy two-inch centers. The head and eyes of the subject are still shown seven times actual size to be identifiable at this scale. When the line of fixation of the eyes is convergent on pin #1 in lane C, the back row of pins extends over an angle of 3.35 degrees. When the line of fixation of the eyes is convergent on pin #1 in lane C, the principal rays passing through the center of the far end of lane B, and lane D, are at an angle of 5.73 degrees to the line of fixation.

Several propositions are offered. The typical subject cannot see the individual pins in lanes A, B & D when fixated on pin #1 in lane C. Each pin group could be replaced by a triangular cardboard box and the subject could not tell the difference if his line of fixation is restricted to pin #1 in lane C. This is due to the significant aberrations in the physiological optics of the eyes. Furthermore, the principal rays drawn between the pin sets in adjacent lanes do not intersect among themselves or with the principal ray to pin #1 of lane C. Thus, no “phantom targets” are created in this configuration. This is also true with regard to the pins in lane C alone as can be seen in the middle view of the figure. Looking at a different case, the caricature of Dow suggests phantom targets could occur (at his location C) if the targets were at significantly different distances relative to the horopter. However, this is unlikely. The accommodation mechanism would cause the potential phantom target to be considerably out of focus and poorly resolved. No phantom targets occur under realistic conditions of human vision.

The middle view expands the left view by 5:1 to illustrate additional details. The nominal view imaged by the foveola of each eye is shown by the 1.2 degree dimension. To image the full set of pins within the foveola requires the instantaneous line of fixation be moved to three positions, the central position and positions 1.2 degrees each side of that position. This is the conventional situation. The visual system is normally required to scan a scene to perceive the relative distances to all of the objects in the scene (Section 7.5.2). When centrally fixated, the foveola can observe the equilateral arrangement of pins #1, 2 & 3. However, the foveola is unable to provide high quality depth information related to the outer pins at #4, 7, 6 & 10. The right view highlights the difficulty in obtaining a high degree of depth perception related to pins #6 & 10 when the eyes are fixated on pins #4 & 7. In fact, most people can only determine which pins are in front by their overlapping images. If smaller diameter “duck Figure 7.4.3-1 Caricature of depth perception at a bowling pins” are used (diameter, 4.125 inches), bowling alley. becomes a much more difficult game. In duck pins, no overlap occurs between images of the pins and most people cannot discern any depth perception related to pins #6 & 10 when fixated on pins #4 & 7. They rely on looking

364Anderson, B. & Nakayama, K. (1994) Toward a general theory of stereopsis: binocular matching, occluding contours, and fusion Psychol Rev vol. 101, no. 3, pp 414-445 Dynamics of Vision 7- 179 at each pair individually, determining their relative positions and then computing their relative positions as a group. This information is stored in their saliency map of the scene (the technical name for the memory portion of the conceptual cyclopean retina of Julesz). The dashed lines in the above figure are drawn with only one apex, essentially the cyclopean eye of Hering. There is no hint in this static figure of how stereopsis is actually achieved. Figure 7.4.3-2 provides an answer to this question using a simpler configuration involving only three pins that fall well within the 1.2 degree field of view of each foveola. Each nominal foveola has a diameter of 175 nominal photoreceptors (pixels for this discussion). The figure shows the instantaneous field of view of only nine pixels (-4, -3 . . . 0, . . . +3, +4) selected from the center of each field of view. The instantaneous fields of view are shown as parallel because of the distance to the 1st principal point of each eye. Each instantaneous pixel field is only 2 arc seconds wide. The vergence angle between the two lines of fixation is also highly exaggerated. The vergence angle is twice what it would be for a real person looking at a scene only 30 cm (11.8 inches) from his eyes. If drawn to scale, there would be virtually no crossing of the field of view of non corresponding pixels for targets at more than 30 cm. Because of this situation, the so-called correspondence problem is a minor problem at best. The correspondence problem will be discussed further after the following thought experiment is defined.

Let the two eyes be converged on the central dot in the fronto-parallel plane of the eyes. Now, introduce a tremor into the line of fixation of each eye and let the motion caused by the tremor be conjunctive. Assume the Figure 7.4.3-2 A caricature introducing tremor to explain eyes are scanning from left to right as shown. The the mechanism providing stereopsis. The vergence angle magnitude of the tremor induced motion is not important shown is at least twice, and typically ten times, that usually as long as it is greater than one pixel diameter. This used in human vision. See text. requirement is due to the fact the pixels act as edge detectors and the interpretation is simpler if the pixels cross edges (or dots) in the field of view completely.

As seen, any dot found in the fronto-parallel plane will appear in the corresponding pixel channels of the two eyes relative to the 0 numbered pixels. This is not true for any dot not located in the fronto-parallel plane. The dot on the left appears in pixel -3L and -4R. As the eyes sweep across the scene, a signal will be transferred to the midbrain representing this single dot. The signals would be deposited in the cells of the associative correlator of the PGN according to their foveola pixel number. Similarly, the right-hand dot would generate signals in cells +3L and +4R. The central dot (located at the point of fixation) would obviously generate signals in channel 0L and 0R. The function of the correlator is to calculate the mean location of each dot in the scene as well as the deviation from that mean location represented by the above differences in cell location. The mean location gives the true location of the individual dot in x,y space and the deviation (which is sign-sensitive) gives the location of the dot in the z-dimension. With this information for each dot (or edge) available in mathematical form, the part of the complete saliency map related to spatial position of objects in the scene can be prepared. 7.4.3.1.2 The practical solution to the correspondence problem

Look more closely at the correspondence problem. Based on the near parallel nature of the instantaneous fields of view of the corresponding and near corresponding pixels of the eyes, very few, if any, false targets (or ghosts) are introduced into the spatial domain of the associative correlator when a scene is at a reasonable level of complexity. At a reasonable level of complexity, the scene can be analyzed by the pulvinar without difficulty. This is the situation in reading, except for the occasional highly illuminated first 180 Processes in Biological Vision

character of a paragraph opening a chapter. It is also the case in most other analytical situations. If, on the other hand, the scene is quite complex, the correlator may report many potential pairings that are not relevant. This situation will require more time for the pulvinar (and possibly higher cortical centers) to interpret the scene content satisfactorily. Alternately, the complexity of the scene may result in a very low contrast ratio in the signal channels related to the different pixels. High contrast signals rely upon distinct edge crossings. Under these conditions, the TRN may instruct the eyes and muscular-skeletal system to establish a different view of the scene to allow easier analysis. This might even result in the reliance upon a microscope or other aid. 7.4.3.2 Comprehensive model of the horizontal vergence system

This section on the horizontal vergence system (HVS) may appear out of place. It will rely heavily upon the models of the brain developed in Section 15.2. The performance of the model, and that of the actual horizontal vergence system, will be shown to be similar to spatial contrast functions of the visual system as a whole (described in Section 17.6.3). The delay aspects of the vergence system, along with those of the pointing system overall, which are dominated by projection delays in the stage 3 circuits, will be discussed in Section 7.4. The material will focus on the human HVS because it is significantly higher in performance than that of most other animals. This higher performance is needed to allow the analytical mode of operation to achieve the performance level it does (especially in reading). The details of the performance achieved by the analytical mode of vision will be discussed in Chapter 19. Justifying the development of the models defined in Section 7.4.3.1 is possible by using the more comprehensive model of this work. Highlighting some problems with the earlier models is also possible. The key new feature introduced here is the fact that the vergence and focus subsystems are entirely dependent on tremor for their operation. This is in spite of the fact the community has largely ignored tremor as unimportant during the last 40 years. This ignorance has prevented the community investigating HVS from defining the source of their conceptual vergence disparity signal.

In the absence of tremor, the explanation of how the eyes can converge to a null condition without the underlying servomechanism becoming unstable has represented a “difficulty” in textbooks.

A major problem is their lack of an appreciation of the importance memory plays in the operation of the pointing system, including the vergence system. It will be shown that the superior colliculus maintains a large memory for converting command instructions, from either the awareness or volition circuits of the brain, into absolute directional commands relative to the local skeletal system. This fact is mentioned to suggest that some reported data may suffer in accuracy if the data were not collected using randomized test stimuli.

It is also important to differentiate between the operation of the three major modes of operation of the precision optical system, POS, of the HVS. These are the autonomous mode associated with such repetitious tasks as reading, the volition mode processing instructions from the higher cognitive centers and the alarm mode processing responses to the detection of change by the LGN/occipital couple. This distinction will make it easier to interpret papers dealing with the psychophysical responses of the visual system. These distinctions are most easily made in Section 7.4 when discussing the time delay related to these different signal paths.

The most comprehensive model of the overall human visual system (HVS), circa 2002, is shown in the figures of Section 15.6.4. A slightly simpler set circa 1998 is shown in Sections 15.2. The operating modes of the servoloops forming the Precision Optical System (POS) are shown in Section 15.3.

The caricature in Figure 7.4.3-3 provides the simplest framework for discussing the operation of the HVS and the origin of the (frequently named) disparity vergence signal of the literature. Recognizing that the eyes are continually scanning object space under the control of the POS. is necessary. When not responding to a pointing command, each eye is responding directly to the instructions of the tremor generator. Defining a typical operating sequence will aid in the following discussion. This sequence will begin with a large pointing command to the POS from the awareness or volition signal paths. If the eyes should fail to bring the desired target in object space to within about two arc minutes of the line of fixation, a second pointing command may be generated. This rarely happens. The next step in the sequence is for the POS to both focus and converge the eyes on the target. It might be assumed that this focusing and convergence process begins from scratch following each major saccade. However, this does not seem true. The operating commands processed by the superior colliculus appear to contain an estimate of the required focus condition and degree of convergence appropriate to the estimated target distance. For the instruction received via the volition signal, this estimate is contained in the saliency map that already exists. For the alarm signal instruction, the information may be inserted by the superior colliculus. Thus, the focus and convergence circuits appear to work routinely in more of an optimization mode. In this mode, the eyes are focused and pointed generally as required when they first observe the target in the foveola. However, an unacceptable error may be contained in the overall response. Because an edge of the target is encountered earlier in one eye than in the other due to scanning, a temporal vergence error is measurable at the PGN. Dynamics of Vision 7- 181 The PGN creates a vergence error signal proportional to the temporal width of this error signal and transmits it to the oculomotor neurons. This signal can occur under two conditions, first where the left eye is leading and second where the right eye is leading. It is likely that these two error signals are transmitted to the oculomotor neurons over separate signal paths. Depending on which signal is dominant, the error is used to drive both of the eyes nasally or temporally to achieve convergence. Following the above action, the POS continues to create signals such as those shown in the figure until the spatial vergence error is forced below an acceptable threshold value. The complete pointing and vergence servomechanism is shown in block form in [Figure 15.3.1-3]. Note the multiple delay terms in this figure associated with the projection neurons connecting the various signal processing engines. By expanding this figure to a circuit level, [Figure 15.2.4-3] is obtained. This figure can be compared with the mathematical model of figure 6 in Krishnan & Stark of 1977365. This model contains four major components, three of which appeared in that paper. The analog elements before the delay term correspond directly to the analog circuitry of the photoreceptor cells (stage 1). The delay term corresponds to the summation of the individual projection delays plus the computational delays associated with each signal processing engine and the delay associated with the plant. The plant corresponds to the more mature plant shown in [Figures 7.5.3-3] and [Figure 7.5.3-4. It is interesting that the complete P/D Equation for a step change in contrast of Section 16.4.1 and 16.4.2 describe precisely the form of the data of figure 5 in Krishnan & Stark. It is only necessary to overlay the two figures to determine the time constants of this model based on their data. The P/D Equation shows a variable amount of overshoot depending on the prior state of adaptation of the photoreceptors and the brightness and contrast of the vergence test stimuli. This is typical of what Krishnan & Stark describe as an integral-derivative controller. It is more commonly called a lead lag network. However, here the “network” represents the photoexcitation/de-excitation process of the chromophores (without any rapid changes in the state of adaptation of the photoreceptors. The rise time of about 0.6 seconds would suggest their 29-year-old subject was looking at a dim or very low contrast stimulus. It is noteworthy that Pobuda & Erkelens did not follow Krishnan & Stark in their 1993 paper366. Without assuming an integral-derivative controller, their model did not have a zero in the numerator. Without a zero, they were unable to account for the overshoot they measured easily. Because of this, they proposed a piece-wise linear circuit as a solution. This work confirms that the Krishnan & Stark model is an adequate precursor to a complete and continuous model. The properties of the P/D Equation can provide the variations in gain, time constant and overshoot that both Krishnan & Stark and Pobuda & Erkelens report in their data.

Pobuda & Erkelens compared their mathematical model (which lacked any circuit model to define the vergence error signal) with the models described in Hung & Cuiffreda up through 1993. This included the two-stage model of Semmlow, et. al367. Pobuda & Erkelens concluded that a two-stage model was unlikely. However, the data in their table 1 appears to support just such a two-stage model. When their test disparity was more than the diameter of the foveola, their data showed a different gain-time delay relationship than for smaller test stimuli. While the data of Pobuda & Erkelens is very useful, it suffers from at least two design difficulties. First, the Figure 7.4.3-3 Caricature of vergence control problem. use of color filters to separate the signal projected to each eye from a monitor nearly insures different time constants in the disparity signal created from the two P/D environments. This is particularly true since one filter was described as red transmitting. The filters were not described in technical terms. They also used a technique of partial

365Krishnan, V. & Stark, L. (1977) A heuristic model for the human vergence eye movement system IEEE Trans Biomed Eng vol. BME-24, no. 1, pp 44-49 366Pobuda, M. & Erkelens, C. (1993) The relationship between absolute disparity and ocular vergence Biol Cyber vol. 68, pp 221-228 367Semmlow, J. Hung. G. & Ciuffreda, K. (1986) Quantitative assessment of disparity vergence components Invest Opthal Vis Sci vol. 27, pp 558-565 182 Processes in Biological Vision

image stabilization to achieve an open loop servomechanism condition without encountering fading of the image as expected under full stabilization conditions. It does not appear that they accounted for this level of stabilization in detail in their calculations. The partial stabilization of the image would have a significant impact on the temporal bandwidth of the signal passing through the limited bandwidth of the adaptation amplifiers found in each photoreceptor cell. The convolution of the signal bandwidth and the bandwidth of the photoreceptor cell could be expected to have a significant impact on their experiment. Pobuda & Erkelens discounted all of the models they reviewed, except their piece-wise linear model, based on four criteria they developed. They proposed that a successful mathematical model met their hypothesis. The hypothesis was that; (1) the vergence loop processes disparity through channels that have low pass filter characteristics; (2) the filter characteristics depend on the amplitude of the disparity input; (3) the vergence loop contains a pure delay of between 80 ms and 120 ms instead of 160 ms as is generally assumed; (4) the vergence loop is insensitive to the rate of disparity change. Their first criterion is clearly inappropriate. If it were correct, they could have completely stabilized their images to obtain a completely open loop condition. As shown by Yarbus, Ditchburn and many others, the image disappears under this condition. The mathematical model must contain a zero in the numerator as zero (temporal) frequency. The second criterion is appropriate but not for the reason they suggest. The time constants of the P/D signal created within the outer segment of the photoreceptors exhibits a strong dependence on amplitude of the stimulus. The third criterion seems argumentative for two reasons. First, the individual delay terms of a more comprehensive model were not defined. Second, the method of defining the individual time constants adding to a net value was not given. The leading edge of the P/D mechanism is not easily defined using a “rise to 63%” criteria. Third, the net delay is a variable in the P/D equation. The delay associated with the P/D mechanism is highly dependent on both the luminance of the stimulus and the state of adaptation of the subject. The eight millisecond value they suggest appears compatible with the long time constant of the P/D equation for σA FAτ > 1.00. The nominal value for the Standard Eye of this work was taken as 12.5 ms. The fourth criterion appears to also be argumentative. Higher quality data is required to prove the point. The vergence loop will fail if the product of brightness and contrast falls below a critical threshold. Because of the reliance upon tremor to transform the spatial sharpness of an edge into a temporal signal, the rate of change of the disparity is highly dependent on the brightness-contrast product.

Noting that the “sine wave response” in figure 3 of Pobuda & Erkelens is actually the response to a sine wave on a pedestal (as shown) is useful. This is significant because of the low frequency characteristics of the actual vision servo loop.

The question of a two-stage model becomes one of semantics when the presence of a significant memory component in the signals delivered to the POS by both the awareness and volition channels of the human visual system. This information could be introduced from a separate stage or be considered a set of initial conditions for the vergence servomechanism. 7.4.3.2.1 The perception of 3D by the neural system

(b) of the figure has been expanded to include a second point, R to the right and more distant than F, for the purpose of discussing the mechanism of fusion and depth perception. Fusion is a two–step process beginning with the calculation of a most optimal convergence angle and state of accommodation based on the determination of the dominant element in object space, taken here as the object at F. This calculation is first performed in the lateral geniculate nucleus (LGN) producing a qualitative signal used to drive both the oculomotor vergence system and an accommodation system. The calculation is then repeated within the perigeniculate nucleus (PGN) after qualitative convergence has been achieved. The resulting precision convergence angle is then used to drive the oculomotor vergence and accommodation systems in vernier mode. The initial state of convergence and accommodation are set by preset values stored in non-declaratory memory. These presets explain why it is so difficult to see an airplane against a clear blue sky even though it is readily heard and tracked by its sound to within a few degrees of its location. By looking at another prominent feature at approximately the same estimated distance to establish appropriate presets, and then looking back at the estimated location, the airplane is frequently perceived suddenly. The process of selecting the dominant element in object space, possibly based on the figure-ground concept, requires additional research. The diameter of the foveola projected into object space largely determines the size of the fronto-parallel plane Dynamics of Vision 7- 183 (FPP) of precision binocular (stereopsis) vision.. These calculations are more complicated than indicated above in the case of the precision binocular, stereopsis, case. The calculations lead to a mean vergence angle for the scene and a mean disparity of the scene (taken here to relate to the point F). They also produce a differential version angle (lateral disparity) and a differential depth disparity value for each element in the scene relative to the mean values. These tags associated with every element within the field of view of the foveola (taken here to be a 1.2 degree diameter in humans) are then used by the LGN’s to prepare a fused perception for the left and right hemispheres of the binocular field and by the PGN to prepare a fused perception of the precision field of view. For a mean disparity distance much greater than the interocular distance, the Vieth-Muller circle can be approximated by a straight line perpendicular to the sagittal plane of the eyes. The straight line and the 0.2 degree tilt of the vertical axis (from a perpendicular to the Vieth-Muller plane) in object space define the so-called fronto-parallel plane of stereopsis vision. A series of fronto-parallel planes parallel to the mean plane can be associated with the resolvable depth disparity values. Following optimum fusion, the resulting preliminary 3D perceptions are passed to the subsequent stage 4 engines within the occipital lobe from the LGN and within the pulvinar from the PGN, for additional information extraction. As a result of the additional tagging, it is likely the information associated with a given scene element is passed to the occipital lobe and pulvinar over address nerves using multiple neurons supporting word-serial/bit parallel encoding. 7.4.3.3 Detailed model of the horizontal vergence system

This work has developed detailed models of each element and circuit used in the HVS vergence system. These individual models have been individually qualified based on the relevant data. By assembling these circuits from other sections of this work, a detailed block diagram of the HVS is shown in Figure 7.4.3-4. To avoid complexity, the two-step process involving both the LGN and PGN is shown as only involving the PGN.

Figure 7.4.3-4 Block diagram of the complete horizontal vergence system of human vision. No stage 2 analog processing is shown following the photoreceptor cells and prior to the analog-to-pulse conversion in stage 3 for signal projection to the pretectum (consisting of both the LGN and PGN. All stage 4 signal manipulation is believed to be performed in analog form (with stage 3 sections interspersed between widely separated (>2 mm) engines. 184 Processes in Biological Vision

This figure is similar to the simpler HVS block diagrams of the literature, but it is greatly expanded. This is particularly true with respect to the sensing circuits that lead to the generation of the vergence signal within the common block labeled the pretectum (or perigeniculate nucleus, PGN, in humans). Compare it with a similar recent diagram by Hung that still lacks any description of how the input function is derived368. The PGN is a very complex computational engine of the midbrain. It will be examined more fully below. The PGN accepts an input signal from each eye that describes the scene viewed by the foveola of that eye. It subtracts the two signals to generate a difference signal. The initial difference signal describes the difference in arrival time between the signals from the two eyes as shown in [Figure 7.4.3-1]. This output may also exhibit a distinct amplitude profile if the signals from the two eyes are not symmetrical. This can happen if the eyes are receiving spectrally different input signals, are operating under different states of adaptation, or the networks connecting the photoreceptors to the PGN have different gain characteristics. The initial difference signal is integrated by passing through a low pass filter. The output of this filter is then used to drive the lateral terminal nucleus in the motorneuron complex of each eye. The signal is applied to the LTN’s with a polarity that causes the two eyes to rotate in opposite directions. This rotation is counter to the rotation introduced by the pointing commands. The pointing commands are applied to the LTN’s with a polarity appropriate to rotate the eye in the same direction. The effect of the vergence signals is to change the angles αR and αL symmetrically such that αR – αL describes the convergence angle required to converge the eyes. This same signal can be used to focus the visual system at a particular distance from the eyes. The signals applied to the PGN are derived from the scene by scanning the line of fixation of each eye across the scene using one of the scanning patterns described in the text accompanying [Figure 7.3.7-1]. These patterns are of very small maximum angle. For purposes of discussion, this angle will be taken as 20 seconds of arc, about the diameter of three 2-micron diameter photoreceptor cells. This scanning is introduced by the twitch portion of the lateral oculomotor muscles in response to the perturbation generator shown in the figure. This twitch signal is introduced into the LTN’s with such a polarity as to cause the two eyes to rotate in the same direction. The result is output signals from the two eyes that are synchronous but exhibit a difference in time depending on the angular rotation of the eyes from parallelism and the distance to the scene ( the angle αR – αL). This is the vergence disparity (or “disparity vergence”) signal. While this time difference is dominated by the position of the scene relative to the eyes, it can be affected by the absolute delay associated with the P/D mechanism of the photoreceptor chromophores. This delay depends on the illumination level applied to the individual spectrally selective photoreceptors, as will be discussed further regarding the operation of the PGN.

While the time difference between the two signals at the output of the photoreceptor cells is quite stable, the amplitude difference can be quite large. This is because of three main factors.

1. The operating state of the chromophores of the individual photoreceptor channels can affect the net signal amplitude. 2. The operating state of the adaptation amplifier within these channels can also affect the net signal amplitude. 3. The logarithmic conversion of the current through the photoreceptors into a voltage by the diode load, labeled (4), at their pedicles can have a lesser impact on this amplitude.

There may also be an impact on the amplitude of these signals if a test set is used partially to compensate for the gross motion of the scene. Such compensation changes the frequency spectrum of the incident image relative to the limited passband of the adaptation amplifier as discussed below.

While the twitch mechanism associated with the oculomotor muscles is critical to the operation of the tremor mechanism, it is not significant in the vergence loop performance. Any contribution to the signal loop is filtered out by the limited high frequency performance of the adaptation amplifier and the projection stage neurons. The most appropriate mathematical model of the vergence subsystem consists of the “integral-derivative” type circuit of the P/D mechanism combined with the bandpass characteristics of the adaptation amplifier and the low pass characteristic of the oculomotor muscles. The resulting equation has a simple zero and a zero expression in the numerator and two poles in the denominator.

7.4.3.3.1 A mathematical model of amplitudes in the sensing portion of the vergence system

368Hung, G. (2001) Models of Oculomotor Control. London: World Sceintific, figure 35 Dynamics of Vision 7- 185 The complexity of the vergence servomechanism makes discussion of a suitable mathematical model awkward. However, it can be addressed initially in segments. This paragraph will concentrate on the amplitude aspects of the vergence disparity signals. The time delay aspects are addressed in the next paragraph. Conceptually, the model consists of a differencing circuit, the PGN, driving a plant, the oculomotor neuron and muscle systems, in response to two sources of vergence information. The two sources provide information concerning their normal, relaxed, condition. This condition is determined both genetically and through learning, as the morphology of the body matures. While the foveola are near the center of the retinas in humans, some animals have displaced, or even multiple foveola, that allow them to provide appropriate signals to their PGN (‘s). The two sources of vergence information are the photoreceptors of the individual eyes. The mathematical model applicable to these photoreceptors has been explored extensively in Chapter 12 of this work. The model includes separate terms related to the P/D mechanism of the chromophores, the variable gain and limited bandwidth of the adaptation amplifiers and finally the logarithmic conversion associated with the pedicles. Without addressing time delays, and under small signal conditions, the voltage at the pedicles of the photoreceptors in the foveola of each eye can be described by:

VLnREd=⋅=∫ ()λλλ () LnA Eq. 7.4.3-1 where R(λ) is the responsivity of the photoreceptor and E(λ) is the radiant flux applied to the photoreceptor. Subtracting the signals from the corresponding photoreceptors in each eye results in an equation of the form:

log AL – log AR which can be rewritten, Log [AL/AR]. Eq. 7.4.3-2 It is interesting to look at the form of Eqs. 7.4.3-1 and 2 for each source and the logarithmic manipulation illustrated above. If the two sources are operating under identical conditions, and complete convergence is achieved, all of the terms within the expression, log[AL/AR], cancel. The variation in amplitude at the output of the correlator, due to the source signals, would be zero. This is the nominal condition. The only output of the correlator is a signal related to the time delay between the two signals as discussed below. This delay may introduce an amplitude term in the output related to the terms AR and AL being shifted in time before complete convergence is achieved. When complete convergence has been achieved, even the output due to the relative delay between the signals is also equal to zero. 7.4.3.3.2 A mathematical model of delays in the vergence system

The time delays associated with the vergence system can be defined using [Figure 7.4.3-2] and figures in Section 15.2.5 which illuminate some of the delays associated with the signal projection circuits of vision.

The following analysis will not discuss delays arising in the awareness and volition portions of the visual system. These delays are distinctly external to the closed loop performance of the vergence system. As noted earlier, these inputs can be considered a separate stage in the overall performance of the vergence capability, or as establishing the initial conditions within the closed loop vergence servomechanism. These additional delays have been discussed in Section 15.2.5. That discussion is consistent with the delays associated with the signaling paths presented in Section 7.4.

Major delays arising within the vergence servomechanism are: 1. The delay (relative to an arbitrary reference) in the light from an edge (or other feature) sweeping across the profile of a given photoreceptor. 2. The delay in the reporting of that edge sweeping across the outer segment of the photoreceptor due to the P/D mechanism. This delay is a function of the excitation state of the chromophores. 3. The delay associated with transferring the signal generated by the chromophores to the pedicles of the photoreceptors. This delay varies with the length of the axons, which vary with position of the photoreceptors within the foveola. 4. The delay associated with the transfer of the signals from the pedicles, through the stage 2 and stage 3 neural circuits, to the output of the stellate cells connecting directly to the nodes of the PGN. 5. The computational delays associated with the PGN and other neural processing within the POS. 6. The delays associated with the transfer of neural signals from the output of the PGN to the oculomotor muscles. 7. The delay inherent in the muscles of the oculomotor system. 8. The delay due to the inertial properties of the ocular globes. The above forms an impressive list. Seeing why a nominal delay of 160 ms is commonly reported in the literature is easy. However, only a few of the terms are variable. Terms 1, 2, 5 and 8 are the dominant variables. The others are 186 Processes in Biological Vision

nominally fixed. The goal of the vergence system is to minimize term 1. Term 2 is a direct function of the illumination level falling on the photoreceptors. This term is under the complete control of the scientific investigator. It can range from less than 1.0 milliseconds to large fractions of a second under laboratory conditions. Whether the value for term 5 is a variable is not currently known. It undoubtedly varies with the complexity of the scene. Term 8 is well documented (Section 7.3.4 through 7.3.6). Under closed loop conditions, terms 1 and term 8 are driven to zero, subject to the limits on loop performance introduced by the other delays (and any compensation provided by other cognitive processes). By accumulating the fixed delays due to terms 3,4,6 & 7, accounting for 120 to 150 ms is easy. Terms 4 & 6 are roughly one or two ms each. Term 3 can be much larger. The remaining delay appears to be associated with the computational time required by the PGN, term 5. The average value of term 5 is not currently known. It could be 750-125 ms. For further details regarding these delays, see the appropriate sections of Chapter 13, signal processing, Chapter 14, signal projection and Chapter 15, higher signal processing. Under closed loop conditions, the remaining delay due primarily to the P/D mechanism brings the total delay into the 160-ms region generally reported in the literature. 7.4.3.3.3 A mathematical model of the plant portion of the vergence system

The plant associated with the vergence servomechanism is the same as that discussed regarding pointing in Section 6.35 & 7.3.5. The only difference is that here the plants associated with each eye are driven differentially rather than in tandem as in pointing. Because the geometry of the ocular globe and muscles introduce transcendental factors (trigonometric terms), this mode may introduce some second order effects when discussing vergence at high angles relative to the rest position of the eyes. However, under closed loop conditions, the net effect will be small. The model from the above sections will be adopted in whole for the discussion in the next section. 7.4.3.3.4 Discussion of the overall mathematical model of the vergence system

Using the definitions in Section 7.3.4, the vergence servomechanism can be described as a Type 2 servomechanism with computational enhancement. This designation applies under normal operating conditions. It exhibits near zero error (noise limited) for fixed objects in the scene. It will exhibit a fixed spatial error when tracking a moving scene. Its ability to track accelerating scene elements is distinctly limited. Through training, the subject can learn to introduce “Kentucky windage” to maintain convergence and focus on objects moving along unusual trajectories. This capability is analogous to that available within the pointing servomechanism.

The servomechanism uses the same stage 1 circuits associated with the photoreception process. It uses a separate G–channel (Y–channel) that involves the direct transfer of signals from the photoreceptors to the mid brain. This channel does not involve any stage 2 signal processing although it still relies upon stage 3 signal projection mechanisms. The system involves a differencing technique within the PGN that makes the system largely immune to amplitude variations in the signals used to create the vergence disparity signal under normal operating conditions. However, laboratory conditions may disrupt this balance and introduce largely spurious (test set generated) amplitude effects. The major limitation on the performance of the system is the total delay due primarily to the length of the neural paths within the stage 3 projection paths. A secondary limitation relates to the time delay associated with the PGN, especially when simple edges are use as test targets. The PGN appears to be optimized for a scene of a specific, as yet un-characterized, complexity. Finally, the performance of the system can be reduced if the scene is not adequately illuminated. Best performance is apparently achieved at light levels similar to those prescribed for a drafting room or for performing fine hand-eye work such as quality sewing. The complexity of the servomechanism makes the presentation of a simple mathematical model, such as those in most of the literature containing less than three variable terms, inappropriate until many test criteria have been established. The partial model associated with just the amplitude response of the photoreceptors of each sensory channel (eq. 7.4.3-1) can be written in expanded form. In expanded form, the amplitude portion of the equation involves a definite integral containing at least seven parameters including the simple zero in the numerator. The latency portion of the equation contains at least four terms containing at least one of these parameters. One parameter appears in both equations and is directly proportional to the absolute intensity of the stimulus at the retina. This dependency makes it very difficult to compare the work of investigators using different test sets. By insuring that the two sensory channels are treated symmetrically in the design of a test set, many terms containing these parameters can be made to cancel completely. Here, the output of the PGN is a simple signal describing the time delay related to the vergence disparity angle, αR – αL. This error signal has a rise time limited primarily by the rise time of the P/D mechanism. Under conditions of unusual scene brightness, the limit may be due to the passband of the adaptation amplifier. Under the assumption that the subject is either emmetropic, or wearing appropriate glasses, these limitations are usually encountered before any other channel restrictions. As addressed in Section 7.3.10, the focus Dynamics of Vision 7- 187 servomechanism and the vergence servomechanism work in parallel and simultaneously. If the eyes are not able to focus properly on the object, the risetime associated with sweeping the scene across the photoreceptors of the retina will be degraded. This will impact the signal-to-noise performance of the vergence servomechanism and its ultimate convergence capability.

Figure 7.4.3-5 provides disparity vergence responses very similar to those predicted in the above discussion369. While the model presented in Zuber & Stark was not confirmed by their data, the data is in excellent agreement with the theory of this work. The initial delay of about 200 msecs is compatible with a relatively low light level stimulus. The curves clearly show the difference between two P/D responses passed through the adaptation and logarithmic conversion processes and then subtracted from each other as described by the model of [Figure 7.4.3-2]. Note the long response exhibits a break at 500 + 200 msec. This is appropriate for a stimulus that is long compared to the impulse response of the network under test. The shorter response does not exhibit such a break. The response is the impulse response of the network when driven by a pulse properly described as an impulse relative to the network. The impulse response is essentially the P/D response of the photoreceptor cell. The model also supports similar response data acquired by Jones (S & C pg 300). In the Jones data, Jones appears to overlook the fact that he is using three different types of stimulation and not two. While he uses a step stimulus, a 200-msec pulse stimulation and a 50-msec pulse stimulation. As in the Zuber & Stark example, the 50-msec stimulation is actually an impulse, not a pulse, from the perspective of the circuit under evaluation. The 50-msec interval is shorter than the impulse response of the circuit by itself. This accounts for the reduction in amplitude of his third waveform without any change in waveshape compared with the second waveform. The quality of the match between theory and measurement depends on the particular configuration of the test set as this affects the balance shown by the signals being differenced. If these are not balanced, the net waveform will reflect this imbalance. The Jones data clearly shows the 160 msec latency associated with th vergence disparity response (regardless of the duration of the stimulus duration). It should be noted that averaging the instantaneous amplitude values associated with five responses is not appropriate if the intent is to preserve the latency associated with the response. Such an average will always show a (less than distinct first order) latency equal to the minimum latency of the set.

Jones found the Fourier transform of his responses exhibited a corner frequency of about 1/3 Hertz. This value is virtually identical to the baseline time constant of the adaptation process of this work. Such a time constant gives a frequency response falling as 6 Db per octave above the corner frequency. Depending on the illumination level of the stimulus, this corner could be a double corner with the frequency response falling at 12 Db per octave above the corner frequency. See similar responses in Section 17.6.3. The response of a circuit to a pulse with a duration shorter than the time constant of the circuit invariably exhibits a peak amplitude proportional to the pulse duration. This is clearly seen in the Zuber & Stark figure. Figure 7.4.3-5 Averaged disparity vergence responses 7.4.3.4 The vergence data available in the obtained for 2-degree convergent disparity pulses of 100 literature msec and 500 msec. Labels have been added and changed to emphasize these are responses. See text. Modified from It is recommended that readers review Sections 7.4.3.2 Zuber & Stark, 1968. & 7.4.3.3 before attempting to review the published data. It is presumed that this will provide the reader a greater understanding of the test configuration affecting the data. The data of Krishnan & Stark form a good starting point370. They describe their model as heuristic. This seems correct from both a pedagogical and a problem solving perspective. They provide very little data on the conditions related to their test equipment (or the test equipment of others they have relied upon in their discussion). Their data in figure 5 and 7 is clearly that of a Type I servomechanism exhibiting significant internal delay. The delay is about 150-200 ms. The dominant time constant is about 1.6 Hz. This value corresponds generally to the time constant of the physical plant, the

369Zuber, B. & Stark, L. (1968) Dynamical characteristics of the fusional vergence eye movements system IEEE Trans Syst Sci Cyber vol. SSC-4, pp 72-79 370Krishnan, V. & Stark, L. (1977) Op. Cit. 188 Processes in Biological Vision

eye muscles combined with the inertia of the ocular globe. They do show some overshoot in their data that leads them to suggest an integral-derivative type of controller. As they show in figure 4, the choice of a conventional integral- derivative controller leads to considerable ringing that is not seen in their data. Under some conditions, the ringing could be filtered out by the time constants associated with the plant. The measured overshoot is completely compatible with the overshoot associated with the P/D mechanism, although other (isolated) pre-emphasis circuits within the neural circuits could contribute to the displayed amount of overshoot. The P/D mechanism does not exhibit any ringing. The time constants shown in their mathematical model for the controller are quite compatible with the time constants of the P/D mechanism under low photopic conditions. The cumulative delay of 160 ms shown in the model is also realistic for low photopic conditions, where the P/D mechanism only contributes a few to 10 milliseconds to the total. As noted above, under higher illumination, the total delay could be reduced by 10-20 ms and the total settling time could be reduced to less than one second. Frame (a) of their figure 5 shows a slight turn of the response away from the zero level near three seconds. This is compatible with the low frequency limit of the adaptation amplifier passband in the model of this work. Their figure 3 is not used in the paper. It is taken from a more extensive model of the plant appearing in an earlier paper. That model appears to have many second order conceptual features.

The data of Pobuda & Erkelens reflects many different test modes371. Their figure 1 suggests the forward bandwidth of the vergence servomechanism is less than 0.6 Hz and is similar to that found for the version servomechanism. This value is much lower than the bandwidth of the oculomotor plant alone.

371Pobuda, M. & Erkelens, C. (1993) Op. Cit. Dynamics of Vision 7- 189

7.4.3.4.1 Transient response of the vergence system

Rashbass & Westheimer have performed a series of experiments designed to measure the forward open loop gain of the vergence subsystem. Their test set did not have the frequency response required to measure the flicks and tremor associated with the analytical mode. For vegence stimuli applied within the field of the foveola, they recorded data suggesting the vergence response was bandlimited to less than 0.6 Hertz. This is compatible with the model of this work. 7.4.3.4.2 Vergence deviation as a function of stimulus location

Schor discussed vergence deviation as a function of forced vergence for conditions where the visual field centered on the point of fixation is blanked out. The discussion centered on the difference in performance between the foveola and the peripheral retina. It focused on the data in two papers by Ogle, et. al372,373. 7.4.3.5 State diagram for the vergence subsystem RESERVED 7.4.3.6 The role of torsion in vergence and version

Pedagogy usually describes the motions of the oculars in terms of an orthogonal system of axes (Section 2.2.1) and suggests the muscles attached to the oculars act orthogonally. This is not the case.

Whenever the two eyes are not looking straight ahead (focused on infinity) in the plane of the optical axes of the two eyes, the two eyes exhibit an independent rotation of their optical axes in the plane of focus. Such rotations would prevent the image from the two eyes to be merged into a single composite image except at the very point of convergence. The role of the torsion subsystem is to correct this condition over the largest region of the object field as possible. In human vision, the goal is to correct the image within the region associated with at least the foveola, 1.8 degrees (2.0 degrees according to Helmholtz in 1867374, 1.5 degrees according to Nakayama in 1983375).

The geometry of the situation and the operation of the torsion subsystem remains a subject of discussion in the literature. Porrill et al, discuss the history of stereo vision beginning with the simple Donder’s Law that focuses on the plane of the horopter and disregards the torsional error. They note Listing’s Law covers the more general requirement376. Contrary to their assertion, this subject has been studied to exhaustion within the field of stereo-photography for aerial and satellite reconnaissance . The subject has been documented extensively in that field where the torsion-angle is called the crab-angle. The geometry has been analyzed for both flat, curved and spherical object surfaces and ahead, behind and lateral to the flight path. These conditions completely encompass the equivalent situation in vision. The Jet Propulsion Laboratory is the current leader in the correction of stereo-images to remove distortion preventing optimum merging. When using photographic film, that does not stretch easily, an average crab-angle must be used to optimize the field. When electronic sensors are employed rather than photographic film, correction is possible on an individual pixel basis before merging. The human imaging system is not believed to employ pixel-by-pixel correction within the lateral geniculate nuclei where merging appears to occur.

Porrill et al, have described both the cycloversion and cyclovergence components of the overall binocular torsion parameter. However, they only explored the vergence condition as a function of field elevation. They discuss the purpose of Listing’s Law as if it is implemented biologically for an unknown reason and constitutes a limitation on the performance of the system (their section 4.2). In fact, Listing’s Law is implemented in order to optimize the performance of the system when viewing the real world external to the subject. Figure 9 of Porrill et al, summarizes the situation well for the conditions they explored. The achieved performance appears compatible with the theoretical requirement. A more complete exploration of both vergence and version with variations in elevation should conform to the situation documented as above.

372Ogle, K. Mussey, F. & Prangen, A. (1949) Fixation disparity and the fusional processes in binocular single vision Am J Ophthalmol vol xxx, pp xxx 373Ogle, K. Martens, xxx. & Dyer, xxx. (1967) xxx 374Helmholtz, H. (Xxx) Southall translation: Treatise on physiological optics, 3rd Ed. NY: Optical Society of America 375Nakayama, K. (1983) Kinematics of normal and strabismic eyes In Schur, C & Ciuffreda, K. ed. Vergence Eye Movements: Basic and Clinical Aspects Boston, MA: Butterworths pp 543-564 376Porrill, J. Inins, J. & Frisby, J. (1999) The variation of torsion with vergence and elevation Vision Res vol 39, pp 3934-3950 190 Processes in Biological Vision

Ostriker et al have provided detailed modeling of the human ocular rotation focusing on the non-orthogonality of the musculatura of the two eyes377. They develop the paths followed by the eyes when commanded to off-axis positions in considerable detail but do not concern themselves with the corrections introduced by the torsional pair of muscles to ensure the ability to merge the two images during convergence. After discussing the prior tendency within the community to consider the ocular movements as orthogonal for convenience, they note, “In general, reducibility is a characteristic feature of orthogonal (e.g. Cartesian x, y, z) multivariable systems, but a decomposition of the multivariable CNS, such as the gaze-stabilization apparatus, into single dimensions along horizontal and vertical directions cannot be taken for granted. Caution is warranted in particular because of the non-orthogonality of both the extraocular motor and vestibular sensory apparatus, and in general because the gaze stabilization involves other systems t hat are even less orthogonal (e.g. neck muscles). Thus, it is preferable to find a general solution for the CNS control of gaze for any system of coordinates, rather than limiting the analysis to special quasi-orthogonal solutions.” Morisita & Yagi have recently addressed the stability of the eye as it relates to vergence and version in three dimensions378.

7.4.4 Filling in of a color

Having established in this work that the photoreceptors do not act as an imaging sensor, but instead as change detectors dependent on relative motion between the retina and the image projected on the retina, how a uniform color field is perceived becomes important. The biological mechanism of color rendition for an area in object space of constant chromatic and luminous intensity appears to be similar to that used in Hollywood to colorize old black and white movies. It involves identifying closed perimeters, or large contrast change contours of figures, and then attaching a specific chromatic parameter to each of them. In the case of the neural system, this process must be achieved without intervention by an outside intelligence. 7.4.4.1 The figure-ground concept of psychology

The psychology community long ago developed the concept of figure-ground within a set of Laws of Perceptual Organization (beginning with the founder of the Gestalt School of psychology, Werthheimer, 1923379).

Closed contours in visual space establish special closed regions in the percept describing that space. These regions are called figures and are perceived as lying in front of the “background.”.

The original concept needs updating in a variety of aspects.

C The idea of the retina as an imaging device is false and archaic. The animal retina is a change detector as proved for humans by Yarbus, by Ditchburn and by others. C The term contour is probably a poor choice of translation from the German. What is meant is either a closed contrast edge or perimeter of sufficient amplitude to be above threshold in stage 4 signal manipulation (information extraction) in the neural system, or a contrast edge forming a closed contour (figure) within a larger ground.. C Such a contrast edge in neural signal space is the result of the motion of the image applied to the retina across the boundary of one or more photoreceptors acting as change detectors. C The contrast edge may be interpreted in luminance space, the R-signal of neural signaling or in spectral space, the UV–, S–, M– or L– signal space. In complex images, the contrast edge in chrominance space is actually evaluated in the chrominance difference, O–, P– and Q–signal spaces. C The basic difference signal is generated as a contrast edge between the figure signal and the ground signal generated within an individual photoreceptor by the image motion (Section 7.3.7). C The basic difference signals are combined and integrated within the lateral geniculate nucleus and the perigeniculate nucleus depending on the area of the retina involved. C The figure-ground concept need not involve a physical edge or border. A contrast change of sufficient amplitude is enoungh.

377Ostriker, G. Pellionisz, A. & Llinas, R. (1985) Tensorial computer model of gaze--I. Oculomotor activity is expressed in non-orthogonal natural coordinates NeuroSci vol 14(2), pp 483-500 378Morisita, M. & Yagi, T. (2001) The stability of human eye orientation during visual fixation and imagined fixation in three dimensions Auris Nasus Larynx vol 283, pp 301-304 379Ellis, W. (1938). A source book of Gestalt psychology (pp. 71-88) in translation London: Routledge & Kegan Dynamics of Vision 7- 191 Contrast edges always belong to the figure, never to the background. The concept is cascadable. The smallest identifiable contrast perimeter, or contrast contour encloses the first figure. If the ground is delineated by a larger closed contrast perimeter, it becomes a secondary figure surrounded by a larger ground, etc. The identifiable contrast perimeter may be formed within the luminance channel or a specific spectral photoreceptor channel of vision. A contrast perimeter or contour may be more prominent if formed within the O, P or Q channels of vision. Yarbus describes the same procedure with regard to filling in a color as suggested by Wertheimer, and as is used in modern computer-based drawing programs. The system first establishes the perimeter of a figure. Once established, a color can be selected from a palette and used to fill in the perimeter. The manner in which the color is selected is very specific in biological vision. If the luminance of the interior of a large field remains constant for more than three seconds, it becomes a null field, in the absence of any other information, the fill color selected to fill that null field is the same as that of the surround (ground). However, if the photoreceptor signals defining the perimeter indicate a change in luminance or chrominance between the outer edge of the figure and the adjacent edge of the ground, the figure is assigned a luminance and chrominance compatible with the integrated luminance and chrominance differences measured by those photoreceptors (relative to the surround). In this work, the notion of Yarbus, that the visual system “identifies” the empty field arising in artificial conditions with the empty field arising in natural conditions, is not needed. The cognitive and control portions of the cortex have instructed the ocular muscles to tremor. The system is an open loop. The cortex has no choice but to assume the muscles did as instructed. Therefore, the cortex has no way of determining whether the empty field is due to natural or artificial conditions. This realization is probably why he chose to use quotation marks around the word identifies.

The definite delay in filling in a color discussed by Yarbus is interesting. It is not obvious whether this delay is related to signal processing in stage 4 or to cognition in stage 5. See the call for additional research in Section 7.4.4.3.

7.4.4.2 Filling in for the blind spot and capillaries on the retina

Ditchburn380 discusses the fact that a stationary image on the retina is not perceived. He points out that this is why the blind spot and blood vessels on the surface of the retina, are not perceived. However, he humanizes the visual system by saying; “The visual system accepts the fact that no signals are received from certain parts of the visual field.” There is no need for this position. The computational and cognitive powers of the visual system work with the information received. Again, the cognitive and control portions of the cortex have instructed the ocular muscles to tremor. The system is an open loop. The cortex has no choice but to assume the muscles performed as instructed. Therefore, they process and evaluate the imagery detected by the retina. It does not happen to include any information about the blind spot or other scotomas. It does include changes in luminance contrast surrounding the blind spot. These changes constitute a perimeter. Therefore, the cortex assigns a color and luminance to the area within the enclosed perimeter in the usual way. Ditchburn does point out another important aspect. The cortex does not just assign a uniform color and luminance to the assigned area within the perimeter. It appears to fill the perimeter with a pattern matching that at the edge of the perimeter, again like “wallpaper” in a computer paint program.

Ditchburn also documents the fact, recognized by many people, that the blood vessels may be seen under two special conditions. If a bright light is shined into the eye from the side of the field of view, the shadow of the blood vessels may move relative to the retina due to normal tremor. They will be perceived under this condition. Some people will occasionally perceive the pattern of some blood vessels due to the expansion or movement of some blood vessels caused by the pulse from cardiac activity. 7.4.4.3 Processing luminance versus chrominance values at a perimeter

It appears that color contrast does not play a major part in whether a perimeter is recognized. If the luminance of the interior of the first perimeter is the same as the luminance found between the first and second perimeters, the first perimeter will not be perceived. The color of the inner perimeter will be chosen to be the same as within the second perimeter. The color found at the inner edge of each perimeter is sensed and used to fill in the rest of the interior of that perimeter. Whether more than one pixel is used to determine this color fill is not clear. However, this technique does explain why Mach bands are not seen in daily life except under very special conditions. 7.4.4.4 A uniform field is dependent on the luminance/chrominance at its perimeter

After a 15 year hiatus, Ditchburn & Foley addressed the question of color rendition in a partially stabilized image in two

380Ditchburn, R. (1973) op. cit. pg. 242 192 Processes in Biological Vision papers in 1985381,382. The first paper addressed step changes in the position of a stabilized image. In the second paper, they employed a colored diamond on a dark background and repetitive motions aligned to the diagonals of the diamonds. Tests for a possible variation with illuminance were made with red light (Chance’s OR]) at retinal illuminances of 640 and 60 id. The results were indistinguishable. It was not therefore considered important to equate the illuminances (for colours of different spectrophotometric composition) very accurately. Retinal illuminances of 500 td for yellow (Chance’s OY3), 370 td for green (Chance’s OGr l ) , 400 td for blue (Chance’s OBIO) and 320 td for blue (interference filter 448 nm) were used. Initially the subject classified the colour appearances into the following 4 grades. Grade 1: Full colour (appearance of the restored image similar to that of the unstabilized image except for some darkening near the centre). Grade 2: Good colour (the target appeared bright and of the same hue as the unstabilized image but there was a noticeable loss of saturation). Grade 3: Poor colour (the hue was distinguishable and, except when the yellow target was displayed, was the same as the hue of the unstabilized target. The target appeared pale in comparison with the unstabilized target and very unsaturated). For yellon the perceived hue was orange-definitely more red than the unstabilized target. Grade 4: No colour (field appeared grey). The results for different hues and for triangular-wave motion are shown as contour maps of frequency f and peak-to-peak movement M in Figs 2-6. Red gives rather larger areas for “full colour” and “good colour” than green or blue. Green and blue require about the same value of M for full colour but green is restricted to a narrower frequency range. The results obtained for blue (Chance’s OBIO) and blue (448 nm interference filter) are similar except that the interference filter gives good colour at a lower value of M and a narrower range off. The artificial oscillatory movements required to restore vision of the boundaries of a dark target on a white background were measured by Ditchburn and Drysdale (1977). The curve from their results giving, for each frequency, the movement required to enable the subject to see the boundaries 50% of the time is shown on Fig. 2. This curve indicates that the movement required for “good” colour vision is at least 5-fold larger than that required for seeing the boundaries. For “full colour” the movement is more than 15-fold that required for seeing the boundaries. Also the frequency range for colour vision is much narrower.

Good graphics were supplied describing these results.

Preliminary observations If the target is moved with a fairly large movement ( I 5 ‘ ) and the frequency is varied from 0.1 to I5 Hz, the following sequence is observed with green light at a retinal illuminance of about 1000 td. (i) At 0.1 - 0.2 Hz the edges of the target are seen intermittently and they are seen to move but the hue is uncertain. (ii) At 0.5 Hz, the green hue appears at the edges of the target but the centre is pale. (iii) At 1 .O - 3.0 Hz the appearance is similar to that of the unstabilized image in regard to brightness, hue and saturation. (iv) At 4.0 Hz the centre goes dark and appears unsaturated. (v) At 6.0 Hz the edges are clear but there is little discrimination of hue. (vi) At 6.0- 10 Hz the edges gradually disappear. (vii) At 12 Hz the edges are not seen but there is a flicker in the field over a vaguely defined area in the region of the centre of the target.

Effects similar to (vii) and (viii) are obtained with the unstabilized target.

If the movement is suddenly stopped when the target is seen (i.e. with a movement of 1.0 Hz to 4.0 Hz), the target disappears in about a second. If the movement is stopped when the target is not seen (13- 15 Hz) the target appears for about 2 s as though it were a freshly presented stabilized image. When the frequency is held at 3 Hz and the movement is gradually increased from zero, the edges appear at a peak-to-peak value of about 1 ’ , then colour at the edges of the target ( - 5 ’ peak-to-peak) and finally colour spreading over the whole target area ( - 10’ peak-to-peak). Similar results were observed with red and blue targets. Optimal frequencies were not quite the same for all colours. SUMMARY AND CONCLUSION ( I ) There is no significant difference between the effects of sinusoidal, triangular and square waves, provided the

381Ditchburn, R. & Foley-Fisher, J. (1985) Effect of imposed step movements and pulse movements of the retinal image on perception of hue with coloured targets Ophthal Physiol Opt vol 5(4), pp107 - 116. 382Ditchburn, R. & Foley–Fisher, J. (1985) Effect on perception of hue of imposed oscillatory movements of a stabilized retinal image Ophthal Physiol Opt Vol 5(4), pp 369 - 382 Dynamics of Vision 7- 193 amplitude and frequency are the same. This applies to frequencies of 1 Hz and above. (2) The movements required to give good colour vision are large compared with the movements required for perception of a boundary. At luminances in the range 300- 2000 td a factor of 10 is involved. (3) The above movements are also large compared with the movements which remain when a well trained subject fixates as accurately as he can. (4) The movements required for perception of red are less than those required for perception of green or blue. Yellow requires even larger movements. ( 5 ) Movements of frequencies above about 4 Hz (i.e. periods of 250 ms) are ineffective in regard to perception of hue. We conclude that, whereas for perception of boundaries it is sufficient to move the boundary through a distance equal to one-inter cone separation, hue perception is obtained only when the boundary sweeps rapidly over many cones. Perception of hue thus involves combination of signals from many cones. Also the signals must be maintained for a time of at least 100 ms. The data of Ditchburn & Foley–Fisher is compatible with, and supportive of the distribution of color photoreceptors in the retina described in this work. The experiments need to be repeated using more detailed protocols and selected narrow band lights in order to discover additional properties of the contrast edge detection (stage 1) and information extraction (stage 4) mechanisms. By using narrower band lights in these experiments, it should be possible to determine the relative utility of using luminance contrast versus various chrominance contrasts in reading and other tasks. (See Chapter 19 on reading.

7.4.5 Mechanism of precision convergence & Stereopsis via PGN-pulvinar

The closely related mechanisms of precision convergence and stereopsis underly the high performance aspects of both the fusion and depth perception phenomena associated with the foveola. These mechanisms will be discussed before these phenomena. Precision convergence and stereopsis are phenomena dependent on the foveola-PGN-pulvinar signal pathway. As of this time, no significant theories of precision convergence and precision stereopsis, fusion, and depth perception have appeared in the literature that successfully explain the operation of the visual system in these areas or along this signal pathway.

There is a problem with the theoretical derivations in the literature. That is their inconsistent treatment of the relationship between stereopsis and fusion. In some cases fusion is treated as a result of stereopsis, in some stereopsis is treated as a result of fusion and in some fusion and stereopsis are treated as distinct mechanisms. A similar situation exists between stereopsis and depth perception. This work will show that a clear difference exists between these concepts. Stereopsis is a specific visual mechanism involving a two-dimensional correlation process performed within the precision optical system and extending over the spatial field associated with the foveola. Both fusion and depth perception are perceived phenomena that result from the stereoptic mechanism. Stereopsis operates in a three-dimensional signal processing environment, x, y & z. Fusion is a perception related to the plane perpendicular to the line of sight and only involving the x,y plane. Depth perception is a perception related to distance along the line of sight and only involves the axis perpendicular to the x,y plane, the z plane.

Both fusion and depth perception has been studied for centuries. However, most of the studies have been qualitative. Most studies have failed to differentiate between performance related to the foveola and that related to the rest of the retina. This situation has been partly due to the precision of the instrumentation available. It has also been due to the lack of an adequate understanding of the underlying mechanisms of vision. Recent texts continue to be qualitative, follow the 19th Century investigators Maddox, Fechner, Panum and Hering, and be based on Gaussian Optics. They further assume the index of refractions of both the aqueous and vitreous humors are equal to 1.000. Schor & Cuffreda describe depth perception as falling off logarithmically with field angle. However, they do not describe how the response reaches a peak near the point of fixation. Neither do they discuss the significant change in depth perception performance at the edge of the foveola. This section will compare the wider literature with a more quantitative model. The two-dimensional correlation process is performed by the combination of the PGN and pulvinar, components of the thalamus in the midbrain. The two-dimensional correlator found in the PGN is the embodiment of the conceptual binocular correlator discussed by Lappin & Bell in 1976 and by Tyler & Julesz in 1980. The combination of the PGN and the pulvinar are the physiological embodiment of the so-called cyclopean retina of Julesz. The label cyclopean retina is misleading in that it suggests the existence of an image within the cortex. The information concerning the scene is actually stored in a tabular file and not as a multi-spatial-dimension image. The tabular file corresponds to (at least a portion of) the saliency map of Treisman in 1986 (Section 15.2.2). The two-dimensional correlator of the thalamus also supports the version, accommodation and analytical function of 194 Processes in Biological Vision vision. When the correlator and the additional memory functions of the pulvinar are combined, the couple are the principal elements responsible for the interpretation and perception mechanisms associated with reading and the study of images containing fine detail. 7.4.5.1 Background

The psychology community has struggled to understand the extraction of 3D information from the visual system for a very long time under the assumption that the retinas of the visual system are image detectors rather than change detectors. Pizlo has recently presented an entire book attempting to show how the parameters of figures in object space can be perceived as three-dimensional based only on a single 2D image projected onto one retina (Section 7.4.1.4.1). His entire discussion assumes the implicit operation of the retina as an imaging device. “Existing psychophysical evidence shows that the human eye is a calibrated camera (page 202).” This statement is patently false based on Yarbus, on ditchburn and on others. His arguments as a group are not convincing at the detailed level. His figure 3.13 can be disregarded. He proceeds to a Neo-Gestaltism and Neo-Empiricism to overcome classical problems found in the psychological explanation for 3D perception (Chapter 2). His discussion in Chapter 5 regarding a new paradigm based on recent work makes interesting, but largely anecdotal reading. He does present an interesting table in Appendix C comparing the goals of human vision research compared to the goals of machine vision. To understand all aspects of 3D imaging and particularly stereopsis, the operation of the visual system as a large array of change detectors (each retina) dependent on the relative motion between those detectors and the images projected separately onto the TWO retinas is mandatory. 7.4.5.1.1 Precision versus qualitative depth perception

Up through the 1950's, the depth perception of human vision had been assumed to involve only a single mechanism. At that time, Ogle stressed the likelihood of two distinct mechanisms but did not isolate them. He introduced the ideas of a high performance area and a low performance area. However, his criterion was the actual level of depth perception achieved rather than a relationship involving different areas of the retina. He defined a high performance area as that of patent stereopsis (threshold disparity less than 10 minutes of arc) and a lower performance area of qualitative stereopsis (threshold disparity greater than 10-15 minutes of arc). He also discussed areas of central and peripheral fusion (pg 99). With time, the area of patent stereopsis, or central fusion, has come to be called foveal stereopsis. This interpretation is rather loose and includes the area currently labeled the foveola, the fovea and the peripheral retina out to about seven degrees from the point of fixation. The more peripheral area of the retina became known as the area of ambient stereopsis or peripheral fusion. In more recent times, improved instrumentation has identified an area of significantly better stereopsis than the inner area defined by Ogle. The area bounded by the edge of the foveola (nominally 0.6 degrees from the point of fixation) consistently exhibits threshold disparities down to less than five seconds of arc.

The area of best disparity threshold could be defined as the area of foveola stereopsis. However, this could easily lead to confusion. It will be shown that this area involves a different mechanism than that used in the more peripheral areas. It becomes easier to speak of the area involving the foveola as the region of true stereopsis and use the alternate term, qualitative depth perception, to describe the performance in the surrounding area. In this way, stereopsis is associated with the foveola and the analytical mode of vision (involving the PGN). This leaves qualitative depth perception as associated with the remainder of the binocular field and with the awareness mode of vision (involving the LGN). Using this notation, stereopsis exhibits a linear relationship between disparity and the perception of depth (it is veridical) while qualitative depth perception does not. As frequently noted, true stereopsis requires geometric similarity between the images provided to each eye. This is to be expected since the stereoptic correlator of the PGN operates primarily upon the contrast edges associated with each object. The image provided to the LGN is much lower in spatial resolution than that provided to the PGN due to the off- axis performance of the lens group. As a result, the LGN is much less sensitive to the detailed geometric similarities of the images provided to individual eyes. The similarity requirement suggests that the term fusion is related primarily to stereopsis as associated with the foveola. Fusion of dissimilar shapes is a bit of an oxymoron. The above definitions are supported by the more recent literature. The comment by McKee is probably the most direct. “Fine stereoacuity is a property only of the foveola383.” Depth perception appears to occur outside the foveola. However,

383McKee, S. (1983) The spatial requirements for fine stereoacuity Vision Res. vol. 23, no. 2, pp 191-198 Dynamics of Vision 7- 195 its precision is much less, and judgements rapidly become impossible, see Ogle for details384. Tyler, writing in Schor & Ciuffreda, has provided a graphical description of stereo-optical performance using a spherical coordinate system but without providing any data points385. They also note the performance falls off exponentially with distance from the point of fixation. Figure 7.4.5-1 illustrates this capability along the horizontal meridian. The square data points and the dashed lines represent measurements made by Rawlings & Shipley using point light sources with a diameter of one minute of arc386. They employed a mirror haploscope adjusted for a fixation point at infinity. The large diameter of their sources obviously limited the threshold stereoacuity they could measure. They specifically noted, “There is a specific binocular function, a central process, without which stereopsis simply does not exist.”

Blakemore reported very similar results at about the same time387. He used multiple vertical slits 2.25 minutes wide and 2.25 degrees long. He notes that his measurements were limited to about 0.5 minutes of threshold disparity by the relatively low illumination levels used. It was probably also limited by the width of his lines and by scatter from the surface quality of the mirrors in his haploscope. He separated his measurements into those related to convergent and divergent disparities relative to the point of fixation. The results showed small but systemic differences between the two. By plotting his results on logarithmic scales, he showed that the disparity did follow an exponential curve for disparities greater than 0.6 minutes of arc.

384Ogle, K. (1953) J Opt Soc Am vol. 43, pp 906- 385Tyler, C. (1983) In Schor, C. & Ciuffreda, K. Op. Cit. pg 240 386Rawlings, S. & Shipley, T. (1969) Stereoscopic acuity and horizontal angular distance from fixation J Opt Soc Am vol 59, no. 8, pp 991-993 387Blakemore, C. (1970) The range and scope of binocular depth discrimination in man J Physiol vol. 211, pp 599-622 196 Processes in Biological Vision

Figure 7.4.5-1 Stereoacuity as a function of horizontal offset. See text for details.

The triangular data point and the solid lines represent the data of McKee. She used narrow vertical lines of variable length generated on Tektronix model 602 monitors using a P4 phosphor (33 msec decay time). Images from the two monitors were combined using a pellicle. Polarizers were used to isolate the image presented to each eye. A series of dots was used to form long lines. These lines were used by the correlator in the PGN to improve the stereoacuity reported. The dots on the screens were also about one minute of arc in diameter and spaced at 30 seconds of arc. The quality of the edges of these lines may have an impact on the performance of correlator within the PGN of the subject. The monitors had a refresh rate of 100 Hertz. The test images were presented for short intervals to avoid voluntary eye movements during each test. She says this configuration limited the minimum change in symmetrical binocular disparity to 3.4 arc seconds. Her subjects were chosen for their stereoacuity. She says the general population seldom exceeds 5 arc seconds stereoacuity under similar conditions. McKee provides considerable discussion related to the need for a physiological summation process that is compatible with the 2-dimensional correlator defined in Section 15.6 of this work. She confirmed that the summation was not based on the absorbed photon flux but on the information carried by the test structure. She also noted the complications encountered when abutting lines were used as a stimulus. Finally, she noted the observation of Westheimer & McKee that “presenting the target as little as five minutes of arc in front or behind the fixation plane elevates the stereo Dynamics of Vision 7- 197 threshold388.” This latter finding suggests a mathematical description of the stereopsis function (within the region of the foveola). It appears the threshold of stereoptic vision relative to the point of fixation can be described by an exponential function with a negative exponent. McKee (1983) and others have frequently discussed the variability in the limit of human stereoacuity among individuals. Textbooks generally give the limit as 2–10 sec of arc for “good” eyes. It appears stereoacuity is a significant function of learning and probably practice. Richards & Kaye have presented data on perceived depth perception as a function of stimulus disparity.389 Their data are for crossed (convergent) disparities only. The images were centered on the point of fixation. The data can also be interpreted as defining the perceived depth perception versus distance from the point of fixation. The perception of depth falls rapidly when part of the stimulus falls outside of the foveola. For stimuli completely within the foveola, the data shows good agreement with the linearity rule corresponding to the veridical condition (although the use of a logarithmic axis obscures this relationship). For stimuli at least partially outside of the foveola, the depth perception performance is only qualitative. Figure 7.4.5-2 is redrawn from xxx [ not from Richards & Kaye, could be Rawlings & Shipley, 1969, probably McKee, 1983 but coord are different. ]

7.4.5.1.2 Block and state diagrams for the process of stereopsis

Figure 7.4.5-3 presents a top level block diagram of the visual modality optimized for discussing stereopsis. It is an expansion of [Figure xxx] that highlights the parallel signal processing occurring in stages 1 & 2 of each eye. It also highlights the matrixing of stage 3 signals passing along the optic nerve in accordance with Section 15.2.5. Note the chiasm of the optic nerve is not reproduced as normally represented because the two LGN are not shown separately as in Section 15.2.5. A secondary bifurcation of the optic nerve is shown here dividing the Figure 7.4.5-2 Stereopsis as a function of field angle within signals of the optic nerve between the LGN and PGN the foveola. engines (Section 2.8.1). The PGN is also called a minor feature of the Brachia of the Superior Colliculus in the morphological literature, Section 15.1.6.

The arrows shown between stages in this figure represent stage 3 pulse signals (action potentials). In the case of the Qian & colleagues papers, it requires careful reading to determine whether their electronic probing of the striate cortex involves sensing pulse signals arriving at the occipital lobe or leaving the occipital lobe. The potential pulse signal paths associated with the cerebral cortex are described in Section 14.4.1. A saliency map, presumed to be located in the parietal lobe of the brain, but potentially of holographic form is shown as a subtitle under the parietal lobe. It is a critical engine between the occipital lobe and the engines of stage 5 cognition. It is presumed, based on discussions in Chapter 19, that the saliency map is initially provided with signals from the LGN- striate signal path and that these are updated by subsequent signals provided by signals from the PGN-pulvinar signal path. Nothing has been found in the academic literature expressing how the perceived depth of a feature in the external field

388Westheimer, G. & McKee, S. (1978) Stereoscopic acuity for moving retinal images J Opt Soc Am vol. 68, pp 450-455 389Richards, W. & Kaye, M. (1974) Local versus global stereopsis: two mechanisms Vision Res. vol. 14, pp 1345-1347 198 Processes in Biological Vision

of view is represented in the saliency map or in stage 5 cognition. The dark band, labeled the TRN represents the non-conscious “command and control” role of the thalamic reticular nucleus of the diencephalon (an outer layer of neurons covering a majority of the thalamus). A neutral density filter is shown in the optical path between the stimulant and the stage 0 optics of the left eye to support the discussion of the Pulfrich Effect in the papers of Qian & colleagues, Section 7.4.5.5. This figure can be interpreted as a detailed extension of the lower left quadrant of the Schematic of the recognition function, [Figure 19.10.6-1]. The signal path leading to precision stereopsis, the foveola–PGN-pulvinar path, is associated with the “What” path of that figure. The signal path leading to coarse stereopsis, the complete fovea–LGN–Occipital lobe, is associated with the “Where” path of that figure. The regions of the retina beyond 5-8 degrees eccentricity are probably not significant to the stereopsis mechanism.

Figure 7.4.5-3 Top level block diagram optimized for stereopsis discussions. The upper left quadrant illustrates the parallelism between the signal path for the two eyes. The upper center illustrates the complex matrixing of the early stage signal paths into the later stage signal paths. At the extreme left is a neutral density filter intercepting the stimulus reaching one of the complete eyes and important when discussing the Pulfrich Effect. See text.

Tyler & Kontsevich have provided a conceptual model of the visual system supporting their conclusions concerning the extraction of several features of a scene, including depth perception. The top level schematics of Section 7.3.1 lead to a much simpler view of the framework for stereopsis and other complex relations within the visual system than the approach of Tyler & Kontsevich390. Figure 7.4.5-4 is drawn to resemble their figure.

390Tyler, C. & Kontsevich, L. (1995) Mechanisms of stereoscopic processing: stereoattention and surface perception in depth reconstruction Perception vol. 24, pp 127-153, fig. 1 Dynamics of Vision 7- 199

Figure 7.4.5-4 A more detailed schematic of the top level block diagram focused on precision stereopsis. The binocular field of view is reduced at the 12 degree vergence angle shown. The oculomotor subsystem, the PGN and the pulvinar are under the operational control of the POS during the analytical mode of operation.

The plane of vision is shown for a typical viewing distance of not less than 30 cm (10 inches). The fields of view of the foveola are shown. When optimally converged, the two foveola image a common area of 1.2 degrees in diameter. The neural signals related to this area are passed over the optic nerves to the PGN of the midbrain. The neural signals from the surrounding field of view are passed over the optic nerves to the LGN. These signals can be further segregated into those related to the binocular field of view and the remaining monocular fields.

The signals related to the foveola are delivered to the PGN acting as a two-dimensional associative cross-correlator. The correlator exhibits an effective diameter equal to that of each foveola. It is typically 175 photoreceptor cells in diameter. Internally, it may exhibit a small multiple of this number of cells to accomplish its mathematical tasks. The main tasks of this correlator are two. The first is to establish a global average vergence disparity error signal for the total image on the foveola. If the deviation relative to this mean is acceptable (less than the required disparity for fusion), no further oculomotor action is required before further analysis. If the deviation is unacceptable, instructions are issued to the oculomotor subsystem, by the POS, to reconverge the lines of fixation. Following this fine tuning, the global mean and deviation are recalculated.

The second is to establish a local vergence disparity signal for each significant object in the image. This signal consists of a mean location of each object and a deviation from that mean describing the depth of the object relative to the global average vergence.

The global average disparity for the scene and the local means and deviations for each object in the scene are transferred to the pulvinar. The pulvinar is a large lookup table and general database storing nearly all of the detailed signatures previously identified by the subject. It attempts to identify each object in the field of view based on experience. Its output is commonly described as an interp, an initial interpretation of a small fraction of the scene (the following terminology is drawn largely from the field of reading research discussed in Chapter 19). Multiple interps may be required before a complete object is recognized at the level of the pulvinar. The result is commonly called a percept. It relates to the content of the instantaneous field of the foveola. By systematically moving the line of fixation around within the binocular field (Section 7.5.2), a group of percepts are assembled that result in a perception of the entire binocular field of view. These movements involve a series of flicks, and some minisaccades, followed by a period of tremor. The tremor period corresponds to the time the PGN absorbs the necessary amount of information about the instantaneous scene for it to establish a new interp. This period is frequently described as a quiescent period between saccades when low spatial and/or temporal resolution instrumentation is used to study eye movements. The binocular perception is passed to the next higher level of the brain for insertion into an even larger saliency map describing the overall perception of the external environment of the subject. This saliency map also receives information from other visual channels and from other sensory modalities. It is the saliency map that can be perceived and acted 200 Processes in Biological Vision upon by the higher cognitive centers. Figure 7.4.5-5 presents a preliminary stereopsis state diagram describing the operation of the stereopsis mechanism as an overlay on the pointing system of the POS. This state diagram will be discussed in terms of analyzing a simple set of geometric primitives combined into a single scene presented to the visual system of a subject. It can also be used to describe the process of reading. It includes inputs from both the awareness and volition modes of the overall visual system shown in the top-level block diagrams. Following the initialization process, the process continues cyclically under control of the precision optical system (POS) controller. Each cycle consists of three steps. The first step involves moving the direction of each major subject in the overall field of view into alignment with the line of fixation. The second step involves analyzing the content of the scene presented to the foveola. The third step repositions the line of fixation to interrogate the next most significant object in the overall scene. The next cycle occurs after the POS directs the line of fixation to the next most significant object. Dynamics of Vision 7- 201

Figure 7.4.5-5 State diagram for the stereopsis mechanism. 202 Processes in Biological Vision

The steps outlined in the preliminary stereopsis state diagram can be listed in more detail. 1. Initial setup A. Start in rest condition B. Accept high level a priori instructions (commands) from awareness or volition mode channels. C. Convert a priori commands to operational commands using the superior colliculus lookup tables 2. Implement operational a priori commands for accommodation, vergence and version to center the best images of a scene at the point of fixation for each foveola. 3. Prepare to transfer images of the scene to the midbrain A. Form an image of the scene on each foveola B. Microscan images using the physiological tremor facility C. Highlight the edges (contours) by differentiation within the adaptation function of the photoreceptor cells. 4. Transfer foveola images to the two-plane, 2-D associative correlator of the perigeniculate nucleus. 5. Perform vergence optimization on the scene presented to the correlator A. Compute global vergence value for scene B. Adjust vergence to minimize the error between computed global vergence and the a priori vergence value C. Adjust global accommodation value based on corrected vergence value and the lookup table in the superior colliculus.

------FUSION IS OBSERVED HERE ------

6. Compute local (mean) position and vergence difference tokens for each significant edge in the scene using the 2-D associative correlator.

7. Transfer all stereopsis tokens to the pulvinar

8. Compare edge tokens with the (experience based) signatures stored in the pulvinar.

9. Assemble the signatures to form an initial interp of the scene

10. Transfer interp of the initial scene to the appropriate cell of the initial saliency map

11. Repeat steps 1 through 10 for each foveola-sized image of the overall scene, under instruction from the POS.

12. Assemble individual interps, stored in individual cells of the initial saliency map, into a more comprehensive visual percept

13. Transfer the comprehensive visual percept to the overall visual saliency map of the subject.

14. Associate the comprehensive visual percept with signals from other sensory channels to form a comprehensive saliency map. 15. Make the comprehensive saliency map available to the higher cognitive centers for cognition and action. The above schematic of stereopsis and state diagrams can be compared with the five-step process offered by Tyler & Kontsevich391. The differences are major because of the basic differences in concept. Tyler & Kontsevich do not address how the information from the retinas is transferred to the CNS or where in the CNS it resides.

7.4.5.2 Background specific to stereopsis

391Tyler, C. & Kontsevich, L. (1995) Mechanisms of stereoscopic processing: stereoattention and surface perception in depth reconstruction Perception, vol. 24, pp 127-153 Dynamics of Vision 7- 203 7.4.5.2.1 Stereopsis as distinct from binocular vision

If as proposed, stereopsis is distinct from binocular vision, and disparity is primarily related to stereopsis, it follows that the term binocular disparity should be replaced by stereoptic, or stereo-optical, disparity. Efforts to define Panum’s Area have been pursued for a long time (Section 7.4.1.6). Leibovic provided a very conceptual plan view of Panum’s area using a Keplerian Projection392. The figure was reproduced from an earlier paper of his in 1970. The figure exhibits three significant problems. First, it does not conform to the preferred definitions of Tyler and of Howard. They define Panum’s area as a two-dimensional plane perpendicular to the line of sight. Such an area would appear as a line in Leibovic’s figure, not an area. Second, it does not recognize what is now known about the topography of the occipital lobes and the role of the PGN (and foveola) in depth perception. If the fields of view of the foveola are drawn on his projection, a useful relationship appears. The small area around the fixation point (1.2 degrees in diameter) associated with the foveola projections is the same area as delineated in the previous figure. This small area is only represented at low resolution in the topography of the occipital lobes as shown in Section 15.2, and especially [Figure 15.2.4-6 ]. The neurons from the foveola deviate from the rest of the optic nerve going to the LGN and eventually the occipital lobes (Section 2.8.1). These neurons proceed to the PGN and the precision optical system. Thus, the lines of fixation should be accompanied by the field of view of each foveola. The area within these fields of view does not appear at full resolution in either hemisphere of the occipital lobe. Third, the target vergence angle shown in the figure is much larger than found in the real world. This distorts the proportions of the figure. The Leibovic caricature also omitted the effect of accommodation and the spatial performance of the optics of the eye. These effects limit the area of high performance imaging to a small circle surrounding the fixation point. This circle is well matched to the circle representing the field of view of the foveola. 7.4.5.2.2 The PGN as a general purpose 2-D associative correlator

General agreement is found in the recent literature of stereopsis that a dedicated computational element associated with the foveola of the retina exists (at least in humans and some higher primates). At this time the literature has not linked any physiological elements of the HVS to such a computational element. The 2-D correlator of the perigeniculate nucleus and the lookup tables of the pulvinar and superior colliculus appear to provide the necessary computational capability and memory to satisfy the requirements of the stereoptic function. These entities are defined in considerable detail in Sections 15.5 & 15.6. They are also shown to be compatible with the requirements of the more sophisticated analytical functions of detail analysis of a scene and reading.

The literature has assigned a variety of conceptual names for the entity that merges the two images into a single “image” in stereopsis. A serious problem exists in associating the term image with an abstract mathematical message that is transferred to a general data base. Thus, while Julesz has used the term cyclopean retina to describe this entity as an image, there is clearly no 3-D spatial image formed within the CNS as part of the stereopsis mechanism. Similarly, Marr & Nishihara have spoken of a 2 ½- D image consisting of a 2-D image surface with attached tokens describing the third dimension. Here again, the word image implies the storage of information in a reasonably conformal surface somehow relatable to the input images. This is not likely to be the case. Tyler, writing in Schor & Ciuffreda, refrains from speaking of an image in this context and uses quotation marks when discussing the three-dimensional “fovea393.” It is most likely that the information from the foveola undergo anatomical computation before reaching the PGN and the organization of the data in the PGN is optimized for feature extraction, not conformal integrity.

Noting that modern commercial and military map-makers no longer maintain a master 2-D map of an area of interest is useful. Nor do they maintain a physical 3-D map of an area. All of the data is stored in a general purpose database in a digital computer. How the data is stored in the computer varies with ease of accessibility by the computers peripherals. It has absolutely nothing to do with a two-dimensional planar projection. The data base can be used to create images of the terrain never before seen by surveyors or anyone else. The convenience of using random dot stereograms has contributed to the concept that the information processed within the CNS is maintained in a recognizable series of planes. It appears from this work that this is unlikely. Even the reconstructed stereographic information appears to be stored as a vector containing a series of tokens. These tokens relate to the angular location and radial range of the original feature in “spherically specified” object space. In line with the above paragraph, it may be that anatomical computation is used to arrange the information supplied to the PGN in a more rectilinear coordinate system. If this aids feature extraction, it would surely be done. However, when discussing

392Leibovic, K. (1990) Perceptual aspects of spatial and temporal relationships Chapter 6 in Science of Vision, NY: Springer-Verlag pg160 393Tyler, C. In Schor, C. & Ciuffreda, K. (1983) Op. Cit. pg 239 204 Processes in Biological Vision feature extraction within the 1.2 degree diameter area imaged by the foveola, the question appears moot. The small angle approximation of trigonometry would suggest minor differences between the two representations. The ability of the associative 2-D correlator proposed in this work to correlate over local regions as well as more globally appears compatible with the concept of Tyler & Kontsevich ( pg 130). The claim by Tyler and Julesz in the last paragraph of their paper appears quite compatible with the model of this work. They are describing the operation of the awareness mode of vision, involving the LGN and occipital cortex. This capability operates without significant inputs from the vergence system and does not use the high quality memory associated with the mechanism of stereopsis associated with the analytical mode of vision. 7.4.5.2.3 The general operation of the PGN as an associative correlator

The PGN is discussed more fully in Section 15.6.6. This section will use Figure 7.4.5-6 to describe the operation of the PGN in its role of supporting the version, vergence and accommodation subsystems. This engine may be shown eventually to be the most functionally complex individual engine in the neural system. It is a two-dimensional, multi- plane associative processor containing some multiple of 23,000 data cells. This number is based on the nominal 23,000 photoreceptors within the foveola of each eye represented in each of the input planes of the correlator. It is likely there are at least three or four times this number of individual cells within the PGN. This larger number would provide a cell for each signal input and a cell for the sum and the difference signals relative to these inputs. This is the configuration shown in the figure.

It appears that the PGN accepts input from all of the photoreceptor channels of each foveola without regard to their spectral selectivity. It also appears that the PGN treats them all equally. As a result, the PGN can operate at the maximum spatial precision provided by the complete retinal array in each foveola. This methodology provides the maximum signal-to-noise ratio for a given scene contrast when the scene is illuminated by a broadband (white) source. In the above sense, the PGN operates in an achromatic regime but does not employ the spectral channel weighting found in the luminance, R–, channel (Sections 13.3 & 13.4 ). This signaling regime, associated only with the foveola, has generally been labeled the Y– or G–channel (depending on the context) in this work. 7.4.5.2.4 The detailed operation of the PGN as an associative correlator

[xxx Stereopsis is one of many mechanisms implemented by the perigeniculate nucleus, a part of the thalamus and within the precision optical system. The perigeniculate nucleus is organizes cytologically as a two-dimensional, multi-plane associative correlator very similar to the organization of the lateral geniculate nuclei (LGN). Like in the LGN, the information from the photoreceptors of Figure 7.4.5-6 The associative correlator of the PGN in its each foveola are inserted into separate planes of the role supporting the version, vergence and accommodation correlator. Unlike in the LGN's, where the planes are subsystems. The two loops highlight the ability of the divided into pairs to support the luminance and correlator to perform local stereopsis in the horizontal and chrominance channels of vision, the perigeniculate nuclei vertical directions. See text. treats all of the photoreceptors achromatically. The PGN uses the remaining planes to hold calculated parameters related to the images. The effective diameter of each plane of the associative memory is proportional to the diameter of the foveola, nominally 175 photoreceptors. The data from corresponding points in the foveola are loaded into similar locations within the x,y space of the correlator. For the PGN, it appears the association process related to stereopsis is largely limited to the horizontal and vertical directions. Dynamics of Vision 7- 205 The stereopsis mechanism mathematically merges the information (but not the images)from the two foveola. The mathematical manipulations of the stereopsis mechanism are carried out in two major steps, one of global stereopsis and one of local stereopsis.

First, using its associative capability (which is similar to that used in a variety of man-made military computers), the PGN attempts to pair up all significant edges in the two images within the effective spatial limits of the correlator. This process is called local stereopsis. The scope of this sub-mechanism is illustrated by the vertical and horizontal loops in the figure. Each loop can interrogate all of the cells in a given row or column related to the nominal cell, xn,yn of both the right and left planes of the correlator. The long dimension of each loop is adjustable under control of either the POS or the thalamic reticular nucleus (TRN). The correlator can institute a pair of such loops for each nominal location, xn,yn, within the correlator. The individual loop attempts to locate significant edges appearing in both the left and right planes of the correlator. When it locates a significant pair, it calculates a mean x,y location for each pair. It also calculates a signed difference in location, delta-x, delta-y for each pair. It places the value of this signed difference in the cell of an auxiliary plane with the x,y address equivalent to the mean calculated value. Having only two eyes arranged in a horizontal plane, the vertical signed difference in the above calculation is usually quite small relative to the horizontal signed difference. Second, the PGN then calculates the 2-dimensional centroid of all the means found above. This establishes the nominal coordinate of the center of the information, x0,y0. This is the first part of the global stereopsis process. It then calculates the mean of all of the differences associated with the above pairs (where the sign of the differences is important). This value is taken as z0.

The original means and differences calculated above are now readdressed relative to the centroid x0,y0,z0. The mean locations become xn,yn and the differences become zn. These values are assembled into a vector called an initial interp. It describes all of the significant edges in the small portion of the original scene imaged on the foveola. This interp is delivered to the pulvinar for further correlation and conversion into an initial percept of vision.

The values xn,yn,zn, obtained in the above process, constitute the mathematical equivalent of the cyclopean image described by Julesz. The value of the centroid, x0,y0,z0 corresponds to the error between the actual target location in object space and the a priori location the eyes were directed to by the POS. The x 0,y0 value corresponds to the eccentricity error and z0 corresponds the disparity error. If either of these errors is excessive, the POS may command a saccade or a change in vergence before proceeding.

The mathematical values xn,yn correspond to the "fused" image of each significant edge in the scene imaged on the foveola. The value z0 describe the distance, the local disparity, of each "fused" edge relative to z0. The purview of the correlator is circumscribed by Panum's limit in correlator space. This limit reduces to Panum's limit in x,y space for z = 0. The area enclosed by this limit is known as Panum's area. This limit can also be expressed by the maximum value of z for the condition x = y = 0. This limit describes the maximum range of depth perception relative to x0,y0,z0. For other values of x,y,z, a volume can be defined that is conceptually equivalent to Panum's two-dimensional area. It defines the combined range of fusion and depth perception achievable by the subject.

As a two-dimensional correlator, the PGN can increase the signal-to-noise ratio of its output signal by correlating over a larger area than that represented by a pair of photoreceptors, one from the same nominal position in each foveola. Thus, the length of the edge projected onto the foveola is important in determining the performance of the PGN, both in vergence signal generation and other signal extraction operations. Further, being a two-dimensional correlator, it can correlate differences over the entire area of the foveola. This explains the ability of the visual system to improve its performance using gratings, even sine-wave gratings, and other patterns relative to its single edge performance. It can even obtain vergence information from a series of dots placed within the object space viewed by the foveola. If the dots are not in the same plane, neural confusion, and even vertigo, can result. Such dots, and various three-dimensional patterns are used in conventional tests of vergence performance in the clinic. The desire for the photoreceptors of each retina to represent similar positions in object space introduces a condition of interest in the literature. Remarks that the two foveola may not be organized as mirror images of each other appear in the literature (Section 3.2.2.2). They exhibit swirl patterns that suggest they are anti-symmetrical, e. g., their patterns would overlay each other if projected into object space. 7.4.5.2.5 Putative use of spatial frequency filters and Fourier transform calculations in vision

Discussions and proposals appear repeatedly in the binocular and stereoptical vision literature concerning the likelihood 206 Processes in Biological Vision

of spatial frequency filters occurring in the visual system. Many authors also have speculated on the use of the Fourier Transform (and occasionally the La Place Transform) in conjunction with computations within the visual process. No data has been uncovered in the physiological or morphological areas supporting the use of such complex transform calculations in vision. Neither have any physiological structures been identified within the neurology of the visual system that could represent multi-section frequency domain filters. Using spatial frequency patterns, both square wave, sine wave, Gabor windowed and orthogonal, as stimuli in the laboratory is obviously easy. For waveforms of only a few cycles, the actual frequency of the underlying frequency is only known precisely because it was generated locally. However, showing any unique response to such patterns that cannot be explained using simple temporal domain (and spatial domain in the case of the PGN correlator) sum and difference circuits is very difficult. As in discussing strings of action potentials, a conceptual problem arises when speaking of “windowed” waveforms as containing a specific spatial (or temporal) frequency. The recovery of the underlying frequency of a waveform is difficult if the waveform consists of less than five cycles. At the level of two or three cycles, the definition of frequency becomes awkward. Under these conditions, it is more appropriate to speak of time differences between the peaks, or other features, in the waveform. No evidence of transcendental calculations being performed within the visual system has been documented. In specific cases, computations leading to similar results are accomplished using convolution integrals instead of transforms and multi-section filters.

While postulating such filters and transform calculations is easy, investigators should refrain from proclaiming the existence of such circuitry in the complete absence of any supporting physiological evidence. 7.4.5.3 The mechanism of stereopsis

The mechanism of stereopsis is associated with an overlay to the basic pointing system. It is one of the most complex mechanisms in the nervous system. Stereopsis can be described as a control-intensive synchronous mechanism involving considerable neurological computation and memory. It involves the entire Precision Optical System in a variety of individual activities. These activities include the computational and memory activities of both the perigeniculate nucleus (PGN) and pulvinar of the thalamus. These two entities, operating in tandem, form the two-dimensional correlator and the lookup table described conceptually as the cyclopean retina by Julesz. All of these activities are under the supervision and control of the thalamic reticular nucleus (TRN). These functional relationships are defined in greater detail in Chapter 15.

The PGN and pulvinar are feature extraction engines. Their output is abstract information. That information does not form a fused image and cannot be described in terms of a retina. The information does form a tabular description of the instantaneous three-dimensional scene presented to the two foveola. This tabulation can be considered an instantaneous or fragmentary saliency map. It includes both version, vergence and accommodation values for each element in the scene imaged on the foveola.

The mechanism of stereopsis is highly dependent on the mechanism of tremor, the continuous, microradian-level, motions of the eyes. Stereopsis involves several individual operational processes. One of these processes calculates an instantaneous global vergence value for the complete scene imaged on the foveola. A second process calculates an instantaneous differential vergence value for each significant element in the scene imaged on the foveola. These values are used to form the instantaneous saliency map associated with that scene. The instantaneous saliency maps resemble the interps and percepts defined when discussing reading. In both cases, these low level perceptual elements can be combined with other similar elements to form a complete saliency map or semantic thought. 7.4.5.3.1 The geometry associated with stereopsis

Figure 7.4.5-7 is an expansion of the right frame of [Figure 7.4.3-1]. It shows an instantaneous field of view similar to that labeled “α” of a bowling alley in that figure. However, the instantaneous field of view has been rotated slightly to be centered on pin #4. The resulting figure shows the detailed geometry involved in stereo-optically imaging and evaluating an instantaneous scene of 1.2 degrees width (requiring a total of few hundred milliseconds). The exquisite precision of the stereopsis mechanism makes it difficult to represent the true geometry of the situation. Therefore, the distance between the eyes has been expanded in the figure to represent a convergence angle for each eye of four degrees. The dashed sector represents the instantaneous field of view of the left foveola. It consists of an array of about 175 Dynamics of Vision 7- 207 photoreceptors arranged in a one-dimensional fan that is symmetrical about the left line of fixation. The dotted sector represents a similar situation for the right foveola. The two lines of fixation are shown intersecting at the instantaneous point of fixation established a priori by the visual system. Note that the two nominal foveola are unable to image the four bowling pins simultaneously. Pin #1 is unknown to the stereopsis process.

Figure 7.4.5-7 The geometry of the stereopsis mechanism in object space.

The stereopsis process begins with the two eyes rotating synchronously by a few seconds of arc (microsaccades) according to the discussion of tremor in Sections 7.3.2 & 7.3.7. For purposes of discussion, let the tremor be represented by a linear sawtooth motion. This motion will convert the spatial positions within object space into a time series of electrotonic signals that can be passed to the midbrain. On arrival at the midbrain, these signals are placed in the two- dimensional correlator of the PGN (Section 15.4.1). The signals from the left foveola are placed in one surface of the correlator and the signals from the right foveola are placed in a second. The correlator subtracts the signals in these two planes to learn where and for what distance the signals in each local area (the local correlation interval) of the correlator differ. It then performs a two-step process. It calculates an average signal that describes the instantaneous global vergence error between the scene and the a priori vergence value. With this value available, it calculates a differential vergence error for each significant object in the image relative to the instantaneous global vergence error. These values are illustrated in the lower right frame of the figure. Since the depth of the scene is symmetrical about pin #4, the instantaneous global vergence error shows the error between the a priori point of fixation and the radial position of pin #4. For the same reason, the differential vergence error for pin #4 is zero. The differential vergence error for pin #7 suggests its additional distance from the a priori point of fixation. The differential vergence error for pin #2 suggests the fact it is nearer to the subject than both the a priori and instantaneous global vergence values. 208 Processes in Biological Vision

The ensemble of the a priori vergence value, the instantaneous global vergence error and the differential vergence errors associated with each significant object in the image constitutes a complete three-dimensional representation of the spatial features associated with the instantaneous scene. This information can be combined with luminance and chrominance channel information to provide a complete instantaneous saliency map of this particular instantaneous field of view. It is proposed that this is a responsibility of the pulvinar. With further manipulation of the data, it can be combined with other instantaneous saliency maps to form a more complete initial saliency map or update any a priori saliency map. Whether the complete saliency map is a responsibility of the pulvinar or a higher cortical center remains uncertain. Under many conditions, the instantaneous global vergence error computed in the above sequence appears to become the a priori vergence value for the next instantaneous scene presented to the two eyes. With the two-dimensional correlator of the PGN performing local correlations over a correlation range corresponding to only a small number of adjacent photoreceptor cells, there is no need to employ complex correlation and evaluation calculations to eliminate extraneous “phantom targets.” 7.4.5.3.3 The local correlation range supporting stereopsis and fusion

Little firm data exists concerning the correlation range of the two-dimensional correlator of the PGN when operating in the vergence signal extraction mode. Sperling has suggested that correlation occurs over a range of +/– 1/8 degree394. This would correspond to a diameter of about 17 photoreceptors in the foveola. It appears likely that the number is more likely five, or less, based on known tremor amplitudes. This would correspond to plus or minus two minutes of arc. Such performance would suggest the alarm mode of visual operation should command the line of fixation to alignment with a target to within this precision.

7.4.5.3.4 The potential variation in tremor amplitude

In the above analysis, tremor is a free variable. Variations in the amplitude and phase of the tremor signals applied to the oculomotor muscles can have a major impact on the above correlation process. These parameters are completely under the control of the POS itself. The amplitude of the horizontal tremor determines what horizontal range of the foveola area is overlaid on the two-dimensional correlator of the PGN. Under some circumstances, such as a very complex scene, changing the amplitude of the tremor may allow the correlator to concentrate on only the very center of the field of view. This would allow the POS to establish a nominal value for an a priori global vergence value. With this value in hand, the POS could then proceed with evaluating the remainder of the image.

7.4.5.3.5 Forms of stereoscopes and autostereograms

A variety of stereoscopes have been defined over the years. There are refractive(Brewster’s prismatic), reflective (Wheatstone’s mirror) and polarizing (polaroid projection) stereoscopes. “An autostereogram is a 2-D display that appears in 3-D without the aid of a stereoscope” according to Howard & Rogers395. They describe a variety of these stereoscopes and presentations, and how they are created. ]xxx 7.4.5.4 Theories of Stereopsis prior to 1995

No definitive theories of stereopsis, separate from discussions of fusion and depth perception, were found in the recent literature. The more general “theories” tend to emanate from either clinical or psychophysical investigations. Most were in the psychology literature. Based on limited physiological data, the theories related to stereopsis developed in the 20th Century were necessarily conceptual and quite general. All these theories assumed the visual system was based on imaging sensors. They therefore assume stationary geometries when they discuss fusion, stereopsis, etc. The assumption of a stationary geometry, along with building caricatures with very large vergence angles, has impeded progress in understanding fusion and stereopsis. Progress has also been impeded by the lack of an understanding of how the spatial information in the scene is converted into electrical signals within the neural system. Noting that many stereograms presented in the literature are about three inches square

394Sperling, G. (1970) Op. Cit. 395Howard, I. & Rogers, B. (2002) Op. Cit. pg 544-549 Dynamics of Vision 7- 209 and are normally viewed at twelve inches is also important. The resultant field of fifteen degrees is much larger than the visual system can analyze at one time. It requires many saccades to explore such a scene in roughly 1.2 degree diameter instantaneous fields (requiring about 200 milliseconds each). Thus, the visual system makes many trips around the state diagram of vergence and stereopsis in analyzing such a complete scene. Making global pronouncements concerning the performance of the visual system in analyzing such a scene is dangerous. Weinshall and Malik provided a very brief review of computational models of stereopsis with many references but little critical content396. They did provide one controversial figure they credited to Krol & van de Grind. It shows “the double nail illusion.” Accepting their illusion as real is difficult based on their description and the author’s personal experience in many similar situations. Howard & Rogers provide the additional caveats required to understand this “illusion397.” The bibliography of Anderson & Julesz provides a better reference list on this subject398. Sperling provided a broad review of what he called binocular vision in 1970399. It was followed by an equally broad review by Nelson in 1975400. The terminology and variations in terminology between these two works are striking and will be addressed below. While Sperling titled his important 1970 paper “a physical and a neural theory,” a more current description would describe his physical theory as a computational theory based conceptually on a physical analog. He states that his neural theory is speculative. He does review the Keplerian view of stereopsis, features of which are adopted later by Julesz. While he employs a smaller vergence angle in his caricatures of stereopsis, he continues to suggest that his and the Keplerian theory of stereopsis recreates a three-dimensional equivalent of the external world withing the cortex.

Sperling treats fusion and vergence as equivalent mechanisms, along with accommodation, and develops highly conceptual interactions between each of these pairs. His separate physical models are not defined at the physiological level. Thus, they could be overlays on a common physiological system.

He does claim his secondary neural binocular field (NBF) provides “outflow” to control the primary NBF. This interpretation is consistent with this work.

Sperling expands on Helmholtz procedure for determining the fusion capabilities of the eyes. He noted that following fusion, the images presented to the two eyes could be increased in vergence by as much as eight degrees before fusion was lost by the subject. Although he did not discuss the initial fusion requirements in the same paragraph, he does suggest that initial fusion requires a vergence error of less than plus or minus one-tenth degree. This value corresponds to six minutes of arc. At another point in the discussion, a figure of one-eighth degree is given.

The Nelson paper of 1975 provides a very detailed outline of what was known then about all aspects of binocular vision. However, it is quite conceptual in nature. He defines “fusion” as meaning sensory fusion and continues. “It is important to distinguish between sensory fusion, which bespeaks unknown cortical processes, and motor fusion, or oculomotor convergence and divergence, which produces apparent contour displacement by trivial eyeball turning.” While conceptually very useful, the expressions must, may, perhaps and apparently appear quite frequently in the paper. The paper treats fusion and stereopsis on a full retina basis. It also assumes a stationary scene geometry. The paper does not define the range of vergence angles it is discussing. It therefore must address the point-to-point correlation task associated with the two complete binocular images and address the resultant ambiguity problem. The paper defines retinal disparity detectors and additional disparity detectors in the cortex (no further delineation) without defining their physiological embodiment. In an analysis on page 31, Nelson stipulates that the visual fields are stabilized without realizing that the visual system becomes totally blind to fully stabilized visual fields. Nelson does define a nonius horopter but the definition is torturous. After the midpoint, the paper provides a wealth of information concerning the observed features of stereopsis and fusion but very little definitive material on the whys and wherefores related thereto. The Summary and conclusions are useful. However, Nelson does not cross the bridge to a two-dimensional correlator. He addresses the elements of such a correlator in terms of individual processes, using terms like inhibitory and mutually facilitatory. The papers of the late 1970's through the early 1990's are all focused on Keplerian projection and develop a variety of

396Weinshall, D & Malik, J. (1995) Review of computational models of stereopsis In Papathomas, T. ed. Early Vision and Beyond Cambridge, MA: MIT Press 397Howard, I. & Rogers, B. (2002) Seeing in Depth. Toronto, Canada: I Porteous pg 65 398Anderson, B. & Julesz, B. (1995) A theoretical analysis of illusory contour formation in stereopsis. Psychological Review vol.102, 705-743. 399Sperling, G. (1970) Op. Cit. 400Nelson, J. (1975) Globality and stereopscopic fusion in binocular vision J Theor Biol vol. 49, pp 1-88 210 Processes in Biological Vision

rationales for solving first the correspondence problem and then the false target problem associated with a stationary geometry. The false target problem is frequently described as the “disambiguity problem.” Several invoke complex mask shapes to be implemented within the neural system. None of the papers discuss the physiology of the visual or neural system. Several papers invoke a two-stage processing algorithm. However, none of these papers quantify the spatial extent of the individual stages. The Keplerian projection adopted in these papers invariably involves targets closely spaced relative to the interocular distance. The result is vergence angles suggestive of targets located less than three to five inches from the eyes. Several papers have invoked a multidimensional memory site in their concept. A few call for coarse motion between the eyes and the scene as an additional tool in solving the false target problem. Marr & Poggio called for the use of stabilized image tests in future work401. They were apparently unaware of the work of Yarbus showing this approach is not a solution. Grossberg and Mingolla were aware of the limitations on stabilized images demonstrated by Yarbus402. However, both Grossberg & Mingolla and Grossberg403 are highly philosophical in content. They frequently reference the paradoxical qualities of visual imaging, introduce the retinex theory of color vision, and focus on area 17 of the cerebral cortex as the location of percept extraction related to depth perception. They are the first to offer a block diagram of the neural signal processing required to satisfy their computational algorithms. Their binocular percept (extraction) stage, ostensibly in area V4 of the prestriate cortex corresponds functionally with the PGN-pulvinar correlation processor of the midbrain proposed in this work. In a detailing of his proposed circuitry, Grossberg introduces the concept of two syncytia. The first syncytium processes monocular information and the second processes binocular information. Grossberg does focus on a boundary contour approach rather than a bulk object approach to signal processing.

Many papers in the 1980's focused on the partial occlusion of targets as a cue in the determination of depth perception. No substantial results were offered. In 1991, Yuille, et. al. presented a highly conceptual and intuitive discussion relying upon probabilities and statistics (the Bayesian approach) to solve both the correspondence and false target problems related to depth perception404. Their conclusions are not in a concrete form that can be implemented.

In their 1994 paper, Anderson & Nakayama continued to assume a stationary geometry in discussing paths “Toward a General Theory of Stereopsis405.” However, they take a refreshing look at the “false target problem” as noted in Section 7.4.3.1.1. Their description of Keplerian projection (pg 416) is one of the clearest found in the literature. They introduce considerable new material related to the perception of occluded scenes and occluded features within scenes. Anderson & Nakayama also address the question of the relationship between binocular fusion and stereopsis. They assert that most vision scientists “would acknowledge that the relationship between stereopsis and fusion is tenuous at best.” They do introduce several comments on the differences between fusion theories and suppression theories as they relate to stereopsis. Their paper concludes without making any strong assertions. Alternately, they “hope that our attempts to unify the stereoscopic phenomena described herein will motivate psychophysical and physiological experiments to discover both the merits and the shortcomings of these ideas.” 7.4.5.5 Recent “physiologically–based” theories of stereopsis by of Qian & colleagues

Qian and colleagues have, during the 1990's to date, been expanding studies begun by Hubel and colleagues of the 1960's. Their studies have been focused on measuring action potentials from stage 4 neurons accessible in the striated visual cortex of cats.

While they use the word physiology–based to support their mathematical models, their concept and background in physiological models is elementary. They do not recognize the dual signal paths involved in stereopsis; the initial coarse and later precision stereopsis achieved in human vision (Section xxx and then Section 15.2.5 for greater detail). The role of the foveola and the role of the perigeniculate nucleus (PGN) and pulvinar are critical in establishing precision stereopsis (after achieving precision convergence.) Their investigations related to the

401Marr, D. & Poggio, T. (1976) Cooperative computation of stereo disparity Science vol. 194, pp 283-287 402Grossberg, S. & Mingolla, E. (1985) Neural dynamics of form perception: boundary completion, illusory figures, and neon color spreading Psychological Rev vol. 92, pp 173-211 403Grossberg, S. (1987) Cortical dynamics of three-dimensional form, color and brightness perception Percept. Psychophys vol. 41, pp 87-116 & 117-123 404Yuille, A. Geiger, D. Bulthoff, H. (1991) Stereo integration, mean field theory and psychophysics Network, vol. 2, pp 423-442 405Anderson, B. & Nakayama, K. (1994) Toward a general theory of stereopsis: binocular matching, occluding contours, and fusion Psychol Rev vol. 101, no. 3, pp 414-445 Dynamics of Vision 7- 211 striate cortex of the occipital lobe are only applicable to coarse stereopsis. Even in this role, they do not recognize the importance of the lateral geniculate nucleus (LGN) in establishing the initial convergence of the eyes upon a specific feature in external environment, or the role of the LGN in the initial merging of the images from the two eyes. They also fail to consider the role of the saliency map of stage 4 in its ability to present a coarse stereopic perception to the stage 5 cognition engines before being updated by the precision stereoptic information from the foveola-PGN–pulvinar signal pathway (Section 4.3.6 and Section 19.6.3 for even more detail). Generally, their investigations have been primarily electro– physical in character (measuring electrical signals within the striate cortex following the binocular projection of patterns onto the retinas of cats. They have largely ignored the physiological role of the LGN and apparently totally ignored the role of the foveola–PGN–pulvinar signal pathway associated with precision convergence and precision stereopsis. When members of the Qian group use the expression receptive field, RF, they are normally referring to the receptive field in the external environment associated with the neurons of the striate cortex, and not the photoreceptors of the retina. In general, they have used extracellular probing of the neurons of the striate cortex and monitored the amplitude of stage 3 action potentials to insure observations were restricted to one neuron. They have categorized both simple and complex neurons of the striate cortex. Categorization was based primarily on whether the neurons showed a relatively narrow bipolar response to adjacent spatially diverse stimulations (simple neurons) or a broad unipolar (analog or demodulated pulse) response to multiple spatially diverse stimulations (complex neurons), as illustrated in figure 1a and 1b of the 1997 Ozhawa et al. paper. While they focus on transforming their data into Cartesian coordinates in that paper, they noted that the striate cortex of cats was not a rectilinear representation of the external visual field. Their stimulation in the Ozhawa et al. paper consisted of 5° by 0.5° bars. These size bars were not conducive to precise convergence and precise stereopis via the foveola–PGN–pulvinar signal path. They are compatible with the peripheral–LGN–occipital lobe signal path. The scales associated with figure 3 of their paper confirms this assertion.

The patterns in figure 3 show a variety of signal intensity patterns but they are not directly relatable to either the coarse vergence or coarse stereopsis tasks. they do appear to map the individual stimulus bars or patterns of bars. They employ a process called a “binocular reverse correlation procedure” similar to a monocular reverse correlation procedure first introduced in the 1960's to further reduce their data. Their Results section does not focus on the advantages of this correlation procedure or develop clearly how coarse stereopsis signals are derived that show how stereopsis is achieved.

The Ozhawa et al. paper introduces their proposed “disparity energy model” after explaining how the name relates to a general concept for “energy models” as those that compute the sum–of–squares as an indication of the desired signal output (as in their example of “the integral over time of the square of a voltage waveform across a resistor is proportional to the energy dissipated within the resistor.”). They assert this notion may be generalized to neural signals with no further justification.

They explore a variety of options relating to their disparity energy model and show it reasonably predicts the wide range of disparity tuning curves derived from their electrophysiological experiments on anaesthetized cats. However, they do not show how this model outputs a signal in either absolute spherical coordinates relative to the external visual environment of the eyes, or in relative coordinates related to a reference distance (of an initial feature forming the point of attention of the brain) from the first principle point of the optics of the cats eyes. 7.4.5.5.1 Mathematical model of Qian & Anderson paper of 1997

[xxx expand intro to this subject ] Qian & Andersen (1997) focused on the Pulfrich Effect; “The classical Pulfrich effect refers to the observation that a pendulum oscillating back and forth in the frontal parallel plane appears to move along an elliptical path in depth when a neutral density filter is placed in front of one of the two eyes. It is known that by reducing the amount of light reaching the covered retina the filter causes a temporal delay in the neuronal transmission from that retina to the cortex. The standard explanation of this effect is that since the pendulum is moving the temporal delay for the covered eye corresponds to a spatial displacement of the pendulum which produces a disparity between the two eyes and therefore a shift in depth This interpretation becomes problematic however when it is observed that the Pulfrich depth effect is present even with dynamic noise patterns since there is no coherent motion in these patterns to convert a temporal delay into a spatial disparity It was further discovered that the effect is still present when a stroboscopic stimulus is used such that the two eyes never see an apparently moving target at the same time and therefore no conventionally defined spatial disparity exists. It has been suggested that more than one mechanism may be responsible for these phenomena. Our mathematical analyses and computer simulations indicates that all of the above observations can be explained in a unified way by our integrated model.” 212 Processes in Biological Vision

Unfortunately, their mathematical model does not address an adequate physiological model. It is suggested that the signal flow diagram of Section 15.2.4 more clearly describes the physiological situation. The Qian & Anderson model does not describe clearly whether the signals they collected by probing the striate cortex represent the stage 3 pulse signals arriving at the striate cortex or leaving the cortex. It does not incorporate the time delay introduced into the signal path of one eye due to the filter. This time delay originates in the photoreceptors of stage 1 according to the P/D Equation (Section 7.2.4) and not in the subsequent signal path. Specifically, their phase parameter difference, equation (4), although defined precisely contains a term not accounted for in their analysis, a delay introduced into one signal path as a function of the density of their optical filter.

The corrected equation (4) should read: Δφ / φl – φr ± φf(D) where the sign of the last term depends on which optical path contains the filter of density, D. The introduction of the filter term changes the apparent straight path of Pulfrich’s pendulum in their figure 1 into an elliptical path with a minor axis proportional to the density, D.

7.4.5.5.2 Mathematical model of Qian & Li paper of 2011

Figure 7.4.5-8 shows an extension of a figure originally by Qian & Li406 which others have modified. The extended caption of the Qian version is worth reviewing. The first principle points of the two eyes are labeled Ol and Or in their figure and the distance between these two points is given by “a” in this figure. Frame b of the modified figure defines several terms as they are defined in the “nomenclature” section below.

There is a significant problem with the figure as modified by unknown authors. They have added a point labeled R to frame B. While, the differential disparity, ΔR shown is reasonable, the binocular disparity shown is not. The dotted triangle in frame A, added by this author, is realistic for the differential disparity. However, the it is not reasonable for the binocular disparity as drawn. The location of the triangle in frame B for the left eye must be necessarily be farther from the mean disparity than that for the right eye. The location of the two small diamonds in frame B must be reversed in order to represent a realistic feature in external space.

Contrary to their assertions in the cited paper, Qian and colleagues have been working primarily on mathematical models of stereopsis. Their references to the physiology of vision are primarily conceptual. They note, “In an effort to address this shortcoming, we have constructed physiologically based algorithms for disparity computation according to the quantitative properties of binocular cells in the visual cortex reported by Ohzawa and coworkers407 (section 2.1).” Two earlier papers by DeAngelis et al408,409. provides details related to the Ohzawa paper. They encounter the behavioral parameters related to but do not identify or differentiate between the precision and coarse signal processing paths of stereopsis leading to the saliency map of stage 4. No description of the actual neural circuitry involved is provided in the paper. The models described are static models; the cited paper and references do not include the terms, tremor, saccades or micro-saccades associated with the critically important motion of the oculars in the mechanism of stereopsis. Later in section 2.1, they introduce a simple photoreceptor cell and its receptive field (RF). They note, “Because of the phase dependence, simple-cell responses cannot explain the fact that we can detect disparities in static stereograms and in complex dynamic stereograms. Their description of the critical features of a simple cell are not supported in this work. Their focus on the cells of the visual cortex is not supported in this work either.

406Qian, N. & Li, Y. (2011) Physiologically based models of binocular depth perception, in Harris, L. & Jenkin, M. eds. Vision in 3D Environments Cambridge: Cambridge University Press, Chapter 2, pp 11-45. 407Ohzawa, I. DeAngelis, G. & Freeman, R. (1997). Encoding of binocular disparity by complex cells in the cat’s visual cortex. J Neurophysiol vol 77(6), pp :2879–2909. 408DeAngelis, G. Ohzawa, I. & Freeman, R. (1993) Spatiotemporal Organization of Simple-Cell Receptive Fields in the Cat’s Striate Cortex. I. General Characteristics and Postnatal Development J Neurophys vol 69(4), pp 1091-1118 409DeAngelis, G. Ohzawa, I. & Freeman, R. (1993) Spatiotemporal Organization of Simple-Cell Receptive Fields in the Cat’s Striate Cortex. II. Linearity of Temporal and Spatial Summation J Neurophys vol 69(4), pp 1118–1135 Dynamics of Vision 7- 213

Figure 7.4.5-8 The geometry of binocular projection and definition of disparity ADD. (a); the geometry within the plane of regard. (b); image points along the curved retina at its intersection with the plane of regard. See text. Modified from Qian & Li, 2012.

“For simplicity, we consider only the plane of regard defined by the instantaneous fixation point (F) and the optical centers, (ol and or) of the two eyes (i.e., the points n the eyes’ optical system through which the light rays can be assumed to pass in straight lines).

This statement includes the implicit assumption that the angle (olFor) is small, less than about one degree. Otherwise, the thick lens equivalent of the lens must be used. Then, the Vieth–Muller circle passes through the first primary point of each lens, located within the lens portion of the ocular (Section 7.4.1.5).

The two foveas (fl and fr) are considered as corresponding to each other and thus have zero disparity. The condition of no disparity between the two retina is only achieved at the point of the Vieth–Muller circle at its intersection with the sagittal plane formed by the eyes. This limitation is due to the muscular mounting associated with the oculars.

To make clear the positional relationship between other locations on the two retinas, one can imagine superimposing the two retinas with the foveas aligned (bottom).

This sentence applies to a non-existent “bottom” portion of the image.

The fixation point F in space projects approximately to the two corresponding foveas (fl and fr) with a near-zero disparity. This condition is a result of the precision steropsis mechanism fine tuning the vergence system.

The disparity of any other point in space can then be defined as φ1 – φ2, which is equal to ψ2 –ψ1. Correct only for within the horizontal plane. It then follows that all zero-disparity points in the plane fall on the so-called Vieth–Muller circle passing through the fixation point and the two optical centers, since all circumference angles corresponding to the same arc (olFor) are equal. Other points in the plane do not project to corresponding locations on the two retinas, and thus have nonzero disparities. 214 Processes in Biological Vision

Each circle passing through the two optical centers defines a set of isodisparity points. Only within the nominal 51 degrees of binocular vision for quality binocular vision. Only within a 8.7 degree circle about the point of fixation for processing within the fovea. Only within one degree for precision (stereopsis) binocular vision (using a nominal 1.2 degree diameter foveola), where the circles are usually approximated by frontoparallel planes. When fixation distance is much larger than the interocular separation and the gaze direction is not very eccentric, the constant–disparity surfaces can be approximated by frontoparllel planes.” This statement corresponds to the small–angle condition required to use the thin lens approximation mentioned above. This approximation only applies to the precision binocular (stereopsis) condition. In summary, the drawing of Qian & Li, and similar drawings by many others, only apply to the region of one degree diameter at the intersection of the sagittal plane and the Vieth–Muller circle. By adding additional points in object space, the operation of the fusion and depth perception mechanisms can be outlined.

7.4.5.6 The proposed physiological theory of stereopsis

[xxx need to discuss here or in section on LGN extracting and/or merging of disparity information leading to the mean disparity and differential disparity values. ]

[xxx need to separate peripheral, qualitative (LGN path) and precision (PGN path) binocular operation ] Based on the above analysis, a concise physiological theory of stereopsis can be proposed.

Stereopsis is primarily a mechanism associated with the precision analysis of imagery presented to the foveola of the visual system. Stereopsis is complemented by a separate mechanism associated with the remaining peripheral retina and providing generally qualitative estimates of depth perception.

Stereopsis defines a mechanism performed within the 2-dimensional associative correlator of the perigeniculate nucleus that mathematically merges the information from the two foveola, calculates the x,y & z parameters associated with each edge presented by the foveola, within the effective spatial limits of the correlator. It then delivers a vector, in the form of an initial interp of the information to the pulvinar. The pulvinar performs additional correlation against data in its memory and delivers an initial percept of vision to the saliency map.

[xxx mean disparity and differential disparity ] [xxx how is this tied to Panum space? ] The stereopsis mechanism calculates the mean and deviation from the mean (both in two dimensions) of edges appearing in the images presented to the foveola within the spatial limits imposed by the effective outer spatial dimensions, the global correlation range, of the correlator. The associative properties of the correlator allow it to examine edges within a local radius, the local correlation range, of each image from the foveola in calculating the mean and deviation from the mean of each edge. Finally, the correlator calculates a global mean in x,y & z associated with the scene. The values, x0, y0, specify the nominal point of fixation of the scene. Deviations from the global mean associated with the local mean of each edge define the mathematical location, xn,yn, of each edge in the original scene. Deviations from the mean z0, of the deviation associated with each edge define the location, zn, of each edge relative to the nominal point of fixation. The stereopsis mechanism transfers these parameters, which as a group constitutes an initial interp of the scene imaged on the foveola, to a local interp map. It then collects these values over a period of time and places them into a larger context associated with the complete scene.

The value z0 is a descriptor of the disparity error associated with the scene compared with the a priori eye disparity used to image the scene initially. The value x0,y0 describes the eccentricity error associated with the scene compared with the a priori pointing angles used to image the scene initially. The associative correlator is organized to accept two-dimensional information from each foveola separated into vertical and horizontal components by the tremor mechanism. The data is stored in logically definable, but not necessarily spatially definable, independent planes before processing. The extent of these two-dimensional planes is determined by the effective global dimensions of the correlator. For purposes of this work, this dimension is nominally 1.2 degrees Dynamics of Vision 7- 215 in diameter in object space, centered on the point of fixation, and corresponds closely with the defined diameter of the foveola of each retina. In cases where the foveola are not converged on a single point of fixation, it is the effective diameter of the correlator, not the combined fields of view of the foveola, that defines the limits of Panum’s area. This difference introduces a limitation similar to physical occlusion in object space into the signal processing of the visual system. Information only loaded into the effective area of the correlator from one foveola is effectively occluded. In this physiological theory of stereopsis, the associative correlator of the PGN is the physiological embodiment of the computational portion of the conceptual cyclopean retina of Julesz. The set of mathematical values, xn, yn, zn for all n comprise the “image” analogous to the image of the cyclopean retina.

7.4.6 Fusion and Depth Perception as phenomena related to stereopsis

The phenomena of fusion and high performance depth perception both rely on the underlying mechanism of stereopsis. Because of this, it requires care to differentiate between the two phenomena in many experimental activities. This interaction is complicated by the use of both dichotic and dichoptic test configurations. It is still further complicated by the frequent use of simple test stimuli, frequently consisting of only two points of light or two simple objects–lines, squares etc. As a result, much of the data in the literature requires very close study to interpret the results and the claims by the authors involved correctly. The use of more complex targets in fusion experiments highlights the importance of the role of the torsion muscles of the oculars in generating superimposable images. Lacking this capability, the eyes would not be able to combine images of scenes distant in angle from when the eyes were pointed straight ahead. See Section 7.4.3.6 and the broader discussions of convergence in Sections 7.4.3 & 7.4.5. See also Section 15.3.2. 7.4.6.1 The phenomenon of fusion

A closer marriage of the physiology of the visual system with the psychophysical data base leads to a more specific interpretation of the fusion phenomenon than that found in the literature. It clearly defines two different fields of view associated with, and two different mechanisms supporting, fusion.

This work will differentiate between fusion associated with images within the field of view of the foveola (and processed by the PGN), and fusion associated with images within the more peripheral retina (and processed within the LGN). For the area imaged upon the foveola, stereoptic fusion is achieved (with its veridical relationship between disparity and perceived depth of field. For the area imaged outside the foveola, a simpler fusion is achieved where depth perception does not produce a significant veridical relationship.

Ogle has provided a good description of the conditions involved in the phenomenon of fusion. “It is the contours, the demarcations between light and less light areas, that provide the pattern of the images and the stimuli for fusion when they exist in both eyes410.” This may be the case both within the field of view of the foveola and in the surrounding field.

A preliminary discussion of peripheral fusion appears in Chapter 9 of Ogle.

Stereoptic fusion as a phenomenon is limited primarily to the imagery presented to the nominally 1.2 degree diameter foveola. The spatial resolution of the visual system is so poor outside this region, fusion becomes academic in the peripheral regions of the retina. The phenomenon is encountered when the following conditions have been met:

1. The vergence system has assumed a rest position or other a priori vergence condition based on experience. 2. The area of the scene imaged on the foveola of the two eyes has been scanned by means of the tremor mechanism and a set of signals relating to each edge of each significant object in the scene has been transferred to the two- dimensional correlator of the thalamus. 3. The two-dimensional correlator and the associated signal processing of the perigeniculate nucleus have calculated a nominal global vergence error for the imaged scene relative to the a priori vergence condition. 4. The precision optical system has rotated the eyes into convergence at a location near the point of fixation. It does this by minimizing the average vergence disparity error calculated using the edges of all of the significant objects imaged on the foveola. This is the condition where the global vergence response most nearly equals the global vergence

410Ogle, K. (1950) Researches in Binocular Vision. London: W. B. Saunders pg 61 216 Processes in Biological Vision

stimulus. 5. The accommodation of the two eyes has been optimized for the same target distance as suggested by the above global vergence condition. 6. The local vergence values (errors), relative to the global vergence condition, have been calculated for each significant edge associated with an object in the scene imaged on the foveola. 7. The local vergence values for each edge of each significant object have been assimilated with other values associated with the object and placed in the saliency map representing the area imaged by the foveola. 8. The saliency map representing the area imaged on the foveola has been accessed by the higher cognitive centers and a multidimensional perception of the scene has been arrived at. This multidimensional perception includes chromatic, brightness, depth, transverse velocity, historical and other information (including that from other sensory channels) concerning the scene. Under the above conditions, the instantaneous image of the scene is commonly said to be fused. To achieve a multidimensional representation of a larger scene, the above process is repeated in a stepwise procedure at intervals of about 200 msecs. Each foveola-size area of the full scene that is within the binocular range of the visual system is assembled. If the head or body is allowed to rotate, a larger visual contribution to the saliency map may be made. After all of the individual contributions to the saliency map have been made, the higher cognitive centers can access the more extensive saliency map and perceive a more complete scene. Note the perceived saliency map may also be larger than the total visual field of view in the sense that it may contain other sensory inputs obtained independently of the visual system.

The question of fusion as it relates to the peripheral retina is largely academic. The higher cognitive centers only access the saliency map for information about the outside world. This map contains information concerning the scale and distance of every object in its surround that the eyes have observed. Wherever the eyes have imaged an area on the foveola, a more precise estimate of the three dimensional aspects of that scene have replaced any previous estimate based on peripheral fusion. It is the final estimates that are accessed by the higher centers. In the absence of such access, the higher cognitive centers are largely unaware of, and unable to identify, objects in the peripheral field of view.

Ogle has provided a discussion of fusion as a function of the specific features of a simple scene411. 7.4.6.1.1 Misconceptions and the Keplerian Projection related to fusion

Several authors have written on the “fusion compulsion412.” The use of the word compulsion suggests the question of will is involved. As seen in this discussion, the circuitry of the pointing system, and the overlays leading to fusion are entirely “hard wired” and deterministic. The phenomenon of fusion occurs spontaneously when appropriate imagery is presented to the visual system. The term is closely associated with the concept and phenomenon of rivalry. Rivalry involves the commonly observed situation where the visual system will continue to change from one perception of a dichoptic scene to another because of the difference between the two images provided (Section 7.4.1.1.2). Rivalry and compulsion are phenomena related to the two-dimensional associative cross-correlator of the PGN. Ogle has provided a set of symbols frequently used to illustrate fusion, compulsion and rivalry phenomena413. No signal processing (other than related to the steps outlined in the state diagram of stereopsis) is required to achieve correspondence between points in the two foveola. The combination of the global vergence value and the individual local vergence values, both the mean and the deviation from the mean, related to each individual object in the scene imaged on the foveola completely describe the spatial parameters of the perceived scene. Julesz reintroduced the Keplerian projection to discussion of stereopsis in the 1960's after a very long hiatus. It has appeared in many technical articles since. However, the caricatures of this projection always exhibit three distinct properties. They always lack scales. They always describe static conditions. They typically describe target vergence angles of greater than 45 degrees. Contrary to this literature, false targets (or ghost targets) are not a significant problem in stereopsis as defined here. Stereopsis only involves a nominal instantaneous field of view of 1.2 degrees. With a

411Ogle, K. (1950) Researches in Binocular Vision. London: W. B. Saunders, pg 61 412Ogle, K. (1950) Op. Cit. pg 60 413Ogle, K. (1950) Op. Cit. pg 62 Dynamics of Vision 7- 217 nominal interocular separation of 6.4 cm and a typical minimum viewing distance of 30 cm or greater, the actual vergence angles of vision are nearly always less than 12 degrees. As a result, virtually no false targets are generated under realistic visual conditions. Unless more realistically drawn, the Keplerian projection should not be used for pedagogical purposes or research. For evaluating the actual world of stereopsis, the Keplerian projection should be limited to two converging 1.2 degree beams intersecting at a target vergence angle of 12 degrees or less. The beams should be labeled to show they each contain about 175 distinct sub-beams associated with the individual fields of view of the photoreceptors of the foveola. The beams should also be labeled to show they are synchronously scanning the scene, in two-dimensions, at approximately 30 Hertz with an amplitude of about 2-6 seconds of arc. It is the scanning mechanism and the small size of the individual beams that insure that no false targets are encountered. 7.4.6.1.2 The fusion phenomenon

Fusion as a phenomenon depends upon the process of stereopsis as performed within the associative correlator of the PGN. It is constrained to the common field of view of the two foveola. Fusion is also dependent on the mechanism of tremor to generate the pair of temporal signals related to each edge in the scene. The phenomenon is therefore limited to a small instantaneous field of view described by foveola vision within the larger shared field described by binocular vision. Through the assembly of a percept of an entire binocular scene, the phenomenon of fusion can appear to apply to a full scene. This however is a cognitive perception based on a large saliency map in memory. It is not an instantaneous event associated with stereopsis as a mechanism. This relationship suggests that fusion as a complete percept and the related phenomenon of rivalry occur after the information from the correlator of the PGN is turned over to the pulvinar.

The phenomenon of fusion relies upon the calculation of a mean and deviation from the mean related to the two images projected onto the foveola from a scene in object space. Therefore, each significant edge in object space must be imaged on both foveola. To obtain a meaningful description of a complete object, its entire perimeter must be found within the instantaneous field of view of both foveola. Because of this last requirement, small objects can appear over a wide area related to the foveola. However, large objects frequently extend outside of the imaged area. The perception of depth related to such large objects is generally constrained to that provided by the qualitative mechanism associated with the awareness mode of vision and the LGN/occipital lobe couple.

The local deviation of the mean associated with an object in a scene is calculated with respect to the global mean vergence error of the entire scene. It is this local deviation that is linearly related to the distance of the object from the global mean vergence of the scene. This linear relationship is described as veridical in the literature. 7.4.6.1.3 Fusion as a routine event

It may seem obvious but a primary requirement must exist before fusion can occur. The instantaneous scene projected onto the two foveola must contain at least two separate edges resolvable into either horizontal or vertical components. In theory, two separate edges are required that are resolvable into both horizontal and vertical components. However, vertical vergence plays a minor role in vision compared with horizontal vergence. While a single edge appearing in both images is adequate for vergence and accommodation, it is not adequate for fusion and the description of the relative locations in depth of the edges. Fusion requires the calculation of a global vergence value for the scene and the calculation of at least two local vergence values relative to that global value. Once these calculations are performed, the location of the edges (and the shape of any associated object) can be defined in three-dimensional space. Fusion will always occur under the above conditions. No need exists to invoke a mechanism related to the will. As long as multiple edges appear in the scene imaged on the foveola, fusion will occur. It usually occurs within less than 100 msecs of PGN processing. The overall time delay following stimulus encountered in the laboratory is likely to be closer to 200 msecs when various transit times and latencies are recognized. To optimize the stereopsis process leading to fusion, the edges in object space should not be too short. If they are very short, the full two-dimensional capability of the cross-correlator is not used effectively. It is also true that if the scene is very complex, such as a random dot stereogram, fusion may become difficult. Ancillary clues may be necessary to achieve an acceptable state of fusion within a finite time. A question is frequently found in the literature concerning how large a target disparity error can be introduced before the phenomenon of fusion will be destroyed. The answer depends heavily on the nature of the scene being viewed. If it contains relatively long high contrast edges, fusion can be maintained up to the point where secondary mechanisms, such as the ability of the eye muscles to create divergent values of eye vergence, are encountered. 218 Processes in Biological Vision

7.4.6.2 Effects related to the phenomenon of fusion 7.4.6.2.1 Spatial hysteresis related to stereopsis and fusion

The fusion phenomenon exhibits an asymmetry sometimes described in terms of hysteresis. Under laboratory conditions, two images of a 3-D scene can be presented to the eyes at a disparity considerably different than the rest disparity of the visual system. Under these conditions, the images will not be fused. The subject will encounter physiological diplopia and perceive a “double image.” If the difference in disparity between the two images and the rest disparity of the subject is gradually reduced, a point will be reached where the correlation function of the PGN can produce a meaningful vergence disparity error. At that point, the pointing system will be enervated to cause fusion of the two images. This enervation will cause the eyes to adopt a disparity as equal to that of the test images as possible. The critical disparity error prior to fusion is about XXX. Following fusion, increasing the disparity between the two images presented to the eyes by a considerable amount is possible before fusion is destroyed. The vergence overlay and the pointing system will attempt to vary the vergence of the eyes to maintain a minimum disparity vergence error. Ogle has noted that the perceived image becomes double at two different conditions414. For horizontal lines in the scene, doubling usually occurs near +/– two degrees. For vertical lines, doubling usually occurs near +/– eight degrees. These early measurements did not give the absolute disparity associated with these values. 7.4.6.2.2 References to peripheral fusion

Ogle has noted, “Fusion has usually been thought of in terms of central vision when the acuity is highest, and in the main, fusion in the central parts of the visual field is considered more important and dominant than fusion in the periphery (pp 94-100).” The discussion is enlightening. However, calibration is largely absent from the data. The only conclusion that can be drawn is limited in its practical utility. If large enough peripheral targets are used (0.5 degree visual angle), fusion of narrow lines at the point of fixation can be disturbed by these targets at angles of 12 degrees from the line of fixation in the vertical direction. The experiment is worthy of repeating under more sophisticated test conditions. 7.4.6.2.3 Temporal hysteresis and other temporal aspects of fusion

Fusion exhibits a temporal hysteresis along with its spatial hysteresis. Records has discussed this hysteresis on page 657. He notes that fusion is achieved rapidly after even a very short exposure such as from a spark source or flash bulb. Of course it is not perceived until later due to the signal propagation delays inherent in the visual system. However, the process appears to be complete within about 30 msecs for simple targets. For complex targets, the required time is much longer. On the other hand, Julesz & Tyler found that if the fusion associated with a scene was disrupted and then immediately reestablished, the disruption could be detected for intervals as short as four msecs. The shortness of the interval suggests that the integration time of the photoexcitation/de-excitation process may have been affected in these experiments.

Records has also provided some comments on visual evoked potentials as a result of fusion experiments. 7.4.3.2.4 Fusion as a function of peripheral angle

Figure 7.4.6-1 reports the general transition between fusion and diplopia as a function of eccentricity for the normal eye using a specifically configured stimulus. At the detailed level, the transition described by Panum’s limit is asymmetrical between vertical and horizontal eccentricities.

414Ogle, K. (1950) Op. Cit. pg 66 Dynamics of Vision 7- 219

Figure 7.4.6-1 Fusion as a function of peripheral angle in the normal eye. No measurements were reported for less than one degree. From Ogle, 1950. 220 Processes in Biological Vision

7.4.6.3 Theories explaining the fusion phenomenon 7.4.6.3.1 Previous theories of image fusion

Only a few theories of fusion appear in the literature. They are generally conceptual and use only loosely defined, and frequently generic, terms. More significantly, they fail to pass the Yarbus Test of applicability (Section 7.4.1.2.1). The discussion in this section borrows heavily from a short section in Schor & Ciuffreda (S & C pp216-210). Quoting Schor & Ciuffreda in 1983, “There have been four classic approaches to the binocular fusion of stimuli in the two eyes into a single percept; the synergy (Panum, 1858), local sign (Hering, 1864), eye movement (Helmholtz, 1866), and the suppression hypotheses (Verhoeff, 1935). Each is subject to serious misgivings and all four have been rendered essentially obsolete by neurophysiological data on binocular responses of cortical neurons, which give rise to a fifth, physiological hypothesis.” Note carefully the dates associated with these theories. None of these theories is based on a realistic physiological model of the visual system.

Considerable speculation arose concerning how the visual system fused the images from the two eyes (binocular fusion) during the 19th Century. Four simple concepts were offered. They were almost totally lacking in experimental support. (1) The synergy hypothesis of Panum (1858) originated the suggestion that binocular fusion is due to the “binocular synergy of single vision by corresponding circles of sensation.” The Panum concept is enshrined in the designation of Panum’s area as the range over which fusion occurs. (2)The local sign hypothesis of Hering (1864) was first applied to both stereopsis and binocular fusion. It assigned an address to each target location imaged on the two retinas. If the two addresses were within a specific tolerance, the images were seen as one. (3) Helmholtz (1866) proposed an alternative to Hering (it seems they always proposed alternatives to the work of the other). He proposed an eye movement hypothesis of fusion, where the imprecise fine movement of the eyes led to an opening of a region in which the two images could be considered one. (4) du Tour proposed a suppression hypothesis in the 18th Century. It focused on the rivalry of two dissimilar targets presented in equivalent areas of the two retinas leading to rivalry. In his hypothesis, the signals from the two retinas attempted to suppress each other through manipulations within the visual cortex.

During the 20th Century, several experiments were performed to validate or refute the above hypotheses. Schor & Ciuffreda summarize the problems associated with each hypothesis. This included the material in both the popular press and technical literature by Hubel and his team. These studies continued to point toward what were described as physiological theories. These were generally associated with the occipital lobe of the cerebral cortex. However, no significant new theories developed during the entire 20th Century. This was basically due to a lack of a comprehensive model.

Chapter 16 of Howard & Rogers, “Linking Binocular Images,” introduces a long series of empirically derived rules concerning fusion rather than a single concise theory of fusion. 7.4.6.3.2 The proposed Physiological Theory of Image Fusion

The hypotheses reviewed above are all relevant to the mechanism of fusion. However, they are all first order characteristics gleaned from a black-box approach. They involve observations of the external parameters associated with the internal operation of a complex system. Each of these hypotheses offers something that will be recognized in the theory proposed in this section with one exception. It will become obvious that the visual cortex plays no direct role in the mechanism. The phenomenon of fusion is focused on mechanisms concentrated in the midbrain and the POS. While signals may be found within the cerebral cortex that can be correlated with events associated with fusion, they are not participants in the mechanisms resulting in the fusion phenomenon. It is proposed that the fusion of images occurs within the two-dimensional correlation capability found anatomically within the thalamus of the midbrain. The visual cortex plays no role in this mechanism. The overall process involves the operation of an outer (coarse) servomechanism associated with the awareness channel of vision and an inner (fine) servomechanism associated with the analytical channel of vision (see Section 7.3.3). It also involves the operation of a two-dimensional correlator of major importance to the operation of the visual system in the higher primates, raptores, and many other arboreal and carnivorous species. It is this correlation capability that provides the superior acuity associated with these species. Here, the higher primates are limited to humans, Hominidae, chimpanzees, Pan, Gorilla, Dynamics of Vision 7- 221 Gorilla, and the Orangutan, Pong. This capability is not shared fully with the other members of Anthropoidea, such as the monkeys. It is interesting that a similar capability appears to have evolved among some higher members of Mollusca through evolutionary convergence. The eyes of squid, Loligo, and octopus among Cephalopoda appear to have introduced such a correlation capability into their visual systems. The outer loop operates as a Type 0 servo due to the limited resolution of the optical system supporting the retina in the periphery. In this role, the outer loop provides steering instructions, based primarily on angular position information. This information is used to point both eyes conjunctively and bring the target to a position along the line of fixation. The tolerance on this operation is about 10 arc minutes. This dimension is well within the 1.18 degree diameter of the foveola. The inner loop operates as a Type 1 servo. It relies upon the tremor generated within the POS to provide the angular velocity required to convert the spatial position of targets imaged onto each retina into electrical signals occurring at specific times. It is the difference in these times associated with a specific small target that are used to develop the first order steering commands used by the inner loop. These commands are used to bring the target to within a few arc seconds of the line of fixation. Simultaneously, with the final positioning of the target using conjunctive eye movements, the correlator provides vergence signals to move the eyes disjunctively and accommodation signals to focus the eyes on the target. These signals are derived from different aspects of the two electrical signals provided by the sensory signals delivered from each eye. The conjunctive pointing signals are based on the average difference between the arrival times of the sensory signals and the “zero crossing time” obtained from the tremor generator. The disjunctive signals are derived from the absolute difference in time between the arrival times of the sensory signals. The accommodation signal is derived by measuring the slope of the leading edge of the signals received from the sensory channels. The goal is to maximize these slopes.

The above actions by the two-dimensional correlator are sufficient to raise the acuity to approximately the diameter of an individual photoreceptor. However, the equivalent spatial range of the correlator is dynamic. It can use the same memory allocation capabilities found in a modern track while scan radar. Initially, a special high amplitude tremor signal (labeled a flick in this work) is generated. This flick causes the entire correlator ( with a capacity of n times 175 cells in diameter) to concentrate on a single small target area (possibly 10-17 elements in diameter). Once the target is brought within an adequate precision of the line of fixation, the tremor signal is changed, along with the switching pattern used to load the correlator. The change causes the correlator to correlate the signals over the lengths of longer elements (in both the horizontal and vertical dimensions). This process increases the signal-to-noise ratio of the signals required to perform the final conjunctive, disjunctive and accommodation adjustments. The final output of the correlator provides the ultimate acuity of the visual system (of about six arc seconds in untrained humans). Training has been known to marginally increase this acuity. This is particularly true for complex targets exhibiting unique patterns.

Because of this mechanism of fusion, the human, and many other, visual systems exhibit multiple levels of acuity as a function of location within the field of view and specific target configuration. Once the target has been positioned near the line of fixation, a separate sequence of steps occurs that can be described as a “pull-in” process. The correlation process optimizes the performance of the overall physiological optics to maximize the acuity of the system for that target. The result of this pull-in process leads to the ultimate condition of “lock-in” during a period of up to 200 msecs before the next flick or mini-saccade. [xxx merge this paragraph with previous on flicks ]

It is quite clear that the ultimate acuity of the visual system is highly dependent on at least four properties of the scene. 1. the light level, 2. the specific shape of the target (a grid offers a much higher correlation coefficient than a small filled circle), 3. the contrast of the target and, 4. the performance of the lens group. Astigmatism, and other physical and neurological disorders are also obvious deterrents to the optimum operation of the overall correlation process. As part of the correlation process, signals are output by the correlator that define the location of every distinct target within the field of the foveola. These signals are used to provide depth of field relative to the reference target nearest the line of fixation. They are also used by the lookup tables associated with the pulvinar to define the meaning of groups of individual target elements in conceptual space. This is the methodology for defining patterns and ultimately achieving the function of reading. The proposed Theory of Image Fusion states: Fusion is achieved among the higher chordates by dynamic merging of the information received from the two eyes through two-dimensional correlation. The process is inherently dependent on the tremor associated with vision in these species. The correlator is physically located within the perigeniculate nucleus of the thalamus, a major element of the POS. The optimum operation of the correlator also creates the final pointing, convergence and accommodation signals that are used by the physical layer, including the plant, to implement optimum acuity in three dimensions. Optimum operation is dependent on the prevailing conditions of illumination, scene contrast and target complexity. The perception of depth for individual targets 222 Processes in Biological Vision

within the field of the foveola is directly associated with the secondary statistics (residual disparity) relating those targets to the reference target. The theory appears to satisfy the majority of the problems associated with the four hypotheses from earlier times (S & C pp 216-218). Signals at each location in the retina are differenced in general accordance with the local sign concept. However, the question of rivalry is associated with, and explained by, the higher level correlation process. Whereas Helmholtz hypothesized small perturbations as an instability degrading the performance of the eyes, these perturbations are in fact the raison d’etre for the unique performance of the human visual system. It is this small perturbation (tremor) that provides the desired acuity through correlation. The du Tour rivalry hypothesis is quite compatible with the above proposal. The correlator can issue signals in time sequence that are quite different. These signals reflect the signal-to-noise limitations of the correlation process. Under certain circumstances, these signals can be interpreted as rivalry. Under other circumstances, they may lead to vertigo.

Inconsistent imagery provided to the two eyes can also introduce problems generally associated with rivalry. The correlator cannot reach an optimum level of performance if the two images are not consistent in their structure. 7.4.6.4 The phenomenon of precision depth perception

The study of depth perception has a less formalized history than the study of fusion. Defining the depth perception phenomenon succinctly has been more difficult for four reasons.

1. No concerted effort has been undertaken to determine the reason for the difference between qualitative depth perception (as associated with the peripheral retina) and the precision depth perception (associated with the foveola of the eyes).

2. The unique role played by the photoreceptor channels connecting to the perigeniculate nucleus and generally associated with the foveola has not been recognized. It is the signal processing associated with this limited group of signaling channels that provides the precise depth perception achieved in human vision.

3. Because of the above, a robust theory of stereopsis has not emerged to build upon.

4. In addition, no natural null point, such as the fixation point along the plane perpendicular to the midpoint of the interocular line, is available when discussing fusion. The distance from the eyes of the fixation point under rest conditions varies substantially among individuals, and apparently with time, for a single individual.

Because of these factors, a considerable empirical literature concerning qualitative depth perception exists. However, a much smaller, almost anecdotal, literature exists concerning precision depth perception.

The difficulty in studying the phenomenon of depth perception is also due to two complications related to its common antecedent mechanism with fusion, stereopsis. This contribution of parameters related to stereopsis, frequently labeled a double-duty linkage, complicates the data reduction process. The second complication relates to the simplicity of introducing dichoptic images to the two eyes. These images frequently introduce information related to both fusion and depth perception. The need to separate the contributions to these two phenomena during data reduction is not always understood by the investigator. Ultimately, the parameter known as Panum’s limit, that describes the spatial extremes of the associative correlator of the PGN, impacts the depth perception phenomenon. It does so in a complicated way that is mathematically related to the instantaneously achievable range of fusion. 7.4.6.4.1 Dichotic versus dichoptic instrumentation

The human visual system is optimized for viewing dichotic scenes, where the same information is presented to each foveola but from a slightly different perspective. The eye vergence associated with these scenes is usually less than 12 degrees and the eyes are usually turned toward each other by the same angle relative to their far-field alignment. The ease with which dichoptic images are presented to the two foveola in the laboratory frequently leads to additional situations that the visual system is not designed to handle. These situations can introduce limitations into the visual Dynamics of Vision 7- 223 signal processing that are difficult for the researcher to appreciate fully and account for. Panum’s limit, when defined at the level of the associative correlator, forms a very important part of this overall limit. 7.4.6.5 Data for perceived depth

The available experimental data on depth perception is very large. Much of it is based primarily on psychophysical experiments where many pertinent parameters were not recorded. While still useful, such shortcomings place the data in the exploratory category with respect to its utility. Some of the most useful data involves the limits on fusion and depth perception combined. This data appears in Section 7.4.6.3. Coutant BE, Westheimer G. have provided some data relating to stereoscopic depth detection415. Their abstract reads: “Of 188 unselected biology students participating in one or both of two tests measuring stereoscopic depth detection ability, 183 (97.3%) were able to see a depth difference at horizontal disparities of 2.3 min arc or smaller. At least 80% could detect depth differences at 30 sec arc disparity. These findings indicate that most people are able to take advantage of the increasing utilization of stereoscopic displays.” Regan reviewed the literature on three aspects of stereo vision including (1) static stereoacuity,(2) motion in depth perception evoked by a rate of change of disparity and (3) cyclopean perception of motion within a frontoparallel plane. 416. 7.4.6.5.1 Depth perception as a function of binocular disparity

Figure 7.4.6-2 is reproduced from Tyler with the addition of the nominal diameter of the foveola417. The original was based on data from Richards and Richards & Kaye using two different experimental techniques under dichoptic conditions. The curve was drawn freehand but shows several pertinent features. The maximum perceived depth occurs at a disparity only slightly below that associated with the diameter of the foveola. More important, this diameter also describes the maximum correlation range of the two-dimensional correlator of the PGN. The perception of depth falls rapidly for targets of greater binocular disparity. It is shown reaching a minimum value at a binocular disparity of about ten degrees. Within the range of the PGN correlator, the perception of depth is directly proportional to the disparity introduced (the perception of depth is veridical). Beyond the functional diameter of the correlator, the perception of depth is associated with the LGN and a variety of cues. Depth perception in this region is described as qualitative.

415Coutant, B. & Westheimer, G. (1993) Population distribution of stereoscopic ability Ophthalmic Physiol Opt vol 13, pp 3-7 416Regan, D. (2000) Human Perception of Objects. Sunderland, Mass: Sinauer Chapter 6. 417Tyler, C. (1983) Sensory processing of binocular disparity Chapter 7 in Schor, C. & Ciuffreda, K. Op. Cit, pg 237 224 Processes in Biological Vision

Figure 7.4.6-2 Relative depth perception as a function of the binocular disparity of a target under dichoptic conditions. The dashed line shows the putative linear relationship between perceived depth and binocular disparity (veridical region). See text for details. Dynamics of Vision 7- 225

7.4.6.5.2 Depth perception associated with spatial frequency (interval)

The reader is reminded that the basic concept of frequency requires a periodic event (consisting of more than two occurrences). Frequency defined as the reciprocal of the period between only two events is a technical and semantic crutch. Figure 7.4.6-3 is reproduced from Tyler418,419. It shows a significantly different correlation ability in the vertical and horizontal directions. While he describes this experiment in terms of a sinusoidal line stimulus, it is only the one line that is sinusoidal in the plane of the image. The two lines were both high contrast. They were presented straddling the point of fixation. Tyler ended his experiments at a minimum frequency of about 0.04 cycles/degree. This is a pitch of 25 degrees per cycle and corresponds to lines at plus and minus 12.5 degrees from the point of fixation. At the other extreme, his minimum period was about 0.25 degrees. Tyler frequently speaks in terms of period or interval in his writings.

Noting the challenge faced by the two-dimensional correlator of the PGN in Tyler’s experiments is useful. Comparing one straight and one wavy line is obviously more difficult than correlating two straight lines. However, the length of the lines was held constant at 15 degrees in object space. Because of this, both lines appeared to be straight within the analytical channel of vision. His data apparently applies predominantly to the awareness channel and the peripheral retina.

Schor & Tyler also studied the fusional area of vision by moving a pair of lines closer and farther apart sinusoidally420. These and other experiments show that fusion occurs within about 33 msec, the nominal sample interval of the POS of human vision. As Helmholtz showed much earlier, fusion will occur for images acquired in a much shorter interval. He used a spark light source to demonstrate this. Figure 7.4.6-3 Region of fusion versus spatial frequency for 7.4.6.6 Theories explaining the depth images centered on the point of fixation. The stimuli for perception phenomenon the lower frame were under each other and not as shown. From Tyler in Schor & Ciuffreda, 1983. The analysis of depth perception can be separated into two distinct fields. The study of precision (or quantitative) depth perception associated with the analytical channels of human vision, and the study of all other aspects of depth perception. This section will only address the subject of precision depth perception associated with the analytical channel of human vision. This channel contains the foveola of the eyes and the associative correlator of the perigeniculate nucleus found in the thalamus of the precision optical system. The PGN is closely associated with the pulvinar. The two, acting as a couple, are critical to the evaluation of the phenomena of fusion, depth perception and the process of reading. 7.4.6.6.1 Previous theories of precision depth perception

The study of depth perception represented by the literature has not evolved to the point of providing succinct theories

418Tyler, C. (1973) Stereoscopic vision: cortical limitations and a disparity scaling effect Science vol. 181, pp 276-278 419Tyler, C. (1975) Spatial organization of binocular disparity sensitivity Vision Res vol. 15, pp 583-590 420Schor, C. & Tyler, C. (1981) Spatio-tempral properties of Panum’s fusional area Vision Res vol. 21, pp 683- 692 226 Processes in Biological Vision of the depth perception phenomenon. Neither the texts by Schor & Ciuffreda or Howard & Rogers directly address or review the theoretical aspects of depth perception. The Psychology Departments of several colleges are still relying upon a theory of depth perception by Gibson of the 1950's. It is too conceptual to be considered here. A series of articles during the 1990's has focused on vertical disparity as the potential source of depth perception. They have generally been based on the “traffic analysis” form of physiology used earlier by Hubel and others. They generally depend on psychophysical experiments on humans combined with electrophysiological experiments on cats and monkeys. The results are conceptual theories that do not rely upon detailed physiology. They generally define unique circuits of the occipital lobe that act as disparity detectors. No attempt has been made to define the actual circuitry of the retina and midbrain that support these putative disparity detectors. Nor has the circuitry of the disparity detectors been defined except to say they take differences between signals from the two retinas. These papers have employed computational analysis as a major tool, rather than physiological events. They have remained qualitative and have not differentiated between different areas of the retina or different signaling channels within the visual system.

Ohzawa, et. al. have provided the clearest description of the “archetypal disparity detector421.” They define both a simple and complex type of disparity detector. They also state the theme of most of the theories described above: “The neural process of stereoscopic depth discrimination is thought to be initiated in the visual cortex. However, the neural mechanisms of this process are not clear.” Their first endnote lists references supporting their statement.

Nakayama & Shimojo have contributed insights based on occlusion experiments422. They note: “depth from binocularly presented targets depends critically on the solution to the correspondence problem” highlighted by Julesz. However, they do not propose a comprehensive single theory of depth perception.

Grossberg & Raizada have provided considerable background and references related to the problem of depth perception423. However, their neural model is at a very high level and their exposition is based primarily on computational analysis. It does include contributions from both the LGN and V1 & V2.

Mathews et. al. have recently proposed a theory of depth perception, labeled physiological but appearing to be primarily a computational theory. It is based primarily on traffic analysis to assign a major role to disparity detectors in the occipital lobe. They did not offer a quantitative description of depth perception nor did they define the portion of the retina or the visual system contributing to their model. Their assumption that no motion occurs within a 200 msec interval following a major saccade cannot be supported. It is well documented that tremor contributes considerable fine motion during this interval (Sections 7.3.3.1.2 & 7.3.5.3).

Their experimental configuration did depend on a priori knowledge to achieve the proper state of vergence of the eyes. It also depended heavily upon their equation (6) which predicts a conversion of a vertical disparity to an equivalent horizontal disparity under certain conditions. It is the assumption that a vertical disparity, “V will be ‘mistaken’ as an equivalent horizontal disparity . . . ” that is the key to their theory. 7.4.6.6.2 A proposed Physiological Theory of Precision Depth Perception

The analysis of depth perception can be separated into two distinct fields. The study of precision (or quantitative) depth perception associated with the analytical channels of human vision, and the study of all other aspects of depth perception. These other aspects are generally associated with the awareness channel and the LGN/occipital couple. They must be separated into individual phenomena to develop appropriate theories related to them. These descriptions and theories of qualitative depth perception will not be pursued further in this section. A clear description of the phenomenon of precision depth perception can be derived from the previous discussions. The description can be thought of as a corollary to the theory of stereopsis in Section 7.4.6.4. Precision depth perception is a phenomenon derived from the mechanism of stereopsis performed within the associative

421Ohzawa, I. DeAngelis, G. & Freeman, R. (1990) Stereoscopic depth discrimination in the visual cortex: neurons ideally suited as disparity detectors Science vol. 249, pp 1037-1041 422Nakayama, K. & Shimojo, S. (1990) Da Vinci stereopsis: depth and subjective occluding contours from unpaired image points Vision Res vol. 30, pp 1811-1825 423Grossberg, S. & Raizada, R. (2000) Contrast-sensitive perceptual grouping and object-based attention in the laminar circuits of primary visual cortex Vision Res vol. 40, pp 1413-1432 Dynamics of Vision 7- 227 correlator of the perigeniculate nucleus within the precision optical subsystem of human vision. Precision depth perception is the perception of depth associated with the local vergence disparity calculated by the correlator for each significant edge within the purview of the correlator. Within the purview of the correlator, the perception of depth is a linear function of the local vergence disparity. This linear characteristic is labeled veridical depth perception. The purview of the correlator is circumscribed by Panum’s limit in correlator space. This limit reduces to Panum’s limit in x,y space for z = 0. The area enclosed by this limit is known as Panum’s area. This limit can also be expressed by the maximum value of z for the condition x = y = 0. This limit describes the maximum range of depth perception relative to x0,y0,z0. For other values of x,y,z, a volume can be defined that is conceptually equivalent to Panum’s area. It defines the combined range of fusion and depth perception achievable by the subject. Precision depth perception is dependent on tremor to convert the images in spatial coordinates into temporal signals that can be projected to the correlator. Tremor provides separate temporal signals related to both the vertical and horizontal components of the images. 7.4.7 Evaluating stereopsis through fusion and depth perception

The current pre-eminent source of information on the subjects of fusion and depth perception is the compendium by Howard & Rogers424. It devotes seven chapters to these subjects. While this work does not support the concept of individual disparity detectors or the involvement of spatial frequency sensing circuits in vision, the data obtained empirically is extensive, comprehensive and valuable. Their Chapter 18 titled “Tokens for stereopsis” is a particularly important starting point. As noted in that chapter, the edges of objects in the scene are important in stereopsis and therefore fusion and depth perception. The next paragraph will express certain cautions involved in interpreting some of their data. Howard & Rogers explored a variety of special dichoptic conditions that will not be explored here. These include various occluded images (the field of view of one eye was restricted compared with the other) and camouflaged images. Similarly the secondary phenomena of rivalry, shimmer and aura will not be discussed here.

Lit has provided a graph describing the threshold of stereopsis as a function of retinal illuminance in Trolands425. For illuminance greater than one Troland, the threshold remains near ten arc-seconds.

When evaluating depth perception in the laboratory, Millodot has described a “pseudoscope” designed to reverse the perception of depth426. It is not clear whether this device reverses the perception of depth with respect to a mean distance or whether it reverses the perception of depth relative to the point at infinity. 7.4.7.1 Discussion

Stereopsis is a mechanism associated with the two-dimensional associative correlator of the perigeniculate nucleus. While defining the effective spatial dimensions of this correlator using its surrogates, the foveola, is frequently convenient, caution is needed here. It is the correlator that is the source of the mechanism.

A similar area of caution involves the test instrumentation used. While generating dichoptic images of a pseudo-scene to explore the performance associated with stereopsis is frequently easier, it is the actual scene and the resulting dichotic images that the visual system was designed to process and evaluate.

Remembering that the results may become discontinuous when either the local or global correlation ranges of the correlator are exceeded is important when defining either dichotic or dichoptic experiments. These common stereoptic limits can be exceeded by either the fusion or depth component of a single measurement. The result is that most experiments exploring either fusion or depth perception contain boundaries associated with the other phenomenon. The expression of the common stereoptic parameters, the local and global correlation ranges of the correlator, in both the fusion and depth perception phenomena is frequently labeled the double-duty linkage. Howard & Rogers describe the exploration of this linkage on pages 132-135. It suggests that Panum’s area is more directly related to the effective dimensions of the correlator of the PGN than the physical dimensions of the foveola. As a result, Panum’s limit describes a limit on a function combining the parameters of fusion and depth perception. Because of this functional relationship, the linkage is more symmetrical than suggested by the references to Gettys & Harker and to Westheimer.

424Howard, I. & Rogers, B. (2002) Seeing in Depth, vol 2, Depth Perception Toronto, Canada: I Porteous, Chapters 18-24 425Reading, R. (1983) Binocular Vision. London: Butterworths pg 121 426Millodot, M. (2009) Dictionary of optometry and visual science NY: Elsevier page 298 228 Processes in Biological Vision

A distinct experimental problem arises when using only a pair of test objects, points, lines, etc., to evaluate either fusion or depth perception. The results invariably exhibit components related to both fusion and depth perception because of the double-duty linkage. The basic problem is that two objects do not define a specific point of fixation from which independent fusion or depth measurements can be made. A minimally sophisticated test environment should include sufficient objects to cause a mean point of fixation to be calculated. From this mean, differences in x,y coordinates related to fusion and in z coordinates related to depth perception can be measured. When defining a test stimulus for exploring precision stereopsis, it is important that the dimensions of the stimulus not exceed the effective dimensions of the associative correlator. This parameter has frequently been overlooked in the past. As an example, figures 18.37 and 18.38 in Howard & Rogers cannot be viewed within the confines of the effective dimensions of the correlator (typically a 1.2 degree diameter circle). To explore these stereograms requires a repositioning of the point of fixation and a calculation of a new mean for the point of fixation of the horopter. In their original publication, the vertical height of the stereo-pair shown in Figure 7.4.7-1 was 50 mm. At the prescribed viewing distance of 12 cm (4.7 inches), the nominal diameter of the associative correlator is only 2.5 mm. As a result, the depth perception associated with most of the stereogram is perceived qualitatively. The selection of 12 cm as a viewing distance in Howard & Rogers appears awkward. Most adults cannot hope to view a scene in proper focus at such a short range. The range may have been chosen to reduce the size of the stereo-pairs reproduced on a typical 216 x 279 mm pages.

The need to reposition the gaze was noted in their caption. It was impossible for this naive viewer to obtain a single interp of the entire scene at the prescribed distance.

A distinct difference in conceptual (and mathematical) complexity arises between scenes of complex random-dot- stereograms (RDS) and simple geometric shapes. It is proposed that the two approaches lead to the same conclusions concerning the mechanisms and phenomena of interest here. The same proposal applies to dichoptic images containing color differences. The results obtained with simple geometric shapes are easier to describe and interpret. The performance of the correlation capability of the correlator is also easier to demonstrate with simple geometrical shapes. The lack of straight edges in RDS’s degrades the performance of the correlator.

7.4.7.2 The plasticity of fusion and depth Figure 7.4.7-1 A stereogram from Howard & Rogers with parameters added parameters. Specified to be viewed at 12 cm with the provided viewer. Small circle shows relative size of Howard has addressed the plasticity of fusion limits from PGN correlator in this experiment. After alignment of the a variety of perspectives on page 272-281. Similarly, Nonius lines, one set of vertical lines appears behind and Howard & Rogers discuss the plasticity of depth the other in front of the vertical bar. The perception of perception limits on pages 132-135. The latter discussion depth is essentially all qualitative in this figure. is in the context of what they describe as “double duty linkage.” This empirical terminology of these discussions is associated with the performance limits determined by the finite effective spatial dimensions of the associative correlator of the PGN. The calculations in the correlator required to establish the mean and the deviation from the mean of edges in the scene presented to the foveola require that these edges be represented in the field of view of the correlator. If not, the calculations cannot be performed within the precision signal processing channels of vision. This requirement is the key to the plasticity of the limits on the fusion and depth perception phenomena. To evaluate the plasticity of fusion and depth perception, using imagery more complex than just two sources that can be moved to emulate motions in either the x,y or the z dimensions is necessary. The reason is that the point of fixation and the vergence of the visual system must be held constant during the experiments. Howard & Rogers noted the need to control the vergence in their discussion. By introducing targets at a distance from x0,y0,z0, understanding the plasticity in the limits of the fusion and depth perception phenomena is possible. For small distances, these targets introduce signals into the associative correlator that are at distinctly different positions within the effective maximum dimensions of the correlator. The correlator can calculate the mean position of a single edge in object space based on the representation of the two edges in the correlator. The correlator is also able to calculate the difference in the position of the two edges in the correlator. These values represent the fused two-dimensional representation of the scene plus Dynamics of Vision 7- 229 its depth relative to the nominal point of fixation.

As the distance of the object in object space increases relative to x0,y0,z0, one of the edges associated with the object will no longer be processed properly. Its image on one of the foveola will no longer fall within the maximum effective dimensions of the correlator. At this point, the calculation of the mean and deviation from the mean for the edge in correlator space will fail. This dimension defines Panum’s limit in correlator space. Clearly edges with a small z- dimension can have a larger x and/or y-dimension without exceeding the capabilities of the correlator. Conversely, objects with small x and/or y-dimension can have a larger z-dimension without exceeding the capabilities of the correlator. This situation is the underlying characteristic leading to the empirical concept of double-duty linkages. 7.4.7.2.1 Use of “double-duty” parameters in depth perception

When imagery is presented to the two foveola that contain edges well separated from each other in both x,y and in z dimensions, the limited capability of the associative correlator may be taxed. In this case, Panum’s limit, expressed in terms of either the dimensions of the foveola or the dimensions of object space, may appear to shrink considerably. 7.4.7.2.2 Chromostereopsis and transverse chromatic abberations

Simonet & Campbell have provided an update on research related to chromostereopsis in 1990427. They define the phenomenon, “Chromostereopsis, also known as chromatic stereoscopy or the colour stereoscopic effect, occurs when two coloured adjacent objects located in the same frontal plane are perceived binocularly as separated in depth. Chromostereopis is positive if the red object is perceived in front of the blue one, the opposite perception gives negative chromostereopsis'. The magnitude of chromostereopsis is measured by the relative physical displacement between coloured objects which gives a perception of coplanarity. ” They note (supported by many citations), “The majority of studies on chromostereopsis have been conducted with a small sample of four subjects or less, or were not performed with natural pupils but with fixed dilated pupil, or with artificial. In studies performed with natural pupils th e wavelength and the luminance of the targets, as well as the pupil diameter of the observers are not known. The illumination of the targets and the ambient illumination reported by Kishto' present substantial variations. As Einthoven' and Kishto' disagree about the most frequent direction of chromostereopsis, there is a need for a study of the direction of chromostereopsis under controlled experimental conditions and to investigate its dependence on illuminance.”

Simonet & Campbell provided data drawn from a cohort of 30 subjects under a wide range of conditions. The discussion is extensive but is generally in the context of a psychology laboratory where a variety of parameters are not controlled adequately and no circuit model of the visual system is employed as a framework. As an example they note, “The ambient illuminance at the level of the cornea is set respectively at 10 and 1000 lx (10 and 1000 cd/m2 respectively) for low and high illumination levels. Ambient illumination, controlled by a rheostat, is provided by two 500 W halogen lamps, located in the temporal visual field of each eye at an eccentricity of 50 degrees in the horizontal meridian.” No color temperature for the ambient illumination was given. The two extremes of their illumination range both occur within the mesopic range where color constancy is not preserved (Section 2.1.1.1). More significantly, they measured their monochromatic light sources using broadband light meters. Their measurements should have been in watts per square meter at the specific wavelengths used. Filters were made from available sheets of ordinary plastic. They did note that the orientation of their two targets, whether positioned vertically or horizontally with respect to each other was important in the literature. They did not take note of Maxwell’s spot (Section 17.3.1.7.2). Taking the expected positive view, they conclude, “This study, with well-defined experimental conditions, has determined, in a relatively large sample, that the colour stereoscopic effect does not have a preferential direction associated with the level of illumination for levels of 10 and 1000 Ix. Our results do not support or reject any theory proposed for explaining the reversal of chromostereopsis perceived by some subjects.” 7.4.7.3 Combined fusion and depth data in the literature ADD EXAMPLES

While not supporting a unique chapter related to fusion, Howard & Rogers dedicate chapters 21 through 23 to material on depth perception (other than that due to parallax and cues unrelated to stereopsis). Although using crude instrumentation by current standards, an early paper by Tyler is informative428. He explored a variety of test conditions beyond those discussed here. Figure 7.4.7-2 provides a modified composite of two of his

427Simonet, P. & Campbell, M. (1990) Effect of illuminance on the directions of chromostereopsis and transverse chromatic aberration observed with natural pupils Ophthal Physiol Opt vol 10, pp 271-279 428Tyler, C. (1975) Spatial organization of binocular disparity sensitivity Vision Res vol. 15, pp 583-590 230 Processes in Biological Vision figures. He used an oscilloscope on its side as a test source. It apparently provided two signal channels with one baseline unmodulated and the other modulated in both amplitude and frequency. Using polarizers, this instrumentation gave the desired dichoptic presentation with the lines presented alternately in time. The measured brightness of the display is within the photopic operating region. The sharpness of the lines generated probably contributed to the limited the performance of the subject in these tests. However, Tyler also provided data on several subjects that varied considerably in perceived minimum disparity difference. Tyler also presented off-axis data showing that the minimum depth perception performance of one subject fell by a factor of about four for stimuli presented in the region between seven and ten-degree eccentricity.

Figure 7.4.7-2 Perceived depth as a function of vertical line interval. Dotted, dash-dot and dash-dot-dot lines added. Left; 15 degree high lines presented at 30 cm. Right 1.5 degree high lines presented at 3 meters. Dashed and dash-dot lines represent slopes of minus one with respect to frequency (plus one with respect to interval). Dash-dot-dot line represents minimum interval achieved for 30 cm presentation. Dotted line represents diameter of foveola. Long dashes on right are image of curve on left. Adapted from tyler, 1975.

The figure presents two situations. The size and orientation of the lines are shown at the upper right in each frame. Most of the data was taken with a single sinusoid as a stimulus. However, other data was also presented in the paper. While the actual eye vergence during the tests was not given, the effective target vergence upon fusion at 30 cm would be 12.3 degrees. At three meters, it would be 1.2 degrees. These values would relate to the stereoptic performance of the correlator in the PGN.

The frame on the left relates generally to the region of qualitative perception of depth. Most of the relevant signal information falls outside the foveola. The figure shows the perception of depth to be constrained between an area of diplopia, an area of no apparent depth and an area of limited spatial resolution. The minimum perceived peak-to-peak depth perception was about one arc minute. The maximum depth perception was about 1000 arc minutes. The minimum represents about one part in 738 at 12.3 degrees target vergence. The maximum represents a peak perception of depth (his maximum depth limit), each side of the effective point of fixation, slightly greater than one half (500/738) the Dynamics of Vision 7- 231 distance to the effective point of fixation (+/– 67% for the peripheral retina). The dashed and dash-dot lines suggest that the ratio of disparity to interval remains constant with respect to both maximum depth perceived and minimum depth perceived. The right frame shows that projecting the stimuli on an area only the size of the foveola increases the spatial resolution performance related to stereopsis significantly, although it is probably still test set limited. It also improves the minimum perception of depth to about 0.3 arc minutes (20 arc seconds). This minimum represents about one part in 240 at 1.2 degrees. The maximum perception of depth is more difficult to define for this case. The images of the bars exceeded the diameter of the putative correlator within the PGN when the two images were fused since the effective target vergence was about 1.2 degrees. Using the value at an interval of 1.2 degrees, this data gives a ratio of one part in five each side of the fixation point (+/– 20% for the foveola alone). The curve at the lower right describes the monocular limit for perceiving the sinusoidal curvature of the one stimulus. The minimum in this curve also suggests a combined subject and test set limit near 20 arc seconds, or about 4-6 times the spatial limit of the normal foveola for small objects.

7.4.7.4 Depth perception experiments of Allison & Howard

Allison & Howard429 have reported on a series of experiments related to changes in disparity associated with apparent “to and fro” motion using the geometry introduced in [Section 7.4.1.4]. They also noted some competitive work being pursued by Shioiri et al. in Japan430. In their discussion, they introduced the terms change of disparity (CD) and interocular velocity difference (IOVD) signal. As an introduction, they presented Figure 7.4.7-3

Figure 7.4.7-3 Potential motion in depth mechanisms. The “change in disparity” operates on the disparity signal (i.e., in the cyclopean domain) to signal a changing disparity. The “difference–of–velocity detector detects a pure interocular velocity difference between monocular motion detectors. The “dynamic disparity detector” is directly sensitive to changing disparity in binocularly matched features. See text. From Allison & Howard, 2012.

429Allison, R. & Howard, I. (2012) Stereoscopic motion in depth In Harris, L. & Jenkin, M. eds Vison in 3D Environments. NY: Cambridge Univ Press Chap 8 430Shioiri, S. Kakehi, D. Tashiro, T & Yaguchi, H. (2009) Integration of monocular motion signals and the analysis of interocular velocity differences for the perception of motion–in–depth J Vis vol 9, pp1-17 232 Processes in Biological Vision

They described “three ways in which the visual system could, in theory, code changes in relative disparity. C First, the change in binocular disparity over time could be registered. We will refer to this as the change–of–disparity (CD) signal. C Second, the opposite motion of the images could be registered. We will refer to this as the interocular velocity difference (IOVD) signal.” C “Third, the motion in depth could be coded by specialized detectors sensitive to changing disparity in the absence of instantaneous disparity signals.” The physiological elements defined in the caption to their figure are only conceptual. They did not cite any material defining these elements (the disparity detector, the “difference–of–velocity” detector or the “dynamic disparity detector” [the quotation marks were in their caption]) in detail. No representation was made as to where in the signal chain these conceptual elements, and the related elements in the figure, would even be found. The flow diagrams in the figure are too simple to relate to any processing of two and three dimensional information by mechanisms involving engines distributed to different regions within the neural system. In their experiments, all motion was related to the motion of the object space presentations. No motion of the oculars as a result of tremor was considered.

- - - -

Allison & Howard conclude (page 179), “ Several lines of evidence favor the existence of a true IOVD mechanism based on monocular motion signals [with citations].” [xxx expand on this from my perspective ]

- - - -.

Bradshaw & Cumming made an observation in their 1997 paper that focuses a basic situation431. A considerable amount of recent empirical evidence suggests that binocular disparity and motion information are combined during visual processing, although the purpose of this combination remains open to question. Here we investigate the suggestion that retinal motion can facilitate the solution of the binocular correspondence problem. The correspondence problem refers to the difficulty faced by the visual system in establishing which features in the left and right eyes' images originate from the same location in physical space. Establishing the correct matches is prerequisite to the measurement of binocular disparity, which in turn specifies the three-dimensional structure of the world. The number of potential matches can be large, particularly when there is a dense distribution of image features as in, for example, a typical random dot stereogram.

From a physiological perspective, it is obvious that the lateral position differences between the two retinal images at a given time are all of the information available to decode into true lateral and axial positions. The derivatives of these two motions clearly describe the lateral and axial motions associated with these positions. The complexity of the scene, particularly the part imaged on the foveola, only defines the processing load on the appropriate stage 4 engines. Figure 2 of Bradshaw & Cumming describe the sub-threhold to resolved stereo imaging situation but without defining the separate situations related to the foveola and the periphery visual fields. They define the transition region as pykno–stereopsis. Their protocol used a random dot pattern slightly larger than the foveola. “The stimulus comprised a 2 degree square (120 x 120 pixels) patch of random dots centered within an 8 degree square of static random dots. A new array of dots was computed for each interval and for each trial. Observers fixated the centre of the stimulus pattern.” They employed a “standard Wheatstone stereoscope” meaning apparently a two–channel dichoptic presentation. They did not describe the tilt of their fronto-parallel planes with respect to the vertical or the subjects diopter. The overall size of their imagery tends to achieve qualitative binocular vision based on the stationary surround pattern, and then evaluate the precision stereopsis achievable with imagery presented to the foveola related portion of the visual system. However, it is possible they achieved precision convergence before the imagery applied to the foveola began moving. The test pattern was generated by a square wave impressed upon the raster lines at a rate of 15 cycles of depth modulation per degree of visual

431Bradshaw, M. Cumming, B. (1997) The direction of retinal motion facilitates binocular stereopsis Proc Royal Soc B vol 264(1387) pp 1421-1427 Dynamics of Vision 7- 233 angle. “This frequency was chosen as it is well beyond the resolution limit of the stereo and motion systems.” However, this frequency is well below the resolution of the transduction process at the photoreceptors.

The typical statement was made, “All three subjects had normal or corrected-to-normal vision.” How their stereopsis capability was determined is not stated. Figure 7.4.7-4 shows an example of their results. “A two-interval forced choice procedure was used. In one interval (chosen at random on each trial) the disparity of the square wave (i.e. its peak-to-peak amplitude: d1– d2) was set at zero and in the other it was set to one of a range of non-zero values. The disparities were chosen so that responses spanned the range from chance (50%) to perfect performance (100%).” In experiment 1, the thresholds for depth segregation were compared with a control condition in which all of the dots moved in the same direction (dashed line and open bars in figure 4). In the correlated condition, thresholds were significantly lower than the control (p50.05 for all three observers), indicating that stereo matching is specific for the direction of motion. Experiment 2 investigated whether there is a similar advantage for correlated disparity and motion if the square-wave was de¢ned by differences in speed rather than by differences in direction. The results (figure c) show that this is not the case, differences in speed alone do not affect performance appreciably.

Figure 7.4.7-4 Example psychometric functions for dot patterns are shown for one observer from experiment 1 (a) and experiment 2 (c). Each symbol represents the mean of at least 80 trials and the error bars indicate s.e. based on the binomial distribution. Correlated disparity-motion is represented by the open circles, uncorrelated disparity-motion by the closed circles and the translation (control condition) by the open squares and dashed fit. From Bradshaw & Cumming, 1997.

“When near threshold, the stimulus with non-zero disparity would appear as a slightly thickened plane.” Bradshaw & Cumming subsequently determine, “Together, these experiments provide clear psychophysical evidence that different directions of motion, but not different speeds, are processed separately in stereopsis.” And, “The results reported here show clearly that different directions of retinal motion, but not different speeds, affect the ability of observers to resolve two spatially overlaid planes, defined by binocular disparity, into two surfaces separated in depth.” These statements are consistent with the physiology that suggests lateral and axial disparity are computed before speeds are determined. 234 Processes in Biological Vision

[xxx discuss Cummings & Parker method of differentiating between CD and IOVD signals, pg 165 in Allison & Howard]

7.4.7.5 Dynamics of stereopsis and role of non-declaratory (implicit) memory

It is not commonly known, but measuring a subjects horopter is highly dependent on his quiescent state of visual performance432. If a subjects performance is measured a natural and then measured again within a short period while employing corrective glasses, his performance usually deteriorates drastically. The numbers are startling; Depth perception (sec of arc) Subject Case 1 Case 2 Case 3 Unaided +8.8 -2.93 +12 Aided -73.49 -67.61 +24 They noted, “Correcting the refractive errors of athletes is a common management in order to maximize their visual performance. It is generally suggested that uncorrected refractive errors would adversely affected the depth perception, which is one of the most important visual performance in sports playing. These cases, however, demonstrated the athletes with well- adapted monovision can show a marked deterioration of depth perception immediately after refractive correction.”

This author encountered this problem while putting on glasses he only wore intermittently during a badminton game. His depth perception deteriorated greatly.

Changes of the above magnitude in stereopsis performance are indicative of the considerable amount of neural computation and rewriting of the scaling parameters associated with stage 4/6 non-declaratory memory (within the cerebellum) involved in stereopsis. See Sections 17.1.1.3 and xxx in Chapter 17 of “The Neuron and Neural system” for more discussion of non-declaratory memory. See Section 4.6.3 in Chapter 4 of that work showing the role of the cerebellum, and potentially the striatum/pallium couple, in the stage 4 to stage 6 reflex arc associated with stereopsis. See Sections 8.5.3 & 8.6.3 in “Hearing: A 21st Century Paradigm” for additional material associated with non-declaratory memory. [xxx reproduced 8.6.2 of early version into 17.1.1. look also at the figure showing reflex arcs via the cerebellum ] 7.4.8 Overall temporal response & latencies

Considerable literature exists concerning the latencies found in the visual system. However, the parameters are described using a variety of definitions and criteria. Zeki & Moutoussis have provided an overview with many references433. They have also highlighted the significance of “the retino-tecto-cortical branch, the latter bypassing V1 and reaching V5 at latencies of about 35 ms, to which it delivers signals from fast-moving objects (>10° s -1).” Vidyasagar has provided a variety of latencies. He generally relates the latency for a signal arriving at the primate visual cortex of from 50 to 100 ms434. The minimum appears to be 50 ms for a signal not intercepted for further processing within the midbrain. He defines the cells exhibiting this latency as AR cells (attention related). He describes the cells exhibiting the longer latencies as AS cells (attention specific). In this work, the signals arriving at the AS cells have undergone processing within an engine of the thalamus before relay to the primate visual cortex. In the last paragraph of his paper, he presents a concept that is analogous to the POS of this work. 7.4.8.1 Definition and tabulation of latencies

Latency has lacked a precise definition throughout the vision literature. It is frequently used quite differently in electrophysiology and psychophysics. In electrophysiology, it is usually used as a generic term for the time between

432Tong,C. & . Ng, V. (2010) http://www.polyu.edu.hk/so/images/research/file/Cecilia%20Tong-Poster%20of%20Depth%20Perception% 20(amended).pdf 433Zeki, S. & Moutoussis, K. (1997) Temporal hierarchy of the visual perceptive systems in the Mondrian world xxx pp 1415-1419 434Vidyasagar, T. (1998) Gating of neuronal responses in macaque primary visual cortex by an attention spotlight Neuroreport vol. 9, no. 9, pp 1947-952 Dynamics of Vision 7- 235 the initiation of an impulse stimulus and the beginning, the leading edge, of the electrical response at a later point in the visual system. In psychophysics, it is frequently used to describe the time between a stimulus and the conclusion of a saccade designed to bring the line of fixation to the location of the stimulus. The latter description obviously includes both a latency (period before any measurable output) and a response time. In electrophysiology usage, ambiguity is frequently found concerning what feature on the rising edge of the response should be used as an indication of the beginning of the response. Since most of the responses are described initially by a first order differential equation, the logical feature is where the response departs from the quiescent level. Although this point is frequently obscured by test set noise, it can be readily determined. Projecting a first line through the midpoints of the quiescent level and then a second line back along the rising waveform to its intersection with the first line will define the start of the signal waveform. This feature provides a latency that is independent of the slope of the response and leaves the description of the response as a separate parameter. For a ganglion cell, the time delay before the response reaches a threshold value is then assigned to the ganglion cell and not to the prior signal path. In the psychophysical case, the situation is a bit more difficult because the instrumentation frequently does not allow measurement of the subsequent fine motions following the saccades. In these measurements an arbitrary velocity threshold is frequently imposed to define the completion, and frequently the start, of the saccades. The above definition has little relevance to a microsaccade that never reaches the conventional velocity threshold. It is also possible that the microsaccade may be the result of a second order mechanism. This makes the start of the saccade, as a function of position, less defined. Either a lower velocity threshold is required or a zero crossing definition may be preferred.

The literature has accumulated many measured latencies but it has not evolved a framework for correlating these many values. By reviewing this material, such a consistent framework can clearly be assembled. Such a framework concerning the motions of the eyes, and the related motions of the skeletal system that will not be addressed here, can be formed if at least four operating modes of the visual system are recognized. These modes can be described as involving;

+ an awareness mode. In the absence of any visual threat to the animal, this mode involves all of the visual field except the foveola of each eye.

+ an analytical mode. This mode involves only the object space imaged on the foveola.

+ a volition mode. This mode involves a spontaneous decision by the cortex to change the line of fixation. The resulting command is independent of the field of view of the eyes.

+ an alarm mode. This mode involves a change in the line of fixation in response to changes in the scene peripheral to the part imaged on the foveola. The change in the line of fixation may exceed the original field of view if inputs from other sensory channels are involved.

Figure 7.4.8-1 presents a flow diagram of the visual system based on these modes. The complete diagram is complex. The individual mode diagrams at the bottom simplify tracking the signals. Note that the total latency, or the total response time through the initial saccades, is determined largely by the length of the signal paths involved. The signal paths involving the optic nerve and the oculomotor signal paths from the superior colliculus do not change with mode. However, the other paths do change. This figure can be compared with a similar one by Reichle focused on eye movements during the reading process. His focus on the thalamus, superior colliculus, oculomotor plant and the intraparietal and frontal eye fields (labeled 5, 6, 7, 8 & BS) is very similar to the focus of this figure. 236 Processes in Biological Vision

Figure 7.4.8-1 Nodes and transit times affecting the latencies and response times of the visual system. Upper frame; full flow diagram. Blue boxes represent areas concerned with interpretation and initial perception of fine detail. Brown boxes represent elements of the awareness mode of vision. Green boxes represent higher cognitive centers. Lower frame; individual flow diagrams by mode. Tan boxes represent the LGN. Blue boxes represent the Pretectum (perigeniculate nucleus and pulvinar). See text for discussion. Dynamics of Vision 7- 237

Figure 7.4.8-2 presents a tabulation of the transport times involved in the various legs of this diagram and the processing time at each node. It presents the best available data describing the time required for the human visual system to respond to a stimulus while in each of these operating modes. For convenience, the various potential delays have been listed on the left of the figure. The right of the figure is divided into three parallel sections of two columns each followed by a comment column. Each pair of columns is arranged to show absolute time delays on the left and cumulative delays on the right. The comment column is necessarily brief and is only used to describe key parameters. The following discussion is more definitive and presents comments on an individual row and column basis using the labels on the left and at the top of the table. The left of the figure follows the general division of the visual system into the stages discussed elsewhere in this work. It begins with signal detection (stage 1) and is followed by retinal signal processing (stage 2), signal projection (stage 3) and the initial signal processing within the brain (stage 4). These stages are followed by higher level information analysis within the brain (stage 5). This stage is then followed by a less defined stage 6 involving the generation of a group of instruction signals to be sent to the thalamus. Implementation of these instructions follows. The first column to the right of the location descriptors is labeled the ROUTINE PATH. This path is meant to encompass the performance of the signaling environment of the eye during its routine operation. This path has been defined in two segments. These segments have been presented in one column for artistic convenience. The upper segment describes the Awareness mode of operation. It can be used to describe the time spans associated with the absorption of information concerning object space within the total field of view of the subject. It concludes with the delivery of the sensed information to the cortex at the level labeled 5-2 on the left. This level is labeled Area (multiple) to suggest that it involves many information processing engines in the dispersed cognitive areas of the frontal lobe. Because of this dispersal of information within the frontal lobe, level 5-1 provides only an average absolute time of transmission over the multiple association fibers operating in parallel. The cumulative time can only be calculated to the point of delivery of the information to the appropriate engines of the frontal lobe. The lower segment of this column describes the Volition mode of operation. In this mode, the mind has decided to change the line of fixation (independent of any external stimulus) to examine some element of the field of view in detail. No time can be assigned to the decision process. The implementation time begins with the travel of the command over the association fibers leading to the initial motor instruction generation area in area 7A of the parietal lobe.

The second column is labeled the Alarm mode. This mode is initiated by the Lateral Geniculate Nuclei in response to a change or changes in object space peripheral to the foveola. Before this initiation, the signal path is the same as for the Awareness mode. The subsequent response is complicated by the nature of the change(s) and may be under the command of the control point (TRN) to be discussed in detail in Section 15.2.2. For a single change in object space, the initial response is transmitted to the cerebellum where a reflex response is generated and passed back to the superior colliculus for implementation. Beyond this point, the responses are essentially identical to those of the Volition mode.

Some evidence exists that the neurons called W-cells in cats are associated with this alarm function. While they are described as connecting the retina to the superior colliculus, this claim appears based mostly on behavior experiments with induced lesions. If they do not actually pass through the LGN, they are probably the neural paths defining areas of the retina in terms of azimuth and elevation coordinates. These signals would allow the superior colliculus to drive the POS without requiring trigonometric calculations.

If multiple changes are sensed in object space within a finite interval, the TRN senses this complication and reroutes the alarm response along an alternate path providing additional analytical capability before creation of a response decision. In the figure, this alternate path is shown being passed to area 7 and area 7A of the parietal lobe. The data may travel further into the frontal lobe on its way from area 7 to area 7A. As a result, two distinct and parallel subpaths are shown along the alarm path to the superior colliculus. It is likely that the TRN participates in the choice of which signal is implemented by the superior colliculus. From this point on, the response follows the same timeline as the Volition mode. The existence of an alternate, and slower, alarm-processing mode is well known. It is important when the system is faced with multiple threats. It explains precisely why “swat teams” and other militaristic actions are timed to present the subject with multiple threats at the same instant. The third column describes the Analysis mode of vision. This is arguably the most important mode of vision and is found in its most sophisticated state primarily in the higher members of Chordata. The signal path is essentially independent of the other modes in that the signal detection function only involves the photoreceptors of the foveola and the PGN/pulvinar couple. Much of the retinal signal processing is bypassed in the signaling path of this mode. In addition, the LGN and the so-called primary visual cortex of Area 17 are also bypassed. 238 Processes in Biological Vision

To achieve the very high speed analytical capability required to evaluate a complex threat or a nutritional opportunity, it is important that the system can analyze a small complex scene quickly. To accomplish this function, the signals from the foveola are projected directly to the PGN (alias pretectum) and pulvinar where they are processed for both scanning information and contextual information. The contextual information is passed directly to Area 7 of the cortex via the Pulvinar pathway for immediate analysis by the information extraction engine located there. If appropriate, this information is then passed to the frontal lobe for analysis by additional engines. This activity may result in commands being passed back to the premotor command generator at Area 7A. There is no requirement that this time critical information be processed through the “primary visual cortex.” Meanwhile, the extracted scanning information is passed back through the superior colliculus, via a parallel path, to the other elements of the oculomotor subsystem to aid the continuation of the scanning process. The signal path from the photoreceptors of the foveola directly to the PGN and then to the superior colliculus and the oculomotor subsystem forms a servomechanism designated the Precision Optical System, POS. This servomechanism includes the elements previously included in the auxiliary optical system. The purpose of the auxiliary optical system was previously unknown. The POS is a very high performance servomechanism exhibiting the minimum possible time delay. The two sets of commands received by the superior colliculus are merged, possibly under the guidance of the TRN, before following the same signal path to the oculomotor subsystem as in the Volition mode. The Analytic mode commands passed directly from the PGN to the superior colliculus are limited to microsaccades and flicks. The commands from Area 7A can call for larger amplitude saccades, especially when merged with commands from the Volition and or Alarm modes. Dynamics of Vision 7- 239

Figure 7.4.8-2 Flow chart of latencies in the human visual system. See text. 240 Processes in Biological Vision

7.4.8.2 Comments by line

In the table, the transport delays associated with phasic signals (employing action potentials) appear under two different designations. When outside the brain, it is labeled signal projection and is associated with the optic nerve. When inside the brain, it is labeled association projection because it is carried over association fibers. Although the line is blurred, for purposes of discussion, the signal processing within the brain that is not related to cognition is labeled signal processing. This includes the reflex responses associated with the Cerebellum and that associated with areas 17 through 22 in the posterior cortex. That associated with cognition is assumed to occur in the parietal and frontal lobes of the cortex and is labeled information analysis. As can be seen from the table, some cumulative numbers are available for some modes of processing although individual values for segments of a given path may not be available from the literature. Line 1. The time delay in the signal detection stage is dominated by the delay associated with the P/D process within the Outer Segment of the photoreceptors. This delay is sensitive to the temperature and the photon flux rate. For humans, the temperature is essentially constant. The variation as a function of photon flux has not been reported under rigorous test conditions. The best data appears in Figure 16.3.6-2 and dates from the 1930's. Based on this data, the delay varies from three milliseconds in the photopic region to 10 ms or more at very low light levels.

Line 2. The signal processing within the retina is carried on in the analog domain and the transport velocity of the signals is quite high. The minimum delay is as shown. For additional processing through the lateral paths, the delay may be marginally longer.

Line 3-1. Reyem’s loop. This variable path distance for phasic signals between the ganglion cells and the lamina cribosa introduces a variable time delay into the awareness path. This delay is proportional to the location of the source of illumination on the retina. This can introduce an additional path length of up to 10 mm or 0.44 ms relative to the fixed delay associated with the analytical path.

Line 3-2. The signal projection over the optic nerve from the lamina cribosa to the midbrain employs the phasic domain and is relatively slow. The nominal signal is for a path length of 75 mm.

Line 4-1. The time delays associated with signal processing in the LGN and the PGN are assumed to be small and approximately equal because of the size of the structures involved. The TRN operates as a major switching point here, directing the signals to different elements of the brain based on a previously defined set of rules (that may be changeable through training).

Line 4-2. The pulvinar pathway is assumed to have a length of 50 mm in humans for purposes of discussion.

Line 4-3. The geniculocalcerine pathway has a variable length due to Meyer’s loops. The minimum pathway length is taken to be 50 mm in humans for purposes of discussion.

Line 4-4. The time delay associated with the cerebellum is only an estimate. Little data was found concerning this parameter.

Line 4-5. The visual signal processing in the cortex is concentrated in the posterior lobe. In transferring signals from area 17 to area 7, it is not clear how much signal propagation is by analog means and how much is by phasic means. The transport delays associated with these modes are quite different. It is also unclear how much time is used in signal processing within areas 17 through 22. Specific values are becoming available from fMRI studies. Line 5. The time delays associated with information processing within the frontal lobes can only be given as estimates since the processing is so dispersed and the variety of techniques for stimulating the system varies so widely. Line 5-1. This line provides an average delay for association fiber projection of 2.0 ms for purposes of discussion. The cumulative values shown on this line are the best available based on the variety of data available in the literature. Line 5-2. This line is meant to hold an average processing time for all of the engines of the frontal and parietal lobes associated with a given experiment. Line 5-3. This line provides an average delay for association fiber projection of 2.0 ms for purposes of discussion. The cumulative values shown on this line are the best available based on the variety of data available in the literature. Dynamics of Vision 7- 241 Line 6-1. The values shown are small reflecting the size of the visual processing engines in Area 7. Line 7-1. The TRN operates as the decision maker and gatekeeper, determining what signals are to be implemented by the superior colliculus Line 7-2. The superior colliculus directs signals to the oculomotor portion of the POS. Line 7-3. The oculomotor portion of the POS implements the prescribed saccade Line 8. The command signals to the oculomotor muscles encounter a constant time delay Line 9-1 The only delay shown on this line is the absolute delay associated with the electrical performance of the oculomotor muscle fiber. The delay associated with the mechanical performance of the fibers depends on their design function. This value has been lumped into Line 9-2. Line 9-2 The total time required to accomplish a specific saccade varies with the starting eccentricity of the eye and the target eccentricity. In the first approximation, this time is proportional to the change in eccentricity angle. The cumulative values shown in Line 9-2 provide important information about the overall performance of the visual process. They highlight the large difference between the time to recognize a threat imaged on the foveola and one not imaged there. For a threat not imaged on the foveola, its evaluation time is highly dependent on whether the threat is single valued or multivalued. 7.4.8.3 Correlation with the literature

When attempting to correlate the times shown in the above figure with those found in the literature, it requires considerable effort to understand precisely the experimental procedure used by various investigators and the corresponding specific path in the above table. Kennedy has provided some data of this type435. Jones has also provided data that can be interpreted in this framework (pp 300-306 in S&C).

Wolfe, et. al. have provided some simple experimental results recently comparing searches involving volition and “anarchic searches436.” The reader is cautioned they use the term rate in place of interval. A rate is an expression with time (or another independent variable) in the denominator, such as frames/ms. They show that the selection of objects in a simple scene is much slower if volition is involved rather than a simple anarchic search (presumably under the control of the TRN and POS). 7.4.8.3.1 Volition mode experiments

Becker & Fuchs give excellent data on a variety of volition mode eye movements437. The primary variant was whether the eyes could fixate on a visible object or not. They generally found that a volition mode movement made without a visual reference reached a lower velocity and therefore took longer than its counterpart with visual cues. The measurements in the dark followed a period where the subject attempted to memorize the location of the desired pre- and post saccade locations. After the light was extinguished (for at least three seconds) the duration of the volition mode saccade was increased by 10 to 35 ms and the maximum velocity was 46 to 108°/sec slower for a desired movement of 40°. Their values were shown to be statistically significant for some of their subjects but not for others. Some subjects were not able to produce a saccade close to the desired 40° without visual cues. Because the dark and illuminated field results were so different, Becker & Fuchs also examined the transition state by varying the dark time between the memorization step and the actual dark saccades. They found a shoulder near 1100 ms that defined the transition between the two modes of operation. They then explored the ability to prevent or interrupt a saccade by changing the environment. Mulligan has recently provided data on the delay in volition tracking movements as a function of the contrast of a Gaussian target spot438. A typical delay of about 100 msec was found before any movement.

435Underwood, G. (1998) Op. Cit. pp 177-179 436Wolfe, J. Alvarez, G. & Horowitz, T. (2000) Attention is fast but volition is slow Nature vol. 406, pg 691 437Becker W. & Fuchs, A. (1969) Op. Cit. 438Mulligan, J. (2002) Sensory processing delays measured with the eye-movement correlogram. Ann. N.Y. Acad. Sci. vol. 956, pp 476-478 242 Processes in Biological Vision

7.4.8.3.2 Alarm mode experiments (including conflict resolution)

In some of the final discussions in Becker & Fuchs, the experiments fall into a more complex mode than the simple volition mode. While useful as an overview, their terminology appears to stretch their assumed baseline. Extracting more data from their paper may be possible but its precision may suffer from their lack of a sufficiently sophisticated model when they were preparing the text. They described this “package hypothesis” and further experiments to explore it in detail on their pages 1253-55. Zeevi & Peli have presented data on the saccadic latency following the movement of a test stimulus within the central 10° field of view of the subject439. Their tests involved a change in location of 4° but the starting and ending locations were unusual. They also defined a reaction time corresponding to the time between the movement of the test stimulus and the first antisaccade. Their results highlight the inhibitory process involved in the “dual target” task suggested by the alternate alarm path between lines 4-4 and 7-1. The initiation of a saccade by the superior colliculus is clearly inhibited until the decision has been made about which target upon which to concentrate. Findlay investigated the dual alarm phenomenon in some detail440. Under the simplest scenario, he said (citing Levy- Schoen) an additional nominal 30-40 ms was required to resolve what saccade was to be implemented. This scenario involved two alarms placed symmetrically relative to the prior fixation point. This decision process involves either an unrecognized area of the midbrain or area 7. Findlay’s own experiments used an 8° eccentricity stimulus and a second stimulus at an eccentricity that varied from 4° to 6° to 8° and occasionally 12° and 16°. A variation is size between the two stimuli was also introduced. Findlay introduced the concept of a saliency map in object space to report his results. Findlay performed experiments to define the saliency due to sudden appearances of stationary lights and then experiments to determine saliency as a function of movement of extra-foveola lights. He rapidly noted that both the velocity and the direction of the stimulus were important. He was also careful to note that he employed apparent motion as opposed to real motion by rippling the illuminated lines on the face of the monitor. The apparent velocities were 6, 12 and 18 degrees/sec. The monitor imposed a variety of limitations on the quality of the simulation. These limitation were related to both the apparent motion of the moving stimulus and the apparent brightness of the fixed stimulus.

Although the title of Findlay’s paper focused on saccadic motion, the results provide significant information about the sensitivity of the eye as a function of eccentricity to both brightness and velocity changes. He also reviews the evidence for a gating of the data from the “deeper cells” of the superior colliculus by the upper cells (page 18). This gating might be controlled by the TRN. He then discusses the fact that cells have been found among the ganglion cells that provide a transient output that may be related to saccade generation. In this work, these cells are a result of spatial diversity before subtraction in the signal processing stage. It should be noted that this mechanism is distinctly different in the central 2° of the retina than in the extra-fovea area. This spatial diversity encoding provides a method for the visual system to differentiate between a flashing source of illumination and a moving source of illumination. His work was inconclusive with respect to biases favoring one direction of saccadic motion over another and in sensitivity to the direction associated with motion in object space. A tendency for saccades to favor a rightward direction in reading is probably not shared with readers of Hebrew and other languages using a right-to-left or top-to-bottom presentation.

The fact that the subject’s performance improves so rapidly with training in the Zeevi & Peli study is interesting. It suggests that the shortest times given by them for the cumulative values applicable to line 9-2(D) of 200 ms +/– SD 30 ms corresponds to the direct variant of the alarm path. It also suggests that their unusual test protocol causes the subject to initially follow the alternative path that employs additional analysis. This path results in an initial value for line 9-2(D) that ranged from 610 +/- 14 ms to 310 +/- 39 ms among three subjects before converging toward the 200 ms value with training. With only three subjects, they were unable to provide a statistically relevant average observer but did demonstrate both the variability and the training aspects of the times involved with Line 9-2(D). Zeevi & Peli made the assumption that the visual system is symmetrical with respect to temporal and nasal saccades. Carpenter & Williams have presented data on the latency of alarm mode signals from 15° eccentricity (either left or right) in object space441. Their thesis assumed an entirely stochastic process after their discussion dismisses synaptic delays and conduction velocities as not providing viable answers. They also assume the visual system is random noise limited under presumably photopic conditions. They used an unusual graph with a probit vertical scale and a logarithmic scale representing “reciprocal latency.” Each figure shows two distinct families of loci that bear further study to determine

439Zeeki, Y. & Peli, E. (1979) Latency of peripheral saccades. J. Opt. Soc. Am. vol. 69, no. 9, pp 1274-1279 440Findlay, J. (1980) The visual stimulus for saccadic eye movements in human observers. Perception, vol. 9, pp 7-21 441Carpenter, R. & Williams, M. (1995) Neural computation of log likelihood in control of saccadic eye movements. Nature, vol. 377, pp 59-62 Dynamics of Vision 7- 243 relevancy. One family extends beyond the boundaries of the graph.

7.4.8.3.3 Interrupted alarm mode and other experiments

Hanes & Carpenter provide excellent data, and a bibliography of recent work, on a strategy of interrupting a normal alarm path response in humans442. They employed a 14 by 23 arc-minute size yellow LED against a “color-matched background” at a uniform luminance of 4.5 cd/m2. The goal was to learn how late in a normal alarm path response, the subject could inhibit his initial saccade. They defined this interval as the stop-signal reaction time (SSRT). A similar set of papers has been presented by Hanes, et. al working with the rhesus monkey443. Lacking a functional model in their presentations, the work of both groups must be considered exploratory. Although the analyses are exemplary in presentation, they do not appear to allow a combination of determinate and stochastic processes in the explanation of the SSRT and other cumulative time intervals. These authors did not address the alternate signal paths through the visual system employed by signals from different locations in the retina developed in this work. Lacking this level of discrimination, determining the precise paths (analytical, awareness or volition) involved in their experiments is difficult based on their text alone. Using the model of this work to describe the signal paths taken by signals resulting from their stimuli, their protocol can be seen to involve more variables than they controlled. When correlated with the Flow and Latency Profile presented above, their experiments emphasize the critical importance of the TRN (or other unrecognized element of the midbrain) in the overall operation of the visual system. The TRN is responsible for controlling the flow of responses through the alternate loops of the cerebellum, and to the parietal areas of the cortex. It is also critically involved in the final selection of which responses received from these elements are carried out. Further correlation would suggest their experiment involved two alarm path responses plus a complex relationship between the two. They used the reappearance of the initial fixation spot as a signal to stop their saccade to the second spot. However, the use of an audio tone to request inhibition may have been more appropriate. In the case instrumented, the stop signal effectively introduced a two-target situation (with one target stimulating the faster, foveola related, signal path) upon which an additional decision had to be made.

Hanes & Carpenter controlled the light level and contrast of their test environment and recognized a small but significant difference in the temporal and nasal saccade implementation times. They determined the difference in saccade time with direction was trivial in their experiments. At high contrast, the average delay for the control experiments, across several thousand tests, was 227+/-0.4 ms for a target presented at 9° eccentricity. The reduced contrast tests introduced an additional delay of 32 +/-0.4 ms.

As for their SSRT experiments, they found “the latency required to inhibit the production of a saccades following presentation of a stop signal is similar across subjects, on average 137 ms, and is approximately 40 ms longer than in rhesus monkeys.” This value is specific to their signaling protocol. They discussed their protocol in relation to others on their page 2790. Below that discussion, they discuss their future plans involving varying many parameters. Certain conclusions can be drawn based on this model and a reading of their published description. The 137 ms value is associated with the difference in time between an initial stimulus traversing the dual alarm mode signal path and a subsequent signal traveling the analytical mode path. Based on the small statistical variation in this value, discussed below, it would suggest that 137 +/- 11 ms can be assigned to a specific path in the latency profile. It appears to match the cumulative value for the analytical path from the photoreceptors of the foveola to the PGN/pulvinar decision point (Line 7-1 in the Latency Profile). It also appears that the 40 ms difference for the rhesus monkey is due primarily to the shorter signal projection distances involved in this smaller animal. Their variation in performance with contrast appears to agree with the threshold level of the ganglion cells in the R- channel of this work. The discussion of their results in terms of a race model can be described with greater relevancy in terms of a delay loop model. In that model, the TRN acts as a gatekeeper for signals passed to the superior colliculus for implementation by the oculomotor subsystem. In this interpretation, their figure 4 is replaced by a graph with an ordinate defining the percent path length (normalized) between the TRN as a signal distributor and the TRN as a decision maker. The abscissa remains a plot of time. Neural signals travel at either of two principal speeds, a high speed over signal projection (action potential) circuits and a low speed over analog (tonic) circuits. The total time delay for a signal to travel out from the

442Hanes, D. & Carpenter, R. (1999) Contermanding saccades in humans. Vision Res. vol. 39, pp 2777-2791 443Hanes, D. Patterson II, W. & Schall, J. (1998) Role of frontal eye fields in countermanding saccades: . . . J. Neurophysiol. vol. 79, pp 817-834 and references 244 Processes in Animal Vision TRN and back primarily depends on the physical length of the signal projection circuits plus the equivalent length of the signal processing circuits. The length of the signal processing circuits within a given engine may change through logic controlled switching. Under this analog, the progress of each signal along its prescribed path can be plotted as a percentage as a function of time. Note, the slope of the normalized curves does not represent a velocity since the curves are normalized. In the time delay analog, the first signal to reach the TRN will theoretically control the decision process. In practice, either the signals take a statistically variable time to travel through some of the signal processing engines or the TRN takes a statistically significant time to make a decision. As a result, a finite statistical range may be associated with the process. Hanes & Carpenter define this range in stop-signal reaction time in their table 2. This range averaged +/-11 ms at high contrast and +/- 6.5 ms at low contrast. Using the model of this work and a modified Hanes & Carpenter protocol should provide additional information about the operation of the visual system. This combination would not require a broadly based Monte Carlo simulation. It would significantly enhance our knowledge of the various signaling paths leaving the LGN and TRN. 7.4.8.3.4 Participation of the frontal eye fields of the cerebrum

The arrival of the MRI and the fMRI has introduced an additional non-invasive research capability of great value. These techniques have illuminated the precise location and size of the frontal eye field (FEF). The large area of this field focuses attention on the approach of Hanes, et. al444 whom have probed this area in search of individual neurons that show activity prior to the onset of a saccade. Based on the area of this field, it is estimated to contain over 100,000 individual neurons. It is likely that a great many of these neurons participate in a complex analog signal manipulation process. This process is best described using boolean algebra. Only a few neurons will be involved in the generation of action potentials to be sent over association fibers to other engines leading to the POS. However, this number may still be in the hundreds to thousands. The method of encoding the action potentials to represent the vector information required by the POS is still not known. It is possible that a group of parallel association fibers must be activated simultaneously to transmit the desired vector. The path these activation signals follow is also poorly understood. It appears to go via area 7A (and possibly via the TRN) to the superior colliculus for further signal processing before generation of the final commands by the Oculomotor Nuclei. Prior to the development of the above techniques, Robinson commented on a measured discrepancy between the shortest latency between stimulation of the FEF and stimulation of the superior colliculus445. They noted that the shortest reported latency from the FEF was 15 ms while the shortest reported latency from the superior colliculus was 20 ms. Again, lacking a detailed model of the signal path options, knowledge of the size of the subjects involved, and a careful comparison of the stimulation techniques used, reconciling this difference is difficult. In the Robinson paper, figure 3 speaks of the mesencephalon instead of the superior colliculus thereby showing the potential variation in experimental results based on the precise location involved. This is an area of current and fertile research. Further discussion of these matters will be found in Chapter 15.

7.4.9 The lens and aperture control subsystems: accommodation

The lens and the aperture control system are elements of two separate closed loop subsystems. Portions of each system are found within the ocular globe and portions are found in the midbrain. These elements are interconnected by nerves projecting distally from the superior colliculus to the interior of the eye (See Section 2.3.4). The circuits also appear to contain an analog computational function associated with the PGN/pulvinar couple. The details of how the neural signals are decoded and converted to an analog signal capable of controlling the appropriate muscles is beyond the scope of this discussion. It is addressed more completely in (Section 15.3.3.2.2). The neural signaling techniques are probably the same as for any other muscle system and similar to those discussed in Chapter 14. Performance errors associated with this system are discussed in Section 18.8.3. The vergence control system is also a closed loop servomechanism. It is a more complicated loop physiologically. The initial information is extracted from the data of the two eyes in the layers of the LGN. This information is fed to the thalamus where correction signals are generated via the superior colliculus. The foveola signals obtained by each eye following the initial vergence are probably used by the PGN/pulvinar couple to perform a second level of vergence. If true, the overall vergence control system is a two level system of considerable expanse in terms of the individual elements of the visual system.

444Hanes, D. Patterson II, W. & Schall, J. (1998) Op. Cit. 445Robinson, D. (1972) Eye movements evoked by collicular stimulation in the alert monkey. Vision Res. vol. 12, pp 1795-1808 Dynamics of Vision 7- 245 Westheimer & Blair have provided a block diagram of the signal paths associated with these three elements of the Precision Optical System446. Because of its complexity, the vergence control subsystem will be discussed in Section 7.4.3 & 7.4.9 following the broader discussion of the pointing system. [xxx does this belong here?] Few images of the plant associated with the lens and aperture subsystems are found in the literature. Figure 7.4.9-1 from Kuwabara & Cogan in Weiss is particularly useful447. Taken from the position of the retina, but with no specifics relative to location of the optical axis or foveola, it clearly shows the gross size of the lens, and both the nominal maximum diameter (note change in shading) and current diameter of the iris. It also shows the radial arrangement of structure (pars plicata) of the ciliary body probably related to the equalizing structure maintaining the tension on the lens. Beyond this structure is the more uniform ciliary body (pars plana). 7.4.9.1 The lens control system

The lens control system involves a broad collection of elements based on different technologies and mechanisms. This section will attempt to place these elements in perspective and note the unique performance limitations imposed by some of them. 7.4.9.1.1 Background

As usual, the discussion of the lens control system is addressed independently and from different perspectives in various segments of the vision community.

The papers of L. Stark448 and Blaker449 are important in understanding the operation of the lens control system. Stark provides a review of earlier attempts at developing a model of the physical plant. Blaker provides some actual data on the operation of the plant. It was collected by Fincham in 1937. One useful piece of information was found in Jennings & Charman450. Coleman has presented a paper analyzing the lens system in terms of a hydraulic model involving pressures between the various humors and the lens. It is described as the “catenary” approach based on the shape of one surface of the lens. As usual, no complete model, either static or dynamic, is provided.

Grosvenor & Flom have edited a significant compendium on the subject of refractive anomalies (with a concentration on the physiology of myopia). Schor, writing in their book (pp 310-317), introduces the subject of a closed loop servomechanism controlling accommodation, based

Figure 7.4.9-1 Image of the lens and pupil taken from the position of the retina LARGE FILE. See text.

446Westheimer, G. & Blair, S. (1973) The parasympathetic pathways to internal eye muscles. Invest. Ophthalmol. Vis. Sci. vol. 12, pp 193-197 447Kuwabara, T. & Cogan, D. (1977) The eye In Weiss, L. & Greep, R. Histology, 4th ed. Chapter 32 448Stark, L. (1988) Presbyopia in light of accommodation. Am. Jour. Optom. Physiol Optics, vol. 65, no. 5, pp 407-416 449Blaker, J. (1980) Toward an adaptive model of the eye. J. Opt. Soc. Am. vol. 70, n0. 2, pp 220-223 450Jennings, J. & Charman, W. (1978) Optical image quality in the peripheral retina. Am. J. Optom. Physiol. Optics vol. 55, no. 8, pp 582-590 246 Processes in Animal Vision on his earlier work in vergence451, but does not pursue it in detail452. Two brief discussions including block diagrams have appeared recently. The same material of Wallman appears in two separate publications453 ,454. The simple and largely conceptual figures in that material suffer from a number of structural problems (note the question mark along one of the signal paths in the servomechanism diagram) and do not address the signal processing associated with the servomechanism at all. The Boolean logic associated with the diagram does not appear to relate to any realizable servomechanism. Jiang has also contributed a largely conceptual floating model455. He introduces two non-linear elements into the basic conceptual servomechanism of the accommodation subsystem without providing any substantive justification or references for the concepts. This work has found no justification for a nonlinear operator in real visual systems describing a symmetrical dead space related to the amplitude of the sensory signals. The stage 3 circuits do include a unidirectional threshold level. However, this level is very low and corresponds to the lowest level of scotopic performance. It is generally not relevant in discussions of accommodation. He incorporates a nonlinear gain element (gain is proportional to the average signal input level) that he associates with the sensory part of the system His words suggest that this operator is actually intrinsic to the input signal. In real visual systems, the adaptation process exhibits a characteristic that is the exact inverse of the suggested operator. As a result, the product of these operators is a null condition throughout the photopic range of vision. The product does become significant in the mesotopic range, but the net result is the opposite of that described by Jiang. Jiang proceeds to develop closed loop equations that ignore the “plant” shown within his servoloop and any time delay associated with signal transmission within the loop. The relevance of the resulting equations are at least questionable.

Recently, a spirited (might I say vituperative) discussion has appeared by Glasser456. He takes considerable exception to the theory presented by Weale457 (without either having the benefit of a physical model of the subject). His discussion highlights the ongoing approach of the vision community to rely upon experiment and intuition and avoiding physical models based on engineering principles.

A parallel school is engaged in analysis of the lens servomechanism from a more physical and materials-based approach458. They have even begun using elementary ray tracing to confirm that their analyses are compatible with the actual operation of the eye.

The fact that very few references are shared between the papers of the groups defined above is interesting. They generally do not reference each other. This shows a lack of thoroughness in aggregating ideas to create a more comprehensive and consistent model. As a positive example, a paradox appeared recently in the Koretz school459. A paradox is similar in concept to a magic trick. It is not understood by one party because of an inadequate model. This paradox was rapidly resolved by more attention to the detail boundary conditions of the situation460. For a good example of aggregating ideas coherently, see Coleman461.

Before proceeding, reviewing the art in the papers of Coleman referenced above and of Koretz462 is useful. They develop

451Schor, C. (1979) The relationship between fusional vergence eye movements and fixation disparity Vision Res vol. 19, pp 1359-1367 452Schor, C. (1991) Effects on the resting states of accommodation and convergence in Grosvenor T. & Flom, M. Ed. Refractive Anomalies. Boston, MA: Butterworth-Heinemann, Chapter 18 453Wallman, J. (1990) Introduction in (Ciba Symposium 155) Myopia and the Control of Eye Growth. NY: John Wiley & Sons 454Wallman, J. (1991) Retinal factors in myopia and emmetropization: clues from research on chicks in Grosvenor, T. & Flom, M. Refractive Anomalies. Boston, MA: Butterworth–Heinemann Chapter 15 455Jiang, B-C. (1997) Integration of a sensory component into the accommodation model reveals differences between emmetropia and late-onset myopia. Invest Ophthal Vis Sci vol. 38, no. 8, pp 1511-1516 456Glasser, A. (2001) On modeling the causes of presbyopia. Vision Res. vol. 41, pp 3083-3087 (Note the last paragraph probably required by the reviewers). 457Weale, R. (2000) Why we need reading-glasses before a zimmer-frame. Vision Res. vol. 40, pp 2233-2240 458Koretz, J. Cook, C. & Kaufman, P. (2002) Aging of the human lens: changes in lens shape upon accommodation and with accommodative loss. J. Opt. Soc. Am. A, vol. 19, no. 1, pp 144-151 459Koretz, J. & Handelman, G. (1986) The lens paradox and image formation in accommodating human eyes. Topics in Aging Research in Europe, vol. 6, pp 57-64 A similar discussion in Scientific American, vol. 259, pp 92-97 460Koretz, J. Cook, C. & Kaufman, P. (2002) Op. Cit. pg 150. 461Coleman, D. (1970) Unified model for accommodative mechanism. Am. J. Opthalmol. vol. 69, no. 6, pp 1063-1079 462Koretz, J. & Handelman, G. (1986) How the human eye focuses. Sci. Am. vol. 259, July pp 92-97 Dynamics of Vision 7- 247 many subtle features of the physical plant associated with the lens servomechanism. A more recent Coleman paper is also useful463. Additional work is required using ray-tracing (or by introducing more optical design experience into the field). Nearly all of the recent papers are struggling to show the surfaces of the physiological optics are either spherical or paraboloid. Occasionally, it is claimed they are hyperboloid. It is proposed that a full ray-trace, even with the precision of the available numbers would show that the cornea is of necessity ellipsoidal in shape. Otherwise, the wide angle coverage of the eye could not be obtained. Some authors have justified their choice based on a 2nd order conic section. All of the above forms can be described as 2nd order conic sections. Fisher’s very careful analyses of the materials of the lenses and humors used the term ellipsoid to describe the anterior surface of the cornea (page 36)464. Dubbelman & Van der Heijde provide information on the great latitude in many parameters of physiological optics465. The scatter diagrams are very broad. They raise the question whether a best fit first order line through the data is meaningful. Maybe additional cross correlation diagrams are needed to discover underlying relationships between pairs of data points. This team could also use the help of an experienced optical design engineer. The requirement on the accommodation servomechanism is to achieve a focus at a given range within the depth of focus of the optical system itself. This value is about ±1/8 diopter for a three-mm aperture466. For a one mm pupil diameter (typical of photopic conditions), the servomechanism needs only meet a ± 1/4 to ± 3/8 diopter requirement. Little data appears in the literature on the performance of the lens control system (see Section 7.4.9.1.3). Fisher & Ciuffreda have provided coarse data on the static response of the lens system to changes in the accommodative stimulus467. They note that the role of the lens control system in distance perception remains controversial. No significant data on the role of the lens control system in depth perception was found in the literature. 7.4.9.1.2 The overall servomechanism of accommodation

The overall servomechanism of accommodation proposed in this work is shown in Figure 7.4.9-2. The diagram begins at the lower right with the creation of the physiological optical system. The brackets behind the integral sign indicate the processes involved in the growth of the eye are essentially unidirectional. As the cornea and lens begin to create optical refraction with respect to light, the ocular globe begins to expand and move the retina away from the optics. Post- partum, these processes proceed at independent rates up through the teen years. The scaled difference between these rates can result in an ammetropic condition under anatomical conditions (the ciliary muscle completely relaxed and having not affect of refraction). This gross refraction error (relative to an image at infinity) can be affected slightly by the tonus of the ciliary muscle. The result is the net anatomical refraction condition. Optimally, this condition images a source at infinity perfectly on the retina. To achieve ideal focus at closer distances, the lens must provide a net change in refractive power that is called accommodation. This change is achieved by the physical distortion of the lens by the ciliary muscle under in-vivo conditions. The resulting accommodated refraction is designed to match perfectly the change in refraction needed to image a source at an arbitrary finite distance from the eye. If it fails to do this, an accommodation error is associated with the image projected on the retina at the plane representing the entrance to the photoreceptor cells.

463Coleman, D. (1986) On the hydraulic suspension theory of accommodation. Trans. Am. Ophthalmol. Soc. vol. 84, pp 846-868 464Fisher, R. (1969) The significance of the shape of the lens and capsular energy changes in accommodation. J. Physiol. vol. 201, pp 1-19 & 21-47 465Dubbelman, M. & Van der Heijde, G. (2001) The shape of the aging human lens: curvature, equivalent refractive indes and the lens paradox. Vision Res. vol. 41, pp 1867-1877 466Smith, W. (2000) Modern Optical Engineering. NY: McGraw-Hill, pg 138 467Fisher, S. & Ciuffreda, K. (1988) Accommodation and apparent distance Perception vol. 17, pp 609-621 248 Processes in Animal Vision

Figure 7.4.9-2 Block diagram of the accommodation servomechanism. Note the plurality of plus signs in the figure. The servomechanism is unidirectional. If the gross anatomical refraction error is significantly positive, the system can not compensate for it. As a result, the subject is hypometropic and is said to suffer from myopia. The stage 1, 2 & 3 circuits are not shown explicitly in this figure but they are important to the dynamic operation of the subsystem. See text.

The error in accommodation sensed at the retina is processed within the two-dimensional correlator of the perigeniculate nucleus which is charged with ascertaining the quality of the edge response within each signal channel due to the tremor of the oculomotor pointing subsystem. An auxiliary calculation is also performed in the lateral geniculate nuclei to provide a coarse accommodation signal derived from the vergence servomechanism. These two signals are shown combined into a net analog accommodation correction signal although these signals are transmitted independently to the signal conditioner within the Edinger-Westphal complex associated with the superior colliculus. This signal processing engine appears to accept a variety of other signals in order to process the appropriate instruction to control the ciliary muscle. These include an accommodation signal obtained from the saliency map suggesting the correct accommodation signal value based on prior examination of the stimulus. It may also accept a bias signal used to establish the quiescent accommodation level (sometime called the dark focus condition) in the absence of any other signal. Changes in the Dynamics of Vision 7- 249 quiescent focal accommodation are known to occur with activity level468. This change may be due to a calculation based on time-integration within the signal processing engine using unknown sensory inputs, or it may be due to the engine continuously referring to the accommodation level updated frequently and stored in the saliency map. The signal processing engine may also accept a copy of the perturbation signal used to cause tremor in the oculomotor subsystem. This would be used to cancel any artifacts of tremor in the net signal sent to the ciliary muscle. The Edinger- Westphal complex processes these signals in the analog domain and then converts the information into action potentials for transmission to the ciliary muscle. The so-called ciliary ganglion, found along the signal path between the brain and the ciliary muscle is used to introduce emergency control signals associated with the Alarm Mode of the visual system. These emergency signals are normally insignificant. The low-pass electrical characteristic of the ciliary muscle integrates the action potential pulse stream from the ciliary ganglion and generates a physical response proportional to the original analog signal from the Edinger- Westphal complex. This physical response distorts the lens, thereby introducing the degree of accommodation required and closing the accommodation servomechanism loop. It is important to note that the accommodation servomechanism is essentially a unidirectional servomechanism. Only a single ciliary muscle is present in each eye. This muscle works with the elasticity of the lens and its support structure as an antagonist. This unitary character is suggested by the number of plus signs shown at the summing junctions within the diagram. A disease (myopia, or hypometropia) arises when the net anatomical refraction (resulting from the growth process) is significantly positive. The unidirectional accommodation servomechanism is unable to compensate for such a condition.

The stage 1, 2 & 3 circuits are not shown explicitly in the figure. They are described more completely in Sections 7.3, 7.4.2 & 7.4.3 and in the chapters related to their specific roles. The performance of these circuits are important to the short term dynamic performance of the accommodation servomechanism. Little research has been reported concerning the short term performance of the accommodation servomechanism.

The sophisticated lens support system is discussed in Section 7.4.9.1.3. Other aspects of the servomechanism will be also addressed in the following sections. 7.4.9.1.3 The performance of the overall accommodation servomechanism

Figure 7.4.9-3 presents a nomograph describing limits imposed by the nature of the above schematic. The nomograph starts at lower left giving the distance from the subject of a stimulus. By projecting horizontally and then vertically to the horizontal axis, from the characteristic representative of the subject, the required accommodation relative to the collimated condition is found. The two bounding curves represent the anatomical refractive error of the subject. A log- normal distribution representing the degree of ametropia found in the general population is shown at the lower right. This distribution shows a slightly sharper edge on the hypermetrope side and properly describes the slightly higher frequency of 4D hypometropes compared to 4D hypermetropes. It suggests plus and minus four diopters are reasonable values to describe the limits of ametropia found in the general population other papers have used six diopters as a criterion. Notice that a four diopter hypermetrope (myope) requires a negative degree of accommodation for stimuli at distances greater than 25 cm. By projecting the value on the horizontal axis up until it reaches the fold line and then horizontally to the vertical axis, the available correction is determined for the stimulus originally postulated. As noted above the accommodation mechanism cannot provide negative accommodation. This area of operation has been shaded.

Normally, it is found that the human visual system operates with a small amount of accommodation associated with the tonus of the ciliary muscle. This tonus may be supported by a neural signal of a nominal value as discussed in Section xxx. The quiescent focal condition due to this level of tonus is indicated by one diopter on the upper left scale. For the emmetrope, this level represents a normal quiescent accommodation to a stimulus distance of 100 cm, whether there is a stimulus present at that distance or not. For the four diopter myope, the quiescent accommodation distance is nominally 20 cm.

The limits of the nominal human eye as a function of aging is shown by the figures along the fold line469. Young eyes normally provide accommodation up to at least 14 diopters. As the subject ages, the maximum accommodation decreases, a condition known as presbyopia. By 50 years, the nominal emmetrope can only accommodate over a range

468Owens, R. & Wolf-Kelly, K. (1987) Near work, visual fatigue, and variation of oculomotor tonus Invest Ophthalmol Vis Sci vol. 28, pp 743-749 469Glasser, A. & Campbell, M. (1999) On the potential causes of presbyopia. Vision Res. vol. 39, pp 1267-1272 250 Processes in Animal Vision of zero to two diopters. The subject can no longer read newspaper size text held within arms length but he can still accommodate to long distances. By the time he is 60 years of age, the lower part of the fold line will also be lost. He will need spectacles to accommodate to long distances as well. As another example, a two diopter hypermetrope will begin requiring spectacles for reading at about 30 years and also require spectacles to aid accommodation at long distances at age 48. This nomograph, along with the data of later sections, clearly supports the definition of “young eyes”. Young eyes for physiological research purposes should be restricted to fully matured emmetropes not over 25 years of age. If necessary, hypermetropes within the same age boundaries can be included. Hypometropes will introduce asymmetrical results into many test protocols. It is interesting to note the ability of a hypermetrope to properly image scenes presented to his eye that involve converging light rays (a situation not found in nature). Within their degree of hypermetropia, young eyes can actually bring such a scene (such as generated by a set of misadjusted binoculars) into correct focus. Dynamics of Vision 7- 251

Figure 7.4.9-3 A nomograph describing the performance of the accommodation subsystem. 252 Processes in Animal Vision

The above nomograph can be extended to also represent the situation when the scene contrast is low. For the ideal emmetrope without any age limitation, the curve is precisely as shown by Owens (pg 324 in Grosvenor & Flom) and reproduced here as Figure 7.4.9-4. For other eyes, the quiescent focal condition must be determined and a similar set of lines drawn. As Owens has noted (pages 326-340), previous clinical investigations have been contradictory concerning the quiescent focal condition (also called the dark focus condition). 7.4.9.1.3.1 Static performance errors in accommodation (clinical)

[xxx this section will move to Section 18.2.4 ] Abbott, et. al. have shown that the first order static performance of the closed loop accommodation subsystem is quite good, particularly at high degrees of accommodation470. Errors with a mean of less than 0.1 D +/– 0.1 D were reported for absolute accommodation demand greater than 1.0 D. The errors were considerably higher in the range of 0.0 to 1.0 accommodation demand. This is the region where the spring force associated with the lens mounting system is minimal and any neural bias is small.

Radhakrishnan, et. al. recently presented data on the second order asymmetry of the accommodation servomechanism based on precise acuity measurements471 and precise contrast measurements 472. Their data applies to the second order anomalies associated with subjects fully corrected for spherical refractive error. The title of their paper does not indicate the fact their subjects were corrected. The cause of these Figure 7.4.9-4 Accommodation performance of the “young” anomalies were not discussed in detail but they emmetrope eye as a function of stimulus intensity. Modified suggested they may be related to the data and from Owens, 1991. The emmetrope cannot correct for mechanisms discussed the papers of He, et. al. stimulus distances beyond infinity (converging light referenced below. bundles). A small negative change in accommodation may be available by allowing the bipolar output signal of the PGN to subtract from a small static bias applied continually to the ciliary muscle as part of a quiescent accommodation signal. This capability can be inferred from some of the literature but it is difficult to confirm. Radhakrishinran, et. al. reported a residual accommodation after cycloplegia was about 0.2D in both hypometropes (myopes) and hypermetropes. Their abstract did not define the reference point for their differential measurements. The literature shows that averaging groups to obtain a quiescent focal accommodation condition is fraught with difficulty. The mean values obtained are associated with a very wide standard deviation. Other literature suggests the quiescent focal accommodation is time and activity dependent.

Gwiazda, et. al. have provided unexpected support for the presence of a two dimensional correlator within the perigeniculate nucleus and the effect of reduced low frequency response of the overall visual system at low spatial frequencies473. They note the unexpected improvement in the operation of the accommodation subsystem when smaller letters are used as a test source. They also surface once again a terminology problem. First, the visual system does not operate based on a phenomenon described in the literature as blur. The system employs edge detection of scene elements aided by the application of tremor to the oculomotor system. The edges are detected using standard electronic techniques that include the measurement of the slope of the edge expressed in volts. The slope in measured indirectly by subtracting

470Abbot, M. Schmid, K. & Strang, N. (1998) Op. Cit. pg 18 471Radhakrishnan, H. Pardhan, S. et. al. (2004) Unequal reduction in visual acuity with positive and negative defocusing lenses in myopes Optom Vis Sci Vol. 81(1) pp 14-17 472Radhakrishnan, H. Pardhan, S. et. al. (2003)Effect of positive and negative defocus on contrast sensitivity in myopes and non-myopes. Ophthal Physiol Optics Vol. xxx pp xxx 473Gwiazda, J. Thorn, F. Bauer, J. & Held, R. (1993) Myopic children show insufficient accommodative response to blur Invest Ophthal Vis Sci vol. 34, no. 3, pp 690-694 Dynamics of Vision 7- 253 the instantaneous value of the correlator output at two points in time separated by one half the period of the tremor signal. The magnitude of this measurement (the strength of the stimulus in the prior figure) is an estimate of the product of the intrinsic sharpness of the image, and the quality of the focus and modulation transfer function of the physiological optical system. Second, the signal to noise performance of the accommodation signal generated as above can be considerably improved if there are more edges in the stimulus scene and a correlation process is used to optimize the resulting signal. The quality of the achieved accommodation is proportional to the number (and length) of sharp edges ( both vertical and horizontal) occurring within the 1.6 degree diameter foveola and supported by the perigeniculate 2-dimensional correlator See Section 15.6.3). The data of Gwiazda, et. al. and of Jiang both show that the accommodation subsystem operates as a high quality type 2 (rate controlled) first order servomechanism. High quality in this case refers to adequate loop gain to insure effective closure of the servoloop under most conditions. To improve the quality of operation at low absolute dioptric values (imaging sources near infinity) would require implementation of an absolute error signal relative to the stimulus intensity. Such a capability is not available in the visual system. He, et. al. have presented data on the errors in accommodation related to aberrations in the physiological optical system474. These are generally small second-order errors when compared to the developmentally related errors. However, they do highlight the movement of the lens along the optical axis and perpendicular to the axis during accommodation. The latter errors introduce asymmetries that may account for the second order errors found to be more dominant in myopes. 7.4.9.1.3.2 Dynamic performance errors in accommodation (clinical)

[xxx this section will move to Section 18.2.4 ] Only a few papers were found discussing the dynamic performance of the accommodation subsystem. The paper by O’Leary & Allen only lists six references dating from 1982475. Their instrumentation was minimal and they reported settling times following changes of 2.0 D using a flipper bar containing only one –2.0 D lens, or changes of 4.0 D using a flipper bar containing +2.0 and – 2.0 lenses, on a continuing basis. The resulting cycle times peaked in the 12-18 cycle per minute range for the 2.0 D changes at long range. At the shorter ranges, the cycle times showed a broader peak in the 6-18 cycle per minute range. These values appear to be compatible with the more detailed physiological experiments discussed below. 7.4.9.1.4 The physical plant of the lens control servomechanism

The physical plant portion of the servomechanism of lens control has been studied in detail by L. Stark476. The situation is more complex than suggested by the block diagram of [Figure 18.2.4-3] because of the unique physical arrangement of the ciliary muscle and associated structure. This arrangement was explored in detail by Helmholtz in the 19th Century. His model has an awkward name because of its precision. It has become known as the “dual indirect, Active Theory of Accommodation.” It is dual because both the lens and its capsule appear to play active roles. It is indirect because the ciliary muscle does not act directly on the lens/capsule. It operates through a complex trusswork (trabecular meshwork within the community) designed to insure equal forces around the perimeter of the lens/capsule. Finally, accommodation is achieved through an active mechanism. Positive accommodation requires contraction of the ciliary muscle. This configuration is interesting in that it can avoid astigmatism due to lens distortion if desired. Alternately, it could be used to introduce astigmatism. The ciliary muscle is a unified muscle as opposed to a sphincter muscle. It is served by multiple nerve endings. If programmed separately, various portions of the muscle could be used to introduce a desired astigmatism. Conversely, if the nerves are not all functioning normally, an astigmatism can be introduced that is not related to distortion in the geometry of the retina or sclera. Stark has provided a static biomechanical model of the lens control plant based on the above truss configuration first explored by Helmholtz. Wyatt has provided some valuable data concerning the movements of the lens capsule477. However, he was not satisfied with his model of the accommodation mechanism plant, within the region labeled O, P & S. His discussion of the accommodation plant elements in this area of the eye will not be reported here. Figure 7.4.9-

474He, J. Burns, S. & Marcos, S. (2000) Monochromatic aberrations in the accommodated human eye Vision Res vol. 40, pp 41-48 475O’Leary, D. & Allen, P. (2001) Facility of accommodation in myopia Ophthal Physiol Opt vo. 21 no. 5 pp 352-355 476Stark, L. (1988) Op. Cit. 477Wyatt, H. (1988) Some aspects of the mechanics of accommodation Vision Res vol. 28, no. 1, pp 75-86 254 Processes in Animal Vision 5 presents a more explicit bio-mechanical model based on Beers & Heidje478. The outer shell of the ocular is shown using solid lines. The interior details related to the retina, lens and iris are shown using thin dashed lines. The vitreous humor in contact with the retina and the anterior humor between the cornea and the lens are not shown. If present, a pressure difference between these two media could be significant in the dynamics of the accommodation system. Most of the literature discounts this difference because of the apparent porosity of the structure surrounding the lens capsule. The lens is suspended from a floating link between the accommodation muscle, the ligament connecting to the capsule of the lens and a ligament connecting to point O. The paucity of data on the accommodation mechanism is illustrated by the age of the references in Stark. Wyatt provides the broadest collection of data relative to accommodation, but does not include the data of Stark published the same year. Stark, Wyatt and Beers & Heidje use significantly different models to analyze this structure. Stark assumes the ocular is a rigid structure while the others do not. Wyatt takes two internal structures, the pars plana and the pars plicata of the ciliary body to be functionally rigid while the others do not. Beers & Heidje treat the lens as a spring and damping element while the others treat it as only a spring. Stark does not address the axial motion of the lens while the others do (to various degrees). While both Wyatt and Beers & Heidje use concepts drawn from the discipline of statics, neither addresses all dimensions of the appropriated free-body model. They both focus on the radial motions of the point P while recognizing axial motions are also present. Future researchers must study the points made by each of these authors carefully before developing more appropriate unified models. Most investigators have discounted the angular geometry associated with this mechanical mechanism. This is unfortunate since the lens will clearly move axially as the ciliary muscle contracts unless it is constrained. Figure 2 in Wyatt expressly describes the axial motion. Both the radial and axial motions are transcendental because of the geometry. Fortunately, the relevant angle does not change significantly in human vision.

The above authors have not addressed the optical equations appropriate to this situation. The fundamental lens equation is shown at upper right in the figure to illustrate a concept. This is also known as the thin lens equation. The distances, f1 and f2 are measured form the centerline of a thin lens. Neither the cornea or the lens are thin lenses. For thick lenses, separate principle planes are defined at the entrance aperture and the exit aperture of the lens. The distances must be measured from the appropriate principle planes of the lens. The principle planes are shown generically for both the cornea and lens. These planes are addressed more thoroughly in Section 2.4.1.2.1. When two separate lenses form a lens group, a different equation applies. Such a group exhibits its own set of principle planes, P1 and P2, as shown in the figure. The equally important equation for a lens group consisting of two thin lenses is shown below the fundamental equation. This equation shows that the Effective Focal Length, EFL, of the group varies with the spacing between the adjacent principle planes, d. In most man-made optical systems, the focus is adjusted exclusively by varying the distance d. Only in physiology is the power of the second lens also varied. The result is a more compact design. However, as the equation shows, both the distance, d, and the focal length of the second lens, Fl, are important in the overall equation. The angles involved in the geometry of the accommodation plant appear to be chosen to recognize the relative effects of the change in position and the change in optical power of the lens in response to contraction of the accommodation muscle. Wyatt gives values for these angles as appropriate to his free-body-diagram.

Early man-made variable focal length optical systems (“zoom” systems) did not include automatic focus mechanisms. The correct focus was achieved by calibrating the lenses and providing a cam-operated mechanism to control the separation of the lenses and their position relative to the focal plane. The physiological system provides a focus sensor that controls the accommodation muscle in a unidirectional servomechanism (as opposed to the push-pull or bidirectional systems used to rotate the eyes). The relative motion of the second lens relative to the focal plane and the power of that lens are adjusted simultaneously by the linkage shown by the heavy dashed lines. The Beers & Heidje model illustrated here is based on the same interpretation of the physiological structure as Helmholtz. The details were provided by those investigators. “The component elements are: the ciliary muscle, the axial and peripheral zonules, the choroid and the lens. . . . The ciliary muscle is a unified muscle with a single physiological action. . . . It has a single origin, the scleral spur, and a single insertion namely the ciliary body epithelium and superficial stroma between the ciliary processes, where the span fibrils of the zonules are attached. The elastic antagonists of the muscle are the choroid and the peripheral zonules.” They also stated the action clearly. “During near- to-far (NF) accommodation, when the ciliary muscle relaxes, the elastic force of the antagonists pulls at the axial zonules and lens capsule, thus flattening the lens. During far-to-near (FN) accommodation ciliary muscle contraction takes up the elastic tension in the antagonists, thus relaxing the axial zonules and allowing the lens capsule to mold the lens into a more convex shape.”

478Beers, A. & van der Heidje, G. (1994) In-vivo Determination of the biomechanical properties of the component elements of the accommodation mechanism Vision Res vol. 34, no. 21, pp 2897-2905 Dynamics of Vision 7- 255

Figure 7.4.9-5 Static biomechanical model of the lens servomechanism and relevant equations. Note the foveola is not on the optical axis and the line of fixation is not parallel to the optical axis. The plant serving the accommodation function is shown between the anchor point, points O, P and S, and the capsule containing the lens. See text. Muscle and ligament arrangement based on Beers & Heidje, 1994. 256 Processes in Animal Vision Stark presents the predicted performance of two conceptual theories that are both based on the Helmholtz model. The Hess-Gullstrand lenticular theory assumes the muscles of the eye remain capable throughout most of life but the lens becomes less compliant. The Duane-Fincham extra-lenticular theory is based on the results of pharmacology intervention. It assumes the capability of the ciliary muscle degrades with age. Stark recognizes the correctness of the results claimed by Duane-Fincham but shows they are not representative of the actual in-vivo operation of the plant. The results predicted by the Hess-Gullstrand approach provide better answers. Accommodation is degraded by changes in the lens/capsule. The change is probably due to the continued increase in volumetric size of the lens although a change in viscosity cannot be ruled out. The debate concerning whether presbyopia is due to lens hardening or muscle aging has been solved. Successful surgical intervention now provides a new, more compliant, lens and relies upon the muscle to operate normally regardless of the age of the subject. The complexity of the accommodation servomechanism is high and careful choice of words is needed in any discussion. In some cases, considerable discussion is needed to explain simple assertions. As an example, Stark makes the statement that “The Hess-Gullstrand theory is a nonlinear theory.” This may be a poor choice of words concerning his interpretation of his figure 6. He is discussing presbyopia, the loss of accommodation with aging. With age, several of the parameters associated with accommodation may change. Over the available operating range, the system operates linearly. Only over an extended operating range does the response show nonlinearity. Technically, the model he presents and the performance the Hess-Gullstrand theory projects are both linear for small signals from a control theory perspective. Wyatt also notes the linear character of his model over its operating range. The values shown for the total force in the different structural elements of the Stark and Wyatt models are based on the literature of the time. They appear large to the uninitiated. Fisher uses values of one to two grams versus Stark’s 30-40 grams. Stark discusses several “ground points” associated with this model. They help explain the character of the performance presented below.

When discussing the loss in performance due to presbyopia, Stark chose to interpret the Hess-Gullstrand Theory using a “piece-wise linear” approach. He describes the major active portion of the response as the manifest zone and the area of falloff, as a function of aging, as the latent zone. Here again, using latent appears to be a poor choice of words. He is speaking of a loss in performance unrelated to any time delay. The true latency in this process will be developed in the section on transient performance. He does not address the transition region. Unfortunately, Beers & Van der Heijde also employ a piecewise linear approach and assume the source impedance of the ciliary muscle is negligible. The criticism concerning using a piecewise linearization must also be extended to the paper of Fisher, particularly regarding his figure 3479.

Based on the assumptions they used, Beers & Van der Heijde have analyzed their data in terms of a simple exponential curve. However, the data in their figure 3 clearly do not follow an exponential curve. Their time constant in figure 5 are actually averages of a continually changing underlying time constants. This is shown in Figure 7.4.9-6.

Coleman has critically reviewed the above Helmholtz model concerning several shortcomings. By adding in a hydraulic component, he proposes to eliminate some of these shortcomings. Koretz has introduced another interesting factor480. She supports the view of Farnsworth that the zonules connecting to the lens change their relative position with age. As a result, the zonules lose much of their mechanical advantage relative to the shape of the lens.

A major problem in discussing the lens servomechanism is the great amount of scatter in the data when multiple subjects are involved. Brown has provided an excellent example of this481. 7.4.9.1.5 The physical parameters of the materials of the servomechanism

Koretz & Handelman have provided some excellent electron micrographs of the mounting of the lens to the zonule. They also provided excellent caricatures of the physical plant, in both perspective and cross section482. Fisher has provided extensive measurements and analyses concerning the physical characteristics of the lens. However, the experimental techniques are now dated and he made several linearizing assumptions. His energy storage calculations also lead to a solution of the paradox developed in the above papers. His papers provide a series of trend lines versus age. They also provide the characteristics of several materials found in the eyes. Glasser and Campbell have recently made a range of similar measurement, using current (1999) computer-augmented

479Fisher, R. (1969) Op. Cit. 480Koretz J. & Handelman, G. (1986) Op. Cit. 481Brown, N. (1974) The change in lens curvature with age. Exp. Eye. Res. vol. 19, pp 175-183 482Koretz, J. & Handelman, G. (1988) How the human eye focuses. Scientific Am. July , pp 92-99 Dynamics of Vision 7- 257 equipment483. They included an extensive source list. Although their sample size was small (19 pairs of eyes) they measured many parameters as a function of age.

7.4.9.1.6 The neural circuitry of the lens servomechanism

The specific nature of the neural circuits supporting accommodation has not been reported in detail. It appears the signal is extracted from the data presented to the PGN/pulvinar couple by the foveola of the eyes. The operation of the system clearly varies with the contrast in the fine detail of the image484. This would suggest the performance is controlled by the sharpness of the edges in the scene presented to the retina, as converted into an electrical signal by the action of tremor. A key question is whether the derived signal is an average of the values for several points in the scene or not. If it is, this implies the signal is extracted by the two-dimensional correlator of the PGN. Simple experiments could determine the relationship between quality of accommodation and the spatial extent of the fine detail within a scene. Many suggestions appear in the literature that the lens control system involves a memory element in its computational unit. As a result, measurements of the time delay associated with a change in distance to the scene are highly dependent on whether the subject anticipating the change to a previously fixated point or whether the change is completely random. This is consistent with the casual experiments of this author. The characteristics of this memory element will be discussed further in the next section. It appears that the system can call on recent data in the short term saliency map (see Section 15.2.1). With this data, a volition command can be generated containing two, or three components. It can return the line of fixation to a given scene, and adjust both the required vergence and required accommodation state during the saccade.

The analog operation of the ciliary muscle is a strong indication that the neural signals controlling it consist of a quasi- continuous stream of action potentials. It can be expected that these signals will have a frequency that increases in proportion to how much accommodation is required. Whether this relationship is a simple linear one or whether it is more complex is unknown. It may include nonlinear compensation to account for the geometry of the physical plant and possibly the nonlinearity of the stress/strain relationships associated with the ciliary muscle and the lens. These signals originate in a nucleus near those that prepare the drive signals for the oculomotor system. The signals pass through a nucleus known as the ciliary ganglion near the eyeball before entering it.

It should be noted that the lens accommodation is not a symmetrical closed loop servomechanism. Instead, it operates from a position of basal accommodation in the absence of a signal to the contrary. Such a system is sensitive to errors in calibration. These errors can be compared to a windage error in pointing a rifle. Since the lens accommodation system involves analog signal computation and signal preparation, it is subject to offset errors similar to those found in color blindness. There are clinical symptoms that suggest that such errors are encountered in practice. A numerically small group of subjects, primarily myopes, have consistently reported seeing scenes in better focus immediately after opening their eyes than one half to one second later. These “flashes of clear vision” are repeatable. They say they attempt to focus on infinity before opening their eyes. For a myope, this is equivalent to going to their basal accommodation level. However, there is a problem. This experiment is suggestive of a significant computational or transmission error involving a DC offset in the analog signal before it is converted to a stream of action potentials. Defining both the refractive and the neural components of the basal accommodation level is necessary. The basal refractive accommodation level is that assumed by the physiological optics of the eye. It is the level reached with the eyes closed and relaxed. Upon opening the eyes, the neural system evaluates the distance to the scene and provides an incorrect signal that is the basal neurological accommodation error. The total basal accommodation level is now in error by an amount that is time dependent. It remains at the basal refractive error level for approximately 200 ms and then transitions to the total error level in another 100-200 ms depending on the light and contrast level of the stimulus.

7.4.9.1.7 Steady state operation of the lens servomechanism

Cieffreda & Kenyon have provided a graph of the accommodation response to the introduction of an artificial accommodation error (their figure 5.1)485. The figure is similar at the detail level to the graph presented in the next paragraph.

483Glasser, A. & Campbell, M. (1999) Biometric, optical and physical changes in the isolated human crystalline lens with age in relation to presbyopia. . vol. 39, pp 1991-2015 484Adams, A. Wong, L. Wong, L.& Gould, B. (1988) Visual acuity changes with age: some new perspectives. Am. J. Optom. Physiol. Optics vol. 65, no 5, pp 403-406 485Ciuffreda, K. & Kenyon, R. (1983) Accommodative vergence and accommodation in normals, amblyopes, and strabismics In Schor, C. & Ciuffreda, K. eds. Vergence Eye Movements: Basic and Clinical Aspects. Boston, Butterworhs, pp 101-173 258 Processes in Animal Vision Blaker has provided data on the change in radius of the anterior surface of the lens of two 20-year-old males in response to a forced accommodation plant486. However, he does not record the change in the radius of the posterior surface. By assuming the posterior surface remains unchanged, the performance of these two eyes can be interpreted as shown in Figure 7.4.9-6. The figure shows the achieved accommodation level in response to a commanded accommodation. The curve tilts up at the left. That curvature is an indication of a residual accommodation level in the absence of any command. This is called the basal level of metropia in this work. Using Blaker’s data, this residual was about 1 D in the two hypermetropic individuals. The one individual, #1, could achieve a linear change in response to a command out to about 10-10.5 D before his performance began to degrade. If this individual were typical based on Blaker’s model, this would occur at a near point of 11.76 cm (4.6 inches). The second individual, #2, did not perform as well. His system exhibited a shoulder near 7 D. The break points in the maximum accommodation of individuals move to the left on this graph with age. Jennings and Charman indicate one 32-year-old female, #3, had an accommodation range of 6.5 D. No details were given as to the criterion for this measurement. This figure begins to show the progression in accommodation performance with biological (not necessarily chronological) age. There is no way to calibrate the absolute efficiency on this type of curve. Stark does observe that the firing rates among the ciliary neurons appear to continue to fire at a rate required to achieve a specific accommodation level even after the individual is no longer able to achieve this level. This would suggest the signal extraction circuits are still working properly but the plant associated with the lens is unable to respond. This continual high signaling rate may account for some of the pain associated with eye strain.

Figure 7.4.9-6 Accommodation efficiency based on data of Fincham in Blaker, 1980.

Saladin & Glasser have provided a few more data points that can be plotted on the above figure487.

486Blaker, J. (1980) Toward an adaptive model of the human eye J. Opt Soc Am vol. 70, no. 2, pp 220-223 487Saladin, J. & Glasser, L. (1975) Presbyopia: New evidence from impedance cyclography supporting the Hess- Gullstrand Theory. Vision Res. vol. 15, pp 537-541 Dynamics of Vision 7- 259

Glasser has recently provided more data that helps fill in the voids in the static database488. Figure 7.4.9-7 reproduces his figure 2. The solid line and data points appear compatible with the above figure. However, they do no provide a clear way to establish a precise comparison based on the protocols used. A similar compilation of data has been provided by Weale489. The reduction in the accommodation range falling to about 1 D at 60 years is consistent with this authors experience.

7.4.9.1.8 Transient operation of the lens servomechanism

A variety of studies have been made of the transient operation of the lens servomechanism at a gross level. The instrumentation is demanding. Phillips, et. al. provided considerable early data and compared it to many other studies490. Unfortunately the data is reduced to histograms rather than to the parameters of transient response curves. They appear to use the term latency to describe the time to peak response following a change in the required accommodation level. This approach lumps many discrete time intervals (including the true latency) before the physical plant begins to respond. Often, they have averaged the time to peak response for different changes in accommodation. It appears they have not Figure 7.4.9-7 Accommodation range of the human eye examined the variation in response time require to versus age. Glasser used the term amplitude instead of respond to a given size accommodation change as a range. From Glasser, 2001. function of the position along the accommodation function. Care is suggested in analyzing the data provided and referenced.

Smith has provided a curve showing the time to accommodate to a 1.3 diopter change versus age491. The value varies from 0.6 seconds at 20 years linearly to 1.6 seconds at 60 years. A change of this nature would suggest, but not specifiy, a change in the time constant of the lens plant rather than a change in the strength of the ciliary muscle.

The discussion in the paper by Phillips, et. al. does not suggest a familiarity with the laws of optics. The shape of the cornea is certainly not spherical. It is most likely not parabolic. Achieving the desired wide field of view requires the optics to be ellipsoidal. However, the geometrical differences between the above three shapes are measured in very small fractions of a millimeter.

Phillips, et. al. do observe a significant difference in time to achieve a change in accommodation in response to a random change in required accommodation than in response to an expected change. This clearly suggests an ability of the subject to call on information in his short term saliency map in order to perform on repetitive tasks. It is also likely this capability is widely used, especially in reading (See above and Chapter 18 on reading). The gross data in Table I of Phillips, et. al. report times to peak response. The means are on the order of 370 ms for non-repetitive tasks and 200 ms for repetitive tasks (three or four subjects per test). Little difference is shown between tasks involving near-to-far and far-to near accommodation. While this is quite possible with the physical plant shown above, it would be unexpected. His remarks may apply to small signal conditions (small changes in accommodation) where it would be expected. It is clear from the simplest behavioral experiments that the performance of the accommodation mechanism is closely tied to the performance of the oculomotor system. For random tasks, accommodation cannot be performed with precision until the line of fixation has settled on the object of interest. Beers & Van der Heijde found it took about 200 msec before the lens accommodation plant began to respond. It then took from 100 to 500 msec for the plant to respond (depending on the change required). This would suggest the total response consists of several time intervals. First, the subject must perceive the need to reestablish the line of sight, the vergence and possibly the accommodation (or at least some of these depending on the test configuration). Fortunately, the computational engine required to make these

488Glasser, A. (2001) Op. Cit. 489Weale, R. (2000) why we need reading-glasses before a zimmer-frame. Vision Res. vol. 40, pp 2233-2240 490Phillips, S. et. al. (1972) Analysis of accommodative response times using histogram information. Am. J. Optom. Arch. vol. 49, no. 5, pp 389-401 491Smith, W. (2000) Modern Optical Engineering NY: McGraw-Hill, pg 137 260 Processes in Animal Vision decisions is found in the POS of the midbrain. This allows this initial perception to be performed within a period of about 50-100 ms. This interval would be shared between the correlation time (correlator cycle time of 50 ms) and the travel time for the signals from the retina to reach the PGN/pulvinar couple (transmission delay of 1.0 ms). The gross commands must then be generated in the superior colliculus. These are then processed further in the various nuclei of the pointing and focus circuits. The commands leave the midbrain following an interval of about 350 ms. The transmission time between midbrain and muscle innervating neuron (omitting any delay at the ciliary ganglion nucleus), is only about 1.0 ms. The response of the muscles generally varies according to the size of the change required. This response is well documented regarding the saccades. However, it is not as well known with respect to the focus mechanism. Careful study of the raw data supporting figure 7 of Phillips, et. al. could give good data on this time as a function of required accommodation change (from various starting accommodation levels). The longer latencies suggested by Phillips, et. al. as a function of age are most likely associated with the response time of the physical plant and not a true temporal latency before the lens begins to deform. The data of Beers & Van der Heijde provide specific delay times as a function of required change in accommodation. A careful review of the records of Phillips, et. al. could provide more data concerning the type of transient response exhibited by the accommodation system. Is it impulse driven as with the oculomotor system or is it continuously driven? Ciuffreda & Krueger have provided detailed data showing that the lens servosystem is essentially a first order system492. It exhibits a simple exponential transient characteristic regardless of amplitude. Beers & Van der Heijde provided a better schematic of the accommodation plant and better data recently but their analysis was too brief493. Their interest was only in the change in shape of the lens under a set of linearized forces. Figure 7.4.9-8 reproduces their figure 2a.

To simplify their analysis, they ignored one of the two orthogonal components of the forces applied at an angle to the lens perimeter. The angle associated with the force is 25 degrees and varies less than +/– 2 degrees during

Figure 7.4.9-8 Dynamic biomechanical model of the accommodation plant, with the angle, θ, added. It remains nearly constant with accommodation. Original shows the model superimposed on the anatomy of the outer eye. From Beers & Heijde, 1994.

492Ciuffreda, K. & Kruger, P. (1988) Dynamics of human voluntary accommodation. Am. J. Optom. Physiol. Optics. Vol. 65, no. 5, pp 365-370 493Beers, A. & Van der Heijde, G. (1994) In vivo Determination of the Biomechanical Properties of the Component elements of the Accommodation Mechanism Vision Res vol 34(21), pp 2897-2905 Dynamics of Vision 7- 261 accommodation according to Wyatt494. These values suggest the sine and cosine of 25 degrees play significant roles in the accommodation mechanism. Their radial force applied to the lens should be multiplied by the cosine of 25 degrees whereas the axial force (with respect to the axis of the eye) applied to the lens perimeter should be given as the applied force times the sine of 25 degrees. Since the angle remains nearly constant, there is an axial motion associated with the lens that is equal to the radial motion at the perimeter of the lens times the tangent of 25 degrees. This axial motion is required to maintain the Petzval surface in congruence with the entrance aperture surface of the photoreceptors of the retina. Without this axial motion, the system will go out of focus with any change in the lens power. While not generally recognized in the vision literature, the combination of the cornea and lens represents a “zoom” type optical system. It is designed, like any zoom system , to achieve optimum focus over a range of operating distances by performing two actions. It first changes the power of one of the elements. It then compensates for the resultant change in focal position relative to the centroid of the optical assembly, by changing the spacing between the optical elements. Both actions are necessary to maintain proper focus at the retina. Figure 7.4.9-9 presents an expanded schematic of the arrangement in Section 7.4.9.1.2 using more convenient symbology. The expansion in A only addresses the overall forces applied to the periphery of the lens. It incorporates the elements of both the Stark and the Beers & Van der Heijde models but provides a more appropriate model of the muscle. The springlike elements are shown as coils rather the zigzag symbols for continuity. The muscle is replaced by an active source in parallel with a mass and a dashpot. These elements are connected to the remainder of the network by a spring, km. The muscle delivers a force to the system through this spring. With the symbols used, this configuration can be considered an electrical analog of the actual physiological circuit. In that case, the force is represented by a voltage. If the muscle is held tonic, a constant voltage is applied to the network. As noted in both Stark and Beers & Van der Heijde, only one half the lens is represented here. The center of the lens is considered a virtual ground for purposes of the analysis on the assumption it does not move radially. In B, the “axial zonular spring, kza, of Beers & Heidje is resolved into two components by the “resolver.” The resolver is merely the geometry associated with the axial zonular spring and the lens capsule. At a nearly constant angle of 25 degrees, the radial component of the force applied to the capsule via the axial zonular spring is given by the total spring force multiplied by the cosine of 25 degrees. The axial component of the force applied to the capsule via the axial zonular spring is given by the total spring force multiplied by the sine of 25 degrees.

Look first at frame A. Without any force due to the ciliary muscle, the position of the edge of the lens is determined by

the tensions generated among the three tension elements, kc, kzp & kza, and the tension of the lens, kL. This position determines the unaccommodated optical power of the lens (lacking pharmacological intervention).

The left portion of frame C shows the forces involved in both distorting and moving the lens. When a force is applied, the perimeter of the lens moves in response to the force applied through the tensile elements, km. The net force is divided between overcoming the tensions associated with kzp & kc and the force available to move the lens perimeter. The more structurally rigid km and kc are, the less force required to distort them. However, they are essential as a restoring force and must exhibit a useful spring constant. On the other hand, if the element kza is not rigid, the muscle will be unable to apply significant force to the periphery of the lens. The distortion of the lens by a change in the radius of its perimeter (the radial position) does not result in a large change in the location of the center of mass. The term, mL, can generally be ignored when computing accommodation.

The right portion of frame C shows the forces involved in moving the lens axially. The motion is small and slow. As above the mass of the lens plays a small role in this motion. The main point is that the axial motion is significant if the angle, θ, is to remain nearly fixed. The primary opposition to this motion is believed to be the resistance, bv, and the springiness, kv, associated with the viscosity of the humors surrounding the lens. Conformation of the axial motion of the lens in the process of accommodation has not been widely discussed in the literature. However, the author recently underwent post-pupil lens replacement in his right eye. Post-pupil lens replacement means the biological lens was removed from its capsule and replaced by an acrylic substitute. Thus, the precise geometrical configuration of the musculatura is maintained but the lens element has been replaced. This had an unanticipated result. On the second day after the replacement procedure, and using my glasses from before the surgery, I observed an unusual event. Immediately after removing my glasses with my left eye closed, I observed a change in the quality of my distant vision over a period of a few seconds. It appears the fixed acrylic lens was being moved axially by the ciliary muscle in an attempt by the focus servomechanism to optimize my vision through the right eye. I discussed this with my ophthalmologist and he confirmed there were reports in the literature of patients exhibiting accommodation of as much as two diopters following post- pupil lens replacement.

494Wyatt, H. (1988) Some aspects of the mechanics of accommodation Vision Res vol. 28, pp 77-86 262 Processes in Animal Vision

Figure 7.4.9-9 Schematic of the plant of the lens servomechanism. A; full diagram that portrays both the actual physiology using an electrical analog of that physiology. The “resolver” is undefined in this view. C; the electrical analog resolved into its radial and axial components. The electrical analog uses the current = force & voltage = radial displacement relationships. See text for discussion. B; data in upper right frame from Beers & Van der Heijde, 1994. Dynamics of Vision 7- 263 Beers & Van der Heijde proposed that the response of the lens accommodation system followed an exponential function. However, frame B of the figure disputes this fact. Two pure exponentials have been overlaid on one of their frames of data. It can be seen that the initial rise is much too fast to be represented by an exponential. This is the situation in most, if not all, of their data frames. Some frames at only a one diopter change are difficult to analyze. In their data, the relaxation of the lens system was always faster than the change to a higher level of accommodation (except possibly for the one diopter change). The force applied by the muscle must be defined, as a function of the innervation, if this network is going to be

completely understood. Similarly, the characteristics of the tensile force kL need to be understood. Muscles do not normally exhibit a constant tensile strength as a function of innervation (or extension). Similarly, the tensile strength of a gel enclosed by a thin walled membrane does not conform to Young’s Law for solids, defining a linear tensile strain in response to a linear tensile stress. The relationship is more complicated. These two nonlinearities, the muscles force/innervation relationship and the strain/stress relationship of the lens, account for the non-exponential relationships shown in the data of Beers & Van der Heijde. They also lead to an explanation of the awkward time constant relationship shown in their paper. The time of response is not given by a single time constant as suggested by Beers & Van der Heijde. Their equations employ several simplifications that may obscure the actual values. Furthermore, the response of the network depends on the precise state of accommodation of the eye in each test run. By introducing more realistic values in this analysis, more realistic time constants can be defined based on the lower portion of the figure. This calls for a more realistic forcing function, a more reasonable tensile performance for the lens, and the recognition of other circuit elements related to the ciliary muscle.

Fisher has provided some early data on the stress/strain relationship for the human lens. However, he was well aware of the delicacy of the measurement and the difficulty of analyzing it precisely. His analyses did rely upon a piecewise linear assumption. As a result, he has provided initial values for the energy stored in the lens as a function of distortion level.

A better understanding of the operation of the lens accommodation system could be obtained by measuring the time required to make a one diopter change (or smaller), in each direction, from positions differing by one diopter throughout the adaptation range. The resulting functions could be integrated to more clearly define the time required to make multiple diopter changes. The resulting responses would be much closer to an exponential form. Assuming the noise component in the recording is not excessive, more meaningful time constants could be read from this data. 7.4.9.2 Short term accommodation errors

Hung, crediting Ong & Ciuffreda, gives the nominal accommodation system error as 0.3 to 0.5 D495. 7.4.9.3 Presbyopia due to refraction errors is a normal consequence of aging

The analysis of this section has provided a clear graphical explanation of presbyopia and its impact on humans beyond the age of 10. Before that age, the individual elements of the physiological optics are subject to such a variety of growth rates that little can be said about their ultimate visual performance. Figure 7.4.9-10 illustrates the situation. The gray band in the figure represents the range from zero to plus 4 diopters. This is the normal range required for focusing from 25 cm (10 inches) to infinity and includes the range of most human activity. The properly functioning accommodation system has more than sufficient range to cover this zone. However, the accommodation system is not symmetrical. The system goes from a fully relaxed condition to a condition of accommodation some 10 diopters more positive. Two problems arise in the visual process. First, if the relaxed accommodation level (called the basal accommodation level here) of an individual is already positive, the eye cannot accommodate to zero diopters. This is the condition shown by the solid triangle. The bottom of the triangle intersects the basal accommodation scale on the right of the figure. Such a person is considered near sighted, refractively myopic, and will normally require glasses of at least minus one diopters to achieve proper distant vision. For a serious myope, a basal accommodation level of plus five is not unusual. Anyone with a basal accommodation level exceeding plus four will require glasses at all ranges throughout his lifetime. Most subjects with a basal accommodation level that is negative do not have a need for glasses during their early years. The accommodation level of such individuals will almost always include the shaded zone as shown by the dashed triangle. However, if he should have a basal accommodation level more negative than minus five, he may require glasses for reading, and other close work, beginning at an early age.

495Hung, G. (2001) Models of Oculomotor Control. London: World Scientific pg 35 264 Processes in Animal Vision

In this discussion, the basal accommodation level is an operational level found when the eye is relaxed by the subject. It is not the level of relaxation achievable by pharmacological means. Most people exhibit an accommodation range of at least ten diopters at age ten. This range decreases monotonically with age as shown, for both hypermetropes and hypometropes (myopes). The results are the same but occur at different times as shown. For the myopic subject, he will require glasses for far vision from an early age. He will begin to suffer a loss of accommodation and require glasses at short range beginning somewhere around 45 years of age. This point will depend on both his basal accommodation level and his initial accommodation range at age 10. The hypermetrope cannot achieve as high a level of absolute peak accommodation as the typical myope, because of Figure 7.4.9-10 Presbyopia as a normal process of aging. the offset introduced by his basal accommodation level. The gray band shows the normal range of accommodation Therefore, the hypermetrope will lose his ability to focus needed to properly image from 25 cm to infinity. This range at short range earlier than the myope. This frequently includes the zero line. See Text. happens around 40 years of age. These people are the ones that need “reading glasses,” particularly for menus in dark restaurants, etc. As his accommodation capability continues to decline, he will begin to have difficulty seeing at a distance. This occurs when his accommodation range becomes numerically less than his basal accommodation level. This latter phenomenon appears to equate to the “distance hyperopia” of Donders. At an age of 45-50, the typical hypermetrope will need positive lenses for seeing both near and distant objects. Bifocals become the order of the day for hypermetropes more than 50 years of age. 7.4.9.4 The aperture control system 7.4.9.4.1 The iris control system

Bremner has described the detailed neural circuits controlling the iris and some examples of their operation, describing both the parasympathetic and sympathetic pathways496.

Phillips, et. al. provide a brief discussion of the change in iris aperture during their accommodation experiments497. However, they did not provide sufficient detail about the light levels involved as a function of required accommodation to show that such a change is correlated with accommodation in the absence a change in light level. Myers & Stark have provided considerable recent material on the operation of the aperture control system498. Surprisingly, they did not investigate the asymmetrical operation of the iris, between increases in light levels versus reductions in light level, in detail. Their model in figure 2 appears to only apply to increases in light level. Their models related to increases in light level “have been made parsimonious to simulate the main experimental findings reported here. . . .” They specifically say their models are not meant to be homologous with the physiology of the visual system. They did, however, recognize the logarithmic transformation occurring in the photoreceptor cells. Careful analysis of their data reveals the operation of several distinct mechanisms. Their test protocol involved relatively long pulses of illumination. The leading edge of the characteristic shows the variation in delay and in initial slope characteristic of the Photoexcitation/De-excitation process discussed in Section 7.2. The droop in the signal during the interval following peak response appears characteristic of the adaptation process. The time constant appears very similar to that expected in that process, about six seconds, and reported in Section 2.4.3.1. Saturation in the adaptation process

496Bremner, F. (1999) Disorders of pupillary function in Acheson, J. & Riordan-Eva, P. Fundamentals of Clinical Ophthalmology: Neuro-ophthalmology. pp 183-190 497Phillips, et. al. (1972) Op. Cit. 498Myers, G. & Stark, L. (1993) Level dependent signal flow in the light pupil reflex Biol Cybern vol. 68, pp 229-246 (three papers) Dynamics of Vision 7- 265 is quite evident in the response to a maximum stimulus of 10,000 foot-Lamberts. Following the end of the stimulus, the trailing edge of the transient exhibits a combined time constant of about five seconds. This value appears to be a composite of the time constants of the adaptation process and the plant associated with the iris. The decay time constant of the P/D process is too short to contribute. It is less than 12.5 msec. Based on the above analyses, a model quite different from that suggested by Myers & Stark appears more appropriate. It separates the small signal mathematical model in the s-plane into three distinct portions. The first portion relates to the P/D process, the second to the adaptation process and the third to the plant associated with the iris. To develop the model further for the large signal case requires introduction of the logarithmic transform (also found in the Myers & Stark model) occurring at the pedicle of the photoreceptor cells. This model suggests the neurological portion of the mechanism is the same as that associated with the photoreceptors of the eye and developed in Sections 7.2 & 12.5. The remainder of the model is associated with the servomechanism plant associated with the iris. Note that in this model, the intrinsic delay is also divided into three portions. A portion is associated with the phototransduction process (stage 1), a portion is associated with the servomechanism (stage 5), and a portion is associated with the transport delay within the signal projection process (stage 3). The minimum total delay encountered by Myers and Stark, about 180 ms, is probably accounted for by the sum of the neural transport delay and the delay associated with the plant. The variable delay of up to an additional 70 ms can be attributed to the P/D process. Although Myers & Stark show inputs into their model from the vergence and accommodation subsystems in concept, they did not discuss these inputs in detail. References to earlier papers are provided. Based on this work, the influence of these two subsystems (if substantive) is primarily through their earlier influence on the stored values within the lookup tables of the superior colliculus. The resting pupil area, and the related a priori command value to the iris plant are also stored in the superior colliculus. These proposals are consistent with the position of Myers & Stark that the control of the iris rests within the midbrain. The control of the iris is “via the third nerve to the ciliary ganglion and then to the iris muscle.”

The model proposed here exhibits a variable term in the phototransduction process that provides a deterministic solution to the overall response that does not require the conceptual “adaptive bucket brigade” mechanism introduced by Myers & Stark.

The comment of Myers & Stark concerning the operating range of the iris control system agrees with other sources (Section 2.4.3.1). They found that below one foot-lambert stimulation, the pupil was fully open (typically 6.4 mm diameter in their case). One foot-lambert corresponds to the bottom of the photopic range of vison for most subjects (within a factor of two). At 10,000 foot-lamberts, the iris became fully open (typically 3 mm diameter) after less than two seconds. Their subject appears to have a narrower than normal range of iris operation.

The third paper of Myers & Stark both summarizes the work of others and explores oscillations related to the iris. 7.4.9.4.2 The shutter control system

The 1st and 2nd shutters are simple curtains operating primarily in a two-state mode, either opened or closed. Humans can modulate this situation to a slight degree by transmitting a rapid sequence of open and close commands. However, this mode of operation is fatiguing. The process appears to be designed to be binary in character. What is described here as the 1st shutter is relatively, but far from completely, opaque. Its purpose is to block high frequency spatial information from reaching the retina. The information handling portion of the visual system is not sensitive to low frequency changes below about three hertz. Therefore, completely blocking illuminance changes occurring in the temporal region from zero to three Hz is not necessary. The 2nd shutter operates in the same manner as the 1st. However, it is normally not an opaque structure. It is usually a nearly transparent structure used in two different modes. It is frequently used by terrestrial animals and birds as a protective device against abrasion by the environment. In semi-aquatic animals, it is normally used as a compensating lens. This compensating lens allows proper operation of the optical portion of the eye in media of significantly different indexes of refraction. It is not used in humans. The neurons controlling these shutters are not associated with the optical fiber bundle of the eye but travel along the temples in humans. 266 Processes in Animal Vision

7.4.10 Interplay of version, vergence and accommodation subsystems

Because of the relative simplicity of the experimental procedures, many investigators have documented the relationships between the various functional overlays on the pointing and accommodation subsystems499,500, 501,502. The relationships most often reported include; Function Independent of Dependent on

Vergence accommodation, accommodation Vergence version version Depth Perception accommodation, accommodation Accommodation vergence, vergence Accommodation depth perception, depth perception Clearly, these reported relationships or lack thereof are inconsistent. However, many of the test protocols used earlier were limiting. This is illustrated by experiments associated with the “Independent” column of the table. Westheimer & Mitchell have documented this situation for the vergence response503. Noting that vergence data acquired with the haploscope is under conditions of fixed accommodation is also important. While Rashbass & Westheimer point out that disparity vergence is continuously controlled, their model is inadequate. At a finer level of detail, the vergence system still relies upon stage 3 projection neurons. These circuits are pulse circuits with a minimum refractory interval (See Chapter 14).

It is only through the participation of the memory elements of the superior colliculus that estimates of the most probable values of the individual functions are provided based on changes in the “Dependent” variable. These estimates are clearly based on experience. 7.4.10.1 Cross-coupling of functional overlays– servomechanisms EMPTY 7.4.10.2 Relationship between vergence and version

Jones & Kerr have made measurements related to the response of the visual system to an alarm mode signal as a function of time in milliseconds504. They note that the version response is usually completed before the vergence response. This would be expected based on the model. While the stimulus causes a rapid version response, and an initial vergence response based on a priori value, the final vergence value requires a few additional tens of milliseconds to establish an optimum value of vergence.

Jones & Kerr quote Rashbass & Westheimer as having “shown that conjugate and disjunctive eye movements are entirely independent; each can accept and respond to stimulation irrespective of whether the other is responding, is being stimulated, is within a reaction time, or is fatigued.”

Rashbass & Westheimer have also shown that version responses are much faster than vergence responses. This would be expected if the vergence subsystem was expected to optimize its performance following a version response where the vergence subsystem was provided an a priori value from memory. The vergence system must await completion of at least the first saccade before going into optimization mode. They also showed that the vergence system suffers from significant fatigue effects (pg 347), even at a forced frequency of only one Hertz.

499Rashbass, C. & Westheimer, G. (1961) Disjunctive eye movements J Physiol vol. 159, pp 149-170 & 361- 364 500Brodkey, J. & Stark, L. (1967) Accommodative convergence: an adaptive nonlinear control system IEEE Trans Systems, Sci Cybernet vol. SSC-3, pp 121-133 501Zuber, B. & Stark, L. (1968) Dynamic characteristics of the fusional vergence eye-movement system IEEE Trans Systems Sci Cybernet vol. SSC-4, pp 72-79 502Myers & Stark, L. (1993) Op. Cit. 503Westheimer, G.& Mitchell, D. (1956) Arch Ophthalmol vol. 55, pp848-856 504Jones, R. & Kerr, K. (1971) Motor responses to conflicting asymmetrical vergence stimulus information Am J Optom vol. 48, no. 12, pg 993 Dynamics of Vision 7- 267 7.4.10.3 Change in accommodation with vergence

Fry has studied the change in accommodation in the presence of only changes in vergence. His results illustrate the small changes in accommodation encountered for changes of vergence of up to +/– 2.5 degrees around the nominal vergence value for a scene505. 7.4.10.4 The ratio of accommodative convergence to accommodation

One of the simplest laboratory experiments related to the functional overlays to the pointing system is the introduction of a change in accommodation. Records details the procedures involved in detail506. Stark has noted a failure in accommodation to respond to vergence changes is due to a deficit in controller signal generation in the central nervous system507. It is proposed that the deficit is most likely in the operation of the lookup table in the superior colliculus. 7.4.10.5 The ratio AC/A

The ratio of accommodative convergence to \accommodation is frequently used diagnostically in the clinical setting. Scott has provided a good exposition on this subject including the relevant dimensions (with standard deviations)508. He notes that the ratio is calculated using “prism diopters” in the numerator and regular diopters in the denominator. Hence, the ratio is not truly dimension-less as usually assumed. Unfortunately, he does not differentiate between these two terms in his discussion. It is left to the reader to understand the difference. The reader is also cautioned that his statement that the difference between distance between the nodal points and the distance between the centers of rotation is negligible only applies in the clinical situation. The difference becomes significant in research. He does refer to the importance of this factor on page 607. 7.4.10.6 Recent research in virtual and augmented reality

With the rise in interest and the resultant increase in laboratory investigations related to virtual and augmented reality for both games and industrial processing, the subject of mental fatigue has arisen and traced to the conflicts in the visual modality between the natural (learned) relationship between accommodation and vergence. In the process of optimally focusing on a scene, a two step process is employed. Initially the signals from the two eyes are compared in time to determine an initial estimate of the vergence angle required to optimize stereo vision (Section xxx). This first order vergence estimate is used to extract an accommodation signal. The accommodation signal is used to focus the two eyes at the approximate distance to the target of interest. With the quality of the images projected onto each retina, a second corrected estimate of the vergence angle is made and implemented via the oculomotor system. At the same time, a second accommodation estimate is made and implemented. In virtual and in some cases augmented vision, the image presented to the visual modality involves a virtual image. This virtual image does not necessarily present the same conditions for both the vergence and accommodation parameters simultaneously. As a result, the previously learned values in the lookup table may not be optimal. This creates a “version-accommodation conflict” within the visual modality. The visual modality will then attempt a third round (or more) of optimization. This causes a degree of fatigue and/or vertigo within the subject being evaluated. 7.4.11 Other models of version, vergence and accommodation in the literature

A wide range of independent models and schematics of the version, vergence, and accommodation subsystems have appeared in the literature. Even a putative fusion subsystem has appeared in the literature. These have generally been floating models that did not relate to, and interconnect with, other major elements of the visual system. Based on this work, these models have suffered from three primary problems. First, they have not addressed how the spatial information presented to the retina is converted into a temporal signal. This failing leaves most of these models as interesting but largely irrelevant models. Second, they have generally not recognized the temporal frequency aspects of the photoexcitation/de-excitation mechanism Figure 35 in Hung (2001) and attributed to Hung (1998) is a good example of this situation509. The model assumes a perfect square pulse is applied to the neurological sections of both

505Fry, G. (1983) in Schor, C. & Ciuffreda, K. Op. Cit. pg 407 506Records, R. ed. (1979) Physiology of the Human Eye and Visual System NY: Harper & Row, pp 605-608 507Stark, L. (1983) Normal and abnormal vergence, In Schor, C. & Ciuffreda, K. Op. Cit. pg 11 508Scott, A. (1979) Ocular motility, Chapter 21 in Records, R. Physiology of the Human Eye and Visual System. NY: Harper & Row pp 605-607 509Hung, G. (1998) Dynamic model of saccade-vergence interactions Med Sci Res vol 26, pp 9-14 268 Processes in Animal Vision the disjunctive and conjunctive servomechanisms. These models have then tried to account for these aspects in a more complicated model associated with the neurological circuits and physical plant associated with these subsystems. Many of these models include adaptive switching of discrete filter elements in an attempt to account for the inherently adaptive, but continuous, nature of the P/D process. Third, they have attempted to provide a hard-wired representation of the interplay between the version, vergence and accommodation subsystems and their excitation by the alarm and volition modes of operation. Often, the models have attempted to define and describe discrete disparity detector circuits (instead of the actual correlation device suggested by and definable at the detailed level by physiology). They have also failed to include a priori inputs from memory in their overall operation. These attempts, which have been primarily at the conceptual level, lead future investigators along a deviant path.

Hung has provided a recent introductory text that includes a variety of floating models of the type described above510. Earlier, Semmlow & Hung summarized a variety of models in Schor & Ciuffreda511. Their discussion centered on the triad of version, vergence and accommodation. Others in historical order include a paper by Zuber & Stark focused on fusional vergence512, a subsequent heuristic model by Krishna & Stark focused on vergence513, Schor514 and a largely conceptual paper by Patel, et. al. focused on potential neural networks and subdivided into seven discrete stages (including discrete disparity detector circuits)515. The basic model developed in Section 7.3 and the overlays developed in Section 7.4 provide a more realistic description of the overall models appropriate to version, vergence, accommodation and amplitude control of the light stimulus. A key element of the model is the presence of the superior colliculus (and cerebellum when required) inside the servomechanism loops. The colliculus provides a complete set of a priori or “most likely” values for the version, vergence, accommodation and amplitude control subsystems. These are used in the absence of more explicit values from the alarm and volition modes of vision. Of similar importance is the general computational capability of the pulvinar, probably shared with the superior colliculus. This capability can provide a wide range of output signals. These complete sets of output signals may be provided in the presence of incomplete sets of input signals. To do this, the system relies upon previous training and the values and the instruction-to-command transforms stored in the superior colliculus.

The general computational capabilities provided by the pulvinar and superior colliculus (computation plus memory) essentially replace the discrete cross-strapping circuits proposed in Hung & Ciuffreda516, Hung, Ciuffreda & Rosenfield517, and Krishnan & Stark (figure 3) and others. 7.5 Higher level functional aspects of the dynamics of the eyes

As noted in the introduction to section 7.3, significant differences exist in the performance of the visual systems of each species within the higher primates and the monkeys. These differences make the use of surrogates in the exploratory and precision performance laboratories less than ideal. Often, and particularly regarding the higher level functions of vision discussed in this section, the use of surrogates is completely inappropriate. No known animal can approach the performance of the human in the areas to be discussed in this section 7.5.

A large empirical literature exists on the operation of the human visual system in perusing a scene and in reading text. However, it invariably ends with the supposition that the eyes remain stationary in a gaze between saccades exploring the scene presented to them. This is a major fallacy in the science of vision. The eye is highly active during the gaze at a scale that cannot be observed without special instrumentation. During each gaze, the eye makes a series (up to a several hundred) of individual small motions (microsaccades and flicks) that support the processing of large amounts

510Hung, G. (2001) Models of Oculomotor Control. London: World Scientific 511Semmlow, J. & Hung, G. (1983) The near response: theories of control Chapter 6 in Schor, C. & Ciuffreda, K. Op. Cit. 512Zuber, B. & Stark, L. (1968) Dynamic characteristics of the fusional vergence eye-movement system IEEE Trans Syst Sci Cyber vol SSC-4, no. 1, pp 72-79 513Krishnan, V. & Stark, L. (1977) A heuristic model for the human vergence eye movement system IEEE Trans Biomed Eng Vol. BME-24, no. 1, pp 44-48 514Schor, C. (1979) The relationship between fusional vergence eye movements and fixation disparity Vision Res vol. 19, pp 1359-1367 515Patel, S. Ogmen, H. White, J. & Jiang, B. (1997) Neural network model of short-term horizontal disparity vergence dynamics Vision Res vol. 37, no. 10, pp 1383-1399 516Hung, G. & Ciuffreda, K. (1999) Adaptation model of nearwork-induced transient myopia Ophthal Physiol. Opt vol. 19,pp 151-158 517Hung, G. Ciuffreda, K. & Rosenfield, M. Proximal contribution to a linear static model of accommodation and vergence Ophthal Physiol Opt vol. 16, pp 34-41 Dynamics of Vision 7- 269 of information within the retina. The information is then transmitted along the individual nerve fibers connecting the 23,000 photoreceptors of the foveola of the retina to the brain. While the rest of the retina is operating in the Awareness mode, the portion of the visual system associated with the foveola is operating in the analytical mode. This section will discuss the operation of the visual system from this perspective. There are two easily differentiated areas of visual system operation. That associated with the viewing of objects in their relationship to a scene is one. That associated with the viewing of semantic symbols in reading (where the symbols carry no relationship to the surrounding scene) is another. This differentiation is so important, it divides the field of perception studies into two distinct classes. Findlay observed this considerable gap between the two types of study in 1980 and said few attempts had been made to bridge it518. This differentiation is still important. The two sides of it will be addressed below in separate sections. Henderson & Hollingworth noted a hiatus in the study of eye-movements during the 1980- 90's. The renewal of interest is almost certainly associated with the advances in technology related to accurately recording eye movement. They also report some of the first experiments designed to bridge the gap between the perception of scenes and text and discuss the relationship between the saliency maps of object space and cortical space519.

The major role played by memory in the perception and interpretation of ones environment has not been adequately addressed when studying the functional aspects of vision. It appears that the visual system avoids analyzing most of the elements of every scene presented to it by bringing forth the previously memorized information about that scene. This is illustrated by the ability of a person to walk into a room and almost instantly notice that one small change has occurred. The ability to walk into a room through different doors with the same result supports the importance of memory. It strongly suggests that a vectorized saliency map of the room is stored in long term memory. Such a vectorized map would be independent of orientation and, to a large extent, scale. It is automatically independent of light level because the visual system normalizes light level prior to the analysis function.

Similarly, the major role of training in the function of perception has not been quantified. There appear to be a series of default routines used in the visual process based primarily on training (in conjunction with memory). This seems true in both reading, where one routine has been traditionally described as a “dumb default,” and the perception of scenes. 7.5.1 Background

In the following material, it is crucial that a consistent set of definitions for different size saccades is relied upon. Figure 7.5.1-1 reviews the terminology to be used here. All references cited in the literature will be converted to this nomenclature. Most of the literature related to the fundamental dynamics of the eyes has concentrated on the large and small saccades. Investigations related to reading and scene analysis have generally concentrated on the realm of the Minisaccades. The area of the flicks and microsaccades have only been explored by a few investigators, primarily in Russia (Yarbus and Shakhnovich), and England (Ditchburn), although Riggs in the USA should not be excluded from mention.

518Findlay, J. (1980) The visual stimulus for saccadic eye movements in human observers. Perception, vol. 9, pp 7-21 519Underwood, G. (1998) Eye Guidance in Reading and Scene Perception. NY: Elsevier. pp 286-291 270 Processes in Animal Vision

Figure 7.5.1-1 Definition of saccades by size. Although shown in polar coordinates ( and on a logarithmic scale for convenience), all indications point toward their description in terms of rectilinear components relative to the surface of the spherical retina. The circles are at 2 minutes of arc, 1.2° and 6°. The term tremor is used synonymously with the term microscaddes and occupies the area of the inner circle. Not shown is tremor with a nominal excursion less than 30 arc seconds peak-to-peak.

7.5.1.1 The continuing philosophical debate

The role of microsaccades (and tremor) in vision has been debated for a very long time, with the detractors (and some supporters) generally assuming the eye operates as an imager as opposed to its fundamental character as a scanner. The Dynamics of Vision 7- 271 most recent published colloquy on the subject was between Ditchburn520 and Kowler & Steinman521. In this colloquy, both sides discuss small saccades in terms of their possible role in stabilizing the image on the retina. Both papers are strong on philosophical arguments and empirical observations but very short on theory and numbers. Ditchburn carefully subdivides and defines the saccadic movements of less than six minutes of arc to frame his arguments. He defines tremor as involving movements of less than one minute of arc. He also says: “The primary visual signal is probably proportional to dL/dt where L is the retinal illuminance.” He also notes that “Very precise fixation would produce the loss of vision that occurs when the retinal image is stabilized.” The opposition offered by Kowler & Steinman obviously involves trying to prove a negative. They recognize the phenomenon exists and they close with the statement: “We believe that this (existence) creates an evolutionary puzzle. Why should human beings, and only human beings, exhibit a penchant for making such small high velocity eye movements if they serve no useful purpose? We do not know.” [emphasis added] They argue that: + Microsaccades can be suppressed without training. + Smooth eye movements, not microsaccades are optimal for maintaining clear vision. + Microsaccades are not needed for visual information processing. + The functional significance of microsaccades remains a mystery.

As both sides stipulated, the visual system is blind in the absence of motion.

Unfortunately, Kowler & Steinman did not define their term microsaccades. Their discussion is in terms of information processing. It refers to counting groups of objects that are individually larger than the resolution threshold of the eye and confined to a circle of 30 arc minutes. They speak of minisaccades, and natural body movement, as providing all of the motion required to overcome any latent blindness of the eye caused by a stabilized image. However, they did not address the staring mode of vision where presumably little normal eye motion exists. What they do not address at all is the subject of reading, the one capability that is not shared with other chordates and that requires the analysis of features much smaller than the objects they were counting. It is the capability of the human to perceive and interpret minute differences in symbols imaged near his/her threshold of resolution that differentiates the human from all other animals. It is this capability that allows us to communicate on paper without carrying around large tablets of characters in 72 point type.

Kowler & Steinman did not address or demonstrate that the visual system is capable of reading in the absence of tremor or that reading was possible during a continuous saccade. 7.5.1.1.1 An alternate view adopted in this work

Ditchburn couches his argument for tremor, as recognized by Kowler & Steinman, as introducing a fine motion that contributes to the change in illumination required by the eye. Kowler & Steinman rebut, in the context of tremor as a mechanism required for image stabilization, the proposition that tremor is not needed for this function and would actually degrade the performance of the imaging capability of the visual system. They argue that such tremor would actually degrade the excellent performance of what they describe as the slow eye movement performance of the system. They are clearly arguing apples and oranges here. It must be said that Ditchburn still did not appreciate the functional role of tremor in 1981. Then, he added a paragraph to a discussion that began: “In passing, we may note that frequency components of eye movement below about 10 Hz operate to maintain vision while the higher frequency components (particularly above the critical flicker frequency) effectively blurs the retinal image and impairs vision522.” He continues: “Thus a damping system that nearly eliminates the high frequency components . . . is advantageous to the visual system. . . .” This passing remark may have been introduced to avoid conflict with Stark & Ellis writing in the same text. They said, “In any case, their amplitude makes them an unsatisfactory candidate for any visual function523.” These are the types of statements that have significantly impeded our understanding of the reading process. The list of unresolved problems on page 235 of that text is instructive. This work views the subject of saccades in a specific context. The photoreceptors act as change detectors and limitations are placed on the signaling channels by the performance characteristics of the adaptation amplifiers associated

520Ditchburn, R. (1980) The function of small saccades. Vision Res. vol. 20, pp 271-272 521Kowler, E. & Steinman, R. (1980) Small saccades serve no useful purpose. Vision Res. vol. 20, pp 273-276 522Fisher, D. Monty, R. & Senders, J. (1981) Op. Cit. pg 229 523Fisher, D. Monty, R. & Senders, J. (1981) Op. Cit. pg 202 272 Processes in Animal Vision with each individual photoreceptor. Within this context, the servomechanisms controlling the movement of the eyes can be interpreted from an entirely different perspective. The presence of a foveola in many advanced chordates also introduces a critical feature. The result is recognition that the performance of the servomechanisms can be divided into two distinct categories responsible for two major functions. The first is to provide an effective alarm function, in association with providing a general situational awareness, to protect the animal. The second is to provide the analytical capability associated with the foveola that supports the needs of the great hunters for precision vision and the needs of humans to communicate through reading and writing. 7.5.1.1.2 Dispersion in the empirical literature

By vectorizing the information involved in the visual process, the system can avoid any reliance on the scale of the imagery presented to it. However, this flexibility has made it difficult to interpret the perceptual performance of the visual system in the laboratory. Because of the flexibility in scale allowed by the system, investigators have employed a large variety of test configurations and image samples. This has inhibited development of a clear model of the perceptual process. It would be very useful if the community could settle on a set of standard test samples or test sample styles for further exploration. It is extremely important that investigators quantify precisely the scale at which material is presented to a subject for interpretation. A reading test using 18 point type may involve distinctly different internal parameters than a similar test using eight point type at the same distance. The development of stylistic rules in writing, and more specifically printing, has highlighted the importance of the space between words, the length of words and similar features in our perception of textual material. Unless required by the nature of the experiment, standardizing the spacing and length of word groups when performing experiments to discover how visual perception is achieved would be useful. Using words of variable length introduces additional statistical variables that are poorly documented into eye movement experiments.

Englebert & Mergenthaler have recently provided a discussion of fine retinal motions524. Their definition of microsaccades corresponds to the minisaccades defined in the above figure. 7.5.1.2 The available laboratory equipment

Leigh & Zee have provided a list of the methods available for tracking eye movements in their Appendix B. The equipment available has always been complicated and of limited precision. Sheena & Borah have provided a discussion of the merits of the different available equipments525. At the current time, one of the most widely used is an infrared pupil tracker. The AmTech ET3/4 provides a precision of 0.1 degrees or six minutes of arc at a sample rate of 400/500 Hz. This sensitivity is not adequate for recording microsaccades at the few seconds of arc level required to analyze the most sophisticated motions of visual interpretation such as reading. 7.5.2 Aspects of examining a scene

Two major modes of examination are employed in viewing a bucolic scene, the awareness mode and the analytical mode. If the scene contains elements that might create a threat to the subject, the alarm mode becomes critical to the subject’s interpretation of the scene. For the bucolic scene, the subject initially creates an awareness of the entire scene and a (probably) prioritized list of high information content regions of the scene. In the scene perception community, a term for the non-quantized level of information content has been labeled “informativeness.” The eye(s) then proceeds to scan the scene in a series of generally small saccades, pausing to gaze at each area of high interest according to its informativeness. These pauses are quite long compared with the time spent performing saccades. What functions the visual system is performing during this interval is not well understood.

If the bucolic scene is interrupted by a sudden change in the scene, characterized by a local change in illumination (that can be and frequently is due to motion in the scene), The above saccade-examination scenario, associated with a bucolic scene, can be abruptly interrupted by an alarm signal generated in the LGN. The alarm signal can be generated by any sudden change in the scene characterized by a local change in illumination (that can be and frequently is due to motion in the scene). This signal causes two probably simultaneous actions, an immediate saccade to bring the line of fixation of the eyes to the point of the dynamic element of the scene, and a reorganization and re-prioritization of the saccade-examination sequence. Such a change in a local element of the scene is frequently labeled a “wiggle” in the scene perception community.

524Engbert, R. & Mergenthaler, K. (2006) Microsaccades are triggered by low retinal image slip PNAS vol 103, pp 7192-7197 525Fisher, D. Monty, R. & Senders, J. (1981) Op. Cit. pg 257-268 Dynamics of Vision 7- 273 A variant on the bucolic scene involves a uniform degree of motion for the entire imaged scene. Such motion causes the visual system to attempt to remove the apparent motion of the image on the retinas by introducing a pursuit signal into the oculomotor commands generated by the POS. As indicated above, this action involves a Type 0 servomechanism tracking a moving object. In the absence of any cognitive input, such a system could track a moving image with a fixed displacement error. It is well within the cognitive powers of the brain to introduce a bias into the POS to compensate for this error (commonly called Kentucky windage in hunting and the military). Although this action has many interesting parameters and its effect can blossom into a variety of ancillary phenomena such as vertigo, it will not be discussed here. 7.5.2.1 Aspects of examining a bucolic scene

The recent text edited by Underwood provides a good jumping off point for the following discussion of the perception of a scene526. However, it can only be considered an introduction. The index to this text does not include the word tremor and the word microsaccades appears on only one page. On page 34, microsaccades are discussed briefly with only a hint of their possible importance. Inhoff & Radach (citing Kowler & Steinman527) say: “they could indicate fine attentional adjustments or reflect an intrinsic tendency to move the eyes after some delay has passed. So far, no consistent functional explanation has emerged . . . to relate them to perceptual or cognitive processes.” The conclusion is drawn that limit criteria can be used to eliminate microsaccades from further experimentation, analysis and discussion. This position has unfortunately been perpetuated in the perceptual portion of the vision community for more than 20 years.

Henderson & Hollingworth have provided a good overview of eye movements during scene viewing in Chapter 12 of Underwood. However it should be noted that their color plates 1a & 1b and 2a & 2b are not of the same scene (note the changed positions of the stoves). For convenience, most laboratory investigations have employed artificial scenes, frequently presented by a television monitor. See Table I of Henderson & Hollingworth for a summary of these investigations. This technique usually excludes a large saccade from the investigation except possibly during the setup interval. It also limits both the vertical and horizontal resolution of the image to a value far above the tremor level of the visual system. Because of this last limitation, the use of a cathode ray tube or LCD monitor cannot be used in exploring the limits of the visual system related to fine detail (whether it involves images or text).

Figure 7.5.2-1 shows a typical presentation and the resultant saccades from Henderson, et. al528. The length of each saccade was typically less than 6° and the average was given as 2.4°. These saccades will be defined as small according to the nomenclature of this work. The subject was given 15 seconds to view this figure. The duration of each saccade was not presented in Henderson, et. al. although the average fixation (gaze?) was given as 327 ms. Their figure 2 presents the proportion as a function of latency for gazes, and for a variety of test materials. No information was provided concerning the microsaccades and flicks occurring within these gazes. As a result, a time line for the activities of the eye during the allotted 15 second time interval contains large voids as noted in the auxiliary figure provided below the original figure. The solid bars show the height and duration of a typical small saccade. Between these bars, no activity is recorded for the eye(s), yet each interval represents sufficient time for more than one hundred flicks and microsaccades. The question is, what is the visual system doing this 96% of its operating cycle?

It is a major premise of this work that the system is performing a detailed analysis of the portion of the image falling within its foveola during this interval. The analysis employs both microsaccades and flicks. These activities cannot be observed with the conventional eye trackers optimized for observing wide angle saccades because of their noise threshold and frequently their quantization levels.

In their color plate 2 and 2B, Henderson, et. al. provide an estimate of the time the gaze remains on a given element of the scene. This time appears to correlate to local scene complexity except for something in the original drawing that cannot be defined from the figure. Henderson introduced the concept of a saliency map (in object space) in 1992 to account for these areas of increased attentiveness. It assigns a weighting to the coordinates of each point of potential (or observed) interest. Henderson, Weeks & Hollingworth defined a set of experiments that appear to surface an additional alarm condition at a very high level of cortical performance529. These experiments focus on uncovering semantically illogical elements

526Underwood, G. (1998) Eye Guidance in Reading and Scene Perception. NY: Elsevier. 527Kowler, E. & Steinman, R. (1980) Small saccades serve no useful purpose. Vision Res. vol. 20, pp 273-276 528Henderson, J.& Hollingworth, A. (1998) Does consistent scene context faciitate object perception? J. Exp. Psychology: General, vol. 127, pp 398-415 529Henderson, J. Weeks, P. & Hollingworth, A. (1999) The effects of semantic consistency on eye movements during complex scene viewing. J. Exp. Psych. vol 25, pp 210-228 274 Processes in Animal Vision within an otherwise bucolic scene. The visual system appears to focus attention on these objects to a greater degree than might be expected. Recently Brewer et al. have provided material similar to that of Hendersen and associates with more emphasis on integration with object recognition530. The work is discussed in more detail in Section19.10.4.1.4. 7.5.2.1.1 The familiarity default procedure in examining a scene

A cursory reading of the chapters in Underwood related to the perception of a scene leads to the conclusion that there is a familiarity based dumb default processing routine similar to that found in reading. The length of time spent analyzing a familiar scene is significantly shorter than that for a new scene of similar complexity. Also, the subject focuses on changed elements of a familiar scene more rapidly than expected by chance. These observations suggest a two step process. First, following the initial review of a new scene by the visual system, it prepares a vectorized saliency spreadsheet (map) of the scene based on areas of high information content. Second, it prepares an initial schedule of saccades with which to analyze the scene. They also suggest that the saliency map is stored as a permanent vector record of that scene. When the same scene is reviewed again, it appears that the system again computes a vectorized saliency map that can be compared with the copy stored in memory. The system rapidly determines the difference between the two maps and arranges the second schedule of saccades to focus on the areas of change before performing a complete scene analysis. The vectorization of the saliency spreadsheet is important. Besides reducing the storage capacity requirement, it makes the stored memory independent of scale and orientation. The subject can be shown the same scene photographed from a significantly different angle and it will still be recognized as familiar.

In the above scenario, the visual system interprets a scene in two steps. It initially prepares a vectorized saliency map that can be compared with those stored in long term memory. If an appropriate match is obtained, the difference between the new and old maps is prepared and the schedule of saccades is prepared that places a priority on the difference. This routine shortens the overall analyses considerably and can be likened to the “dumb default” of reading. In the absence of a match, the system prepares a saliency map and a more extended schedule of saccades designed to analyze the entire scene at an appropriate level of detail. The saliency map is stored and the schedule is started in due course. This can be considered the normal analytical procedure.

530Brewer, A. Press, W. Logothetis, N. & Wandell, B. (2002) Visual areas in macaque cortex measured using functional magnetic resonance imaging J Neurosci vol 22(23), pp 10416–10426 Dynamics of Vision 7- 275

Figure 7.5.2-1 Viewing pattern for a complex line drawing. Green dots represent discrete fixations within a 10° x 14.5° field of view in ordinal order beginning at the center of the field. Green lines represent individual saccades. The vertical resolution of the image was quantized to one minute of arc by the monitor. Top frame from Henderson, et. al. 1999.

7.5.2.2 Aspects of examining scenes of finer structure Empty

[xxx this area should include the classic faces of Yarbus and include the limitations of the foveola & PGN/pulvinar when 276 Processes in Animal Vision examining specific scenes. ]

7.5.2.3 Aspects of examining a scene containing a local transient event

If an element of a scene should change rapidly and substantially in illumination, the visual system will be alerted to such a change by the LGN. This will cause a major re-prioritizing of the operations of the eyes. A new saccade will be introduced into the operating schedule to examine the appropriate area of the image immediately. This phenomenon has been studied in some detail without an underlying explanation for its occurrence. Figure 7.5.2-2 is the typical result of one of these investigations reproduced from De Graef writing in Underwood531. The initial experimental sequence was similar to the previous protocol involving a bucolic scene. However, after a given interval, an element or elements of the scene are caused to vary in intensity. The protocol used for this scene is described as: + The subject fixates on a cross on a blank monitor screen for at least 200 ms. + The image is changed to a scene presented for eight seconds that always contained an object in place of the cross.

+ After 160 ms, an object in the scene began moving up and down through four minutes of arc, performing two cycles within 120 ms. Four minutes of arc is near the commonly accepted resolution limit of five minutes of arc corresponding to the height of the E on a Stellen Chart at 20/20.

The wiggle is intended to elicit a saccade from the prime object to the wiggled object.

De Graef provided an additional degree of complexity to the experiments to analyze the results more effectively. He provided prime objects of two distinct types, those that fit semantically within the context of the larger scene and those that were obviously out of contextual place.

The statistical results of De Graef are not of interest here. Describing the above sequence of events, in terms of the awareness, analysis and alarm modes of visual operation is useful. While this description does not affect the empirical results obtained by De Greaf, it does provide a foundation for the discussion of the results and possibly suggest alternate follow-on experiments. Figure 7.5.2-2 A time sequence for viewing a scene on a monitor. See text. From De Graf, 1998. 7.5.2.3.1 The simultaneous dual alarm scenario

Findlay has examined the introduction of two signals into the visual scene simultaneously and studied the resultant propensity of the system to align the line of fixation to one or the other of the two events532. Considering the time line of such activity to determine if it suggests the course of the signals moving through the visual system is instructive. Certain time lines may suggest activity limited to the POS, to the POS in conjunction with the cerebellum or to the POS in conjunction with either area 7 or the posterior areas of the cerebrum.

531Underwood, G. (1998) Op. Cit. pp 313-336 532Findlay, J. (1980) The visual stimulus for saccadic eye movements in human observers. Perception, vol. 9, pp 7-21 Dynamics of Vision 7- 277 7.5.2.4 Strategies Employed in Scene Perception and Interpretation

Koch & Ullman have explored the options used by the visual system to explore a scene533. They introduce several concepts regarding visual attention and build on the earlier work of Treisman relative to the saliency map. They examine the potentials of both a serial and a parallel computing machine at the heart of the visual process. They discuss selective visual attention and the conspicuity of a scene element. Two rules are presented based on the assumption of a three-stage mechanism. When viewing a natural scene, the awareness channel employs a primitive set of learned rules that relate to the total scene. These rules are largely independent of the size and orientation of the scene relative to the observer. The rules aid in the determination of the major objects within the scene and the instruction of the analytical channel to image, perceive and interpret each of these objects in turn. The strategy is roughly as follows. 1. The awareness channel decomposes the scene into a list of probable objects with their associated coordinates within the scene. 2. It prepares an initial sequence of saccades required to perceive the details related to each of these objects. This sequence is used by the analytical channel of vision. 3. It performs each saccade in the above list in turn to present the image of each object to the foveola. 4. The image of the object is held on the foveola for a finite interval during which it appears the eye is fixated. • The eye is not actually fixated, it would be better to say the eye gazes at the object. 5. The Precision Optical System causes the image to be scanned rapidly through a series of microsaccades (tremor) that generates a signal within the analysis channel associated with the foveola. • These microsaccades are not visible to the typical clinician or researcher. 6. The PGN/pulvinar pair of the midbrain extracts the perceivable features of the signal and projects the resulting vectorized signal to area 7 of the cortex. 7. The vectorized analytical signals received at area 7 are interpreted and an initial file is created relating that object to the overall scene. This interpretation is a reiterative two-step process. • The tentative outside contour of each object is determined. • The texture of the object inside the tentative contour is determined. • IF the texture is not uniform, or of a recognizable pattern, the object is determined to contain additional internal objects and the above process is repeated at a finer level of detail. • The initial file for that object defined above is expanded to include the details derived by the scanning, perception and interpretation steps. • The initial file is integrated with the saliency map within the cortex of the individual. • It is at this point that the cortex examines the file for any conflicts with the expected context of the overall scene.

Steps 3 through 7 are repeated until all of the objects on the original saccades sequence list have been perceived and interpreted. Following the above series of steps, the mind of the subject has a fully interpreted understanding of the scene presented. It can then take any cognitive actions it desires.

7.5.3 Oculomotor aspects of reading

Because of the global involvement of the visual system in reading, the overall process will be described at the detail level in Section 17.8 and at a more functional level in Chapter 19. Only the dynamic mechanical aspects associated with reading will be discussed here. It is a premise of this work that reading specifically involves the phenomenon of tremor and its resolution into horizontal and vertical microsaccades and flicks. It will be shown in the following section that most research on reading has not progressed beyond the stage of measuring the duration of the “fixation intervals” composing the gaze interval allotted

533Koch, C. & Ullman, S. (1985) Shifts in selective visual attention: toward the underlying circuitry Human Neurobiol. vol. 4, pp 219-227 278 Processes in Animal Vision to each significant word of text. A great deal of literature exists at this level. However, very little, if any, literature exists at the level describing the motions of the eyes within these periods of nominal fixation. Following the background discussion, the thesis of this work regarding the phenomenon and mechanism of reading will be presented. It is obvious that the human is adept at reading almost regardless of the scale of the symbolic characters relative to the resolution limit of the eyes. However, an argument can be made that above a certain scale, the characters may be viewed as an element of a bucolic scene and not strictly symbolic notation. Nevertheless, recognizing this independence of scale can be important when discussing the details of the reading process. Recognizing the difficulty of using a television or computer monitor in the evaluation of the reading phenomenon at its most fundamental level is important. The displayed imagery must not be limited by the monitor used. The refresh rate and phosphor decay characteristic of the monitor can have a significant impact on the quality of the investigation and the conclusions drawn. To explore the reading phenomenon in detail even requires the careful choice of type font for the imagery. The serifs used at the corners of letters are there to improve character recognition. They are not there to make the text look pretty. They affect the net contrast of small features of individual characters when the characters are projected on the retina at a scale near the resolution limit of the eye. High integrity investigations should either employ high quality printed material in hard copy or projected form or employ carefully selected monitor equipment. The screen of the monitor should be demagnified sufficiently to eliminate any interference with the tests due to the parameters of the monitor. The selection of the type face used in the experiments should be declared and justified in the design of the experiments.

Although scale does not play a major role in casual reading, it does play a significant role in how effectively we read and on research into reading. Figure 7.5.3-1 illustrates the number of characters that can be placed on the nominal foveola at one time as a function of the scale of the characters. At the scale of 20/20 vision, approximately fifteen characters (three five-character groups separated by one-character spaces) can occupy the diameter of the foveola at one time. At 20/40, only about eight characters can occupy this diameter. At 20/80, less than five characters can be imaged. 20/80, or characters that are 20 minutes of arc high at 15 inches, corresponds to 6.28 point type. This is a small size not usually found in books except possibly for footnotes. 9.4 point type corresponds to 20/120 or characters that are 30 minutes of arc high--a typical document size for reading.

This figure shows that the typical subject only images about three characters at a time on his/her foveola when reading nine point type at 15 inches. He may image as many as sixteen characters across the width of the fovea. Casual review of text at these sizes suggests that the typical subject cannot perceive eight characters in a single fixation and requires a second saccade to determine the precise meaning of an eight-character word. An example is deciding whether a suffix relates to a case or a tense in the absence of other cues. Based on these numbers, the size of the type used and the font of that type plays a significant role in the experiments usually carried out and reported in the literature. In addition, a preliminary conclusion would be that the space between letters plays a significant role in the planning of subsequent saccades, even if this space occurs outside of the foveola but within the fovea. It appears such a space plays a larger role in planning subsequent saccades than does the length of the subsequent words. Looking at smaller type suggests the importance of having the individual characters separated by more space to perceive them individually.

Figure 7.5.3-1 The characters of text imaged on the foveola (black bar) and the fovea (Gray) as a function of scale. The black area actually consists of individual dots representing the individual photoreceptors of the retina as will be shown in a later figure.

7.5.3.1 Background from the recent literature Dynamics of Vision 7- 279 Chapters 3 through 12 of Underwood provide a broad overview of the field of perception during reading as of 1998. As indicated earlier, the words tremor and microsaccades do not appear in any of these chapters. The focus is on words, word string length and semantic usage in and out of context. Kennedy in Chapter 7 does explore the influence of parafoveal words on foveal inspection time. The precise definition of the foveola, fovea and parafovea are key to a clear understanding of the reading function. Such a definition was not provided in Kennedy. He describes his test configuration as using a “high resolution (8x16) monopitch font in negative polarity (i.e., white characters on a black background) on a Manitron display driven at a refresh rate of 100 Hz.” At a viewing distance of 525 mm, three characters of this font subtended approximately 1° of visual angle (20 minutes of arc per character). This scale is far above the resolution limit of the normal 20/20 eye where the height of the E is defined as subtending five minutes of arc in object space. Note that a refresh rate of 100 Hz, if not essentially filtered out by the decay time of the phosphor of the display, may introduce a significant conflict with the natural small scale tremor of the eye.

Becker, et. al. have also provided empirical material on the reading mechanism as of 1999534. Both Underwood and Becker, et. al. concentrate on eye movement as the key to our understanding of reading. This work takes an entirely different view. It treats observable eye movement as merely a mechanism for imaging individual scene features, symbols and character groups onto the foveola where the actual process of perception is initiated. This perception involves eye movement at a level not normally observed by the clinician or academician. Based on the conceptualization developed in this work, the following definition is offerred. Reading can be defined as the act of assembling a sequence of perceptions acquired through the sequential analysis of individual symbols or character groups and interpreting these perceptions according to a set of syntactical rules. In this definition, symbols include hieroglyphics and other glyphs. The initial interpretation of each symbol or symbol group by the POS results in the generation of an individual “interp.” When a series of interps are combined, the resulting interpretation will be called a percept.

Within this work, the foveola and the fovea are defined based on the morphological characteristics of the retina. The foveola is defined as 1.18° in diameter, the fovea is defined as 6.26° in diameter in object space and the parafovea is defined as beyond 6.26° from the fixation point. Therefore the fovea of Kennedy conforms closely to the foveola of this work and his parafovea will be assumed to equal the fovea of this work.

Kennedy also states “In reading, each word is inspected by an initial fixation at a particular position resulting from an ‘entry saccade’ of a given size, launched from a particular location in another word.” The conditions, variations and significance of this entry saccade are discussed in some detail on page 152-153. The relationship of such saccades on the empirical model of Rayner, Reichle and Pollatsek in Chapter 11 is also reviewed. The referral to a “location in another word” is about as close to the discussion of characters in a word achieved in the overall Underwood text. More material is presented on regressive saccades, from one word back to a previously examined word, than is given to the examination of the characters within a word.

Kennedy discusses the term prompt, with the word gaze in parenthesis following it, as the sum of all fixations before the first excursion outside its boundary (page 157). In the Abstract to the chapter, he says the time to process one of two possible foveal “prompt” words was examined using measure of gaze and fixation duration.

Figure 7.5.3-2 presents a modified, semi-standard figure from Liversedge, Paterson & Pickering in Underwood. It should not be inferred that most of each time interval shown relates to the latency before the next saccade. These saccades may be a part of a planned saccade sequence. In Chapter 19, the PEEP procedure is defined (Programmed element evaluation in Perception) as the serial sequence of saccades and fixations employed by the stage 4 engines and the oculomotor servo mechanisms prior to extracting information from the signals thereby generated. Such a PEEP procedure is use at all levels of visual modality object recognition, including reading, face recognition, etc. This sequence need not involve a significant latency between the end of analysis within one gaze, and the beginning of a saccade to the next gaze location. Thus, a more specific set of subdivisions of the term latency is probably called for here. The times do define the maximum length of time required for the visual system to analyze the structure of the symbols within the foveola adequately. Adequately here means sufficient time to ascertain their semantic content (probably via a lookup table). The above authors did not address the size of the type (subtense of the height of the characters) used in their experiments but they did say the tracking data was quantized every millisecond.

534Becker,W. Deubel, H. & Mergner, T. (1999) Current Oculomotor Research. NY: Plenum 280 Processes in Animal Vision

Inhoff & Radach, writing in Chapter 2 of Underwood, reported on eye movements when viewing long strings of printed characters. Their data provides good information on the precise nature of the small saccades related to eye movement during reading. However, they did not provide information at the microsaccade level or on the nature of the saccades used to perceive word meaning in Figure 7.5.3-2 Hypothetical eye movement record showing the context of reading. Their text introduces many the time in milliseconds spent in a gaze between saccades. potential experimental variables (flexibility of the eyeball The text being read is shown above the line. The zig in the leading to transient movements of the lens group–cornea line is indicative of a regressive saccade. and/or lens) but does not provide a foundation for overcoming or controlling them. Their figure 1shows many minisaccades at the 0.1-0.2 character width level between saccades of one and seven character widths. However, the noise level of their equipment was not specified or shown and they may have excluded microsaccades from their analyses. Their discussion includes presentation of an interesting dichotomy (citing Deubel & Bridgman535). First, they say the eye is an imager and that small “post saccadic movements will smear the retinal image.” Second, they note that the post saccadic motions are relatively small and principled, and the reader may be able to extract useful information during that movement period [emphasis added]. Radach & McConkie prepared Chapter 4 in Underwood. It discusses the determination of fixation positions in words during reading. One of their conclusions is that, “In all cases (where there are spaces between the character groups), eye movement control during reading appears to be word-based.” This control appears to involve two distinct mechanisms, a selection mechanism and a performance mechanism. It is proposed that the selection mechanism determines how one word rather than another is selected as the target of a saccade, and the performance mechanism determines where the eyes actually land given the above selection. The discussion centers on the general likelihood that the saccade is aimed at the center of the selected word. In the data presented, the imagery was presented (a German translation of the initial text in Gulliver’s Travels) in page format with five to seven double-spaced lines of up to 72 ASCII characters each on a 15 inch VGA monitor. At 80 cm viewing distance, each letter corresponded to approximately 0.25° of visual angle (15 minutes of arc or three times the 20/20 character definition).

A conclusion of the above authors is that “where the eyes go with respect to selected saccade target words, is the result of low-level visuo-oculomotor control factors, almost completely unaffected by higher cognitive processes.” This is probably true for relatively familiar words not calling for regressive saccades or multiple saccades within one word. The above authors go on to caveat their statement. One of their caveats is “in the case of regressive inter-word saccades, the saccade parameters we have looked at suggest a control mode different from the low-level default routines.” The intraword characteristics of the reading process were not discussed in their chapter. Here again, the word inter-word appears in the index to Underwood but the expression intra-word does not. Heller & Radach made note of two important facts536. They noted the work of Dunn-Rankin in 1978 that showed that the initial fixation point on words was not at their center but at positions left of center. They also noted the work of Rayner & Pollatsek in 1981 that showed that the final decision concerning the direction and magnitude of the next saccade was made during a given fixation interval.

Rayner, Reichle & Pollatsek presented Chapter 11 in Underwood. They discuss the effect of limiting the length of time available within a gaze to analyze the text characters. They show that if a given gaze (the traditional fixation interval by name, even if it involves tremor) is interrupted before 50-70 ms have elapsed, the reading process itself is interrupted and comprehension suffers or is lost. They also suggest that a preview of a word, while it is imaged in the area outside the foveola, has a positive impact on reducing the time required to interpret it when it is moved into the foveola. The above authors review several conceptual models of the eye movement control system required to implement reading. A process model by Morrison is summarized. This model is still conceptual but includes the concept of some preanalysis of a word before it enters the foveola or before it is brought to the point of fixation within the foveola. A logic is provided that controls the length of the gaze and/or the series of interim fixations associated with each word. The model explains two aspects of the eye movement phenomenon in the reading process: (1) the fact that there are fixations that are much shorter than the nominal 175-200 ms saccade latency in simple oculomotor tasks and (2) the occurrence of unusual landing positions, such as between words. A competing model by O’Regan, that they describe as a strategy- tactics model, is also summarized along with a critic of its features.

535Deubel, H. & Bridgeman, B. (1995) Perceptual consequences of ocular lens overshoot during saccadic eye movements. Vision Res. vol. 35, 529-538 & 2897-902. 536Becker, et. al. (1999) Op Cit. pg. 341 Dynamics of Vision 7- 281 Finally, the authors summarize their proposed E-Z Reader Model. They describe it as similar to Morrison’s process model except more refined through the use of two additional facets. First, it decouples the signal to shift attention from the signal to program a saccade. Second, it is better specified in that it has been implemented as a computer simulation program. They refer to the fifth generation of this program as the program under current discussion. They describe the simulation with a schematic representing five basic processes: 1. A familiarity check on a word. 2. Completion of lexical access. 3. An early, labile stage of saccadic programming, which can be cancelled by subsequent saccadic programming. 4. A later, non-labile, stage of saccadic programming. 5. The actual saccadic eye movement. They define the first two steps as products of a single cognitive process. This process occurs during a preview while the word is still in the fovea (not the foveola) and occurs before the movement of the word to the center of the foveola. They develop the fact that completion of the familiarity check depends on two additional factors. These factors slow the rate of processing as the eccentricity of the word, relative to the point of fixation becomes larger. This factor was added to recognize the rapid falloff of the resolution (acuity?) of the eye with eccentricity. They also discuss the operational distinction between an interword and an intraword saccade. They suggest the basic eye movement strategy conforms to the “dumb” default strategy: That strategy is to plan to fixate each word from more than one viewing location unless the word’s familiarity indicates that a refixation is unnecessary. The dual fixation strategy could obviously be useful in words that frequently have unusual or multiple suffixes (syllables).

The dumb default strategy emphasizes the importance of having a large vocabulary in the field encompassing the text being read. It also suggests that the average reading rate is dependent on the frequency of occurrence of words found in the vocabulary.

Some authors have argued that no “magic moment” of word identification exists; that identification (of individual words or a total thought) only comes with a growing amount of data collected on a continuous basis. This may be a question of semantics between authors since the motion of the eyes is clearly not continuous with time. 7.5.3.1.1 The familiarity default procedure in reading

The above “dumb default” strategy suggests a major change in the operating mode of the POS during reading. When a word is recognized cognitively (the magic moment?), the POS initiates a saccade and the line of fixation is moved to the next fixation point. This suggests that the visual system incorporates a feature similar to the “auto complete” feature of an INTERNET browser. The browser compares the initial key strokes of an entry with its short term memory and suggests the appropriate completion of the typed entry. If the suggestion is wrong, the typist is free to enter an alternative. An equivalent scenario can be defined for the visual system. After analyzing only a few symbols, initially in the fovea area surrounding the foveola, the system may believe with high probability that it knows what the entire symbol group means. In that case, it will instruct the POS to proceed to the next symbol group. If the subsequent symbol group is recognized but it does not fit logically into a recognized syntax with the first group, the POS may be instructed to perform a regression saccade. This will allow a further review of the previous symbol group to see if an alternate suffix, or other difference from the assumed meaning, exists. This procedure is illustrated in Figure 7.5.3-3. The system attempts to interpret a short sentence. It can methodically perceive and interpret each character group as in A. Alternately, it can adopt a more aggressive approach and make a guess based on the likelihood that the character group “dres” is part of the longer word “dressed.” This results in the sequence shown in B and some time is saved as one gaze (fixation) is eliminated from the initial saccades sequence. If however, the assumption was made that “dres” was part of the word “dresses,” the same procedure can be followed until the second or third character group in “yesterday” is reached. At this point in the interpretation, a context conflict is recognized through comparison of the initial concept file with the saliency map of the individual. As a result, a regression saccade is called for back to the word that was actually “dressed” and not “dresses.” 282 Processes in Animal Vision

Considering the time line of the above activity, to decide if it suggests the course of the signals moving through the visual system, is instructive. Certain time lines may suggest activity limited to the POS, to the POS in conjunction with the cerebellum or to the POS in conjunction with either area 7 or the posterior areas of the cerebrum. Figure 7.5.3-4, from Becker, et. al. addresses this subject directly537. A statistically shorter latency can be associated with the regressive saccade than with the progressive scan. Apparently, when the system notes an inconsistency in the proposed syntax, it cancels the analysis of that character group and calls for a regressive saccade to reestablish a viable baseline. These experiments were carried out with considerable statistical precision and the original source should be reviewed before proceeding.

Figure 7.5.3-3 The procedure of perceiving and interpreting a sentence showing three alternatives.

537Becker, W. Deubel, H. & Mergner, T. (1999) Current Oculomotor Research. NY: Plenum Publishers, pg 320-325. Dynamics of Vision 7- 283

7.5.3.1.2 The question of identification of character groups outside the foveola

Although relying on limited precision in their definition of the foveola, fovea and parafovea, a school of thought has developed that the eye can perform two critical functions. It can identify, to some degree, character groups while they are in the parafovea. This initial identification can affect the subsequent identification activity (including the saccadic sequence). Murray addresses this possibility in Chapter 8 of Underwood while still avoiding a geometrically specific definition of the areas of the retina. He stresses the continuous nature of the retina, based on the apparent continuous loss in acuity with angle from the line of fixation.

7.5.4 Statistics of recentering in vision

Glezer has expanded the earlier work of Cornsweet investigating the tendency of the human visual system to recenter on a target feature of interest following a gradual decentering of the point of fixation538. A small, three arc minute source at 100 nit on a dark background was used as a target. Figure 7.5.4-1 shows his results for seven subjects. No description of the time course of the decentering was provided. Neither was there any discussion of the phenomenon when examining more complex scenes. Figure 7.5.3-4 Distribution of fixation duration for two subjects in a standard reading experiment. Top, cumulative distribution. Middle, fixation duration following syntax 7.5.5 Empirical data on eye movements failure at fixation after a “dumb default.” Bottom, distribution of fixations before normal progression to next There is a vast archive of empirical data on eye fixation. See text. movements taken without the benefit of any theoretical model of the neural system involved. Specific aspects of this data are incorporated into the discussion of the performance of the visual modality in Sections 17.6 to 17.9 of this work. 7.5.5.1 Data from Masson & Perrinet

Masson & Perrinet provided considerable data in a 2012 review on eye movements with the receptive field as a parameter539. [xxx The paper has not been obtained or filed in my archive. It is available on line for viewing only ] The was abstract of the paper notes, “Short-latency ocular following are reflexive, tracking eye movements that are observed in human and non-human primates in response to a sudden and brief translation of the image. Initial, open-loop part of the eye acceleration reflects many of the properties attributed to low-level motion processing. We review a very large set of behavioral data demonstrating several key properties of motion detection and integration stages and their dynamics.”

538Glezer, V. (1965) The receptive fields of the retina Vision Res vol 5, pp 497-525 539Masson, G. & Perrinet, L. (2012) The behavioral receptive field underlying motion integration for primate tracking eye movements Neurosci Biobehav Rev vol 36, pp 1-25 284 Processes in Animal Vision Dynamics of Vision 7- 285

7.6 Mechanical dynamics of the PC/RPE interface

Figure 7.6.1-1 illustrates the photoreceptor cell/RPE interface, from both the morphological and functional perspective. It consolidates many features and issues discussed earlier. Some of these features appear in a figure by Ong540. The static morphology includes both the topology and topography. The dynamic functional aspects focus on the topology, disk life cycle, and signaling. The figure is quite complex, but it tells many stories. 7.6.1 Description of the interface

The Figure segregates the material into three named columns and eight distinct rows. Four background Figure 7.5.5-1 Development of the oculomotor reflex (in regions are defined. From left to right, they are; (a) the percent probability of occurrence) as a function of interneural matrix (INM), (b) the inter-photoreceptor displacement of the fixation point (in arc minutes). Shaded matrix (IPM), (c) the unnamed material shown between area-standard deviation. Interrupted line is the frequency of the RPE cells and Bruch’s Membrane, and (d) the the development of flicks from Cornsweet, 1956. From Vascular Matrix supported by the choroid artery. These Glezer, 1965. regions are separated from each other by distinct structures that prevent diffusion, and usually prevent electrical conduction, between regions.

On the left, the barrier structure consists of the Inner Segments of the photoreceptor cells and the material between the cells, usually defined as Muller type glial Cells. This combination is frequently labeled the external or outer limiting “membrane” based on early work at low magnification. Boycott & Dowling generally define a separate and distinct outer limiting membrane proximal from the Inner Segment-Muller cell combination. On the right, Bruch’s Membrane provides a barrier between the Vascular Matrix and the region extending to the RPE cells. This region is in turn isolated from the Inter-photoreceptor matrix by the RPE cells and any material between these cells acting as a barrier. The literature usually describes the RPE cells as so tightly packed that no separate material is needed between them to provide this isolation. A confetti pattern has been used to define all three of the named matrices. The unnamed material is shown as a black region fading to white at the barrier. The IPM is the material surrounding the Outer Segments in the central column of the drawing.

---[xxx next three paragraphs need editing]

The photoreceptor cells are drawn to emphasize the difference between the Inner Segments and the Outer Segments. Each Inner Segment exhibits a cup-shaped feature in which the protein material, opsin is initially formed by exocytosis. The specific shape of the extrusion cup breaks and forms the protein into individual disks. This breaking and forming is suggested by the line emanating from the black dots in rows one and two. The resulting disks of opsin form the framework of the Outer Segment. The Outer Segment is a dynamic structure external to the photoreceptor cell. Following formation of the individual disks, the disks are coated with chromophoric material from the IPM as shown in row two. The process of continual extrusion forces the disk stack to move continuously toward the right. The rate of movement is about one disk per hour in humans. A similar rate is found in all warm blooded chordates. The Inner Segments are shown without regard to the cell nuclei, which may be enclosed by the Inner Segments or may be found remotely and closer to the pedicles of the cells. Two primary functions of the Inner Segments are shown explicitly. The first is the extrusion cup on the right of each Inner Segment. Many mitochondria and ribosomes are found to the left of the extrusion cup. They are believed to produce the protein material passed through the cell wall into the extrusion cup. The process is one of exocytosis. This material is shown initially as a black dot that is formed into a ribbon. This ribbon is fractured and formed into disks of protein by the extrusion cup. The material of these disks is

540Ong, D. (1985) Vitamin A-binding proteins. Nutrition Reviews vol. 43, no. 8, Aug. pp. 286 Processes in Animal Vision known as Opsin. Second, an electrical path is shown from the dendrites (generally nine in number and shown by the heavy black lines) surrounding the Outer Segments to the pedicles found within the INM (but not shown explicitly). These dendrites enter the Inner Segments from the IPM near the extrusion cup in an area known as the calyx, or collar (shown along the top edge of the Inner Segments of row four and five). The RPE cells are shown containing four different types of chromophoric granules. The granules containing the chromophores of vision are stored as palmitates. In color photography based on reflected light, these granules appear in their complementary color since they are highly absorbent. The complement to a narrow band absorber is a wide band transmitter. Using conventional color photography, the S-channel chromophores appear yellow, the L-channel materials appear cyan, and the M-channel material appears magenta. The UV-channel material does not absorb any radiation when illuminated by conventional light. It appears transparent when imaged onto conventional color film designed to record images as seen by normal human vision. Wolken appears to have captured three of these granule types (UV-, S-, & M-) in a single photomicrograph from a swamp turtle.541 He also presents spectrums for three of the granules from a chicken. Unfortunately, his laboratory methods were harsh and his terminology is now a bit archaic. His choice of color names may be due partially to the wide spectral width of his spectrometer. He labels the S-channel absorber as yellow by reflected light. The spectrum associated with the UV-channel absorber is only partially recorded and is labeled green although it shows little absorption at any wavelength longer than 500 nm. It appears transparent in his figure. The spectrum of the M-channel absorber is labeled red but is more appropriately labeled magenta as it appears in his figure. No record of a L-channel absorber, which would record as aqua, appears in his figure. A full color version of Figure 7.6.1-1 is available on the author’s Web Site. In the color version, the disks of Row 2 are transparent and sensitive in the Ultraviolet. The disks of Row 3 are cyan and sensitive to red. The disks of Row 4 are magenta and sensitive to green. Finally, the disks of Row 5 are yellow and sensitive to blue.

Row 1 has been drawn in simplified form to illustrate how the disks are formed into a stack that is pushed away from the Inner Segment. The disks are formed at the rate of about one per hour in warm-blooded animals. When the stack in a fully mature human eye reaches about 2000 disks, the outer most disks reach the region of the RPE cells. The RPE cells also form a cup like structure around each disk stack. The RPE cell uses this structure to phagocytize the disks within the cup. The broken disks within the cup in rows four through seven are to symbolize this process. A stack of disks as shown in Row 1 is not photosensitive and the situation as described can be considered pathological.

541Wolken, J. (1966) Vision: biophysics and biochemistry of the retinal photoreceptors, Springfield IL: Charles C. Thomas pp. 71-74 Dynamics of Vision 7- 287

Figure 7.6.1-1 Illustrative cross section of the PC/RPE interface. See Text. 288 Processes in Animal Vision Row 2 shows the activation of the photo-receptive structure. A primary purpose of the RPE cells is to create the chromophoric material, store it in chromatic granules until needed and then release these chromophores into the IPM. Once released into the IPM, they diffuse to the region of the extrusion cup of each Inner Segment. There are indications that these cups have passages between the zone of exocytosis and the extrusion die. These passages allow the chromatic materials, the Rhodonines, to enter the cup from the IPM and coat the protein substrates before they complete the extrusion process. Once the disks are coated with a liquid crystalline chromophore, they are photosensitive. The coated disk stack of Row 3, and also Row 4 & 5, is shown surrounded by another material, the bio-energetic fuel that provides electrical energy for all neural cells. The presence of this material has frequently been confused with a putative extension of the Inner Segment cell wall surrounding the Outer Segment. However, in high resolution imagery, no sign of a bilayer cell wall surrounding the Outer Segment is found in this region. The bio-energetic material can diffuse into the region between disks and into the grooves along the disks where the microtubules associated with the dendritic structure are found. However, these diffusion paths are very limited in their capacity to diffuse bio-energetic material. Looking at Row 2 and 3 simultaneously, note that both (all Inner Segments) exhibit exocytosis but the process is only shown explicitly for a few cells. Note also that there is no one-to-one correlation between the RPE cells and the Inner Segments. When a disk stack reaches the region of the RPE cells, the RPE cells form a cup around the stack, or part of a stack, in order to carry out phagocytosis. The entire coated disk is phagocytized. The chromophoric material is absorbed and probably recycled within the RPE. The protein material is absorbed and probably returned to the Vascular Matrix for recycling within the animals body. Four different types of chromophoric granules are shown within the RPE cells, even for humans. Although the ultraviolet spectral capability of the human eye may be only residual, it is clearly present as discussed earlier regarding aphakics. No correlation is known between the type of chromophore in a given RPE cell and the absorption spectrum of a specific Inner Segment. Most RPE cells contain a variety of chromophoric granules.

The signal path between the coated disks of the Outer Segment and the pedicles of the cells have been shown explicitly in Rows 4 and 5. What has not been shown explicitly is the location of the distribution amplifier within each Inner Segment. These Activa may be arranged so their podites contact the IPM or the INM. Furthermore, some of them may be located outside the Inner Segment per se. They may be located in a hillock at the junction of the Inner Segment and the Axon. Here, the podite terminal would be on the surface of the initial segment (of the axon) as it is frequently in bipolar cells.

Rows six and seven illustrate the condition of a tear in the retina. Such a tear frequently causes a shearing of a large group of Outer Segments. In the case shown in row seven, the tear results in a displacement of the alignment of the Outer Segments. If the tear is physically repaired by a physician before excessive time has passed, the disk stacks will continue to grow normally and the physical damaged material will be swept into the cups of the RPE cells. A danger always exists that a tear will destroy the protective environment of the IPM and allow oxygen and/or other oxidizers to enter the IPM space. This can damage the chromophores coating the disks and cause further disease.

Row eight shows an important special case. This case illustrates the formation of a new disk stack by an immature photoreceptor cell. This is the only way that a conically shaped disk stack can exist within a normal retina. Until a complete disk stack reaches the RPE, the conditions within the IS extrusion cup are not nominal and undersized disks may be formed. After the disks reach the RPE, the pressure within the IS extrusion cup is raised to the point that full diameter disks are formed routinely. Once these full size disks have traveled to the RPE, the disk stack maintains a constant diameter after that. It is such an immature photoreceptor cell that is sometimes labeled a “cone” although this designation would have no relevance to the absorption spectrum of the particular photoreceptor. 7.6.2 Summary of the dynamics

Rows 1 through 5 are seen to involve a variety of active processes that must occur if vision is to be achieved on a continuing basis: + The protein substrates must be formed by the photoreceptor cell on a continuing basis, approximately one disk per hour. + The RPE must extract Vitamin A from the vascular matrix, convert it into one of four chromophores, the Rhodonine, and release these materials into the IPM on a continuing basis. +The RPE must extract bio-energetic material, primarily that associated with the glutamate cycle, from the vascular matrix and release it into the IPM on a continuing basis. Dynamics of Vision 7- 289 + Both the Rhodonines and the bio-energetic materials must diffuse through the IPM to their target locations on a continuous basis. + The reactant products of the bio-energetic processes must diffuse away from the reaction location and be disposed of on a continuous basis. + The disks, upon reaching the vicinity of the RPE, must be disposed of properly. The chromophoric material can be recycled within the RPE. The protein material must be transported to the vascular matrix where it may be recycled back to the photoreceptor cells or used for other purposes in the body. Failure of any of the above actions to occur will result in a pathological condition, if not blindness. Rows 6 & 7 describe a common event in vision, the shearing of the retina at a plane through the Outer Segments. This condition, described as a retinal tear, has an interesting pathological course. The immediate symptom is a localized de-focus in the visual image. If not repaired promptly, the IPM is subject to mixing with the vitreous humor and/or the INM. This can significantly disrupt the diffusion processes described above. It does not significantly affect exocytosis and extrusion of the protein substrates. If the tear is repaired promptly, the long term prognosis is interesting. The integrity of the IPM is restored. Diffusion of the bio-energetics and chromophores is restored. The formation of new functional disks re- starts. The disk stacks continue to proceed toward the RPE and the death of the oldest disks. After approximately seven days, all signs of physical damage to the retina are removed. Whereas Row 6 shows a broken stack, Row 7 shows a disrupted stack. Because of the close packing of Outer Segments, such a disruption is limited in extent. The surrounding disk stacks confine the disrupted stack in-vivo and phagocytosis occurs in due time. It is only in-vitro that a drastic disruption of a disk stack is likely.

Row 8 shows an interesting special case. A short conical shaped Outer Segment is shown. It has not reached the region of the RPE. Such a situation may be a sign of an immature growth process or a pathological abnormality. If it is an immature situation, the Outer Segment will continue to grow, achieve a more constant diameter and eventually reach the RPE. Upon reaching the RPE, the disks will enter the region of phagocytosis. After about seven days in humans, such a Outer Segment will no longer exhibit a conical shape. If the reason for the conical shape is pathological or genetic, the effect on the operation of that overall photoreceptor cell is currently unreported.

7.6.3 The visual cycle of the Rhodonines

As discussed in an earlier chapter, this Section illustrates the normal closed cycle use of the Rhodonines. They are formed within the RPE cells from Vitamin A received from the vascular matrix and stored in the chromophore granules as esters until needed. When needed, they diffuse to the extrusion region of the Inner Segments of the photoreceptors where they are deposited as a liquid crystalline coating in their final chemical form on the protein substrates. They are physically transported back to the RPE cells by the growth of new disks pushing the disk stacks into the cup of the RPE cells where phagocytosis takes place. Following phagocytosis, the Rhodonines are returned to the chromophore granules ready for reuse. This cycle takes about seven days in the human. Additional Vitamin A is only required as a makeup material when the Rhodonines are degraded beyond recovery. Such degradation is apparently significant and new Vitamin A is required continuously.

7.7 The hydraulic, metabolic features associated with the electrostenolytic system

The bulk of Section 7.7 has been moved to Section 8.6.

7.7.5 Tracking respiration related to the neural system

Early work on understanding the operation of the neural system was based largely on tracking the flow of a few nuclear species contained in molecules able to cross the blood brain barrier. The work was focused on the use of labeled deoxyglucose. More recent work has expanded these studies using PET and MRI techniques. These latter techniques will be discussed in Section 7.7.6. 7.7.5.1 Use of the radionucleotide, [14C]deoxyglucose

Kennedy, et. al. explored a method of using [14C] deoxyglucose, abbreviated to [14C]DG, as a tracer to follow the 290 Processes in Animal Vision conversion of glucose into glucose-6-phosphate in neural tissue542. Sokoloff, et. al. have provided a mathematical demonstration of the efficacy, and a method of calibration of, the proposed method543. Their work discussed a number of conditions related to the effective use of the technique. However, the conditions appear acceptable within the laboratory environment. The method accurately represents the cumulative conversion of glucose into glucose-6-phosphate, a part of the stage 1 process described above. The process involves the integration of a rate sensitive mechanism. Calibration is most effective over a period of tens of minutes after injection of a charge of the radionucleotide into the blood stream of an animal. The technique is useful because the radionucleotide is initially processed just like glucose into a hexo-6- phosphate by the hexokinase enzyme. However, it becomes trapped within the most active tissue for a short period since it is not enzymatically processed beyond the 6-phosphate stage. Thus the animals had to be sacrificed in a timely manner and the neural tissue frozen immediately to prevent the rate sensitive information from being obscured. Radiographic techniques are then used to determine the location of the accumulation of the radionucleotides and the relative amount of the material at each location. Unfortunately, the method only applies to the initial portion of stage 1 of the process of powering the neural system. It does not allow the process to be traced through the additional steps associated with stage 2, 3 or 3.5. As a result, the technique is unable to differentiate between glucose-6-phosphate that is ultimately used to power the electrolytic portion of the neuron from any material used for respiration and possibly the physical formation of new synapses. Even with this shortcoming, the nucleotide has been extremely useful. It allows recording the differential respiration of thin layers of cerebral material in animals following light stimulation of the retina. These results will be summarized in Section 15.2.8. Film exposures involved in the process have usually been measured in days.

As Sokoloff, et. al. point out, with the development of computerized emission-based tomography, the technique offers potential for in-vivo experiments on humans as well as other animals. 7.7.5.2 Potential use of the radionucleotide, [14C]L-Dopa

Based on this work, it appears that another useful radionucleotide would be [14C]L-Dopa. This material is a mono- carboxylic amino acid with a dihydroxyphenyl ring. It is known to pass through the blood-brain barrier and has been shown to participate in the electrostenolytic process in place of glutamic acid. Therefore, this material offers the potential for tracing the flow of glutamic acid directly to the site of electrostenolytic power generation on an individual neuron and confirming the electrostenolytic reaction of glutamic acid to form GABA and carbon dioxide. Both exploratory activities similar to those of Kennedy, et. al. and an analysis similar to that of Sokoloff would be needed to determine the utility of this radionucleotide.

If the location of the radionucleotide could be specified on L-Dopa (that it was not associated with the carboxyl radical near the amino group), tracing the conversion of L-Dopa to a residue similar to GABA might be possible.

By tracing the L-Dopa to various local areas on an individual neuron and confirming the production of the L-Dopa residue, clear evidence for the role of glutamate in the powering of the neurons would be available. The tracing would represent the lower right most reaction in [Figure 7.7.2-3].

7.7.6 Physiological uncoupling between cerebral blood flow and metabolic oxygen consumption

The advent of PET, MRI and fMRI procedures has introduced a new era in imaging of the brain, and other parts of the body. These studies have demonstrated the critical role that glutamate plays in the operation of the neural system through its relative abundance in neural tissue. They have also uncovered the highly unexpected fact that the relative consumption of oxygen by the brain does not rise in proportion to the relative consumption of glycogen during neural activity. While glycogen consumption may rise by 40-50%, the rise in oxygen consumption rises by 5% or less. This finding highlights the fact the neurons of the brain do not rely upon oxidative metabolism during their short term operation. Oxygen’s primary role is in restoring the reactants used in the electrostenolytic process over a longer interval.

542Kennedy, C. Des Rosiers, M. Reivich, M. Sharp, F. Jehle, J. & Sokoloff, L. (1975) Mapping of functional neural pathways by autoradiographic survey of local metabolic rate with [14C]Deoxyglucose. Science, vol. 187, pp 850-853 543Sokoloff, L. Reivich, M. Kennedy, C. et. al. (1977) The [14C]Deoxyglucose method for the measurement of local cerebral glucose utilization. J. Neurochem. vol. 28, pp 897-916 Dynamics of Vision 7- 291 7.7.6.1 Background

Until 1986, the conventional wisdom was that the brain operated on an oxidative metabolism related directly to oxygen in its operation. In that year, Fox & Raichle demonstrated a significant decoupling between the rate of cerebral blood 544 flow (CBF) and the cerebral metabolic rate of oxygen consumption (CMRO2) . They hypothesized that the CBF was controlled by a mechanism independent of the cerebral metabolic rate of oxygen. Subsequent studies confirmed this startling decoupling545. Both Fujita and Vafee546 have provided some transient data on oxygen consumption both during and following neural activation. Confirmation of these results has led to a variety of explanations attempting to rationalize this coupling relationship between the CBF and CMRO2. To date, these studies have not recognized the role of glutamate in providing power to the neurons independent of the rate of oxidative metabolism. Until the advent of PET and MRI techniques, nearly all studies of brain activity were based on global calculations, frequently influenced by the restrictions introduced by the blood-brain-barrier. Since then, the studies have been much more local in nature. They are currently limited largely by the resolution of the PET and MRI techniques. This resolution is described in terms of the voxel ( the volumetric pixel).

The complexity of measuring the CBF and CMRO2 should not be underestimated. As will be shown below, indirect means are used followed by complex calculations. The investigators should be given great credit for achieving the precision illustrated in their data.

547 Buxton has provided a readable, but complex description of the methods required to determine the CBF and CMRO2 . His presentation is limited to the conventional wisdom concerning neural system operation. The following sections will deviate from his presentation in order to provide a broader context. This context will explain the decoupling found experimentally and suggest additional experimental activity.

7.7.6.2 Framework

The PET and MRI techniques rely upon the interactions of various constituents of organic tissue with crossed magnetic and radio frequency fields. These constituents incorporate a molecule that exhibits distinctive magnetic characteristics that can be recognized easily. The techniques used, and the associated mathematical processing, have advanced rapidly since the early 1990's. As with sonography in medicine, PET and MRI currently employ most of the techniques found in modern radar and sonar equipments. This level of sophistication makes it difficult to describe all of the optimization techniques used in PET and MRI. Buxton has provided an introduction to many of these specialized techniques as well. The signal sensed by the simplest equipments is in what is generally called the spatial frequency domain. The more advanced machines sense signals in the spatial frequency domain under transient temporal conditions. Converting these signals into a spatial position domain is necessary prior to interpretation. This requires the use of the two-dimensional Fourier Transform. This in turn requires considerable computational capability only available with the largest available computers. To conserve on computational power, or computational time until an answer is available, the Fourier Transform process is frequently truncated. This results in the Gibbs phenomenon familiar to all electrical circuit designers. Edges are emphasized in the imagery in spite of the underlying data.

Because of the four-dimensional nature of the most desirable signals, presentation of the resulting data is frequently a problem. The most common presentations present the desired transient data overlaid on a static presentation to provide visual reference to the voxels of interest. Frequently, the overlay is in a contrasting color for clarity.

7.7.6.2.1 Nomenclature

The nomenclature used in PET and MRI revolves around the notion of the voxel, the volumetric pixel of organic tissue.

544Fox, P. & Raichle, M. (1986) focal physiological uncoupling of cerebral blood flow and oxidative metabolism during somatosensory stimulation in human subjects. Proc Natl Acad Sci USA vol. 83, pp 1140- 1144 545Fujita, H. Hiroto, K. Reutens, D. & Gjedde, A. (1999) Oxygen consumption of cerebral cortex fails to increase during continued vibrotactile stimulation J. Cereb Blood Flow Metab vol. 19(3) pp 266-271 546MVafee, M. Meyer, E. Marrett, S. Paus, T. Evans, A & Gjedde, A. (1999) Frequency-dependent changes in cerebral metabolic rate of oxygen during activation of human visual cortex. J Cereb Blood Flow Metab vol 19(3) pp 272-277 547Buxton, R. (2002) Introduction to Functional Magnetic Resonance Imaging. Cambridge: Cambridge University Press 292 Processes in Animal Vision It is the metabolism of the tissue enclosed within an individual voxel that is of interest. To study this subject, the flow of nutrients into and out of the voxel must be quantified. The complexity of the tissue involved within an arbitrary voxel makes this process quite difficult. Some volume of the voxel is occupied by blood vessels transporting bulk blood. Some volume is occupied by the capillary bed providing blood components to the neural tissue. Finally, some volume is occupied by neural and other types of cellular material. The general approach involves two steps. The first step is to describe the cellular blood flow, CBF, into and out of the voxel. The second step is to describe the specific components of interest in the blood both entering and leaving the voxel. To date, the additional specific components have been primarily related to the flow of glycogen and oxygen into and out of the voxel. As a result, the two most prominent measurements have been of the cerebral metabolic rate of glycogen introduction, CMRGlc, and the cerebral metabolic rate of oxygen consumption, CMRO2. Both of these are typically expressed on a local basis. This analysis will show that two other components are critical to the understanding of the local metabolism of the brain. The first is the cerebral metabolic rate of glutamate consumption, CMRGlu. Glutamic acid (or glutamate) is the fuel most directly involved in neural operation. The second is the cerebral metabolic rate of GABA consumption, CMRGABA. This is the primary waste produce of neural operation. These two materials participate in an electrostenolytic process on the surface of every conduit associated with every neuron.

7.7.6.2.2 Parameters involved in the Fourier transforms

Although not of great importance here, being aware of certain parameters used in optimizing MRI images is useful. The white matter (primarily myelinated stage 3 neurons), the grey matter (primarily unmyelinated stage 2 neurons and the necessary capillary beds) and the bulk cerebral fluids (frequently described as the cerebral spinal fluids, CSF) exhibit different time constants when relaxing after excitation by a radio frequency, RF, field. The nominal values for these factors are given in TABLE 7.7.6-1.

TABLE 7.7.6-1 TYPICAL MAGNETIC PARAMETERS FOR COMPONENTS OF THE BRAIN

Material M0 (arb. units) T1 (ms) T2 (ms) Gray matter 85 950 95 White matter 80 700 80 CSF 100 2500 250

M0 is defined as the equilibrium magnetization of the material. T1 is defined as the longitudinal relaxation time associated with the dominant magnetic species within the material. T2 is defined as the transverse relaxation time associated with the dominant magnetic species.

Also important are certain factors associated with the MRI apparatus and the transform calculations. One is the time between repetitions of the scanning operation, TR. The second is the echo time, TE. The relationship between these factors is presented in Edelman, et. al548. They are basically used in optimizing the power of the machine and in “weighting” the Fourier Transform calculations described above. The result is higher contrast in the reconstructed images for the features of interest. These weightings are frequently related to the following Figure 7.7.6-1.

7.7.6.2.3 Metabolic Mechanisms and Figure 7.7.6-1 Simplified diagram of image contrast as a function of TR & TE. From Edelman, et. al. 1996.

548Edelman, R. Hesselink, J. & Zlatkin, M. (1996) Clinical Magnetic Resonance Imaging, 2nd Ed. vol. 1. London: W. B. Saunders. Chapter 1 Dynamics of Vision 7- 293 Nomenclature

Workers in the field of developing PET and MRI techniques have used a simplistic view of the metabolic processes of the body. The initial assumption that complete oxidation of glycogen was to be expected within the brain does not recognize the multitude of serial steps involved in metabolism, or the multitude of alternate paths supporting that metabolism. A more complete illumination of the steps in metabolism and the results of this work highlighting the role of glutamate in neural operation leads to a different interpretation of metabolism in the neural system. The overall concept of the utilization of food and oxygen in the support of life is generally defined as respiration. Metabolism is an important phase of respiration centered at the operations at the cellular level. Our understanding of metabolism, although remarkable, remains at a primitive level. The complexity is recognized inthe nomenclature of Lehninger549. His chapter 14 begins with a discussion of “Intermediate metabolism.” By this term, he means metabolism as the sum total of an immense variety of intermediate steps and residues. He does not mean some intermediate stage of metabolism. Metabolism has historically been defined as the sum total of the enzymatic reactions occurring in the cell. These have been divided into four categories. 1. Extract chemical energy from the environment (food or sunlight). 2. Convert exogenous nutrients into building blocks of macromolecular components of cells.

3. Assemble the macromolecules into proteins, nucleic acids, lipids and other components.

4. Form and degrade those biomolecules required in specialized functions of cells.

The last category is very important in the neural system. One of those degradation processes is used to power each individual neuron. It is the process of electrostenolysis occurring on the surface of every neuron and involving the conversion of glutamic acid into GABA.

Within metabolism, the processes can also be broken down into functional categories. The two most common are:

1. Catabolism– An enzymatic degradation, largely by oxidative reactions of relatively large molecules. Process releases free energy as ATP.

2. Anabolism– An enzymatic synthesis of larger molecular components of cells from similar precursors. Requires energy in the form of ATP.

A third form of metabolism will be discussed below.

Fermentation is a major function within metabolism. It takes on a number of forms that are generally associated with catabolism. However, it includes both aerobic and anaerobic variants. In general, fermentation involves oxidation- reduction reactions that do not involve oxygen. No net oxidation of the fuel and residue of these reactions occurs. These reactions involve primarily rearrangement accompanied by the release of hydrogen, water or carbon dioxide. They frequently involve amination. The changes in energy level of the constituents are frequently small.

Glycolysis is a major activity within the fermentation function. It involves a myriad of individual steps that are difficult to annotate in a single figure. Glycolysis involves two major chemical sequences terminating in the generation of pyruvate or lactate (depending on the personal interests of the investigator). Lactate is particularly important in muscular activity because of its creation in an anaerobic environment and its ability to pass easily through cell walls. The interest here is focused more on pyruvate. Glycolysis can be explained if the processes involved are divided into three types. 1. Degradation of glucose to lactate (the carbon pathway). 2. Introduction of phosphate group (the phosphate pathway). 3. Oxidation-reduction (the electron pathway). The participation chemical energy, ATP, NAD, etc., supporting the conversions associated with the steps in the above

549Lehninger, A. (1972) Op. Cit 294 Processes in Animal Vision pathways must also be considered in a complete analysis. Following glycolysis, the chemistry of metabolism broadens immeasurably in complexity. The framework for this broadening was first defined by Krebs. He proposed the citric acid cycle as the basic element of the framework. This cycle begins with the creation of citric acid from pyruvate via an incredibly complex enzymatic molecule known to this day as “coenzyme A.” The cycle includes a variety of side loops, shunts and other poorly defined processes. One of particular interest here is the glutamate shunt discussed below. The general nature of the process of metabolism centered on the citric acid cycle of Krebs, also known as the Tri-Carboxylic-Acid (TCA) cycle, and glutamate is illustrated in Figure 7.7.6-2. The number of carbon atoms in each molecule is given in parentheses. Noting that the cycle involves multiple decarboxylations along the right hand side is important. The reconstitution of the six carbon citrate from the four carbon succinates and oxaloacetate is more complex than generally addressed in discussions of the cycle. The steps leading to the formation of pyruvate occur within individual cells. The tri-carboxylic-acid cycle occurs within the mitochondria. The glutamate-to-GABA transformation occurs on the surface of the membranes of the neuron as part of the electrostenolytic mechanism.

The glutamate shunt begins with an amination of α-ketoglutarate. The resulting α-amino-glutarate is more commonly known as glutamate. The second step in this shunt is unique in that it does not involve an enzyme in the molecular sense. It involves the decarboxylation of the glutamate on a special area of a cell membrane acting as a substrate. Whether this process requires the presence of pyridoxal phosphate as a coenzyme is unknown. However, lack of this material generally inhibits neural activity (and presumably other reactions associated with glycolysis or the TCA cycle) within the brain.

The two walls of the bilayer membrane forming the wall are asymmetrical at the molecular level. As a result, the membrane acts as an electrical diode. The electrostenolytic decarboxylation process occurring on this membrane generates a free electron on one side of the membrane. This electrostenolytic process is the power source for each electrical conduit within a neuron. It is the physical analog of the putative ion-pump proposed by Hodgkin & Huxley (1952).

Glutamate participates in a wide variety of enzymatic reactions within the organism. These reactions and the glutamate shunt are well documented550. Its support in powering the neurons has not been previously reported. According to Harper, the shunt is particularly important in the gray matter of the brain. The glutamate shunt includes a sub-loop of interest. The sub-loop involves the removal of ammonia from GABA by a pyridoxal-dependent enzyme. This ammonia can participate in a transamination of α-ketoglutarate to glutamate. This sub-loop is shown by the dashed line in the figure.

550Harper, H. (1975) Review of Physiological Chemistry, 15th Ed. Los Altos, CA: Lange Medical Publ. pg 378 Dynamics of Vision 7- 295

Figure 7.7.6-2 The Citric Acid Cycle focused on the glutamate shunt. The shunt involves a decarboxylation but no involvement of oxygen. The decarboxylation is part of the electrostenolytic process powering the neurons. This process generates a free electron. The reported close coupling between the amination of α-ketoglutarate and the transamination of GABA is shown by the dashed line.

Estimates appear in the literature suggesting as much as 80% of the glucose delivered to the brain is used in neuronal activity. The high utilization of glutamate within a neuron suggests the processes of glycolysis and glutamate formation may tax the capability of a single cell. Both Frahm, et. al551. and Magistreeti & Pellerin552 have suggested that a major role for astrocytes within the brain (and their familial neuroglia, Schwann cells in the peripheral neural system) is to aid the neurons by providing additional lactate. The solubility of lactate in intercellular space would allow easy movement of lactate from the astrocytes to the neurons. Although the views of these authors are conventional, their conclusions are compatible with the Activa and neuron of this work. Particularly in stage 3 projection neurons, there is a need for glutamate at locations quite distant from the soma of the neuron itself. Figure 7.7.6-3 provides a more detailed description of the events related to the glutamate shunt variant of the tri- carboxylic-acid cycle. It is divided into four major sections. The shaded area on the left describes the chemical activity on the surface of a neuron upon excitation. The basic event is the (reversible) conversion of a small part of a pool of glutamate into GABA with the release of an electron into the plasma of the neuron and the release of CO2. The steps related to the tri-carboxylic-acid (TCA or Krebs) cycle required to remove the GABA and restore the glutamate supply are enclosed in the large dashed box. These steps normally occur within the mitochondria of the cell. The extraction

551Frahm, J. Kruger, G. et. al. (1996) Dynamic uncoupling and recoupling of perfusion and oxidative metabolism during focal brain activation in man Magn Resonance Med vol. 35, pp 143-148 552Magistretti, P. & Pellerin, L. (1999) Astrocytes couple synaptic activity to glucose utilization in the brain New Physiol Sci vol. 14, pp 177-182 296 Processes in Animal Vision of glucose from the bloodstream and the formation of pyruvate are shown at upper right. These steps normally occur within the larger volume of the cell. The extraction of oxygen from the bloodstream is shown at the lower right. Synaptic excitation of any neuron results in a change in the potential of at least one of the signaling related plasmas within the neuron. If the activity leads to a reduction of the negative potential in any of the plasmas, the electrostenolytic power source for that plasma will attempt to restore the nominal potential by injecting additional electrons into the plasma. This process converts glutamate to GABA. If the potential has risen, the electrostenolytic process has the theoretical capability of extracting electrons from the plasma and causing GABA to be converted back to glutamate (as discussed below). The TCA box contains two distinctly different paths. The top row describes the replacement of glutamate through the extraction of glucose from the bloodstream. This is the path usually considered when discussing BOLD signal generation in the fMRI technique. However, it should be noted that a second path exists as shown in the second row. This path reconstitutes the glutamate from GABA without using any new material derived from glucose. Here, no BOLD signal, related to direct extraction of glucose from the blood stream and its conversion into glutamate, is found.

Figure 7.7.6-3 Trail of events supporting the electrostenolytic process in neurons. The two boxes at lower right support the BOLD Effect in fMRI experiments. The dashed subloop shows how ammonia can be removed from GABA and used immediately in the amination process forming glutamate.

Note that the extraction of glucose from the bloodstream and its conversion into glutamate does not involve oxidation involving oxygen. It merely involves a series of oxidative-reductions (which actually releases oxygen in the form of CO2).

The glutamate to GABA reaction releases CO2. The replacement of the glutamate in the pool necessarily involves the replacement of the lost CO2. However, the top path in the figure shows that the glutamate can be replaced using glucose from the bloodstream without any involvement of oxygen from the bloodstream. In this case, no CMRO2 signal associated with the BOLD signal is seen. Alternately, the glutamate can be regenerated without the participation of new glucose using the middle and lower paths of the figure. This method does require the acquisition of oxygen from some source. The lower row of the figure shows the potential source of oxygen via the bloodstream. The process involves the reconstitution of the reactant used in the transamination mechanism shown. This path has not been documented. However, it involves the removal of oxygen from hemoglobin within the capillary bed supporting the neuron. 7.7.6.3 Energy calculations related to metabolism Dynamics of Vision 7- 297 The normal tendency in PET and MRI studies is to expect the complete metabolism of glucose. This involves the consumption of considerable oxygen in the creation of 38 units of ATP, an energy carrier.

Glucose + 6O2 —> 6CO2 + 6H2O (+38 ATP)

Each unit of ATP in the above calculations contain 7500 calories. The various phases of metabolism involved in supporting neuronal activity are quite different situations. They do not involve the complete reduction of glucose. The goal of glycolysis is the production of pyruvate or lactate (a more easily stored form) that can be used within the tri-carboxylic-acid cycle (TCA). Glucose —> 2 Lactate Similarly, each cycle of the Krebs TCA cycle consumes no oxygen.

CH3COOH + H20 —>2CO2 + 8H These reactions only involve small changes in energy compared with that involved in complete oxidation. 298 Processes in Animal Vision

TABLE OF CONTENTS--4/30/17

7 Dynamics of Vision ...... 1 7.1 Characteristics & Dynamics of Retinoids in the body ...... 1 7.1.1 Introduction ...... 1 7.1.1.1 Overall Baseline ...... 1 7.1.1.1.1 Scenario requirements ...... 2 7.1.1.1.2 Reinterpretation of Data Base ...... 2 7.1.1.1.3 The BIG QUESTION–What is the shape of retinol in various environments...... 3 7.1.1.2 Terminology ...... 4 7.1.1.2.1 Enzymatic activity...... 5 7.1.1.2.2 Naming enzymatic Proteins...... 6 7.1.1.2.3 Transport Proteins ...... 6 7.1.1.2.4 Specifics related to TTR and its relationship to RBP...... 9 7.1.1.2.5 The CRBP’s of vision ...... 9 7.1.1.2.6 Structural Proteins ...... 10 7.1.1.3 Properties of the fundamental chromogens ...... 10 7.1.1.4 State of the ART in crystallography versus molecular modeling ...... 10 7.1.1.4.1 Critical role of disorder and delocalization in chromophore transport ADD ...... 11 7.1.1.4.2 Lack of chromophore planarity plays a critical role in crystallographyADD...... 12 7.1.1.4.3 Application of molecular modeling and visualization versus crystallography ...... 12 7.1.1.4.4 Unique 3rd order protein structures described via crystallography . 12 7.1.2 Transport Scenario for the retinoids...... 13 7.1.2.1 Transport of the visual modality retinoids within the body ...... 17 7.1.2.1.1 Summary premises of retinoid transport within the vascular system ...... 17 7.1.2.1.2 Ingestion or manufacture of Vitamin A...... 18 7.1.2.1.3 Visual retinoid transport within the vascular system EDIT...... 24 7.1.2.2 Transport of the retinoids within the retina...... 35 7.1.2.2.1 Transfer of the retinines from the SRBP + TTR to the RPE cells . . 37 7.1.2.2.2 Clearance of the residues of SRBP & TTR from the RPE cell area ...... 38 7.1.2.3 Potential buildup of drusen resulting in macular degeneration ...... 38 7.1.2.4 Transport of Rhodonine within the RPE/IPM space of the retina ...... 43 7.1.2.4.1 Nature of the RBP’s in the RPE...... 46 7.1.2.4.2 Criticality of IRBP based on genetic mutation testing ...... 46 7.1.2.4.3 Nature of IRBP in the IPM...... 47 7.1.2.4.4 Proportion of IRBP in the IPM ...... 48 7.1.2.4.5 Sources of IRBP in the IPM...... 48 7.1.2.5 Background: SRBP +TTR complex in non-visual applications ...... 49 7.1.2.6 Important extraneous material related to retinoic acid...... 51 7.1.3 A precise redefinition of the aspects of the Visual Cycle involving retinoids GOOD/EDIT ...... 51 7.1.3.1 Gross retinoid transport in vision ...... 52 7.1.3.2 The overall visual cycle related to homeostasis...... 55 7.2 Dynamics of radiation-chemistry and the photoreceptor cell ...... 61 7.2.1 Radiation Chemistry...... 63 7.2.2 Excitation of a Liquid Crystal...... 63 7.2.3 De-excitation of a Liquid Crystal ...... 64 7.2.4 Dynamics of excitation...... 64 7.2.4.1 The dynamics of photon absorption ...... 64 7.2.4.2 The dynamics of excitation/de-excitation (small signal case) ...... 64 7.2.4.2.1 Excitation/de-excitation with transport delay, the P/D Equation . . . 67 7.2.4.2.2 The Hodgkin Solution to the P/D Equation for s •F•t = 1.00 ..... 69 7.2.4.2.3 Other attempts to obtain a P/D Equation ...... 70 7.2.4.3 The dynamics of excitation/de-excitation (large signal case) ...... 71 Dynamics of Vision 7- 299 7.2.5 The dynamics of transduction to a current ...... 75 7.2.5.1 Details of the two-exciton process...... 77 7.2.6 Analysis of the excitation equation ...... 77 7.2.6.1 Parametric analysis of the P/D equation...... 78 7.2.6.1.1 Effect of temperature and incident flux on the delay in the response ...... 78 7.2.6.1.2 Determination of the effective absorption cross section of a photoreceptor ...... 79 7.2.6.1.3 Determination of the effective time constant...... 79 7.2.6.1.4 Effect of the experimental configuration on the time constant..... 80 7.2.6.2 Comparison with the literature ...... 80 7.2.6.2.1 Detailed comparisons...... 81 7.2.7 Noise measurements of Baylor et al...... 85 7.3 Dynamics of the physiological optics of vision ...... 85 7.3.1 Background ...... 85 7.3.1.1 The literature of the physiological optics ...... 85 7.3.1.2 Overview of the dynamics of the physiological optics ...... 86 7.3.1.3 Subsystems of physiological optics ...... 89 7.3.1.4 Block diagrams of the physiological optical system ...... 89 7.3.2 Terminology...... 95 7.3.2.1 Classification of the rotational motion of the eyes ...... 95 7.3.2.1.1 Classification of eye movement syndromes or complexes ...... 97 7.3.2.1.2 Classification of clinically observed eye movements–saccades .... 98 7.3.2.1.3 Classification of clinically unobserved eye movements–flicks and tremor ...... 99 7.3.2.2 Classification of the temporal characteristics of the motion of the eyes ...... 99 7.3.2.3 Defining the operating modes within the physiological optics subsystem ..... 99 7.3.2.3.1 Framework for discussions of pointing and the triad...... 99 7.3.2.3.2 Defining precedence within the physiological optics subsystem . . 101 7.3.2.4 The Law of Equal Innervation...... 102 7.3.2.5 Defining the motor unit...... 102 7.3.2.5.1 A functional description of the oculomotor muscle...... 102 7.3.2.5.2 Defining the fatigue factor in the oculomotor response ...... 103 7.3.2.6 Glossary...... 103 7.3.3 The physiology of the pointing subsystem ...... 107 7.3.3.1 Basic operating scenarios of different species...... 107 7.3.3.1.1 Static mode...... 107 7.3.3.1.2 Tremor mode ...... 108 7.3.3.1.3 Modified tremor mode...... 108 7.3.3.2 Fixation motion ...... 108 7.3.3.2.1 Continuous Tremor ...... 108 7.3.3.2.2 Slow Drift...... 109 7.3.3.2.3 “Flicking movements”...... 109 7.3.3.3 Saccadic Motion ...... 109 7.3.3.3.1 Small saccadic mode ...... 109 7.3.3.3.2 Large saccadic mode ...... 109 7.3.3.3.3 Pursuit Motion...... 109 7.3.3.3.4 Blanking of visual channels during large saccades ...... 110 7.3.3.4 Operating modes of the visual system associated with the physiological optics ...... 110 7.3.3.4.1 The coarse, type 0, (& autonomous) awareness mode ...... 111 7.3.3.4.2 The coarse, type 0, (& semi-autonomous) alarm mode ...... 111 7.3.3.4.3 The fine, type 1, (& autonomous) analytical mode ...... 111 7.3.3.4.4 The (sympathetic and instruction oriented) volition mode ...... 111 7.3.3.4.5 Accommodation, a fixed reference controlled servomechanism . . 112 7.3.3.4.6 The critical role of memory in POS operation...... 112 7.3.3.5 The pointing system in humans...... 112 7.3.3.5.1 The general scenario...... 113 7.3.3.5.2 Alternate models of the oculomotor portion of the POS ...... 113 7.3.3.5.3 Data related to physiological tremor ...... 113 7.3.3.5.4 Effect of stabilization...... 114 7.3.3.5.5 Inertial aspects of pointing...... 116 300 Processes in Animal Vision 7.3.4 Modeling the dynamics of the pointing system ...... 117 7.3.4.1 Developing the model...... 118 7.3.4.1.1 The Cook & Stark model as a point of departure ...... 119 7.3.4.1.2 The expanded oculomotor plant model ...... 121 7.3.4.1.3 The performance characteristics of the subsystem...... 125 7.3.4.2 Expansion of the Top Level Schematic of Vision...... 126 7.3.5 Measured open and closed loop performance of the oculomotor system ...... 126 7.3.5.1 The low frequency, wide angle(>6.2° diam.), case...... 126 7.3.5.2 The mid frequency, mid angle (1.2°

List of Figures

Figure 7.1.1-1 Goodman’s cartoon model of alternate loadings of the SRBP-TTR complex ...... 3 Figure 7.1.1-2 View of an a helix ...... 12 Figure 7.1.2-1 The currently identified retinoid binding proteins, RBP’s ADD...... 15 Figure 7.1.2-2 Breakdown of triglycerides by pancreatic lipase ...... 20 Figure 7.1.2-3 The general metabolism of Vitamin A...... 21 Figure 7.1.2-4 The general flow of Retinoids within the animal body ...... 22 Figure 7.1.2-5 Model of the structure of the hexameric complex (RBP)2-TTR containing retinol...... 26 Figure 7.1.2-6 Ribbon representation of the plasma holo-RBP (RBP4) molecule ...... 27 Figure 7.1.2-7 Ribbon representation of 3D holo-RBP in Blomhoff ...... 29 Figure 7.1.2-8 Peptide sequence of human RBP from Rask...... 30 Figure 7.1.2-9 Schematic outline of the various fragments and peptides ...... 31 Figure 7.1.2-10 Primary structure sequence alignment of the RBPs of six vertebrate species ...... 35 Figure 7.1.2-11 Proposed flow of chromogens (-phores) between the bloodstream and the disks ...... 37 Figure 7.1.2-13 Conceptual schematic of potential disease conditions (drusin buildup) associated with the vascular/RPE interface...... 40 Figure 7.1.2-14 Proposed transport of the retinoids within the RPE...... 43 Figure 7.1.2-15 Overall scheme for retinoid transformation for both chromophore formation and operation .... 45 Figure 7.1.3-1 Gross caricature of retinoid transport in vision...... 53 Figure 7.1.3-2 Details of the flow of retinoids supporting the outer segments via the RPE ADD ...... 56 Figure 7.1.3-3 A schematic of the homeostatic and transduction visual cycles...... 59 Figure 7.1.3-4 Block diagram of proposed homeostasis visual cycle in the chordate eye ...... 60 Figure 7.2.1-1 The morphology and electrophysiology of the photoreceptor cell from Section 10.8.5.3...... 62 Figure 7.2.4-1 Basic flow diagram, equivalent electronic circuit and applicable equations...... 66 Figure 7.2.4-2 Theoretical responses to an impulse as predicted by the photoexcitation/de-excitation equation . . 68 Figure 7.2.4-4 The idealized quantum efficiency of a photoreceptor cell as a function of irradiance ...... 72 Figure 7.2.5-1 The circuit diagram of the combined P/D and transduction process...... 76 Figure 7.2.6-1 Collage of delay data versus flux level and temperature...... 79 Figure 7.2.6-2 Fundamental current paths in a photoreceptor cell ...... 81 Figure 7.2.6-3 A comparison of the theoretical and measure OS currents...... 83 Figure 7.2.6-4 CR The dynamic characteristic of the current collected from the OS...... 84 Figure 7.3.1-1 Top level block diagram of the visual system of Chordata, particularly of man ...... 90 Figure 7.3.1-2 Simplified top level schematic of Chordata focused on vertical oculomotor functions...... 91 Figure 7.3.1-3 The dual nature of the pointing system seen from above ...... 92 Figure 7.3.1-4 The luminance, chrominance and appearance channels of the eye of normal and aphakic humans...... 94 Figure 7.3.2-1 A conceptual framework for discussing saccade amplitudes and temporal frequencies ...... 96 Figure 7.3.2-2 A conceptual framework for saccade angular rates and amplitudes ...... 97 Figure 7.3.2-3 Mirror stereoscope used in disparity vergence experiments...... 101 Figure 7.3.4-1 An initial block diagram of the oculomotor plant ...... 119 Figure 7.3.4-2 A caricature of the static push-pull operation of the ocular muscles...... 120 Figure 7.3.4-3 An expanded model of the oculomotor subsystem ...... 121 Figure 7.3.4-4 The oculomotor servo plant with driving neurons...... 124 Figure 7.3.4-5 The displacement and velocity profile of large-angle human optokinetics ...... 125 Figure 7.3.4-6 RECOPY Action potential firing rate required to maintain an angular position ...... 126 Figure 7.3.5-1 Saccadic duration (A) and maximum velocity (B) of human eye movement ...... 127 Figure 7.3.5-2 Spatial orientation of fine movements (<3°) of the two eyes...... 128 Figure 7.3.5-3 RESCAN Record of eye movements during steady fixation...... 129 Figure 7.3.5-4 Waveforms of tremor resolved into vertical and horizontal components ...... 130 Figure 7.3.5-5 Bandpass recording of tremor in the human...... 130 Figure 7.3.7-1 Potential scanning modes associated with the analytical mode...... 134 Figure 7.3.7-2 Caricature of photoreceptors scanning an edge near the visual acuity limit ...... 135 Figure 7.3.7-3 Candidate tremor waveforms...... 136 Figure 7.3.7-4 Amplitude spectrum calculated with Matlab’s FFT function from AOSLO ...... 138 Figure 7.4.1-1 Geometry of horizontal disparity...... 141 Figure 7.4.1-2 Figure from Blakemore with dashed arcs of best focus added...... 143 Figure 7.4.1-3 The visual fields of monocular, binocular and stereoptic vision ...... 144 Figure 7.4.1-4 A summary of the on-axis parameters of vision in object space ...... 145 Dynamics of Vision 7- 305 Figure 7.4.1-5 Plan view from above of the right ocular ...... 147 Figure 7.4.1-6 A simplified pointing schematic based on the revised Functional Diagram of human vision, ca 2002 ...... 151 Figure 7.4.1-7 A Re-classification of cues found in depth perception from Howard ...... 154 Figure 7.4.1-8 The just–discriminable depth threshold (detectable difference in depth ...... 158 Figure 7.4.1-9 An overview of 3D Information Extraction from two schools of thought ...... 159 Figure 7.4.1-10 Basic stimulus arrangement of Allison & Howard ...... 162 Figure 7.4.1-11 A horizontal horopter showing effect of accommodation and rotation of the eyes ...... 165 Figure 7.4.1-12 A simple horopter test set ...... 166 Figure 7.4.1-13 A typical empirical horopter frequently used in pedagogy ...... 167 Figure 7.4.1-14 Theoretical framework for displaying empirical horopter data ...... 168 Figure 7.4.1-15 Caricature of an empirical horopter based on stereoacuity ...... 169 Figure 7.4.1-16 The optimal horopter for stereopsis discussions ADD...... 170 Figure 7.4.3-1 Caricature of depth perception at a bowling alley...... 178 Figure 7.4.3-2 A caricature introducing tremor to explain the mechanism providing stereopsis ...... 179 Figure 7.4.3-3 Caricature of vergence control problem...... 181 Figure 7.4.3-4 Block diagram of the complete horizontal vergence system of human vision...... 183 Figure 7.4.3-5 Averaged disparity vergence responses obtained for 2-degree convergent disparity pulses ..... 187 Figure 7.4.5-1 Stereoacuity as a function of horizontal offset...... 196 Figure 7.4.5-2 Stereopsis as a function of field angle within the foveola...... 197 Figure 7.4.5-3 Top level block diagram optimized for stereopsis...... 198 Figure 7.4.5-4 A more detailed schematic of the top level block diagram focused on precision stereopsis ..... 199 Figure 7.4.5-5 State diagram for the stereopsis mechanism ...... 201 Figure 7.4.5-6 The associative correlator of the PGN...... 204 Figure 7.4.5-7 The geometry of the stereopsis mechanism in object space...... 207 Figure 7.4.5-8 The geometry of binocular projection and definition of disparity ADD ...... 213 Figure 7.4.6-1 Fusion as a function of peripheral angle in the normal eye...... 218 Figure 7.4.6-2 Relative depth perception as a function of the binocular disparity of a target under dichoptic conditions ...... 224 Figure 7.4.6-3 Region of fusion versus spatial frequency...... 225 Figure 7.4.7-1 A stereogram from Howard & Rogers with added parameters ...... 228 Figure 7.4.7-2 Perceived depth as a function of vertical line interval...... 230 Figure 7.4.7-3 Potential motion in depth mechanisms...... 231 Figure 7.4.7-4 Example psychometric functions for dot patterns are shown for one observer ...... 233 Figure 7.4.8-1 Nodes and transit times affecting the latencies and response times ...... 236 Figure 7.4.8-2 Flow chart of latencies in the human visual system...... 239 Figure 7.4.9-1 Image of the lens and pupil taken from the position of the retina LARGE FILE...... 245 Figure 7.4.9-2 Block diagram of the accommodation servomechanism ...... 248 Figure 7.4.9-3 A nomograph describing the performance of the accommodation subsystem ...... 251 Figure 7.4.9-4 Accommodation performance of the “young” emmetrope eye ...... 252 Figure 7.4.9-5 Static biomechanical model of the lens servomechanism ...... 255 Figure 7.4.9-6 Accommodation efficiency based on data of Fincham ...... 258 Figure 7.4.9-7 Accommodation range of the human eye versus age...... 259 Figure 7.4.9-8 Dynamic biomechanical model of the accommodation plant ...... 260 Figure 7.4.9-9 Schematic of the plant of the lens servomechanism...... 262 Figure 7.4.9-10 Presbyopia as a normal process of aging ...... 263 Figure 7.5.1-1 Definition of saccades by size...... 270 Figure 7.5.2-1 Viewing pattern for a complex line drawing...... 275 Figure 7.5.3-1 The characters of text imaged on the foveola (black bar) and the fovea ...... 278 Figure 7.5.3-2 Hypothetical eye movement record showing the time in milliseconds ...... 280 Figure 7.5.3-3 The procedure of perceiving and interpreting a sentence ...... 281 Figure 7.5.3-4 Distribution of fixation duration for two subjects ...... 283 Figure 7.6.1-1 Illustrative cross section of the PC/RPE interface...... 287 Figure 7.7.6-2 The Citric Acid Cycle focused on the glutamate shunt ...... 295 Figure 7.7.6-3 Trail of events supporting the electrostenolytic process in neurons. The two boxes at lower right support the BOLD Effect ...... 296 306 Processes in Animal Vision

(Active) SUBJECT INDEX (using advanced indexing option) 2-exciton...... 71 3D...... 3, 4, 10-12, 29, 137, 155-161, 163, 164, 182, 183, 194, 212, 231 3-D...... 165, 203, 208, 218 95% ...... 74 action potential...... 102, 103, 118, 122, 126, 135-137, 243, 249 Activa...... 18, 57, 61, 64, 72, 74, 75, 78, 83, 84, 288, 295 adaptation . . . 1, 52, 57, 61, 63, 71, 72, 74, 75, 83-85, 87, 88, 116, 136, 163, 174, 181, 182, 184-188, 202, 246, 263- 265, 268, 271 adaptation amplifier...... 57, 61, 72, 74, 83, 84, 116, 163, 184, 186, 188 alarm mode...... 92, 105, 111, 112, 127, 145, 172, 173, 180, 208, 235, 237, 242, 243, 249, 266, 272 ammonia...... 294, 296 analytical mode ...... 85, 88, 92, 111, 112, 134, 145, 148, 172-174, 180, 189, 194, 199, 204, 235, 243, 269, 272 anatomical computation...... 203 area 7 ...... 237, 238, 240-242, 276, 277, 282 area 7a ...... 95, 237, 238, 244 associative correlator...... 105, 148, 171, 173, 179, 202-204, 214, 215, 217, 222, 223, 225, 227-229 associative memory ...... 204 astigmatism ...... 221, 253 attention...... 3, 84, 96, 97, 100, 103, 140, 158, 175, 177, 211, 226, 234, 241, 244, 246, 273, 277, 281 awareness mode...... 85, 88, 111, 144, 172-174, 194, 204, 217, 235-237, 269, 272 band gap...... 18, 76 Bayesian...... 4, 156, 210 Bayesian trap ...... 4 BBB...... 23, 24 bifurcation ...... 197 bilayer...... 288, 294 bilayer membrane...... 294 bit-serial...... 103 bleaching ...... 52, 64, 71, 73, 75, 85 blood brain barrier ...... 289 blood-brain barrier ...... 290 BOLD...... 296 broadband...... 204, 229 calibration...... 85, 101, 218, 257, 290 canonical forms ...... 2 cerebellum ...... 111, 151-153, 234, 237, 240, 243, 268, 276, 282 cerebrum...... 95, 106, 175, 244, 276, 282 chromostereopsis ...... 229 cis- ...... 5, 32, 47, 52, 53, 57 citric acid cycle ...... 294, 295 colliculus .... 87, 92, 95, 100, 102, 103, 105, 111, 112, 132, 133, 151-153, 172, 180, 197, 202, 203, 235, 237, 238, 241-244, 248, 260, 265-268 commissure ...... 151 compensation...... 115, 118, 184, 186, 257 complex neurons ...... 211 computation ...... 87, 160, 203, 206, 210, 212, 234, 242, 257, 268 computational.... 87, 99, 104, 108, 112, 113, 119, 139, 144, 156, 160, 181, 184-186, 191, 203, 206, 209, 210, 215, 226, 244, 257, 259, 268, 291 computational anatomy ...... 87 confirmation...... 85, 291 correlogram ...... 241 critical flicker frequency ...... 108, 271 cross section...... 18, 47, 63, 65, 66, 69, 71, 78, 79, 117, 256, 287 cyclopean...... 88, 92, 93, 99, 144, 146, 148, 161, 168, 172, 177, 179, 193, 203, 205, 206, 215, 223, 231 dark adaptation...... 71, 72, 75, 85 data base...... 2, 5, 203, 215 database ...... 4, 12, 32, 61, 89, 96, 111, 131, 199, 203, 259 declaratory memory...... 182, 234 Dynamics of Vision 7- 307 deoxyglucose ...... 289 depth perception . . 88, 93, 100, 104, 106, 139, 145, 146, 148-151, 153, 154, 161, 164, 167, 169-172, 177, 178, 182, 193, 194, 197, 198, 203, 205, 208, 210, 212, 214, 215, 222-231, 234, 247, 266 diencephalon ...... 198 dihedral...... 4, 10, 32, 33 diode...... 184, 294 diol...... 2, 17, 38 disparity . 87, 100-107, 139-141, 144-146, 148, 154, 156-161, 165, 166, 169, 171, 175-177, 180-187, 189, 194-197, 199, 203, 205, 209-215, 217, 218, 222-227, 229-233, 246, 266, 268 drusen...... 18, 32, 38, 39, 42 dynamic range ...... 73, 74 Edinger-Westphal...... 248, 249 efferent copy...... 132 electrostenolytic process ...... 123, 290, 292, 294-296 EOG...... 127 equilibrium ...... 2, 292 ERG...... 48, 80 evoked potentials ...... 218 evolution...... 10, 23, 35, 51, 52, 119 exocytosis...... 285, 288, 289 expanded ...... 7, 39, 40, 55, 78, 97, 121, 146, 158, 182, 184, 186, 206, 261, 277, 283, 289 exposine ...... 72 external feedback...... 85 feedback...... 85, 95, 99, 102, 115, 177 figure-ground ...... 156, 182, 190 flicker frequency ...... 108, 271 fMRI...... 133, 149-151, 240, 244, 290, 296 Fourier transform...... 187, 205, 206, 291, 292 free energy...... 293 frequency of occurrence...... 281 fuscin ...... 18 fusion frequency...... 96, 134 GABA ...... 290, 292-296 Gaussian...... 104, 139-144, 163, 164, 177, 193, 241 genetics...... 57 Gestalt ...... 156, 190 glutamate ...... 288, 290-296 glutamate shunt ...... 294, 295 glycolysis...... 293-295, 297 g-protein...... 81 half-amplitude ...... 132 Helmholtz Effect ...... 168 Hodgkin solution ...... 68, 69 horopter ...... 105, 141, 143, 144, 146, 155, 156, 163-172, 178, 189, 209, 228, 234 hydrogen bond ...... 25, 30-32, 36 hydronium ...... 64, 75, 78 hydronium liquid crystal ...... 64, 75 hyperacuity ...... 155, 156, 163 illusion ...... 209 intelligence ...... 190 interp ...... 105, 199, 202, 205, 214, 228, 279 intrafusal...... 118 inverse problem ...... 157 ion-pump ...... 294 knockout ...... 47, 48 lactate...... 293, 295, 297 larynx...... 190 latency ...... 68, 98, 114, 125, 149, 150, 186, 187, 234, 235, 242-244, 256, 259, 260, 273, 279, 280, 282, 283 lateral geniculate ...... 85, 149, 160, 182, 189, 190, 204, 211, 237, 248 lgn/occipital...... 111, 150-152, 172, 180, 217, 226 light adaptation...... 72 Limulus ...... 70, 80, 87 308 Processes in Animal Vision locomotion...... 108 long term memory ...... 112, 153, 269, 274 lookup table ...... 92, 199, 202, 206, 267, 279 machine vision...... 157, 194 macular degeneration...... 18, 32, 38-42 masking ...... 138 Maxwell’s Spot ...... 229 mean disparity ...... 157, 160, 183, 212, 214 mesencephalon...... 244 mesotopic...... 74, 246 microtubule ...... 75 microvilli ...... 4 midbrain . 85, 86, 94, 95, 100, 104, 111, 132, 133, 149, 179, 184, 193, 199, 202, 207, 210, 220, 226, 234, 240, 242- 244, 260, 265, 277 modulation ...... 232, 253 morphogenesis ...... 14 motif...... 14 MRI ...... 110, 244, 289-293, 297 multi-dimensional ...... 136 myelinated ...... 118, 292 narrow band...... 134, 193, 286 neurite...... 52, 57 neurites...... 57 neuroglia...... 295 night blindness ...... 22 nodal points ...... 104, 140, 142, 165, 170, 267 noise . . . 64, 70, 71, 83, 85, 99, 108, 114, 117, 129, 130, 135, 136, 138, 148, 150, 156, 160, 170, 174, 178, 186, 187, 204, 205, 211, 221, 222, 235, 242, 253, 263, 273, 280 non-declaratory memory ...... 182, 234 OCT...... 41, 137 Orangutan ...... 85, 221 orbital...... 35, 130 orbitals...... 4, 35 P/D equation...... 67-71, 78-80, 82-84, 160, 181, 182, 212 pain...... 258 parametric...... 74, 78 parietal lobe ...... 152, 161, 197, 237 pedestal...... 182 PEEP ...... 279 percept ...... 105, 152, 156, 190, 199, 202, 205, 210, 214, 217, 220, 279 perceptual space...... 106 perigeniculate . . 85, 87, 92, 94, 103, 105, 133, 139, 148, 149, 160, 163, 182, 184, 190, 202-204, 206, 210, 214, 215, 221, 222, 225, 227, 236, 248, 252, 253 perigeniculate nucleus . . . 85, 87, 92, 94, 103, 105, 133, 139, 148, 149, 163, 182, 184, 190, 202-204, 206, 210, 214, 215, 221, 222, 225, 227, 236, 248, 252 PET...... 289-291, 293, 297 pgn/pulvinar ...... 105, 111, 150-152, 237, 243, 244, 257, 260, 275, 277 phase velocity...... 132 phylogenic tree ...... 89 piezoelectric...... 123, 124 plasticity...... 228 podites ...... 288 point of regard ...... 146 POS . . 86-88, 90, 99, 100, 102, 103, 105, 106, 110-113, 117-119, 121, 127, 128, 131, 132, 136, 144, 148, 152, 165, 173-175, 180-182, 185, 199, 200, 202, 205, 208, 220, 221, 225, 234, 237, 238, 241, 244, 260, 273, 276, 279, 281, 282 post-holo-SRBP...... 7, 8, 38, 41, 54 Pretectal...... 91, 107 Pretectum...... 87, 105, 160, 183, 184, 236, 238 pre-holo-SRBP...... 7, 8, 55 probabilistic ...... 64 propagation velocity...... 118, 133 Dynamics of Vision 7- 309 protocol ...... 88, 112, 138, 162, 174, 232, 242-244, 264, 276 Pulfrich Effect ...... 160, 198, 211 pulvinar . 85, 87, 105, 110-112, 133, 146, 150-153, 161, 163, 173, 179, 180, 183, 193, 194, 197-199, 202, 203, 205, 206, 208, 210, 214, 217, 221, 225, 236-238, 240, 243, 244, 257, 260, 268, 275, 277 Pulvinar pathway...... 133, 238, 240 pyruvate...... 293, 294, 296, 297 quantum-mechanical ...... 55, 64 reading.....1, 11, 42, 85, 88, 128, 139, 149, 152, 179, 180, 193, 194, 197, 199, 200, 203, 206, 221, 225, 227, 235, 242, 243, 246, 250, 259, 263, 264, 268, 269, 271-274, 277-281, 283 recruitment...... 133 reflex arc ...... 234 residue ...... 7, 8, 12, 28, 35, 44, 290, 293 resonance...... 2, 17, 274, 291, 292, 295 retinine...... 2, 3, 6, 16-18, 28, 30-38, 49 retinitis pigmentosa ...... 39 ringing ...... 188 saliency map....102, 105-107, 110-112, 140, 152, 153, 155, 161, 172, 173, 179, 180, 193, 197-199, 202, 206, 208, 211, 212, 214, 216, 217, 242, 248, 249, 257, 259, 269, 273, 274, 277, 281 scotoma ...... 42 segregation...... 233 servo loop...... 92, 182 servomechanism.... 87, 88, 92, 99, 100, 102, 105, 107, 108, 111-113, 118-120, 123, 131, 132, 137, 138, 148, 172- 177, 180-182, 185-188, 220, 238, 244-249, 252-257, 259, 261, 262, 265, 268, 273 signal-to-noise ...... 64, 114, 136, 170, 187, 204, 205, 221, 222 signal-to-noise ratio...... 114, 136, 204, 205, 221 simple neurons...... 211 sleep...... 133 sphincter...... 253 square law ...... 77 SRBP ...... 2-4, 7-9, 14, 17, 21-25, 28, 31, 33, 36-42, 44, 49-51, 53-57, 60 stage 0 ...... 72, 198 stage 1 ...... 41, 74, 158, 160, 181, 186, 193, 212, 237, 248, 249, 265, 290 stage 2 ...... 152, 160, 170, 183, 185, 186, 237, 290, 292 stage 3 ...... 88, 112, 135-137, 177, 180, 183, 185, 186, 197, 211, 212, 237, 246, 265, 266, 292, 295 stage 4 ...... 146, 158, 160, 161, 172, 183, 190, 191, 193, 210-212, 232, 234, 237, 279 stage 5 ...... 161, 191, 197, 198, 211, 237, 265 stage 6 ...... 234, 237 stellate ...... 185 stereopsis . 85, 88, 91, 100, 103, 106, 111, 112, 139, 140, 142-146, 148-150, 153, 154, 156, 157, 161-166, 168-172, 175, 177-179, 183, 193-195, 197-212, 214-218, 220, 222, 226-229, 231-234 Stiles-Crawford ...... 140 stress...... 5, 9, 43, 48, 95, 113, 133, 156, 257, 263 striatum...... 234 superior colliculus . 87, 92, 95, 100, 102, 103, 105, 111, 112, 132, 133, 151-153, 172, 180, 197, 202, 203, 235, 237, 238, 241-244, 248, 260, 265-268 synapse...... 123, 124 syncytium...... 210 thalamic reticular nucleus...... 92, 105, 111, 149, 152, 161, 173, 198, 205, 206 thalamus.....85, 87, 94, 104, 110, 139, 140, 148, 152, 160, 193, 198, 204, 206, 215, 220, 221, 225, 234, 235, 237, 244 threshold...... 77, 134, 157, 158, 181, 182, 190, 194, 195, 197, 227, 233, 235, 243, 246, 271, 273 tomography ...... 41, 137, 290 topography ...... 203, 285 topology ...... 61, 285 torsion...... 33, 189, 215 transduction ...... 10, 18, 36, 52, 57, 59, 60, 63, 75-78, 80, 81, 233 transistor action ...... 75 translation...... 12, 116, 189, 190, 233, 280, 283 trans-...... 5, 7-10, 18, 32, 47, 51-53 tremor....85, 87-89, 95, 96, 99, 104-111, 113-115, 117, 119, 121-123, 129-139, 142, 155-157, 163, 173, 177, 179, 180, 182, 184, 189, 191, 199, 202, 206-208, 212, 214, 215, 217, 221, 222, 226, 227, 232, 248, 249, 252, 253, 257, 270, 271, 273, 277, 279, 280 310 Processes in Animal Vision tri-carboxylic-acid cycle ...... 294, 295, 297 TTR ...... 2, 3, 7-9, 14, 17, 18, 21-26, 28, 30, 31, 33, 35-42, 44, 49-51, 53, 54, 56, 57, 60 type 0 servomechanism ...... 87, 123, 131, 173, 273 type 1...... 111, 119, 173, 221 type 1 servomechanism ...... 173 type 2...... 119, 186, 253 type I ...... 42, 87, 187 type III...... 42 type IV...... 42 V2...... 226 VEP ...... 150 Verhoeff’s ...... 39, 41, 42 vestibular system ...... 95, 131, 132 Vieth-Muller...... 92, 103, 106, 140-143, 160, 164, 165, 167, 168, 170, 183 visual acuity...... 88, 135, 252, 257 visual cortex...... 85, 94, 150, 210, 212, 220, 226, 234, 237, 238, 291 vitamin A1...... 10, 16 vitamin A2...... 10, 16, 24 vitamin A3...... 10 volition mode ...... 92, 94, 102, 105, 111, 112, 127, 131, 137, 172, 175, 180, 202, 235, 237, 238, 241, 242 white matter...... 292 Wikipedia...... 2, 13, 18 xxx . . 4, 25, 49, 52, 55, 74, 79, 101, 104, 110, 130, 132, 137-139, 157, 160, 161, 164, 170, 174, 189, 197, 210, 218, 234, 249, 252, 267 X-ray ...... 3, 10-12, 26, 32, 33 [xxx . . 24, 25, 32, 35, 37, 38, 53, 74, 85, 109, 110, 128, 137, 162, 172-174, 204, 211, 214, 221, 232, 234, 245, 252, 253, 275, 283, 285