electronics Article May a Pair of ‘Eyes’ Be Optimal for Vehicles Too? Ernst D. Dickmanns Independent Researcher, 85649 Brunnthal, Germany; [email protected] Received: 13 March 2020; Accepted: 23 April 2020; Published: 5 May 2020 Abstract: Following a very brief look at the human vision system, an extended summary of our own elemental steps towards future vision systems for ground vehicles is given, leading to the proposal made in the main part. The question is raised of why the predominant solution in biological vision systems, namely pairs of eyes (very often multi-focal and gaze-controllable), has not been found in technical systems up to now, though it may be a useful or even optimal solution for vehicles too. Two potential candidates with perception capabilities closer to the human sense of vision are discussed in some detail: one with all cameras mounted in a fixed way onto the body of the vehicle, and one with a multi-focal gaze-controllable set of cameras. Such compact systems are considered advantageous for many types of vehicles if a human level of performance in dynamic real-time vision and detailed scene understanding is the goal. Increasingly general realizations of these types of vision systems may take all of the 21st century to be developed. The big challenge for such systems with the capability of learning while seeing will be more on the software side than on the hardware side required for sensing and computing. Keywords: automotive vehicles; autonomous driving; vision systems 1. Introduction The acceptance of technical vision systems by the human driver depends much on the differences in performance the overall system shows in comparison to his own capabilities. Therefore, in Section 1.1, a brief survey is given on some characteristic parameters of the human vision system. Section 1.2 then presents a review of the elemental steps towards future vision systems of ground vehicles, leading to the proposal made in Section4. In Section2, a brief survey of the reasons behind the di fferent developments in biology and technology is laid out. Section3 contains some conclusions for further developments. An outlook on potential future paths of development in automotive vision systems is given in Sections4 and5. Based on observations made, both in many biological species and in technical developments, two (of many possible) designs are looked at in some more detail. Neural net approaches are not taken into account for various reasons. The author is convinced that—as the 20th century has seen a wide spread of designs for ground vehicles—the 21st century will see the development of a wide variety of technical vision systems for vehicles, including deep neural nets. These developments are not a matter of years but of decades. 1.1. Characteristics of the Human Eye and Brain The larger part of the brain of highly developed vertebrates (including ourselves) is devoted to the processing of visual data for scene understanding, behavior decision and motion control [1–4]. Many biologists consider vision as one of the driving factors for developing intelligence, evolved by individuals with complex brains over millions of years [5]. Many different types and numbers of eyes are found per individual in a wide range of classes of animals. It is hard to believe that only chance is behind the fact that most animals with high-performance locomotion capabilities on land, in water and in the air have pairs of (mostly gaze-controllable) eyes with radial areas of different resolutions in Electronics 2020, 9, 759; doi:10.3390/electronics9050759 www.mdpi.com/journal/electronics Electronics 20202020,, 99,, 759x FOR PEER REVIEW 2 of 2827 resolutions in the images they provide. This modality of vision must have advantages (at least for thecarbon images-based they systems) provide. that This modalityhave evolved of vision over must many have generations. advantages (atFigure least for1 carbon-basedshows some systems)characteristics that haveof one evolved of these over solutions, many the generations. human eye Figure (from1 p.shows 5 in [6 some]). The characteristics total horizontal of onefield of of theseview solutions,(FoV) of a the single human eye eye is about (from p.145°; 5 in the [6]). image The total resolution horizontal decreases field of from view the (FoV) center of a singleto the eyeperiphery. is about The 145 central◦; the image area, dubbed resolution ‘fovea’, decreases a region from of the slightly center elliptical to the periphery. size, has a The visual central range area, of dubbedabout 1.5° ‘fovea’, vertically a region to ofabout slightly 2° ellipticalhorizontally. size, hasThere a visual are two range different of about types 1.5◦ vertically of sensor to elements about 2◦ horizontally.present in the There eye: are Color two sensitive different types‘cones’ of sensorand only elements-intensity present-sensitive in the ‘rods’. eye: Color The sensitivefovea contains ‘cones’ andexclusively only-intensity-sensitive cones, about 7 million ‘rods’. The(blue fovea curve contains in Figure exclusively 1) with cones,a high about density 7 million of about (blue 150 curve,000 inreceptors Figure1 per) with mm a2 high; this densityallows a of ‘Vernier about 150,000 acuity’ receptors in edge detection per mm 2in; this the allowsrange of a ‘Vernier 0.06 to 0.25 acuity’ mrad in edge(individual detection variations). in the range For of angles 0.06 to radially 0.25 mrad away (individual from the variations). fovea > 15° For (in anglesthe peripheral radially awayregions from of thethe fovearetina),> the15◦ density(in the peripheralof cones is regionsabout 1/30 of the of the retina), peak the value; density that ofmeans cones that is about the distance 1/30 of thebetween peak value;cones here that meansis five to that six the times distance that between between rods. cones here is five to six times that between rods. Figure 1.1. Distribution ofof rodsrods andand conescones alongalong aa line line passing passing through through the the fovea fovea and and the the blind blind spot spot of of a humana human eye eye (p. (p. 5 in5 in [6 []).6]). The fovea is employed for accurate vision in the direction toward where it is pointed by the The fovea is employed for accurate vision in the direction toward where it is pointed by the rotation of the eye. It comprises less than 1% of the retinal size, but uses over 50% of the visual cortex rotation of the eye. It comprises less than 1% of the retinal size, but uses over 50% of the visual cortex in the brain [7]. The ratio of ganglion cells to photoreceptors in the eye is about 2.5; almost every in the brain [7]. The ratio of ganglion cells to photoreceptors in the eye is about 2.5; almost every ganglion cell in the eye receives data from a single cone, and each cone feeds into one to three ganglion ganglion cell in the eye receives data from a single cone, and each cone feeds into one to three cells. Image preprocessing is performed in the eye so that the number of nerves feeding the primary ganglion cells. Image preprocessing is performed in the eye so that the number of nerves feeding the visual cortex in the back side of the human brain is reduced by two orders of magnitude compared to primary visual cortex in the back side of the human brain is reduced by two orders of magnitude the number of photoreceptors in the eye; edge orientations are coded directly in the eye to achieve this compared to the number of photoreceptors in the eye; edge orientations are coded directly in the eye reduction [8]. The eye does not take a photographic image of the environment, but rather produces in to achieve this reduction [8]. The eye does not take a photographic image of the environment, but parallel up to 100 extremely compressed full images of low precision for the peripheral FoV; in addition, rather produces in parallel up to 100 extremely compressed full images of low precision for the three to four small areas with high resolution are generated by foveal perception [9]. These data are peripheral FoV; in addition, three to four small areas with high resolution are generated by foveal compared with existing images in the actual imagination process, and both together are transformed perception [9]. These data are compared with existing images in the actual imagination process, and into the actual perception of the environment. both together are transformed into the actual perception of the environment. The human eye has three angular degrees of freedom: Two for gaze directions of the line of sight The human eye has three angular degrees of freedom: Two for gaze directions of the line of and one for a rotation around it. The latter, called roll angle, will be neglected here. The downward sight and one for a rotation around it. The latter, called roll angle, will be neglected here. The looking angle (infraduction or depression) may go to 60 , the upward looking angle (supraduction downward looking angle (infraduction or depression)− may◦ go to −60°, the upward looking angle or elevation) to +45 . The capability of the horizontal rotation may go on average up to 50 both (supraduction or elevation)◦ to +45°. The capability of the horizontal rotation may go on average◦ up towards the nose and away from it; however, in general the angles hardly exceed 20 . These changes to 50° both towards the nose and away from it; however, in general the angles ±hardly◦ exceed ±20°. in the viewing direction are supported by rotations of the head. Fast eye movements may go up to These changes in the viewing direction are supported by rotations of the head.