Eurographics Symposium on Rendering 2015 Volume 34 (2015), Number 4 J. Lehtinen and D. Nowrouzezahrai (Guest Editors) Modeling Luminance Perception at Absolute Threshold Petr Kellnhofer1 Tobias Ritschel1;2 Karol Myszkowski1 Elmar Eisemann3 Hans-Peter Seidel1 1MPI Informatik 2Saarland University 3Delft University of Technology 2 cd / m2 10-4 cd / m2 Figure 1: We simulate the change of image appearance between photopic conditions (left) and appearance in scotopic conditions close to the absolute threshold (right), where consistent vision fades into temporally varying (not reproducible in print) noise. Abstract When human luminance perception operates close to its absolute threshold, i. e., the lowest perceivable absolute values, appearance changes substantially compared to common photopic or scotopic vision. In particular, most observers report perceiving temporally-varying noise. Two reasons are physiologically plausible; quantum noise (due to the low absolute number of photons) and spontaneous photochemical reactions. Previously, static noise with a normal distribution and no account for absolute values was combined with blue hue shift and blur to simulate scotopic appearance on a photopic display for movies and interactive applications (e.g., games). We present a computational model to reproduce the specific distribution and dynamics of “scotopic noise” for specific absolute values. It automatically introduces a perceptually-calibrated amount of noise for a specific luminance level and supports animated imagery. Our simulation runs in milliseconds at HD resolution using graphics hardware and favorably compares to simpler alternatives in a perceptual experiment. Categories and Subject Descriptors (according to ACM CCS): I.3.3 [Computer Graphics]: Picture/Image Generation— Viewing algorithms 1. Introduction (e.g., games), the change of appearance is often simulated to The human visual system (HVS) adapts to absolute lumi- convey the illusion of a certain adaptation level despite being nance through several orders of magnitude; we can perceive in a different display condition. A skillful painter is able to a bright daylight scene as well as a moonless night. Appear- depict a scene shown on a photopic canvas as if it actually ance drastically changes for different absolute levels: at night was scotopic. The same holds for movies, where the so-called (scotopic) color and acuity are reduced and a shift towards “Day-for-night” effect is used since the early days of cinema. blue tones is perceived, when compared to the same scene in For computer applications, techniques like tone mapping can daylight (photopic) conditions. convey a scotopic impression. In all cases, it is important to point out that adaptation effects are qualitatively reproduced In visual arts, cinematography or interactive applications c 2015 The Author(s) Computer Graphics Forum c 2015 The Eurographics Association and John Wiley & Sons Ltd. Published by John Wiley & Sons Ltd. Kellnhofer et al. / Modeling Luminance Perception at Absolute Threshold and might differ in quantity: night scenes are blurred only In photopic conditions, cones are active. They have an enough to become noticeable and not as much as a strict HVS uneven distribution with a strong peak in the fovea and con- simulation would require, which would lead to an unpleasant tribute to color vision. They are inactive at night [Pal99], and viewing experience. we will assume they do not contribute to the scotopic effects modeled in this work. We also do not consider the mesopic Computer graphics has now routinely modeled the shift range, in which both rods and cones are active. from photopic over in-between mesopic to scotopic con- ditions [FPSG96, PFFG98, DD00, PTYG00, TSF02, KP04, In scotopic night vision, only rods are active. They have a KO11, WM14] but the scotopic regime (vision close to its different response to light of different wavelengths [Wal45] absolute threshold, e. g., a moonless night), has received little and do not contribute to color vision. Their peak density is attention. Remarkably, the absolute threshold is close to the lower than for cones, but their distribution over the retina physical limitations of light itself; most dark-adapted sub- is more uniform and shows a slower falloff with the retinal jects reliably detect flashes of light resulting in as little as 5 eccentricity. Modelling these differences is the foundation of to 10 photons total on the retina during an integration time of many day-for-night tone mappers [DD00,PTYG00,TSF02, 100 ms [HSP42]. Appearance under such conditions is sub- KP04, KO11, WM14]. stantially different from all other conditions. While scotopic vision can still rely on the retina as a classic noise-free sensor Rods convert light into nerve signals using a photo- described by scalar ray optics, for close to absolute threshold, chemical cascade. Each rod contains rhodopsin, that is isomer- receptor noise due to the particle nature of light becomes ized by exposure to light, resulting in a small change of poten- apparent and requires accounting for quantum statistics. tial to become a nerve signal [Alp71]. In each following step of the cascade, non-linear functions amplify the signal, while In this paper, we complement day-for-night tone mapping at the same time suppressing noise. The temporal aspects of to account for the effects encountered close to the absolute photo-transduction are the cause of afterimages [RE12]. threshold. After explaining the background in physics, neuro- science, and human vision (Sec. 2), as well as reviewing the Not all photons hitting a receptor are actually transduced state of the art in modeling human scotopic perception in com- into an electrical signal (false negative) because it might hap- puter graphics (Sec. 3), we propose a neurophysiologically- pen that the photon does not hit a rhodopsin molecule. The ra- motivated model of rod receptor noise which adds temporal tio of transduction (ca. 0.06–0.2, [HSP42]) is called quantum variations to the image, as expected to be experienced in efficiency. At the same time, it happens that rhodopsin is trans- scotopic conditions close to the absolute threshold (Sec. 4). duced in the absence of light (false positive) [Bar56]. These We then present the related computational aspects, involving aspects will be detailed in our treatment of near-absolute- a photon-accurate retinal image representation, an efficient threshold light levels in Sec. 4. rod noise generation drawn from image content-dependent Finally, other entoptic phenomena, which are not di- distributions, and a temporal rod-signal integration (refSec- rectly caused by light in the common way, such as floaters, OurApproach). Using graphics hardware, our model requires phosphenes, visual snow, the blue-field entoptic effect [RP80], 18 ms for an HD image and we compare our results to differ- or afterimages [RE12], can occur under specific conditions ent alternatives in a perceptual evaluation (Sec. 6). but are not related to scotopic vision and will not be modelled in this work. 2. Background Human vision is based on translating light into nerve signals. Light can be modeled as rays, waves, individual particles, or 3. Previous work as their quantum statistics. In this article, different from a In this section, we discuss tone-mapping solutions for night commonly taken viewpoint in computer graphics and vision, scenes. In particular, we focus on the role of perceived noise we choose the quantum-statistics point of view. Here, “light” in scotopic vision and, in this context, we overview other for a space-time interval is not a single value anymore, but sources of noise in images such as sensor noise, film grain, modeled as a distribution of probabilities to observe a certain and synthetically generated noise, which under certain con- number of quanta. ditions can improve perceived realism and quality. Finally, Light enters the human eye through the pupil, and is sub- we discuss photon-accurate eye modeling as is required near ject to different scattering and absorption events, before it absolute threshold, which is central for this work. reaches the retina, which is covered by receptors converting incoming light to electric signals. Two types of receptors, Tone mapping: Night scene depiction A key goal of tone cones and rods, exist, whose performance varies drastically mapping operators (TMO) is to reproduce scotopic scene with absolute luminance [Shl37, KWK09]. We have to distin- appearance on a photopic display [RWD∗10] by simulat- guish photopic (108 to 3 cd / m2), mesopic,(3 to 0:1 cd / m2), ing a blue-shift and the loss of color vision, visual acu- scotopic (0:1 to 10−6 cd / m2) vision, and scotopic vision ity, contrast and brightness characteristic for night vision close to absolute threshold (less than 10−3 cd / m2). [FPSG96, PFFG98, KP04, KO11, WM14]. Typically, such c 2015 The Author(s) Computer Graphics Forum c 2015 The Eurographics Association and John Wiley & Sons Ltd. Kellnhofer et al. / Modeling Luminance Perception at Absolute Threshold simulations cover higher levels of scotopic luminance (0.001– as an analog of retinal circuitry processing beyond rods and 0.1 cd / m2) including the transition to mesopic conditions, cones, which we ignore in this work. while luminance levels near absolute thresholds are not specif- The exposure and development of silver-halide crystals ically addressed. Furthermore, the time-course of adaptation dispersed in the emulsion of analog films results in forming [DD00, PTYG00], the bleaching phenomenon [GAMS05], tiny blobs of metallic silver or dye clouds, which creates the or stereovision in darkness [KRV∗14] were modeled in com- familiar film grain effect whose density tends to follow a puter graphics. Gaussian distribution [Alt77]. Film grain as a form of noise Nightly impressions have been convincingly reproduced might be manipulated or even intentionally added for tech- in painting [Liv02], digital arts, computer games, and feature nical or artistic reasons in movie post-production [Sey11] films without referring to any rigorous simulation of scotopic and can be acquired through film-stock scanning or synthe- vision.