BACHELOR THESIS Martin Safko Implementation of a tone mapping operator for scotopic viewing conditions Department of Software and Computer Science Education Supervisor of the bachelor thesis: doc. Alexander Wilkie, Dr. Study programme: Computer Science Study branch: Programming and Software Systems Prague 2019 I declare that I carried out this bachelor thesis independently, and only with the cited sources, literature and other professional sources. I understand that my work relates to the rights and obligations under the Act No. 121/2000 Sb., the Copyright Act, as amended, in particular the fact that the Charles University has the right to conclude a license agreement on the use of this work as a school work pursuant to Section 60 subsection 1 of the Copyright Act. In ........ date ............ signature of the author i I would like to thank my parents for supporting me during my studies. ii Title: Implementation of a tone mapping operator for scotopic viewing conditions Author: Martin Safko Department: Department of Software and Computer Science Education Supervisor: doc. Alexander Wilkie, Dr., Department of Software and Computer Science Education Abstract: Creating night-time images and movies that look plausible has been a problem in the industry since the creation of camera. To capture an image we need enough light to create a measurable quantity on a camera sensor. For this reason, shooting at night was not possible until sensors sensitive enough were developed and even then the captured images do not look realistic. Movie industry circumvent these issues by manually color correcting the footage in post- production. We implement an algorithm presented in a 2011 SIGGRAPH paper capable of solving this problem in a psycho-physically plausible and consistent way for spectral images and also augment it by a technique taken from a paper by INRIA. Keywords: human visual system tone mapping low light conditions iii Contents Introduction 2 Motivation . .2 Goals . .2 1 Human Visual System 3 1.1 Eye...................................3 1.2 Trichromacy . .4 1.3 Opponency model . .5 1.4 Rods . .7 1.5 Luminous efficiency function . .7 1.6 Visual acuity . .8 2 Implementation 9 2.1 Overview . .9 2.2 Algorithm . 10 2.2.1 Bilateral filter . 12 2.3 ART . 13 3 Results 14 Conclusion 17 Bibliography 18 List of Figures 19 A Attachments 20 1 Introduction In low-light environments, humans perceive color differently than in daylight conditions. However, devices and software that try to capture reality as we see it, for example cameras or photo-realistic renderers, are optimized for a daylight use and in dark environments produce unnatural looking images. This thesis aims to implement a tone mapping operator producing images that are perceptually similar to real life experience. The algorithm is based on a 2011 SIGGRAPH paper[1]. Motivation Creating plausible night-time looking images or movie scenes is not easy. Night shootings requires long exposures which is suitable only for static scenes. Other possibility is to use more sensitive equipment. Nevertheless, both of these tech- niques are either impractical or require very expensive equipment. Movie indus- try has come up with a process known as day for night to overcome these issues. The idea is to shoot in well lit scenes and later color grade the footage in post- production. Although the result looks good, it requires manual control and is typically not perceptually correct. From the scientific point of view, simulating human vision under different conditions can also help us better understand human visual system, especially perception of colors. By comparing the result of an experiment to the simulation we can build a model which can give us an insight into the inner workings of our brain. Goals The goal of this thesis is to implement a psycho-physically plausible tone mapping operator for low light spectral images and use it in an existing spectral renderer. We would also like to simulate loss of spatial resolution. 2 1. Human Visual System In this section we describe basic theory of vision and general colorimetry termi- nology. Out of all human senses, vision is the most complex one. Figure 1.1 shows the abstract path from light to color sensation. Light entering our eye is converted to a nerve signal which travels all the way to the back side of the brain where it is processed and perceived as a color sensation. We can study the physics of light and the biology and chemistry of an eye in great detail. Unfortunately, human brain is still mostly a black box. What we mean is that we do not yet have a complete understanding of neural pathways in the same way as we do with electrical circuits. Another thing to note is that no two people are the same and so everyone’s vision is a little different. In this text we shall focus on visual system of an average person with no defects. Figure 1.1: Abstract path from light to color perception 1.1 Eye Simplified diagram of an eye is shown in Figure 1.2. Light entering theeyeis focused by the lens onto retina, a light-sensitive layer located at the back of an eye. It is covered by two types of special cells, rods and cones. The distribution of rods and cones across retina in not uniform but follows the curve in Figure 1.3. Most of the cones are located at a spot called fovea, which is where our sharp vision is established. The other special point on the retina is the blind spot. It is a place where optical nerve is connected to the eye and so no rods and cones are residing there. 3 Figure 1.2: Diagram of an eye 1.2 Trichromacy There are three types of cone cells, L(long), M(medium) and S(short). Each one is sensitive to a different part of visible spectrum. Figure 1.4 shows the spectral sensitivity function of each cone type. Sometimes they are also referred to as red, green and blue cones given by the visible part of the spectrum they occupy. Interestingly, L and M types have very similar spectral response and slightly differ only at higher wavelengths. Given the spectrum entering an eye we can calculate response of each cone type by integrating the product of spectral power distribution and response function, as you can see in eq. 1.1 ∫︂ L = R(λ) l(λ) dλ λ ∫︂ M = R(λ) m(λ) dλ (1.1) λ ∫︂ S = R(λ) s(λ) dλ λ where l, m, s are the response functions, R is the spectrum of light and we integrate over the visible spectrum wavelengths. Human eyes have only three cone types which means the incoming spectrum is reduced to three quantities. As a result, different spectra can produce the same 4 Figure 1.3: Cone and rod distribution around fovea color response. In other words, two colored objects could look the same under a table lamp but be completely different under the sunlight. This phenomenon is called metameric failure[2]. This is generally undesirable which is why a color quality testing needs to be done under specific lighting conditions. On the other hand, trichromacy is very useful. From storing colored images to manufacturing display devices, we only need three quantities to specify color. 1.3 Opponency model Trichromacy is not enough to explain all phenomena of colored vision. One such example are color-blind people. Most common form is the red-green color blindness which means such a person has difficulty distinguishing between those colors. Other forms include blue-yellow and total color blindness. Other example is the absence of bluish yellow or reddish green colors. Similar ideas lead to the development of opponency model in the 19th century. It states that there are three channels of opposing values. Light / dark, red / green and blue / yellow channel. We can imagine instead of only positive signal there is negative and positive. We can use an analogy from electrical engineering with 0V/5V signals transformed into -5V/5V with 0V representing neutral value. Opponency model and trichromacy are not mutually exclusive theories but rather complement each 5 1.0 0.8 0.6 0.4 0.2 400 500 600 700 Figure 1.4: Normalized spectral sensitivity curves From left to right: short, medium and long wavelength cone response functions. Note that even though the long wavelength cones are sometimes referred to as ”red” cones, their integrated sensitivity corresponds to a greenish yellow color that is actually not very different from that of the medium wavelength cones. other. Trichromacy states how the signals are created and the opponency model specifies how those signals are transformed and sent to the brain. Diagramin Figure 1.5 shows how cone signals are combined. Although this is just a model, it explains many observed visual phenomena. Figure 1.5: Opponency model Note that rods also contribute to opponency model. Signal from rods is wired to the L+M channel, so we do not need a separate neural pathway for night vision. 6 1.4 Rods Similar to cones, rods also have a specific spectrum response function. However, the difference between rods and cones is in the amount of light at which theyare active. Humans use cones during the bright light situations. We call it photopic vision. On the other hand, during the night, we rely only on our rods vision, which we call scotopic vision. There is also a middle ground, mesopic vision, when both cones and rods contribute to visual perception. One thing to note is that we do not switch between scotopic and photopic vision instantly.