DOCSLIB.ORG

A Practical Bayer Demosaicing Algorithm for Gphoto

Total Page：16

File Type：pdf, Size：1020Kb

Download full-text PDF Read full-text
A Practical Bayer Demosaicing Algorithm for Gphoto A Practical Bayer Demosaicing Algorithm for Gphoto Theodore Kilgore Mathematics and Statistics Auburn University, Alabama 36849 Abstract The Gphoto project exists to provide compression has already taken place. With only few free and open-source software support for USB exceptions, a consumer-grade digital camera pro- and serial digital still cameras which use miscel- duces a color photo by having a grid of color filters laneous proprietary interface protocols and image placed between the microsensors and the lens. Of- data formats. The cameras supported range from ten, the color filters are also microlenses which better very cheap to very expensive, but one thing that capture the intensity of the light, as well as giving many of them need is a good and practical Bayer preference to a certain color. A standard practice demosaicing algorithm. The algorithm should get is to use red (R), green (G), and blue (B). The mi- good results with cheap cameras. The algorithm crosensors with their filters on them are arranged in should not require a memory footprint which is a pattern often called a Bayer array, named after unduly heavy, and the algorithm should not re- its inventor, B. E. Bayer. There are four possible quire excessive time to run, because gphoto should variants called tilings. A typical one looks like run even on fairly small computers. However, the priority is to support still cameras, not pccams row 0 RGRGRG ... (until w is reached) or webcams, which makes the execution time to row 1 GBGBGB ... (until w is reached) be a secondary issue, not a primary one. Re- and then the two-row pattern is repeated down- cently, a practical implementation of the adap- wards, until there are h rows in all. This pattern tive homogeneity-directed Bayer demosaicing al- is called RGGB, because it consists of repetitions of gorithm has been written, which meets the above a basic 2x2 tile which looks like criteria and seems to give very nice results. R G G B Keywords: Visual imaging, Bayer interpolation, demosaicing, Gphoto project Three other patterns, called BGGR and GRBG and GBRG, can achieve similar results. In all of these four Bayer tiling patterns there is twice as much 1 Introduction green data as there is of either red or blue. The reason for this is based upon known visual qualities Behind the lens of a digital camera is a sensor chip. of the human eye. In any event, the raw, uncom- On the surface of the sensor chip, a large number of pressed data from many digital cameras consists of photosensitive microsensors are arranged in a rect- precisely w ∗ h bytes of data arranged in one of the angular grid. For an uncompressed bitmapped im- standard Bayer tiling patterns. age, the rectangular array is w sensors wide and h Now, there is a fundamental problem. To con- sensors high, where w is the pixel width of the fin- struct an image requires data about each of the three ished image and h is its pixel height. Thus, for ex- colors at each pixel. In one of the basic, simple for- ample, a 640x480 image will be produced by an array mats used both in BMP image data and in PPM containing 480 rows of microsensors, in which each (Portable PixMap), one byte of data is required for row is 640 microsensors wide. each of the colors R, G, and B, at each pixel lo- At each of these microsensors, the intensity of cation in the image. Intensity is measured at each light striking it is recorded when a picture is microsensor, then, on a scale from 0 to 0xff for the snapped. The data may then be compressed. How- color which is sampled there. Consequently, we have ever, for the present purpose we will assume either only w∗h pieces of data, and we need to have 3∗w∗h that no such compression has taken place or that de- pieces of data (one byte for each color at the given pixel location) in our finished image. How to do more complicated procedures, such as cubic spline that? The short answer is: Use educated guesswork. interpolation instead of piecewise-linear averaging. We now discuss some possible methods for doing In practice, though, the greater complexity does not that. In fact we have a problem which is known seem always to do a very much better job. from the outset to have no exact solution, which to Some of the more sophisticated methods for do- some extent accounts for the huge proliferation of ing the interpolation involve such things as gradient methods for solving it. searches for the likeliest weighted linear combina- tion of readings at nearby pixels, or involve some basic schemes for edge detection and a method for 2 Methods of doing Bayer in- deciding to use some of the nearby readings and to terpolation. ignore others, in case there is suspicion of an edge. The method currently in use in the Gphoto project The most primitive method for supplying a missing was introduced in [6], in 2007. That method in- color at a pixel location is just to take one of the volves some vertical and horizontal edge detection nearest actual readings of the same color. Suffice and also gives a special role to the green data, as it to say that the crudity of the results thereby ob- there is twice as much green data as red or blue. tained is at least equal to the crudity of the method. Improvement over the previous bilinear interpola- tion method was obvious when this edge detection The most obvious approach is to use an averag- was introduced. Moreover, this improvement in ar- ing of nearest-neighbor data, called bilinear inter- tifact reduction and image clarity was done without polation. This method requires special treatment great increase in processing complexity or memory at edges and corners of the image, obviously, be- footprint. cause the number of nearest neighbors is not the same there. To describe the method with simplicity, therefore, let us assume we are well away from edges and corners. We might, for example, need to supply 3 Adaptive Homogeneity- a missing datum for G in the center of directed Demosaicing G1 Somewhere, while studying about the interpolation G2 C G3 or demosaicing problem, someone stepped back and G4 noticed that, in spite of the difficulties involved in in which C can signify that either R or B is the color re-creating the different colors at their missing loca- actually sampled. Other possible arrangements can tions all over an image, data about the entire image arise if we need to supply a missing datum for ei- is present nevertheless. For, at every pixel location ther R or B, giving minor but irritating variants on one piece of data has been gathered. However, for a the required procedure. However, in the situation long time it seems not to have been clear at all how just described the way to do bilinear interpolation this fact might actually be helpful toward a better is simply to take the average of G1, G2, G3, G4. Ob- reconstruction of the image. Indeed, it could be said viously, this procedure can give very nice results if without exaggerating very much, that the interpola- done in a region of an image where things are fairly tion methods described above have lost sight of the homogeneous, but the procedure can also act quite fact that data is present for the whole image, while badly, causing fuzziness, jaggedness, or other arti- concentrating on reconstructing the missing data for facting at edges in an image. One of the ugliest of the individual colors, viewed in isolation from one these artifacts is called the “zipper” effect, or the another. “checkerboard” effect, in which the edge comes out One method which has put these observations to as alternating pixels of different, contrasting colors good use is given in [1] of Hirakawa and Parks, in or light intensity. An example of the zipper effect 2005. In briefly describing what is done there, we may be seen in the first of the images in Section 5. will assume that the raw data from the camera has Bilinear interpolation was in use in the Gphoto been spread out already to a size of 3 ∗ w ∗ h, so that project for quite some time, the basic algorithm ap- three bytes are allocated for each pixel; the data al- pearing in libgphoto2/libgphoto2/bayer.c, [5] writ- ready present has been put into its proper, respective ten by Lutz M¨uller in 2001. In commercial camera locations, and the “missing” data is represented by software such as OEM camera driver software, bi- filler bytes which are still 0. This then is the input linear interpolation is often replaced by similar but of the procedure, which will fill in the 0 bytes with data and will give back the same data as an output very resource-intensive to implement. The first step but with the blank spaces now filled in. Then, the in the algorithm is to convert image h and image v, AHD algorithm proceeds as follows: each of which has been completely filled out in ac- First, make two additional copies of the image. cord with the previous steps, from RGB colorspace Let us call them image v and image h, because we to CIELAB colorspace, and the choice algorithm is are going to work in one of the copies only in the carried out there. The details have no place here, vertical direction and in the other one only in the and so they are omitted.
Load more

Recommended publications
  • Camera Raw Workflows
    RAW WORKFLOWS: FROM CAMERA TO POST Copyright 2007, Jason Rodriguez, Silicon Imaging, Inc. Introduction What is a RAW file format, and what cameras shoot to these formats? How does working with RAW file-format cameras change the way I shoot? What changes are happening inside the camera I need to be aware of, and what happens when I go into post? What are the available post paths? Is there just one, or are there many ways to reach my end goals? What post tools support RAW file format workflows? How do RAW codecs like CineForm RAW enable me to work faster and with more efficiency? What is a RAW file? In simplest terms is the native digital data off the sensor's A/D converter with no further destructive DSP processing applied Derived from a photometrically linear data source, or can be reconstructed to produce data that directly correspond to the light that was captured by the sensor at the time of exposure (i.e., LOG->Lin reverse LUT) Photometrically Linear 1:1 Photons Digital Values Doubling of light means doubling of digitally encoded value What is a RAW file? In film-analogy would be termed a “digital negative” because it is a latent representation of the light that was captured by the sensor (up to the limit of the full-well capacity of the sensor) “RAW” cameras include Thomson Viper, Arri D-20, Dalsa Evolution 4K, Silicon Imaging SI-2K, Red One, Vision Research Phantom, noXHD, Reel-Stream “Quasi-RAW” cameras include the Panavision Genesis In-Camera Processing Most non-RAW cameras on the market record to 8-bit YUV formats
    [Show full text]
  • Monitoring Glacial Advance and Retreat of the Skaftafellsjökull Glacier, Iceland
    Monitoring Glacial Advance and Retreat of the Skaftafellsjökull Glacier, Iceland Bryce L. Carmichael Amber E. Smith Elizabeth City State University North Carolina Central University Abstract- Visual documentation of glaciers can provide package gPhoto2 was installed to the Linux board daily, seasonal, and yearly statistics concerning their [www.gphoto.org]. Next, we wrote a set of custom scripts advance and retreat, as well as contribute to historical that ran periodically to capture photographs and archive them. record. Recognizing how glaciers change will improve glacier models, which leads to a better understanding of An all-weather case for the camera had to be climate and ice-sheet interactions. Obtaining frequent constructed in order to provide the device with protection and images of glaciers can be difficult since they are often power when out in the field. The picture feedback was located in remote locations with rugged terrain and harsh required to have protection also, so silica packs were placed in weather conditions. Access can be arduous even during the case to prevent moisture build-up on the camera lens. warm weather months. To overcome this obstacle, we propose building an autonomous imaging device that is II. BACKGROUND powered by solar panels and can withstand the harsh weather. The imaging device will have the ability to The Center for Remote Sensing of Ice Sheets capture images of the glacier at specified times, store them (CReSIS) Science and Technology Center was established by for uploading, and send them over a radio link to an the National Science Foundation (NSF) in 2005. The Internet access point.
    [Show full text]
  • Measuring Camera Shannon Information Capacity with a Siemens Star Image
    https://doi.org/10.2352/ISSN.2470-1173.2020.9.IQSP-347 © 2020, Society for Imaging Science and Technology Measuring camera Shannon Information Capacity with a Siemens Star Image Norman L. Koren, Imatest LLC, Boulder, Colorado, USA Abstract Measurement background Shannon information capacity, which can be expressed as bits per To measure signal and noise at the same location, we use an pixel or megabits per image, is an excellent figure of merit for pre- image of a sinusoidal Siemens-star test chart consisting of ncycles dicting camera performance for a variety of machine vision appli- total cycles, which we analyze by dividing the star into k radial cations, including medical and automotive imaging systems. Its segments (32 or 64), each of which is subdivided into m angular strength is that is combines the effects of sharpness (MTF) and segments (8, 16, or 24) of length Pseg. The number sine wave noise, but it has not been widely adopted because it has been cycles in each angular segment is 푛 = 푛푐푦푐푙푒푠/푚. difficult to measure and has never been standardized. We have developed a method for conveniently measuring inform- ation capacity from images of the familiar sinusoidal Siemens Star chart. The key is that noise is measured in the presence of the image signal, rather than in a separate location where image processing may be different—a commonplace occurrence with bilateral filters. The method also enables measurement of SNRI, which is a key performance metric for object detection. Information capacity is strongly affected by sensor noise, lens quality, ISO speed (Exposure Index), and the demosaicing algo- rithm, which affects aliasing.
    [Show full text]
  • Multi-Frame Demosaicing and Super-Resolution of Color Images
    1 Multi-Frame Demosaicing and Super-Resolution of Color Images Sina Farsiu∗, Michael Elad ‡ , Peyman Milanfar § Abstract In the last two decades, two related categories of problems have been studied independently in the image restoration literature: super-resolution and demosaicing. A closer look at these problems reveals the relation between them, and as conventional color digital cameras suffer from both low-spatial resolution and color-filtering, it is reasonable to address them in a unified context. In this paper, we propose a fast and robust hybrid method of super-resolution and demosaicing, based on a MAP estimation technique by minimizing a multi-term cost function. The L 1 norm is used for measuring the difference between the projected estimate of the high-resolution image and each low-resolution image, removing outliers in the data and errors due to possibly inaccurate motion estimation. Bilateral regularization is used for spatially regularizing the luminance component, resulting in sharp edges and forcing interpolation along the edges and not across them. Simultaneously, Tikhonov regularization is used to smooth the chrominance components. Finally, an additional regularization term is used to force similar edge location and orientation in different color channels. We show that the minimization of the total cost function is relatively easy and fast. Experimental results on synthetic and real data sets confirm the effectiveness of our method. ∗Corresponding author: Electrical Engineering Department, University of California Santa Cruz, Santa Cruz CA. 95064 USA. Email: [email protected], Phone:(831)-459-4141, Fax: (831)-459-4829 ‡ Computer Science Department, The Technion, Israel Institute of Technology, Israel.
    [Show full text]
  • Demosaicking: Color Filter Array Interpolation [Exploring the Imaging
    [Bahadir K. Gunturk, John Glotzbach, Yucel Altunbasak, Ronald W. Schafer, and Russel M. Mersereau] FLOWER PHOTO © PHOTO FLOWER MEDIA, 1991 21ST CENTURY PHOTO:CAMERA AND BACKGROUND ©VISION LTD. DIGITAL Demosaicking: Color Filter Array Interpolation [Exploring the imaging process and the correlations among three color planes in single-chip digital cameras] igital cameras have become popular, and many people are choosing to take their pic- tures with digital cameras instead of film cameras. When a digital image is recorded, the camera needs to perform a significant amount of processing to provide the user with a viewable image. This processing includes correction for sensor nonlinearities and nonuniformities, white balance adjustment, compression, and more. An important Dpart of this image processing chain is color filter array (CFA) interpolation or demosaicking. A color image requires at least three color samples at each pixel location. Computer images often use red (R), green (G), and blue (B). A camera would need three separate sensors to completely meas- ure the image. In a three-chip color camera, the light entering the camera is split and projected onto each spectral sensor. Each sensor requires its proper driving electronics, and the sensors have to be registered precisely. These additional requirements add a large expense to the system. Thus, many cameras use a single sensor covered with a CFA. The CFA allows only one color to be measured at each pixel. This means that the camera must estimate the missing two color values at each pixel. This estimation process is known as demosaicking. Several patterns exist for the filter array.
    [Show full text]
  • Linux Hardware Compatibility HOWTO
    Linux Hardware Compatibility HOWTO Steven Pritchard Southern Illinois Linux Users Group [email protected] 3.1.5 Copyright © 2001−2002 by Steven Pritchard Copyright © 1997−1999 by Patrick Reijnen 2002−03−28 This document attempts to list most of the hardware known to be either supported or unsupported under Linux. Linux Hardware Compatibility HOWTO Table of Contents 1. Introduction.....................................................................................................................................................1 1.1. Notes on binary−only drivers...........................................................................................................1 1.2. Notes on commercial drivers............................................................................................................1 1.3. System architectures.........................................................................................................................1 1.4. Related sources of information.........................................................................................................2 1.5. Known problems with this document...............................................................................................2 1.6. New versions of this document.........................................................................................................2 1.7. Feedback and corrections..................................................................................................................3 1.8. Acknowledgments.............................................................................................................................3
    [Show full text]
  • Proceedings of the 4Th Annual Linux Showcase & Conference, Atlanta
    USENIX Association Proceedings of the 4th Annual Linux Showcase & Conference, Atlanta Atlanta, Georgia, USA October 10 –14, 2000 THE ADVANCED COMPUTING SYSTEMS ASSOCIATION © 2000 by The USENIX Association All Rights Reserved For more information about the USENIX Association: Phone: 1 510 528 8649 FAX: 1 510 548 5738 Email: [email protected] WWW: http://www.usenix.org Rights to individual papers remain with the author or the author's employer. Permission is granted for noncommercial reproduction of the work for educational or research purposes. This copyright notice must be included in the reproduced paper. USENIX acknowledges all trademarks herein. The State of the Arts - Linux Tools for the Graphic Artist Atlanta Linux Showcase 2000 by Michael J. Hammel - The Graphics Muse [email protected] A long time ago, in a nerd’s life far, far away.... I can trace certain parts of my life back to certain events. I remember how getting fired from a dormitory cafeteria job forced me to focus more on completing my CS degree, which in turn led to working in high tech instead of food service. I can thank the short, fat, bald egomaniac in charge there for pushing my life in the right direction. I can also be thankful he doesn’t read papers like this. Like that fortuitous event, the first time I got my hands on a Macintosh and MacPaint - I don’t even remember if that was the tools name - was when my love affair with computer art was formed. I’m not trained in art. In fact, I can’t draw worth beans.
    [Show full text]
  • An Overview of State-Of-The-Art Denoising and Demosaicking Techniques: Toward a Unified Framework for Handling Artifacts During Image Reconstruction
    AN OVERVIEW OF STATE-OF-THE-ART DENOISING AND DEMOSAICKING TECHNIQUES: TOWARD A UNIFIED FRAMEWORK FOR HANDLING ARTIFACTS DURING IMAGE RECONSTRUCTION Bart Goossens∗, Hiep Luong, Jan Aelterman, Aleksandra Pižurica and Wilfried Philips Ghent University - TELIN - IPI - iMinds Sint-Pietersnieuwstraat 41, B-9000 Ghent, Belgium [email protected] ABSTRACT RAW Updated input image RAW image Pushing the physical limits of the camera sensors brings significant challenges to image post-processing techniques, Degradation model Impose prior knowledge which often have to work under various constraints, such as (blur, signal-dependent noise, demosaicking…) w.r.t. undegraded image (e.g., wavelets, shearlets) low-power, limited memory and computational resources. Other Iterative platforms, e.g., desktop PCs suffer less from these issues, solver Reconstructed allowing extra room for improved reconstruction. Moreover, the image captured images are subject to many sources of imperfection (e.g., noise, blur, saturation) which complicate the processing. In this paper, we give an overview of some recent work aimed at Fig. 1. A unified reconstruction framework. overcoming these problems. We focus especially on denoising, deblurring and demosaicking techniques. techniques have been developed that jointly solve several degradations (e.g., noise, demosaicking, blur, saturation) at I. INTRODUCTION once, using the mathematical framework of estimation theory, leading to much better reconstruction results. Several notable In the past decade, there has been a significant increase in examples are: joint denoising and demosaicking [1]–[4], joint diversity of both display devices and image acquisition devices, demosaicking and deconvolution [5]–[8], high dynamic range going from tiny cameras in mobile phones to 100 megapixel reconstruction and denoising [9], [10] and declipping and (MP) digital cameras.
    [Show full text]
  • Comparison of Color Demosaicing Methods Olivier Losson, Ludovic Macaire, Yanqin Yang
    Comparison of color demosaicing methods Olivier Losson, Ludovic Macaire, Yanqin Yang To cite this version: Olivier Losson, Ludovic Macaire, Yanqin Yang. Comparison of color demosaicing methods. Advances in Imaging and Electron Physics, Elsevier, 2010, 162, pp.173-265. 10.1016/S1076-5670(10)62005-8. hal-00683233 HAL Id: hal-00683233 https://hal.archives-ouvertes.fr/hal-00683233 Submitted on 28 Mar 2012 HAL is a multi-disciplinary open access L’archive ouverte pluridisciplinaire HAL, est archive for the deposit and dissemination of sci- destinée au dépôt et à la diffusion de documents entific research documents, whether they are pub- scientifiques de niveau recherche, publiés ou non, lished or not. The documents may come from émanant des établissements d’enseignement et de teaching and research institutions in France or recherche français ou étrangers, des laboratoires abroad, or from public or private research centers. publics ou privés. Comparison of color demosaicing methods a, a a O. Losson ∗, L. Macaire , Y. Yang a Laboratoire LAGIS UMR CNRS 8146 – Bâtiment P2 Université Lille1 – Sciences et Technologies, 59655 Villeneuve d’Ascq Cedex, France Keywords: Demosaicing, Color image, Quality evaluation, Comparison criteria 1. Introduction Today, the majority of color cameras are equipped with a single CCD (Charge- Coupled Device) sensor. The surface of such a sensor is covered by a color filter array (CFA), which consists in a mosaic of spectrally selective filters, so that each CCD ele- ment samples only one of the three color components Red (R), Green (G) or Blue (B). The Bayer CFA is the most widely used one to provide the CFA image where each pixel is characterized by only one single color component.
    [Show full text]
  • Index Images Download 2006 News Crack Serial Warez Full 12 Contact
    index images download 2006 news crack serial warez full 12 contact about search spacer privacy 11 logo blog new 10 cgi-bin faq rss home img default 2005 products sitemap archives 1 09 links 01 08 06 2 07 login articles support 05 keygen article 04 03 help events archive 02 register en forum software downloads 3 security 13 category 4 content 14 main 15 press media templates services icons resources info profile 16 2004 18 docs contactus files features html 20 21 5 22 page 6 misc 19 partners 24 terms 2007 23 17 i 27 top 26 9 legal 30 banners xml 29 28 7 tools projects 25 0 user feed themes linux forums jobs business 8 video email books banner reviews view graphics research feedback pdf print ads modules 2003 company blank pub games copyright common site comments people aboutus product sports logos buttons english story image uploads 31 subscribe blogs atom gallery newsletter stats careers music pages publications technology calendar stories photos papers community data history arrow submit www s web library wiki header education go internet b in advertise spam a nav mail users Images members topics disclaimer store clear feeds c awards 2002 Default general pics dir signup solutions map News public doc de weblog index2 shop contacts fr homepage travel button pixel list viewtopic documents overview tips adclick contact_us movies wp-content catalog us p staff hardware wireless global screenshots apps online version directory mobile other advertising tech welcome admin t policy faqs link 2001 training releases space member static join health
    [Show full text]
  • Learning Deep Convolutional Networks for Demosaicing Nai-Sheng Syu∗, Yu-Sheng Chen∗, Yung-Yu Chuang
    1 Learning Deep Convolutional Networks for Demosaicing Nai-Sheng Syu∗, Yu-Sheng Chen∗, Yung-Yu Chuang Abstract—This paper presents a comprehensive study of ap- for reducing visual artifacts as much as possible. However, plying the convolutional neural network (CNN) to solving the most researches only focus on one of them. demosaicing problem. The paper presents two CNN models that The Bayer filter is the most popular CFA [5] and has been learn end-to-end mappings between the mosaic samples and the original image patches with full information. In the case the widely used in both academic researches and real camera Bayer color filter array (CFA) is used, an evaluation on popular manufacturing. It samples the green channel with a quincunx benchmarks confirms that the data-driven, automatically learned grid while sampling red and blue channels by a rectangular features by the CNN models are very effective and our best grid. The higher sampling rate for the green component is con- proposed CNN model outperforms the current state-of-the-art sidered consistent with the human visual system. Most demo- algorithms. Experiments show that the proposed CNN models can perform equally well in both the sRGB space and the saicing algorithms are designed specifically for the Bayer CFA. linear space. It is also demonstrated that the CNN model can They can be roughly divided into two groups, interpolation- perform joint denoising and demosaicing. The CNN model is based methods [6], [7], [8], [9], [10], [11], [12], [13], [14] very flexible and can be easily adopted for demosaicing with any and dictionary-based methods [15], [16].
    [Show full text]
  • LINUX: Rute User's Tutorial and Exposition
    LINUX: Rute User’s Tutorial and Exposition Paul Sheer August 14, 2001 Pages up to and including this page are not included by Prentice Hall. 2 “The reason we don’t sell billions and billions of Guides,” continued Harl, after wiping his mouth, “is the expense. What we do is we sell one Guide billions and billions of times. We exploit the multidimensional nature of the Universe to cut down on manufacturing costs. And we don’t sell to penniless hitchhikers. What a stupid notion that was! Find the one section of the market that, more or less by definition, doesn’t have any money, and try to sell to it. No. We sell to the affluent business traveler and his vacationing wife in a billion, billion different futures. This is the most radical, dynamic and thrusting business venture in the entire multidimensional infinity of space-time-probability ever.” ... Ford was completely at a loss for what to do next. “Look,” he said in a stern voice. But he wasn’t certain how far saying things like “Look” in a stern voice was necessarily going to get him, and time was not on his side. What the hell, he thought, you’re only young once, and threw himself out of the window. That would at least keep the element of surprise on his side. ... In a spirit of scientific inquiry he hurled himself out of the window again. Douglas Adams Mostly Harmless Strangely, the thing that least intrigued me was how they’d managed to get it all done. I suppose I sort of knew.
    [Show full text]