DOCSLIB.ORG

United States Patent (10) Patent No.: US 8,094.208 B2 Myhrvold (45) Date of Patent: Jan

Total Page：16

File Type：pdf, Size：1020Kb

Download full-text PDF Read full-text
United States Patent (10) Patent No.: US 8,094.208 B2 Myhrvold (45) Date of Patent: Jan USOO8094208B2 (12) United States Patent (10) Patent No.: US 8,094.208 B2 Myhrvold (45) Date of Patent: Jan. 10, 2012 (54) COLOR FILTERS AND DEMOSAICING 6,025,213 A * 2/2000 Nemoto et al. ............... 438,122 TECHNIQUES FOR DIGITAL IMAGING 2002fOO97492 A1* 7, 2002 Cobb et al. .................... 359,577 2004/00325 17 A1 2/2004 Walmsley et al. (75) Inventor: Nathan P. Myhrvold, Bellevue, WA 2004/0095531 A1 5/2004 Jiang et al. (US) 2004/0263994 Al 12/2004 Sayag 2007/O145273 A1 6/2007 Chang (73) Assignee: The Invention Sciennce Fund I, LLC 2008.0137215 A1 6/2008 Nurishi ......................... 359/698 (*) Notice: Subject to any disclaimer, the term of this 2009/017.4638 A1 7, 2009 Elliott et al. patent is extended or adjusted under 35 OTHER PUBLICATIONS U.S.C. 154(b) by 273 days. -- “Tutorials: Diffraction & Photography': Diffraction Limited Photog (21) Appl. No.: 12/6.53,172 raphy: Pixel Size, Aperture and Airy Disks;printed on Oct. 29, 2009; (22) Filed: Dec. 8, 2009 pp. 1-5, located at: http://www.cambridgeincolour.com/tutorials, dif O O fraction-photography.htm. (65) Prior Publication Data PCT International Search Report; International App. No. PCT/US US 2011 FOO 13054A1 Jan. 20, 2011 10/02018; Nov. 10, 2010; pp. 1-4. Related U.S. Application Data * cited by examiner (62) Division of application No. 12/590,041, filed on Oct. 30, 2009. Primary Examiner —Yogesh Aggarwal (60) Provisional application No. 61/271,195, filed on Jul. (57) ABSTRACT 17, 2009. Color filter arrays or mosaics are provided for imaging a (51) Int. Cl. scene with diffraction limited optics. A distribution of color H04N 5/228 (2006.01) types in a color filter array is biased toward Smaller wave (52) U.S. Cl. ..................... 348/222.1; 348/273: 348/279; lengths to avoid or reduce loss of spatial resolution informa 348/280 tion at higher wavelengths due to a larger extent of diffraction (58) Field of Classification Search .................. 348/273, at the higher wavelengths. Demosaicing methods for recon 348/274, 279, 280 structing a partial or full color image from raw image data See application file for complete search history. involve applying correction factors to account for diffraction. The correction factors are based on pixel size and/or a mea (56) References Cited Sure of the extent of diffraction (e.g., an Airy disk diameter) for each wavelength in the color filter array. U.S. PATENT DOCUMENTS 5,159,650 A * 10/1992 Nishiwaki et al. .............. 385/37 32 Claims, 10 Drawing Sheets Method 800 810 Provide an array of N pixels (or pixel data set) having light intensity values of a scene sampled by an imaging device having an f-stop (f), wherein a first sub array of N1 pixels in the array of N pixels has light intensity sample values corresponding to a first spectral wavelength M 820 Modify the array of N pixels by modifying the light intensity sample values in one or more of the N1 pixels in the first sub array as a function of an Airy disk diameter 1.22"f X associated with light diffraction by the imaging device at wavelength X 830 Outputting the modified array of N pixels as an image representation of a SCere U.S. Patent Jan. 10, 2012 Sheet 1 of 10 US 8,094.208 B2 U.S. Patent Jan. 10, 2012 Sheet 2 of 10 US 8,094.208 B2 O009 U.S. Patent Jan. 10, 2012 Sheet 3 of 10 US 8,094.208 B2 U.S. Patent Jan. 10, 2012 Sheet 4 of 10 US 8,094.208 B2 N ?ueOS O r ======= U.S. Patent US 8,094.208 B2 U.S. Patent Jan. 10, 2012 Sheet 6 of 10 US 8,094.208 B2 00 CO U.S. Patent Jan. 10, 2012 Sheet 7 of 10 US 8,094.208 B2 O U.S. Patent Jan. 10, 2012 Sheet 8 of 10 US 8,094.208 B2 8"SDIH U.S. Patent Jan. 10, 2012 Sheet 9 of 10 US 8,094.208 B2 O U.S. Patent Jan. 10, 2012 Sheet 10 of 10 US 8,094.208 B2 O US 8,094,208 B2 1. 2 COLOR FILTERS AND DEMOSAICING categories and physical specifications of color are also asso TECHNIQUES FOR DIGITAL IMAGING ciated with objects, materials, light sources, etc., based on their physical properties such as light absorption, reflection, CROSS-REFERENCE TO RELATED or emission spectra. APPLICATIONS In the physical sense, a color is the ratio of intensities measured in different spectral components of the electromag The present application is related to and claims the benefit netic spectrum. In the physiological sense, in humans, the of the earliest available effective filing date(s) from the fol different spectral components are defined by the spectral lowing listed application(s) (the “Related Applications') sensitivity curves of three different types of receptors in the (e.g., claims earliest available priority dates for other than 10 eye (i.e., the so-called red, blue, and green cones). The human provisional patent applications or claims benefits under 35 brain processes and combines signals from the red, blue and USC S 119(e) for provisional patent applications, for any and green cones to create a composite impression or image of a all parent, grandparent, great-grandparent, etc. applications scene. All colors in the scene are perceived as combinations of of the Related Application(s)). All subject matter of the 15 the red, blue and green cone signals. The range or gamut of Related Applications and of any and all parent, grandparent, colors that are perceived by humans is represented, for great-grandparent, etc. applications of the Related Applica example, by the CIE 1931 chromaticity diagram. tions is incorporated herein by reference to the extent such Man-made color image sensors (e.g., color film, or digital Subject matter is not inconsistent herewith. cameras using CCD or CMOS sensors) also sense light inten RELATED APPLICATIONS sities in a finite number of distinct spectral components. Vari ous types of color image sensors differ in how they separate For purposes of the USPTO extra-statutory requirements, and measure the distinct spectral components. For example, a the present application constitutes a divisional of U.S. patent color film may have a stack of three different emulsion layers application No. 12/590,041, entitled COLOR FILTERS AND that are exposed by red, green, and blue components of light, DEMOSAICING TECHNIQUES FOR DIGITAL IMAG 25 respectively. A digital camera may use an array of layered ING, naming Nathan P. Myhrvold as the inventor, filed 30 sensors so that every pixel, like a color film, contains a stack Oct. 2009, which is currently co-pending, or is an application of sensors sensitive to individual colors (e.g., sensors avail of which a currently co-pending application is entitled to the able under the Foveon trademark). More commonly, digital benefit of the filing date. cameras or other man-made color image sensor arrangements The United States Patent Office (USPTO) has published a 30 use a spatial color filter array (e.g., a Bayer color filter array) notice to the effect that the USPTO's computer programs positioned on top of a CMOS or CCD sensor to capture require that patent applicants reference both a serial number different spectral components of light in corresponding nomi and indicate whether an application is a continuation or con nal pixel types. tinuation-in-part. Stephen G. Kunin, Benefit of Prior-Filed The man-made color image sensor arrangements sense and Application, USPTO Official Gazette Mar. 18, 2003, avail 35 collect intensity data for each distinct spectral component able at http://www.uspto.gov/web/offices/com/sol/og/2003/ received from a scene. The data for each spectral component week 1 1/patbene.htm. The present Applicant Entity (herein is monochrome, i.e., it includes only intensity information, after Applicant”) has provided above a specific reference to but no color information. To create an approximate color the application(s) from which priority is being claimed as image or rendition of a scene, the different intensity data for recited by statute. Applicant understands that the statute is 40 the distinct spectral components are processed, encoded to unambiguous in its specific reference language and does not certain colors and intensities, and combined. require either a serial number or any characterization, Such as A three-color Bayer color filter array is used in most single “continuation' or “continuation-in-part for claiming prior chip digital image sensors (in digital cameras, camcorders, ity to U.S. patent applications. Notwithstanding the forego and Scanners) to create a color image. A Bayer filter mosaic is ing, Applicant understands that the USPTO's computer pro 45 a color filter array for arranging red (R), green (G), blue (B) grams have certain data entry requirements, and hence color filters on a square grid of photosensors. The filter pattern Applicant is designating the present application as a continu is typically 50% green, 25% red and 25% blue. FIG.1, which ation-in-part of its parent applications as set forth above, but is adapted from Bayer U.S. Pat. No. 3,971,065, shows another expressly points out that such designations are not to be exemplary Bayer filter mosaic pattern with red, blue and construed in any way as any type of commentary and/or 50 green color transmissive filters arranged in a repeating pat admission as to whether or not the present application con tern. In this pattern, luminance elements (green, denoted by a tains any new matter in addition to the matter of its parent “G”) assume every other array position.
Load more

Recommended publications
  • Sensitive Text Icon Classification for Android Apps
    SENSITIVE TEXT ICON CLASSIFICATION FOR ANDROID APPS by ZHIHAO CAO Submitted in partial fulfillment of the requirements for the degree of Master of Science Thesis Advisor: Dr. Xusheng Xiao Department of Electrical Engineering and Computer Science CASE WESTERN RESERVE UNIVERSITY January, 2018 CASE WESTERN RESERVE UNIVERSITY SCHOOL OF GRADUATE STUDIES We hereby approve the thesis/dissertation of Zhihao Cao candidate for the degree of Master of Science. Committee Chair Xusheng Xiao Committee Member Andy Podgurski Committee Member Ming-Chun Huang Date of Defense Nov. 30 2017 *We also certify that written approval has been obtained for any proprietary material contained therein. Contents List of Tables 3 List of Figures 4 List of Abbreviations 7 Abstract 8 1 Introduction 9 2 Background 16 2.1 Permission System in Android…………………………………………………..16 2.2 Sensitive UI Detection in Android……………………………………………….17 2.3 Pixel and Color Model…………………………………………………………...19 2.4 Optical Character Recognition…………………………………………………...21 3 Design of DroidIcon 23 3.1 Overview…………………………………………………………..……………..23 3.2 Image Mutation…………………………………………………………………..24 3.2.1 Image Scaling………………………………………………………………24 3.2.2 Color Inversion…………………………………………………………….31 1 3.2.3 Opacity Conversion………………………………………………………..33 3.2.4 Grayscale Conversion………..…………………………………………….37 3.2.5 Contrast Adjustment……………………………………………………….42 3.3 Text Icon Classification 48 3.3.1 Text Cleaning……………………...……………………………………….48 3.3.2 Keyword Dataset Construction…………………………………………….49 3.3.3 Classification Algorithm…………………………………………………...50
    [Show full text]
  • Deep Sampling Networks
    Deep Sampling Networks Bolun Cai1?, Xiangmin Xu1??, Kailing Guo1, Kui Jia1, and Dacheng Tao2 1South China University of Technology, China 2UBTECH Sydney AI Centre, FEIT, The University of Sydney, Australia [email protected], fxmxu,kuijia,[email protected], [email protected] Abstract. Deep convolutional neural networks achieve excellent im- age up-sampling performance. However, CNN-based methods tend to restore high-resolution results highly depending on traditional interpo- lations (e.g. bicubic). In this paper, we present a deep sampling network (DSN) for down-sampling and up-sampling without any cheap interpo- lation. First, the down-sampling subnetwork is trained without supervi- sion, thereby preserving more information and producing better visual effects in the low-resolution image. Second, the up-sampling subnetwork learns a sub-pixel residual with dense connections to accelerate conver- gence and improve performance. DSN's down-sampling subnetwork can be used to generate photo-realistic low-resolution images and replace tra- ditional down-sampling method in image processing. With the powerful down-sampling process, the co-training DSN set a new state-of-the-art performance for image super-resolution. Moreover, DSN is compatible with existing image codecs to improve image compression. Keywords: Image sampling, deep convolutional networks, down/up- sampling. arXiv:1712.00926v2 [cs.CV] 26 Mar 2018 1 Introduction The aim of the image sampling is to generate a low-resolution (LR) image from a high-resolution (HR) image or reconstruct the HR image in reverse. Single- image up-sampling is widely used in computer vision applications including HDTV [1], medical imaging [2], satellite imaging [3], and surveillance [4], where high-frequency details are required on demand.
    [Show full text]
  • Deep Learning for Image Super-Resolution: a Survey
    1 Deep Learning for Image Super-resolution: A Survey Zhihao Wang, Jian Chen, Steven C.H. Hoi, Fellow, IEEE Abstract—Image Super-Resolution (SR) is an important class of image processing techniques to enhance the resolution of images and videos in computer vision. Recent years have witnessed remarkable progress of image super-resolution using deep learning techniques. This article aims to provide a comprehensive survey on recent advances of image super-resolution using deep learning approaches. In general, we can roughly group the existing studies of SR techniques into three major categories: supervised SR, unsupervised SR, and domain-specific SR. In addition, we also cover some other important issues, such as publicly available benchmark datasets and performance evaluation metrics. Finally, we conclude this survey by highlighting several future directions and open issues which should be further addressed by the community in the future. Index Terms—Image Super-resolution, Deep Learning, Convolutional Neural Networks (CNN), Generative Adversarial Nets (GAN) F 1 INTRODUCTION MAGE super-resolution (SR), which refers to the process and strategies [8], [31], [32], etc. I of recovering high-resolution (HR) images from low- In this paper, we give a comprehensive overview of re- resolution (LR) images, is an important class of image cent advances in image super-resolution with deep learning. processing techniques in computer vision and image pro- Although there are some existing SR surveys in literature, cessing. It enjoys a wide range of real-world applications, our work differs in that we are focused in deep learning such as medical imaging [1], [2], [3], surveillance and secu- based SR techniques, while most of the earlier works [33], rity [4], [5]), amongst others.
    [Show full text]
  • Movimenc(1) the Movim Video Codec Movimenc(1)
    movimenc(1) The MovIm video codec movimenc(1) NAME movimenc - MovIm encoder SYNOPSIS movimenc [input_options] -i input_file [encoding_options] [output_options] -o output_file movimenc -h DESCRIPTION MovIm is a video codec specifically designed for both conservation and restoration of moving images. The MovIm package includes the libmovim Clibrary implementing MovIm and its associated movimenc, movimdec and movimplay utilities, as well as the openmovim Bash command-line interface allowing to encode, decode, play and analyse virtually anymoving images. movimenc is a MovIm encoder. OPTIONS GENERAL OPTIONS -i input_file, --input=input_file All container formats and video codecs supported by FFmpegshould work. -o output_file, --output=output_file The uncompressed or lossless compressed MovIm data can be used directly as a file (.movim). This format is directly inspired from FFmpeg’sNUT container. INPUT AND OUTPUT OPTIONS --flip=(vertical|horizontal) flip the image on the vertical or horizontal axis This option may be repeated. --rotate=angle angle of counterclockwise rotation in degrees, expressed as an integer or a real number This option may be repeated. --lut[:channel]=path path to an 1D LUT or to a 3D LUT to apply LUTs can be applied to the input file and/or the output file. MoreoveraLUT can be applied to the whole file (default) or only to a single channel. This option may be repeated. For1DLUT,which transforms e.g. from floating-point scene linear into camera log or a display- referred space, the maximum allowed size is currently 16’777’216, that is 24-bit precision. ENCODING OPTIONS The following list is not exhaustive. --bit-depth[:channel]=bit_depth bit_depth can be anypositive integer We hav e tested mainly with 10, 12, 16 (default) or 24 per channel.
    [Show full text]
  • 3D LANCZOS INTERPOLATION for MEDICAL VOLUMES Thiago
    15th International Symposium on Computer Methods in Biomechanics and Biomedical Engineering and 3rd Conference on Imaging and Visualization CMBBE 2018 P. R. Fernandes and J. M. Tavares (Editors) 3D LANCZOS INTERPOLATION FOR MEDICAL VOLUMES Thiago Moraes1, Paulo Amorim1, Jorge Silva1 and Helio Pedrini2 1Division of 3D Technologies, Center for Information Technology Renato Archer Campinas-SP, Brazil, 13069-901 2Institute of Computing, University of Campinas Campinas, SP, Brazil, 13083-852 Keywords: Image Interpolation, Resampling, Medical Imaging. Abstract: Medical imaging techniques have been widely used in the diagnosis and treatment of many diseases, such as magnetic resonance imaging, computed tomography, mammography, ultrasonography, and photon emission tomography. Their main purpose is to enable the visu- alization of internal anatomic structures, such as organs and tissues, for clinical procedures. Resampling is required for certain medical tasks, for instance, registration and reconstruction, which occur when images or volumes are scaled, translated or rotated. The result of the re- sampling depends on the interpolation filter. In this paper, we develop and analyze a novel three-dimensional Lanczos resampling method in the context of medical imaging. 1. INTRODUCTION In the context of numerical analysis, interpolation techniques [12–15, 19, 21–23] aim to con- struct new data points from a set of known samples. In the field of image analysis, resampling techniques [9, 21, 23] transform a sampled image from one coordinate system to another. Resampling is typically employed to create a new version of an image with a different dimension. The increase and reduction of an image are called upsampling and downsampling, respectively. Image interpolation and resampling techniques play an important role in several biomed- ical applications, such as data reslicing, image registration, tomographic reconstruction, and volumetric rendering [23].
    [Show full text]
  • SATELLITE IMAGE ENHANCEMENT TECHNIQUE BASED on LANCZOS INTERPOLATION and NLFMT FILTERING Jino AR 1* , Jayaraj V 2 1M.E Student, Dept
    Jino and Jayaraj. UJEAS 2014, 02 (02): Page 103-107 ISSN 2348 -375X Unique Journal of Engineering and Advanced Sciences Available online: www.ujconline.net Research Article SATELLITE IMAGE ENHANCEMENT TECHNIQUE BASED ON LANCZOS INTERPOLATION AND NLFMT FILTERING Jino AR 1* , Jayaraj V 2 1M.E Student, Dept. of ECE, Nehru Institute of Engineering & Technology, Coimbatore, India 2Professor, Dept. of ECE, Nehru Institute of Engineering & Technology, Coimbatore, India Received: 28-02-2014; Revised: 26-03-2014; Accepted: 25-04-2014 *Corresponding Author : A. R. Jino M.E Student, 2Professor, Dept. of ECE, Nehru Institute of Engineering & Technology, Coimbatore [email protected], [email protected] ABSTRACT Satellite images are being used now a days in various fields such as weather forecasting, remote sensing, cartography, oceanography etc. Obtaining high quality images are of prime importance in such cases. But it may not be always possible due to the sensor limitations, atmospheric conditions and due to transmission. Thus the images suffer from the drawback of losing high frequency contents (which results in blurring). A wavelet domain approach based on method noise thresholding and non local means (NLM) is proposed for the enhancement of the images. An input image is passed through a NLM filter, then its method noise is thresholded. The NLM filtered image is summed with the thresholded method noise to cater for the image details removed by the NLM filter. The denoised image is interpolated using the Lanczos interpolator for resolution enhancement. Objective and subjective analyses reveal superiority of the proposed technique over the conventional and state-of-the-art image enhancement techniques.
    [Show full text]