IMAGE SENSORS and SIGNAL PROCESSING for DIGITAL STILL CAMERAS Edited by Junichi Nakamura Boca Raton London New York Singapore A CRC title, part of the Taylor & Francis imprint, a member of the Taylor & Francis Group, the academic division of T&F Informa plc. Copyright © 2006 Taylor & Francis Group, LLC DK545X_Discl.fm Page 1 Friday, June 24, 2005 12:47 PM Published in 2006 by CRC Press Taylor & Francis Group 6000 Broken Sound Parkway NW, Suite 300 Boca Raton, FL 33487-2742 © 2006 by Taylor & Francis Group, LLC CRC Press is an imprint of Taylor & Francis Group No claim to original U.S. Government works Printed in the United States of America on acid-free paper 10 987654321 International Standard Book Number-10: 0-8493-3545-0 (Hardcover) International Standard Book Number-13: 978-0-8493-3545-7 (Hardcover) Library of Congress Card Number 2005041776 This book contains information obtained from authentic and highly regarded sources. Reprinted material is quoted with permission, and sources are indicated. A wide variety of references are listed. Reasonable efforts have been made to publish reliable data and information, but the author and the publisher cannot assume responsibility for the validity of all materials or for the consequences of their use. No part of this book may be reprinted, reproduced, transmitted, or utilized in any form by any electronic, mechanical, or other means, now known or hereafter invented, including photocopying, microfilming, and recording, or in any information storage or retrieval system, without written permission from the publishers. For permission to photocopy or use material electronically from this work, please access www.copyright.com (http://www.copyright.com/) or contact the Copyright Clearance Center, Inc. (CCC) 222 Rosewood Drive, Danvers, MA 01923, 978-750-8400. CCC is a not-for-profit organization that provides licenses and registration for a variety of users. For organizations that have been granted a photocopy license by the CCC, a separate system of payment has been arranged. Trademark Notice: Product or corporate names may be trademarks or registered trademarks, and are used only for identification and explanation without intent to infringe. Library of Congress Cataloging-in-Publication Data Image sensors and signal processing for digital still cameras / edited by Junichi Nakamura. p. cm. Includes bibliographical references and index. ISBN 0-8493-3545-0 (alk. paper) 1. Image processing—Digital techniques. 2. Signal processing—Digital techniques. 3. Digital cameras. I. Nakamura, Junichi. TA1637.1448 2005 681'.418—dc22 2005041776 Visit the Taylor & Francis Web site at http://www.taylorandfrancis.com Taylor & Francis Group and the CRC Press Web site at is the Academic Division of T&F Informa plc. http://www.crcpress.com Copyright © 2006 Taylor & Francis Group, LLC DK545X_C000.fm Page 5 Wednesday, July 6, 2005 8:12 AM Preface Since the introduction of the first consumer digital camera in 1995, the digital still camera (DSC) market has grown rapidly. The first DSC used a charge-coupled device (CCD) image sensor that had only 250,000 pixels. Ten years later, several models of consumer point-and-shoot DSCs with 8 million pixels are available, and profes- sional digital single-lens reflex (DSLR) cameras are available with 17 million pixels. Unlike video camera applications in which the output is intended for a TV monitor and, thus, the vertical resolution of the image sensors is standardized, there is no standard output or resolution for DSCs, so the sensor pixel count continues to grow. Continuing improvements in sensor technology have allowed this ever increasing number of pixels to be fit onto smaller and smaller sensors until pixels measuring just 2.3 mm ¥ 2.3 mm are now available in consumer cameras. Even with this dramatic shrinking of pixels, the sensitivity of sensors has improved to the point that shirt- pocket-sized consumer DSCs take good quality pictures with the exposure values equivalent to 400 ISO film. Improvements in optics and electronics technologies have also been remarkable. As a result, the image quality of pictures taken by a DSC has become comparable to that normally expected of silver-halide film cameras. Consistent with these improvements in performance, the market for DSCs has grown to the point that shipments of DSCs in 2003 surpassed those of film cameras. Image Sensors and Signal Processing for Digital Still Cameras focuses on image acquisition and signal-processing technologies in DSCs. From the perspective of the flow of the image information, a DSC consists of imaging optics, an image sensor, and a signal-processing block that receives a signal from the image sensor and generates digital data that are eventually compressed and stored on a memory device in the DSC. The image acquisition part of that flow includes the optics, the sensor, and front-end section of the signal-processing block that transforms photons into digital bits. The remainder of the signal-processing block is responsible for gener- ating the image data stored on the memory device. Other technologies used in a DSC, such as mechanics, data compression, user interface, and the output-processing block that provides the signals to output devices, such as an LCD, a TV monitor, a printer, etc., are beyond the scope of this book. Graduate students in electronics engineering and engineers working on DSCs should find this book especially valuable. However, I believe it offers interesting reading for technical professionals in the image sensor and signal-processing fields as well. The book consists of 11 chapters: •Chapter 1, “Digital Still Cameras at a Glance,” introduces the historical background and current status of DSCs. Readers will be able to understand Copyright © 2006 Taylor & Francis Group, LLC DK545X_C000.fm Page 6 Wednesday, July 6, 2005 8:12 AM what DSCs are, how they have evolved, types of modern DSCs, their basic structure, and their applications. •In Chapter 2, “Optics in Digital Still Cameras,” a wide range of topics explains the imaging optics used in DSCs. It is obvious that high image quality cannot be obtained without a high-performance imaging optical system. It is also true that requirements for imaging optics have become higher as pixel counts have increased and pixel sizes have decreased. •Reviews of image sensor technologies for DSC applications can be found in Chapter 3 through Chapter 6. •First, in Chapter 3, “Basics of Image Sensors,” the functions and performance parameters common to CCD and complementary metal- oxide semiconductor (CMOS) image sensors are explained. •Chapter 4, “CCD Image Sensors,” describes in detail the CCD image sensors widely used in imaging applications. The chapter ranges from a discussion of basic CCD operating principles to descriptions of CCD image sensors specifically designed for DSC applications. •Chapter 5, “CMOS Image Sensors,” discusses the relatively new CMOS image sensor technology, whose predecessor, the MOS type of image sensor, was released into the market almost 40 years ago, even before the CCD image sensor. •Following these descriptions of image sensors, methods for evaluating image sensor performances relative to DSC requirements are presented in Chapter 6, “Evaluation of Image Sensors.” •Chapter 7 and Chapter 8 provide the basic knowledge needed to imple- ment image processing algorithms. •The topics discussed in Chapter 7, “Color Theory and Its Application to Digital Still Cameras,” could easily fill an entire book; however, the emphasis here is on how color theory affects the practical uses of DSC applications. •Chapter 8, “Image-Processing Algorithms,” presents the algorithms utilized by the software or hardware in a DSC. Basic image-processing and camera control algorithms are provided along with advanced image-processing examples. •This provides the framework for the description of the image-processing hardware engine in Chapter 9, “Image-Processing Engines.” The required performance parameters for DSCs and digital video cameras are reviewed, followed by descriptions of the architectures of signal-processing engines. Examples of analog front-end and digital back-end designs are introduced. •In Chapter 10, “Evaluation of Image Quality,” readers learn how each component described in the previous chapters affects image quality. Image quality-related standards are also given. •In Chapter 11, “Some Thoughts on Future Digital Still Cameras,” Eric Fossum, the pioneer of CMOS image sensor technology, discusses future DSC image sensors with a linear extrapolation of current technology and then explores a new paradigm for image sensors. Future digital camera concepts are also addressed. Copyright © 2006 Taylor & Francis Group, LLC DK545X_C000.fm Page 7 Wednesday, July 6, 2005 8:12 AM I would like to extend my gratitude to the contributors to this book, many of whom work actively in the industry, for their efforts and time given to making this book possible. Also, I am grateful to all of my coworkers who reviewed draft copies of the manuscripts: Dan Morrow, Scott Smith, Roger Panicacci, Marty Agan, Gennnady Agranov, John Sasinowski, Graham Kirsch, Haruhisa Ando, Toshinori Otaka, Toshiki Suzuki, Shinichiro Matsuo, and Hidetoshi Fukuda. Sincere thanks also go to Jim Lane, Deena Orton, Erin Willis, Cheryl Holman, Nicole Fredrichs, Nancy Fowler, Valerie Robertson, and John Waddell of the Mar- Com group at Micron Technology, Inc. for their efforts in reviewing manuscripts, appendices, and the table of contents; creating drawings for Chapter 3 and Chapter 5; and preparing the tables and figures for publication. Junichi Nakamura, Ph.D. Japan Imaging Design Center Micron Japan, LTD Copyright © 2006 Taylor & Francis Group, LLC DK545X_C000.fm Page 9 Wednesday, July 6, 2005 8:12 AM Editor Junichi Nakamura received his B.S. and M.S. in electronics engineering from Tokyo Institute of Technology, Tokyo, Japan, in 1979 and 1981, respectively, and his Ph.D. in electronics engineering from the University of Tokyo, Tokyo, Japan, in 2000. He joined Olympus Optical Co., Tokyo, in 1981. After working on optical image processing, he was involved in developments of active pixel sensors.