US 20030234866A1 (19) United States (12) Patent Application Publication (10) Pub. No.: US 2003/0234866 A1 Cutler (43) Pub. Date: Dec. 25, 2003 (54) SYSTEM AND METHOD FOR CAMERA Publication Classi?cation COLOR CALIBRATION AND IMAGE STITCHING (51) Int. Cl? ................................................... ..H04N 5/225 (52) US. Cl. ........................................................ .. 348/2071 (76) Inventor: Ross Cutler, Duvall, WA (US) (57) ABSTRACT Correspondence Address: LYON & HARR, LLP A practical, real-time calibration of digital omnidirectional 300 ESPLANADE DRIVE, SUITE 800 cameras in the areas of de-vignetting, brightness, contrast, OXNARD, CA 93036 (US) and White balance control. Novel solutions for the color calibration of an omnidirectional camera rig, and an ef?cient (21) Appl, No,: 10/177,315 method for devignetting images are presented. Additionally, a context-speci?c method of stitching images together into (22) Filed: Jun. 21, 2002 a panorama or a mosaic is provided. - - — — - - — — — — - — — — — — -I 193 -\ F I ° ° ‘' I:I CAMERA I SYSTEM M_E_MgRl _ ‘j I | _ _ _ _ — _ =_ _ 192/ I (ROM) 131 130 120 '- — - - — — — — — - — — — — — 1 I : BIOS I33 /' 194 I 190 /‘195 | MONITOR — PROCESSING OUTPUT J I _ _ _1';2_ _ _ _ _ UNIT CAMERA INT‘I/E'EEZCE PERIPHERAL I191 \\\\ | (RAM) INTERFACE INTERFACE I OPERATING I II SYSTEM 1_3_4 ? jT ? T I 196 PRINTER | APPLICATION JSYSTEM BUS PR GRAMS I OTHER PROGRAM° REMOVABLE 16°\U Il 171 I MODULES 13s NON'REMOVABLE NON-VOL. USER NETWORK l — NON-VOL. MEMORY MEMORY INPUT INTERFACE I INTERFACE INTERFACE INTERFACE LOCAL AREA NETWORK I PROGRAM @140 150 | DATA 137 . | — IIIIII ..II _ _ _ _ _ _ _ --_.£ \-__________ ____ K / / \ 151 155 WIDE AREA NETWORK 110 / \ I / / 141 \ \ / / ’ \ 152 8) REMOTE / / \ \ 156 COMPUTER KEYBOARD 180 OPERATING APPLICATION OTHER PROGRAM PROGRAM sYsTEM PROGRAMS MODuLEs DATA I61 144 15 E 147 REMOTE / 181 —— MOUSE APPLICATION / J PROGRAMS @ 10o Patent Application Publication Dec. 25, 2003 Sheet 1 0f 7 US 2003/0234866 A1 mFOEwm mm: \J02)z/Eoomu\/Qt9: 02n 5m2/Br\ mm"H<20MMIF \awzéoomlw6_u_ |/“‘\IIV.I|lIl lI 0:/\ Patent Application Publication Dec. 25, 2003 Sheet 2 0f 7 US 2003/0234866 A1 A Camera1 Cameraz FIG. 2 Patent Application Publication Dec. 25, 2003 Sheet 3 0f 7 US 2003/0234866 A1 Select a Reference Camera from I A Series of Cameras "' 302 l Compute the Desired Brightness /" 304 for the Reference Camera and Set the Brightness at the Reference Camera. l Compute the Desired Brightness and Set the Contrast at the e) 306 Reference Camera t FIG. 3 Set the WhiteBalanceREd and WhiteBalanceBlue Registers to Make the Mean of the Red, /308 Green and Blue Channels the Same l Propagate Settings from Reference Camera to the the / 310 Other Cameras for the Overlapping image Areas Patent Application Publication Dec. 25, 2003 Sheet 4 0f 7 US 2003/0234866 A1 “2.53.5...“ uat.“nFE@B: Patent Application Publication Dec. 25, 2003 Sheet 5 0f 7 US 2003/0234866 A1 Input lmage(s) of a Uniformly Lit WhIte Surface. /__ 502 l Create a Table for Each Pixel in Input lmage(s) by Determining Average Pixel Brightness for f 504 each Pixel Location (u,v). i Create a Correction Matrix by Taking the Inverse of the Q 506 Average Brightness for Each Pixel Location (U. V)- F I G 5 + . Multiply Each Pixel Value by the Corresponding Element of the [508 Correction Matrix to Remove Vignetting Effect for Each Input Image. Patent Application Publication Dec. 25, 2003 Sheet 6 0f 7 US 2003/0234866 A1 Input a Series of Images. f‘ 702 I Correct Each Image for Radial /" 704 Distortion. I Map Each Image to a Cylindrical Coordinate System i FIG. 7 Use Translation, Scale and Rotation of Each Image to Align /70a Each Image With Its Neighbors. /710 f 712 Stitch Images into Panorama Stitch Image Into Panorama Blending Overlapping Regions Using Context Specific Stitching. Patent Application Publication Dec. 25, 2003 Sheet 7 0f 7 US 2003/0234866 A1 Input A Series of Images /* 802 l Determine Regions of Image Overlap /’ 804 l Determine High Cost Objects in Areas of Image Overlap V4306 Assigning High Cost to Associated+ Pixels. FIG. 8 Determine Vertical Cut Path in 808 Region of Image Overlap Using / Dijkstra's Algorithm to Determine the Least Cost Path. l Use Cut Path to Determine Which Area of Each Image Should be —— 810 Rendered in Overlapping Area. ___-_----_--------------------—----- US 2003/0234866 A1 Dec. 25, 2003 SYSTEM AND METHOD FOR CAMERA COLOR the color uniformity is taken for granted, something that is CALIBRATION AND IMAGE STITCHING not alWays true as discussed above. Others have used techniques to make the change gradual rather than abrupt, BACKGROUND techniques knoWn as feathering or blending Addition ally, some research has been done to analyZe, classify and [0001] 1. Technical Field correct photometric non-uniformities in and across tilted [0002] This invention is directed toward a color calibra projectors The Work that has been done in the ?eld of tion technique and a stitching technique for use With an radiometric calibration has mainly dealt With noise estima omni-directional camera. More speci?cally, this invention is tion and modeling [5, 4] or With single camera self calibra directed toWards a system and method for improving the tion Radiometric distortion, the effects of lens vignett color uniformity of a composite image generated from a set ing, needs to be accounted for to ensure uniform intensity of images depicting different but overlapping portions of the across panoramic images. same scene. In addition, it includes a context-speci?c image [0007] In addition to the problem of color differences, stitching system and method that uses domain knoWledge to other factors affect the desired seamlessness of a mosaiced improve the quality of image stitching. image. For example, When cameras are arranged in the [0003] 2. Background Art back-to-back circular fashion discussed above and shoWn in FIG. 2, the center of projection in each camera is the point [0004] An image mosaic or panorama can be generated by betWeen the CCD and outer lens such that rotations about aligning and “stitching” input images that are acquired from this point result in a pure rotation transformation on images a single camera. To create an optimal panoramic stitched captured (i.e., no motion parallax is induced). The distance image, several types of color calibrations need to be taken betWeen these centers of projection is labeled D. If D Were into account. Differences in exposure, brightness, contrast Zero, then stitching the images together is relatively easy. and White balance can cause signi?cant discontinuities HoWever, in some omnidirectional camera con?gurations, D betWeen images. When acquiring the input images for a is much greater than Zero, Which results in motion parallax panorama from a single camera, the camera is adjusted to in the captured images. As a result, stitching the images have as near as possible the same settings, including color together seamlessly requires knoWing the depth from the settings, for all images acquired. Nevertheless, there are still cameras to the object being stitched. Computing reliable sometimes differences in color betWeen images because object depth With stereo image pairs is a very difficult lighting and other conditions may change over the course of problem, and has not been solved generally (for arbitrary time and adjustments to settings may be necessary When objects, texture, lighting, etc.). photographing images from different angles or perspectives. Furthermore, even When all other conditions remain con [0008] It is noted that in the preceding paragraphs, as Well stant, settings can be inadvertently changed betWeen pic as in the remainder of this speci?cation, the description tures. As a result, When the input images are stitched refers to various individual publications identi?ed by a together, the images can exhibit differences in color levels, numeric designator contained Within a pair of brackets. For making the edges or boundaries of the stitching in the example, such a reference may be identi?ed by reciting, composite image obvious to the vieWer. “reference [1]” or simply “[1]”. A listing of the publications corresponding to each designator can be found at the end of [0005] The problem of color differences betWeen input the Detailed Description section. images in a composite, mosaiced image is compounded When multiple cameras are used to acquire the input images. SUMMARY For example, this is the case When multiple cameras are used to generate a panoramic video. Essentially, the creation of a [0009] The present invention is directed toWard a system panoramic video entails acquiring multiple videos depicting and process that overcomes the aforementioned limitations a full 360-degree vieW of the surrounding scene. A camera in systems for color calibration of images and image stitch rig, consisting of multiple cameras disposed in a back-to ing. back fashion, is sometimes employed for the purpose of [0010] In the color calibration technique according to the capturing these videos. There is great dif?culty in adjusting invention, digital camera controls are used to achieve the settings of the multiple cameras used in this con?gura images that are similar in color to start With. Seamless tion to have exactly the same color settings. Although the panoramic images can thus be stitched together, optionally parameters may be identically set, variations in Charge applying standard blending techniques. Coupled Devices (CCD), Analog to Digital convert ers, and other analog components still cause signi?cant [0011] This general process of color calibrating a series of image variations betWeen cameras. As a result, the stitched cameras can be described as folloWs. A reference camera is or mosaiced composite image Will often exhibit distinct selected from the series of cameras.
Details
-
File Typepdf
-
Upload Time-
-
Content LanguagesEnglish
-
Upload UserAnonymous/Not logged-in
-
File Pages18 Page
-
File Size-