Method, Medium and System for Optimizing Capture Device Settings
Total Page:16
File Type:pdf, Size:1020Kb
(19) *EP002290949B1* (11) EP 2 290 949 B1 (12) EUROPEAN PATENT SPECIFICATION (45) Date of publication and mention (51) Int Cl.: of the grant of the patent: H04N 5/232 (2006.01) H04N 5/235 (2006.01) 04.12.2019 Bulletin 2019/49 (21) Application number: 10012182.1 (22) Date of filing: 17.12.2004 (54) Method, medium and system for optimizing capture device settings through depth information Verfahren, Medium und System zur Optimierung der Einstellungen einer Erfassungsvorrichtung mittels Tiefeninformation Procédé, medium et système pour optimiser les paramètres de dispositif de capture par des informations de profondeur (84) Designated Contracting States: (72) Inventor: Marks, Richard AT BE BG CH CY CZ DE DK EE ES FI FR GB GR Foster City HU IE IS IT LI LT LU MC NL PL PT RO SE SI SK TR CA 94404-2175 (US) (30) Priority: 16.01.2004 US 758817 (74) Representative: D Young & Co LLP 120 Holborn (43) Date of publication of application: London EC1N 2DY (GB) 02.03.2011 Bulletin 2011/09 (56) References cited: (62) Document number(s) of the earlier application(s) in EP-A- 0 750 202 WO-A-01/18563 accordance with Art. 76 EPC: FR-A- 2 832 892 US-A- 6 057 909 04814607.0 / 1 704 711 US-A1- 2003 100 363 (73) Proprietor: Sony Interactive Entertainment Inc. Tokyo 108-8270 (JP) Note: Within nine months of the publication of the mention of the grant of the European patent in the European Patent Bulletin, any person may give notice to the European Patent Office of opposition to that patent, in accordance with the Implementing Regulations. Notice of opposition shall not be deemed to have been filed until the opposition fee has been paid. (Art. 99(1) European Patent Convention). EP 2 290 949 B1 Printed by Jouve, 75001 PARIS (FR) 1 EP 2 290 949 B1 2 Description producing an image that has the exposure/gain and other related parameters adjusted for both the foreground and BACKGROUND OF THE INVENTION background. [0006] US 6057909 discloses an apparatus for creat- 1. Field of the Invention 5 ing an image indicating distances to objects in a scene, comprising: a modulated source of radiation, having a [0001] This invention relates generally to image cap- first modulation function, which directs radiation toward ture techniques, and more particularly to enhancing a a scene; a detector, which detects radiation reflected captured image of a scene by adjustments enabled from the scene, modulated by a second modulation func- through depth information. 10 tion, and generates, responsive to said detected modu- lated radiation, signals responsive to the distance to re- 2. Description of the Related Art gions of the scene; a processor, which receives signals from the detector and forms an image, based on the sig- [0002] Image capture devices, whether cameras or nals, having an intensity value distribution indicative of video based devices, typically have a limited contrast ra- 15 the distance of objects from the apparatus; and a con- tio, which is a measure of the difference between the troller, which varies at least one of the first and second darkest and lightest parts of a scene. One exemplary modulation functions, responsive to the intensity value scene may include a person in the shade and the back- distribution of the image formed by the processor. One ground having bright sunlight. When the background of implementation of US 6057909 includes a camera, an this scene is exposed correctly for the image capture 20 image analyzer, a system control unit and a video display. device, there is little or no detail in the shaded person’s face. SUMMARY OF THE INVENTION [0003] Auto-exposure and auto-gain features are com- monly used to set brightness levels for the capture de- [0007] Broadly speaking, the present invention fills vice. These features tend to take the entire scene and 25 these needs by providing a method and system that en- average it to apply a certain exposure or gain setting. ables adjustment of segments of a scene, e.g., fore- While the averaging may work well for a scene having a ground and background images, where the foreground great deal of images and colors, this scheme quickly and background images are identified through a depth breaks down as the scene has less variety. mask. According to the present invention there is provid- [0004] One attempt to address the limited contrast ratio 30 ed a method for adjusting image capture settings for a of current capture devices is through the use of a back- video capture device during execution of an interactive light feature. For instance, where there is a bright light gaming application according to claim 1. According to source in the background, e.g., sunlight, backlight com- further aspects there are provided a corresponding me- pensation will take the center of the scene and use that dium claim 10 and a corresponding system claim 11. region as the average. By doing this, the center of the 35 [0008] Other aspects and advantages of the invention scene may be brightened while the bright sunlight on the will become apparent from the following detailed descrip- edges becomes washed out or darkened. The shortcom- tion, taken in conjunction with the accompanying draw- ing with backlight compensation is that the object to be ings, illustrating by way of example the principles of the brightened must be in the center of the scene. In addition, invention. a region of the scene is used for computing the average, 40 rather than the actual object itself, which may cause some BRIEF DESCRIPTION OF THE DRAWINGS display artifacts. Furthermore, backlight compensation does not provide a solution where there are multiple fore- [0009] The invention, together with further advantages ground images in different regions of the scene. Addi- thereof, may best be understood by reference to the fol- tionally, with backlight compensation, the foreground ob- 45 lowing description taken in conjunction with the accom- ject is brightened, however, this is done at the expense panying drawings. of the detail in the background. Thus, a user is required to choose between foreground detail and background Figure 1 is a simplified schematic diagram illustrating detail. Some of these shortcomings may be extended to a scene having foreground and background objects, video capture devices which may be used for interactive 50 which is captured through an image capture device entertainment applications. For example, where an im- and subsequently displayed in accordance with one age of a user is incorporated into a video game, a bright embodiment of the invention. light source may adversely the displayed image as de- Figure 2 is a simplified schematic diagram illustrating scribed above. This adverse impact may prevent the the generation of a depth mask for use in discerning tracking of an object of the image in addition to displaying 55 between foreground and background objects in ac- a poor quality image. cordance with one embodiment of the invention. [0005] Accordingly, there is a need to solve the prob- Figures 3A and 3B are simplified schematic dia- lems of the prior art to provide a system and method for grams illustrating the amount of detail enabled in de- 2 3 EP 2 290 949 B1 4 fining foreground objects in accordance with one em- scribed below. The segmentation based upon depth is bodiment of the invention. captured through a foreground/background depth mask Figure 4 is a simplified schematic diagram illustrating which may be created through an image capture device a captured image which is enhanced through a gen- having depth capability or through a light pulse/flash with erated mask to define background and foreground 5 a time of flight cut-off technique, both discussed in more images in accordance with one embodiment of the detail-below. invention. [0012] Figure 1 is a simplified schematic diagram illus- Figure 5 is a simplified schematic diagram of an in- trating a scene having foreground and background ob- teractive entertainment system which utilizes the jects which is captured through an image capture device mask generation in order to more effectively track a 10 and subsequently displayed in accordance with one em- user in accordance with one embodiment of the in- bodiment of the invention. Image capture device 100 is vention. configured to capture an image of a scene in which a Figure 6 is a simplified schematic diagram of an im- person 102 is in the foreground and background scenery age capture device in accordance with one embod- 104. The captured image of the scene is then displayed iment of the invention. 15 on display panel 106. Display panel 106 may be a display Figure 7 is an alternative schematic diagram of an panel affixed to image capture device 100, e.g., a liquid image capture device having logic configured to dif- crystal display (LCD) panel where the image capture de- ferentiate between foreground and background im- vice is a digital camera or camcorder. Alternatively, dis- ages in the invention. play panel 106 may be remote from image captures de- Figure 8 is a flow chart diagram illustrating the meth- 20 vice 100, e.g., a television screen where the image cap- od operations for adjusting image capture settings ture device is a webcam used in conjunction with a com- for an image capture device in accordance with one puting device, such as a game console. As will be de- embodiment of the invention. scribed in more detail below, foreground image 102 and background scenery 104 are capable of having their cor- DETAILED DESCRIPTION OF THE PREFERRED EM- 25 responding image or video characteristics independently BODIMENTS compensated irrespective of their position in either the foreground or the background.