USOO924,5314B2
(12) United States Patent (10) Patent No.: US 9.245,314 B2 Jannard et al. (45) Date of Patent: *Jan. 26, 2016
(54) VIDEO CAMERA 13/0257; H04N 19/00315; H04N 19/00763; H04N 19/00903; H04N 1/648; H04N 5/225; (71) Applicant: RED.COM, INC. Irvine, CA (US) G06T 7/2006; G06T 3/4015; G06T 9/007 (72) Inventors: James H. Jannard, Las Vegas, NV USPC ...... 348/240.2, 222.1, 223.1, 273–280; (US); Thomas Graeme Nattress, Acton 375/240.2, 240.25, 240.26, 340.29: (CA) 382/166 167 See application file for complete search history. (73) Assignee: RED.COM, Inc., Irvine, CA (US) (56) References Cited (*) Notice: Subject to any disclaimer, the term of this patent is extended or adjusted under 35 U.S. PATENT DOCUMENTS U.S.C. 154(b) by 0 days. 3,972,010 A 7/1976 Dolby This patent is Subject to a terminal dis 4,200,889 A 4, 1980 Strobele claimer. (Continued) (21) Appl. No.: 14/485,612 FOREIGN PATENT DOCUMENTS
(22) Filed: Sep. 12, 2014 CA 2831 698 10, 2008 CA 26836.36 1, 2014 (65) Prior Publication Data (Continued) US 2015/OOO2695A1 Jan. 1, 2015 OTHER PUBLICATIONS Related U.S. Application Data (63) Continuation of application No. 13/464,803, filed on 2K Digital Cinema Camera Streamlines Movie and HD Production, May 4, 2012, now Pat. No. 8,872,933, which is a Silicon Imaging Digital Cinema, Press News Releases, Hollywood, continuation of application No. 12/101.882, filed on California, date listed Nov. 1, 2006, in 2 pages. (www.siliconimag Apr. 11, 2008, now Pat. No. 8,174,560. ing.com DigitalCinema News Pr 1 1 0 1 06 1). (Continued) (60) Provisional application No. 60/911,196, filed on Apr. 11, 2007, provisional application No. 61/017.406, filed on Dec. 28, 2007. Primary Examiner — Trung Diep (74) Attorney, Agent, or Firm — Knobbe, Martens, Olson & (51) Int. C. Bear LLP H04N 5/228 (2006.01) HO)4N 9/73 (2006.01) (57) ABSTRACT (Continued) Embodiments provide a video camera configured to capture, (52) U.S. C. compress, and store video image data in a memory of the CPC ...... G06T3/4015 (2013.01); GIIB 27/031 video camera at a rate of at least about twenty three frames per (2013.01); H04N I/648 (2013.01); second. The video image data can be mosaiced image data, and the compressed, mosaiced image data may remain Sub (Continued) stantially visually lossless upon decompression and demosa (58) Field of Classification Search 1C1ng. CPC. G08B 13/19628; H04N 9/43; H04N 9/045; H04N 2209/046; H04N 5/23235; H04N 30 Claims, 18 Drawing Sheets
-50
Obtain Sensor Data
Pre-emphasize Data
Transform Red and Blue Picture elements
Apply Compression Algorithm US 9.245,314 B2 Page 2
(51) Int. Cl. 6,990,240 B2 1/2006 Hagiwara G06K 9/36 (2006.01) 6,995,793 B1 2/2006 Albadawi et al. G06T3/40 (2006.01) 7,038,7196,995,794 B2 2/20065/2006 HiraiHsu et al. H04N L/64 (2006.01) 7,050,642 B2 5/2006 Graffagnino H04N 9/04 (2006.01) 29:29: 3: Milet al. H04N 9/86 (2014.01) 7,1W-1 10,605 - B2 9/2006 MarcellinWa et al. H04N 9/85 (2014.01) 7,113,645 B2 9/2006 Sano et al. GIB 27/03 (2006.01) 7,126,634 B2 10/2006 Kato
H04N 5/225 (2006.01) 2: 1938: SteinaW et al. H04N 5/374 (2011.01) 7,174,045 B2 2/2007 Yokonuma HO)4N 5/77 (2006.01) 7.212,313 B1 5/2007 Hoel H04N 9/9 (2014.01) 7,253,836 B1 8, 2007 Suzuki et al. 52) U.S. C 7.312,821 B2 12/2007 Voss (52) U.S. Cl. 7.313.286 B2 12/2007 Schwartz et all CPC ...... H04N 5/2252 (2013.01); H04N 5/374 7.3344 B2 1/2008 Kubo et all (2013.01); H04N5/772 (2013.01); H04N 9/045 7343,043 B2 3, 2008 Yokonuma (2013.01); H04N 19/186 (2014.11); H04N 7,349,579 B2 3/2008 Kadowaki et al. 19/85 (2014.11); H04N 19/91 (2014.11) 7,365,658 B2 4/2008 Todorov et al. 7,369,161 B2 5/2008 Easwar et al. 7,376, 183 B2 5/2008 Weigand et al. (56) References Cited 7,385,647 B2 6/2008 Park 7.388,992 B2 6/2008 Atsumi et al. U.S. PATENT DOCUMENTS 7,394,485 B2 7, 2008 Kim 7,477,781 B1 1/2009 Tanbakuchi 4,316,213 A 38. Sharon et al. 7,480,417 B2 1/2009 Malvar is, A SE Ripon 7.483,909 B2 1/2009 Sena et al. - W - T.512.283 B2 3, 2009 B 5,016,107 A 5/1991f Sasson et1 al. 7,526,134- K. 4 B2 4/2009 MatsubaraOWe 3. A 63 Social 7,577,689 B1 8/2009 Masinter et al. W - T590.301 B2 9, 2009 W. 5,172,227 A 12/1992 Tsai et al. 7778.473 B2 is 2010 Kodama 5.249,0534 I-1 A 9, 1993 Jain 7.796,186w B2 9, 2010 Osh 5.255,083. A 10/1993 Capitant et al. 7,830,967- w B1 1 1/2010 JannardSa et al. 5,303,062 A SE: Sai et al. 7,868,879 B2 1/2011 Rizko A g Sh etal 7,898,575 B2* 3/2011 Ishii ...... HO4N 3,1562 I k 4 348.222.1 . A &E Sys: et al. 7,902,512 B1 3/2011 Chang et al. 5.535.346 A T. 1996 t O 7,907,791 B2 3/2011 Kinrot - - - T.936,919 B2 5, 2011 K 3: A 1985 shop 8,014,597 B1 * 9/2011 Newman ...... HO4N 1,648 -- 4 382,166 5,592,237 A 1/1997 Greenway 8,125,547 B2 2/2012 Oda et al. 31: A 1988 Saya 8, 174,560 B2 5/2012 Jannard et al. 5,949,468w - A 9/1999 Asahina et al. 8.237.830853; B2; 8, 5852012 JannardANN et al. 33. A E R. 23, 223 royer &E A 258 AR al 8,872,933 B2 10/2014 Jannard et al. 6,169,317k - I B1 1/2001 Sawada et al. SS658.878,952 B2 11/2014A5. JannardNEN et al. 6, 192,086 B1 2/2001 Darr 2001/0048477 A1 12/2001 Misawa 6,262,7631953 B1R. 72001339; TotsukaIt al. 2002fOO417072002/0039142 A1 4/20024, 2002 ZhangNewman et al. 6,269,217 B1 7/2001 Rodriguez 2002fOO63787 A1 5, 2002 Watanabe 6,275,263E3: B1 8/200139, HuWashino et al. 2002/01963542002/0167602 A1 12/20021 1/2002 ChangNguyen et al. R 2.58: Sergiev et al. 2003/0007567 A1 1/2003 Newman et al. 6,466,699- . . . B1 10/2002 Schwartz et al. 2003,36,875. OO 11747 A1A 1/200356 LGoet al. RE38,079 E 4/2003 Washino et al. 2003/0038885 A1 2/2003 Rodriguez 6,567,988 B1 5/2003 Okawa 2003/0122037 A1 7/2003 Hyde et al. 6,597,860 B2 258. Song et al. 2003/O122937 A1 7/2003 Guarnera et al. 6,697,106 B1 $38. s 2003/0156188 A1 8/2003 Abrams, Jr. 37.8 R 838. 1. Netal 2003/0185302 Al 10/2003 Abrams, Jr. 6.825.876 B 11/2004 E.as a 2003/0202106 A1 10/2003 Kanleinsberger et al. 6859,226 B2 22005 Kawamura et al 2004/00325 16 A1 2/2004 Kakarala 6867,717 B1 3/2005 on 2004/0051793 A1 3, 2004 Tecu 6878,977 B1 4, 2005 Kozuka et al. 2004/OO95477 A1 5, 2004 Maki et al. 6,937.276 B2 & 2005 Chung 2004/0131274 A1 7/2004 Perlmutter et al. 6,944,349 B1 9, 2005 Onno et al. 2004O165080 A1 8, 2004 Burks et al. 6,958,774 B2 10/2005 Kuroiwa 2004O169746 A1 9, 2004 Chen et al. 6,983,074 B1 1/2006 Clauson et al. 2004O169751 A1 9/2004 Takemura et al. 6,989,773 B2 1/2006 Wee et al. 2004/0196389 A1 10, 2004 Honda US 9.245,314 B2 Page 3
(56) References Cited FOREIGN PATENT DOCUMENTS U.S. PATENT DOCUMENTS CN 101689357 3, 2015 CN 1047O2926 6, 2015 2004/0201701 A1 10/2004 Takagi EP 1605 403 12/2005 2004/02O1760 A1 10, 2004 Ota et al. EP 2145330 1, 2010 2004/0213472 A1 10, 2004 Kodama et al. JP 06-054239 2, 1994 2004/0218812 A1 1 1/2004 Douglass JP 2000-069488 3, 2000 2004/024634.6 A1 12/2004 Kim et al. JP 2001-5.15318 9, 2001 2005/004111.6 A1 2/2005 Tsukioka JP 2002-051266 2, 2002 2005/0182972 A1 8/2005 Apostolopoulos et al. JP 2004-038693 2, 2004 JP 2004-24.8061 9, 2004 2005, 0183118 A1 8, 2005 Wee et al. JP 2004-260821 9, 2004 2005/0276496 Al 12/2005 Molgaard et al. JP 2004-282780 10, 2004 2006, OOO7324 A1 1/2006 Takei JP 2005-210216 8, 2005 2006, OO12694 A1 1/2006 Yoneda et al. JP 2006-171524 6, 2006 2006, OO61659 A1 3, 2006 Niwa JP 2006-311314 11, 2006 2006,0061822 A1 3/2006 Sung et al. JP 2007-26.7072 10/2007 2006.0114987 A1 6, 2006 Roman JP 2008-124976 5, 2008 2006, O165178 A1 7/2006 Ma et al. JP 2012-52.3790 10, 2012 2006/O1651.79 A1 7/2006 Feuer et al. KR 10-2002-0041778 6, 2002 2006/0170786 A1 8, 2006 Won KR 10-2009-0035204 4/2009 2006/0221199 A1 10/2006 Nakajima KR 10-1478380 12/2014 2006/0221203 A1 10, 2006 Abe et al. WO WO91/O1613 2, 1991 2006/022 1230 A1 10, 2006 Dutta et al. WO WO92, 10911 6, 1992 2006/0244842 A1 11/2006 Hatano WO WO 97.09818 3, 1997 2007.0035636 A1 2/2007 Wu WO WO99, 12345 3, 1999 2007/0041634 A1 2/2007 Sugimori WO WO 2008/128112 10, 2008 9. WO WO 2014f1271.53 8, 2014 2007/0051817 A1 3, 2007 Yano 2007/009 1187 A1 4, 2007 Lin OTHER PUBLICATIONS 2007/0092149 A1 4/2007 Sung 2007, 010931.6 A1 5, 2007 Fainstain 4:4:412-bit Uncompressed DVX100, date listed May 11-16, 2004, in 2007/O127095 A1 6/2007 Sugimori 9 pages. (http://www.dvinfo.net/forum/archive/index.php/t-20332 2007/O133902 A1 6, 2007 Kumar p-13.html). 2007. O153093 A1 7, 2007 Lin et al. Abel Cine, “Abel North American Agent for Phantom Cameras.” Feb. 2007/0160142 A1 7/2007 Abrams, Jr. 7, 2007, http://web.archive.org/web/20120523003248/http://about. 2007, 0164335 A1 7, 2007 McKee abelcine.conn/2007/02/07/abel-north-american-agent-for-phantom 2007. O165116 A1 7/2007 Hung et al. cameras in 2 pageS. 2007/02O6852 A1 9, 2007 McGee Arriflex D-20 Preliminary Specifications, achive.org indicates avail 2007/0216782 A1 9, 2007 Chernoff able on-line on May 31, 2005. www.arri.com, online). http://web. 2007/0285517 A1 12, 2007 Ishikuro archive.org/web/20050531010626/www.arri.com/entry/products. 2008.0002035 A1 1/2008 Yoshida htm, pp. 1-2. 2008/0012953 A1 1/2008 Yang et al. Arriflex D-21: The Film Style Digital Camera, date listed Jan. 4. 2008, OO1874.6 A1 1/2008 Kawanami 2008, www.arri.de, online http://www.arri.de/press/press/press 2008/00554262008.0062272 A1 3/2008 PertSeletal.Kuroiwa. ttnewsEFTSE's backPid=1781& cHash=e89c.9b0855e89c9b0855. Sis: b 2008, OO63070 A1 3/2008 Schwartz et al. Bruner, Guy, Silicon Imaging Shows 1920x1080P Camera System. 2008, OO63269 A1 3, 2008 Chiu Camcorder News, Las Vegas, NAB, date listed Apr. 25, 2006, in 8 2008, 00798.18 A1 4/2008 Takahashi pages. (http://www.camcorderinfo.com/content/Silicon-Imaging 2008/0084581 A1 4/2008 Kobayashi et al. Shows-1920x1080P-Camera-System.htm). 2008. O131013 A1 6, 2008 Suino et al. CineForm Insider, blog post dated Nov. 13, 2007, http://cineform. 2008/0259180 A1 10, 2008 Ovsiannikov blogspot.com/2007/11/cineform-on-chip.html, in 3 pages. 2008/0273809 A1 11, 2008 Demos CineForm Insider, date listed as Jan. through Dec. 2006, in 17 pages. 2008/02858712008/0284485 A1 11,1 1/2008 2008 SchillingIshikawa SENSETsha-Gasoos.:UU:UU-US:UU&updated-max=ZUU f-Ul- :UUUU-US: 2008/030 1315 A1 12/2008 Cheng et al. max-results-22). 2009/0033752 A1 2/2009 Bodnar et al. CineForm Online Workflow Solutions for Film and Video—Nov. 1,
2009 OO52797 A1 2/2009 Matsushita (AC. O Raw—DalsaRaw—aSat anda VisionWSO ResearchReSeaC RawW FileFlle ConvertUonverters, 38885 A. 53. s printed Aug. 16, 2010, www.cineform.com, online. 2009 OO86817 A1 4/2009 Matsuoka et al. systichology Overview and Workflow, date listed 2009, O141140 A1 6, 2009 Robinson pr. 15, , in 5 pages. 2010, OO14590 A1 1/2010 Smith CinemaTechnic Camera Profiles | ARRI 16SR, date listed 2001. 2010/011 1489 A1 5, 2010 Presler Dalsa Origin Brochure, document indicates that it was printed Apr. 2010/0225795 A1 9, 2010 Suzuki et al. 2004, in 2 pages. 2011 0194763 A1 8, 2011 Moon et al. “Dalsa Technology with Vision.” Presentation, Mar. 2003, pp. 35. 2012fO294582 A1 11/2012 Jannard et al. Defendant's Answer, Affirmative Defenses and Counterclaims; 2013/0113951 A1 5/2013 Jannard et al. Demand for Jury Trial; red.com, Inc. v. Sony Corporation of America 2014, OO63297 A1 3/2014 Yamura and Sony Electronics Inc., Case No. 13CV0334-DMS-BGS, dated 2014/O161367 A1 6/2014 Ridenour et al. Jun. 20, 2013. 2014/0218580 A1 8/2014 Mayer et al. Digital Negative (DNG) Specification, date listed Apr. 2008. 2014/0226036 A1 8/2014 Jannard et al. Doutreet al., “An Efficient Compression Scheme for Colour Filter 2014/033381.0 A1 11/2014 Nakaseko Array Images. Using Estimated Colour Difference”, IEEE Canadian 2015,0003801 A1 1/2015 Jannard et al. conference on Electrical and Computer Engineering, date listed 2015,0092094 A1 4/2015 Itonaga et al. 2007, pp. 24-27. US 9.245,314 B2 Page 4
(56) References Cited Silicon Imaging, Press News Releases, www.siliconimaging.com/ DigitalCinema/SI Press.html, printed Nov. 5, 2012. OTHER PUBLICATIONS Silicon Imaging Support: Frequently-Asked-Questions, archive.org indicates available on-line Dec. 12, 2007, www.siliconimaging.com, Ion, Lucian, et al., High Dynamic Range Data Centric Workflow online. http://web.archive.org/web/2007 1212165310/www. System, Dalsa Digital Cinema, this paper reported to be originally siliconimaging.com/DigitalCinema SiliconImaging facq.html, in 12 presented at SMPTE Technical Conference and Exhibit, New York, pageS. date listed Nov. 2005, in 14 pages. SI-1920HDVR Key Features, Silicon Imaging Digital Cinema, in 2 Ion, Lucian, et al., White Paper: 4K Digital Capture and Postproduc pages. (http://www.siliconimaging.com/DigitalCinema key fea tion Workflow, Dalsa Digital Cinema, in 5 pages. tures.html). ISO Standard 15444 (part 1): Information technology—JPEG 2000 Smith, et al.: Constant quality JPEG2000 rate control for digital image coding system: Core coding system, pp. ii-V, xiv. 1-11, 120-122. cinema; Source: Proceedings of SPIE The International Society for copyright date listed is 2004. Optical Engineering, v 6508, in Part 1, 2007, Conference: Visual JPEG2000 still image coding versus other standards, date listed Jul. Communications and Image Processing 2007, Jan. 30, 2007-Feb. 1. 2000. 2007. Lukac et al.: Single-sensor camera image compression, date listed Smith, et al., Image Resolution of the One-CCD Palomar Motion May 2006, pp. 299-307. Picture Camera, 37th Advance Motion Imaging Conference, Seattle, Menon et al., “On the Dependency Between Compression and Washington, date listed Feb. 27-Mar. 1, 2003, in 8 pages. Demosaicing in Digital Cinema'. Visual Media Production. The 2nd Some Like It Raw, Silicon Imaging D-Cinema Camera with IEEE European Conference, Nov. 30-Dec. 1, 2005, pp. 104-111. Cineform Raw Codec, Studio Daily, date listed May 8, 2006, on Mitani, et al.; A 4 KX 2 K-pixel color image pickup system; IEICE line http://www.studiodaily.com/2006/05/some-like-it-rawl. Transactions on Information and Systems; E82D (8): 1219-1227; The Red One Camera 4K Resolution, various dates listed, starting Aug. 1999. from Feb. 7, 2007, URL:http://www.vnnforum.com/showthread. Mitani, et al.; Ultrahigh-definition color video camera system with php?t-44489 retrieved on Aug. 3, 2012). 4K-Scanning lines; Sensors and Camera Systems for Scientific, Wilt, Adam, Camera Log, NAB 2009—SI-2K, date listed Apr. 19. Industrial, and Digital Photography Applications IV, 5017: 159-166, 2009, in 5 pages. (http://provideocoalition.com/index.php/awilt? 2003. story/nab 2009 si 2k/). New Camcorder from Silicon Imaging, (C) 2006-2008 Digital Zeng, Jianfen, et al., Video Coding Techniques for Digital Cinema, (C) Camcorder News, date listed Apr. 19, 2006, in 2 pages. http://www. 2004 IEEE International Conference on Multimedia and Expo digitalcamcordernews.com/2006/04/new-camcorder-from-silicon (ICME), pp. 415-418, vol. 1. imaging). Zhang et al. Real-time lossless compression of mosaic video Nordhauser, Steve, Silicon Imaging Announces World's First Digital sequences, Zhang, L: Wu, X: Bao, P. date listed Aug. 10, 2005, in 8 Cinema Camera with Direct-to-Disk 10-bit CineForm RawTM pageS. Recording and Adobe RProduction Studio Integration, Silicon Imag Complaint for Patent Infringement; red.com, Inc., Inc. v. Sony Cor ing, Inc., Albany, New York, date listed Jun. 26, 2006, in 3 pages. poration of America and Sony Electronics, Inc., U.S. District Court (http://www.filmmakers.com/news/digital/article 713.shtml). for the Southern District of California, Case No. 3:13-cv-00334 Notes from the field: Silicon Imaging SI-1920HDVR camera in DMS-BGS, dated Feb. 12, 2013. actual use, FreshDV, date listed May 18, 2006, in 2 pages. (http:// Joint Motion for Dismissal Without Prejudice; red.com, Inc. v. Sony www.freshdv.com/2006/05/notes-from-field-silicon-imaging-si. Corporation of America and Sony Electronics Inc., Case No. html). 13CV0334-DMS-BGS, dated Jul. 19, 2013. Phantom 65 the world's first 65mm digital cinema, date listed Nov. On-line discussion thread from www.dvXuser.com, first post in 22, 2006. thread dated May 1, 2006, retrieved from http://www.dvXuser.com/ Phantom 65, archive.org indicates available on-line Feb. 4, 2007. V6/showthread.php?55590-Worried-about-depending-on-red www.vision research.com, online. http://web.archive.org/web/ codec. 200702041 10551/www.visionresearch.com/index.cfm?sector=html On-line discussion thread from www.dvXuser.com, first post in files&page-camera 65 new, pp. 1-2. thread dated Sep. 8, 2006, retrieved from http://www.dvXuser.com/ "Phantom HD”. (http://www.alfavisionsrl.com.ar/espanol/alguiler? V6/showthread.php?70333-Workflow-(good)-News. camera/info/manuals/DS phantomHD.pdf>, dated Mar. 30, 2007. On-line discussion thread from www.dvXuser.com, first post in pp. 2. thread dated Sep. 8, 2006, retrieved from http://www.dvXuser.com/ Compiled by Puhovski, Nenad, High Definition Report from Cilect V6/showthread.php?70412-First-video-from-the-red-4K-demo Standing Committee for New Technologies, Madrid, date listed On-line discussion thread from www.dvXuser.com, first post in 2006, in 146 pages. thread dated Sep. 8, 2006, retrieved from http://www.dvXuser.com/ “Red Digital Cinema'.
(56) References Cited Official Communication in Japanese Application No. 2012-506053, dated Oct. 16, 2013. OTHER PUBLICATIONS International Search Report and Written Opinion in PCT Application No. PCT/US2010/028808, dated Aug. 3, 2010. On-line discussion thread from www.dvXuser.com, first post in Examination Report in Australian Application No. 2008240144. thread dated Oct. 9, 2006, retrieved from http://www.dvXuser.com/ dated Dec. 23, 2010. V6/showthread.php?74232-1k-Bubble-Girl-video-up. Official Communication in Chinese Application No. 200880018570. On-line discussion thread from www.dvXuser.com, first post in thread dated Oct. 31, 2006, retrieved from http://www.dvXuser.com/ 6, dated Mar. 31, 2014. V6/showthread.php?76711-First-Redcode-image! Official Communication in European Application No. 08745686.9, On-line discussion thread from www.dvXuser.com, first post in dated Mar. 30, 2010. thread dated Nov. 3, 2006, retrieved from http://www.dvXuser.com/ Extended European Search Report in European Application No. V6/showthread.php?76954-Red-still-gallery-updated-with-new-4k 08745686.9, dated Aug. 4, 2011. Still Office Action in European Application No. 08745686.9, dated Aug. On-line discussion thread from www.dvXuser.com, first post in 10, 2012. thread dated Nov. 4, 2006, retrieved from http://www.dvXuser.com/ Summons to Attend Oral Proceedings in European Application No. V6/showthread.php?77032-Raw-vs-Redcode-Raw. 08745686.9, dated Oct. 31, 2013. On-line discussion thread from www.dvXuser.com, first post in Official Communication in European Application No. 08745686.9, thread dated Nov. 5, 2006, retrieved from http://www.dvXuser.com/ dated Feb. 5, 2014. V6/showthread.php?771 17-Slo-Mo-and-Redcode-Raw-questions. Official Communication in European Application No. 08745686.9, On-line discussion thread from www.dvXuser.com, first post in dated Mar. 18, 2014. thread dated Nov. 6, 2006, retrieved from http://www.dvXuser.com/ Office Action in Mexican Application No. MX/a/2009/010926, dated V6/showthread.php?77216-120fps-at-4K. May 16, 2012. On-line discussion thread from www.dvXuser.com, first post in thread dated Nov. 13, 2006, retrieved from http://www.dvXuser.com/ Office Action in Japanese Application No. 2010-503253, dated Jun. V6/showthread.php?78010-David-Stump-on-Red. 26, 2012. On-line discussion thread from www.dvXuser.com, first post in Office Action in Korean Application No. 10-2009-7023045, dated thread dated Nov. 14, 2006, retrieved from http://www.dvXuser.com/ Feb. 6, 2014. V6/showthread.php?78150-Red-L-A-photos-what-have-you-s. Examination Report in New Zealand Application No. 580171, dated On-line discussion thread from www.dvXuser.com, first post in Feb. 22, 2011. thread dated Nov. 15, 2006, retrieved from http://www.dvXuser.com/ Examination Report in New Zealand Application No. 601474, dated V6/showthread.php?78290-Red-Camera-first-test-with-Still-Lens Aug. 1, 2012. (-Nikon-). On-line discussion thread from www.dvXuser.com, first post in Examination Report in New Zealand Application No. 620333, dated thread dated Nov. 19, 2006, retrieved from http://www.dvXuser.com/ Feb. 14, 2014. V6/showthread.php?78623-Red-compression-and-matrix-tests. Written Opinion in PCT Application No. PCT/US2008/060 126, On-line discussion thread from www.dvXuser.com, first post in dated Jul. 7, 2008. thread dated Nov. 20, 2006, retrieved from http://www.dvXuser.com/ International Preliminary Reporton Patentability in PCT Application V6/showthread.php?78823-Image-links-fixed. No. PCT/US2008/060.126, dated Oct. 13, 2009. On-line discussion thread from www.dvXuser.com, first post in Official Communication in Taiwanese Application No. 0971 13289. thread dated Nov. 21, 2006, retrieved from http://www.dvXuser.com/ V6/showthread.php?78934-redcode-amazingly-good dated Aug. 29, 2013. On-line discussion thread from www.dvXuser.com, first post in Final Office Action in Re-Examination of U.S. Pat. No. 8, 174,560, thread dated Nov. 24, 2006, retrieved from http://www.dvXuser.com/ dated Oct. 31, 2013. V6/showthread.php?79130-More-footage. Notice of Intent to Issue Ex Parte Reexamination Certificate in Re On-line discussion thread from www.dvXuser.com, first post in Examination of U.S. Pat. No. 8, 174,560, dated Mar. 5, 2014. thread dated Dec. 11, 2006, retrieved from http://www.dvXuser.com/ International Search Report and Written Opinion in PCT Application V6/showthread.php?80963-New-Video!!!-Bus-Video-1080p-clip No. PCT/US2010/060851, dated Aug. 24, 2011. online-Redcode. International Search Report and Written Opinion in PCT Application On-line discussion thread from www.dvXuser.com, first post in No. PCT/US2014/016301, dated May 21, 2014. thread dated Dec. 18, 2006, retrieved from http://www.dvXuser.com/ V6/showthread.php?81686-Specs-changes. Official Communication in Taiwanese Application No. 0971 13289. On-line discussion thread from www.hdforindies.com, first post in dated Jul. 15, 2014. thread dated Sep. 8, 2006, retrieved from http://www.hdforindies. Official Communication in European Application No. 14177071.9, com/2006/09/amsterdam-ibc-2006-red-news-redcode-4k.html. dated Aug. 22, 2014. On-line discussion thread from www.hdforindies.com, first post in Examination Report in Australian Application No. 2012216606, thread dated Dec. 19, 2006, retrieved from http://www.hdforindies. dated Jul. 31, 2014. com/2006/12/milkes-conjecture-on-redcode-data-rates.html. Official Communication in European Application No. 107266884, Order Granting Joint Motion for Dismissal Without Prejudice; red. dated Jul. 14, 2014. com, Inc. v. Sony Corporation of America and Sony Electronics Inc., Poynton, Charles, "A Technical Introduction to DigitalVideo.” 1996, Case No. 13CV0334-DMS-BGS, dated Jul. 29, 2013. Ch. 6 (Gamma), pp. 91-114. Red Digital Cinema, “Introducing Redcode'. Sep. 2006, Interna Official Communication in Korean Application No. 10-2014 tional Broadcasting Convention, Amsterdam, the Netherlands, in 1 7021892, dated Oct. 10, 2014. page. Notice of Opposition in European Application No. 08745686.9, Red Digital Cinema, “Mysterium Sensor'. Sep. 2006, International dated Apr. 22, 2015. Broadcasting Convention, Amsterdam, the Netherlands, in 1 page. Red Digital Cinema, “Preliminary Specifications'. Sep. 2006, Inter Official Communication in European Application No. 14177071.9, national Broadcasting Convention, Amsterdam, the Netherlands, in 1 dated Jul 30, 2015. page. Examination Report in New Zealand Application No. 710813, dated Red Digital Cinema, “Preliminary Specifications”, Apr. 14-19, 2007. Aug. 12, 2015. Las Vegas, Nevada, in 1 page. Official Communication in Taiwanese Application No. 991 11497, Red Digital Cinema, "Simple. 4K to Anything”, Sep. 2006, Interna dated Jul. 24, 2015. tional Broadcasting Convention, Amsterdam, the Netherlands, in 1 International Preliminary Report on Patentability and Written Opin page. ion in PCT Application No. PCT/US2014/016301, dated Aug. 27. Request for Re-Examination of U.S. Pat. No. 8,174.560, dated Sep. 2015. 13, 2012. U.S. Appl. No. 14/609,090, filed Jan. 29, 2015, Jannard et al. Re-Examination Grant in U.S. Pat. No. 8,174,560, dated Dec. 6, 2012. * cited by examiner U.S. Patent Jan. 26, 2016 Sheet 1 of 18 US 9.245,314 B2
I6J)
ETTOJOW 927
U.S. Patent Jan. 26, 2016 Sheet 2 of 18 US 9.245,314 B2
U.S. Patent Jan. 26, 2016 Sheet 3 of 18 US 9.245,314 B2
S S ra s $4-uff-9+u U.S. Patent Jan. 26, 2016 Sheet 4 of 18 US 9.245,314 B2
s U.S. Patent Jan. 26, 2016 Sheet 5 of 18 US 9.245,314 B2
|
|+u
U.S. Patent Jan. 26, 2016 Sheet 7 of 18 US 9.245,314 B2
·
-?! +
[NGITHO U.S. Patent Jan. 26, 2016 Sheet 8 of 18 US 9.245,314 B2
-50
Obtain Sensor Data
Pre-emphasize Data
Transform Red dnd Blue Picture elements
Apply Compression Algorithm
FIG. & U.S. Patent Jan. 26, 2016 Sheet 9 of 18 US 9.245,314 B2
/1 6(7
Obtain Sensor Data
Transform Red and Blue Picture elements
Apply Compression Algorithm
F/G, 84 U.S. Patent Jan. 26, 2016 Sheet 10 of 18 US 9.245,314 B2
U.S. Patent Jan. 26, 2016 Sheet 11 of 18 US 9.245,314 B2
U.S. Patent Jan. 26, 2016 Sheet 12 of 18 US 9.245,314 B2
U.S. Patent Jan. 26, 2016 Sheet 13 of 18 US 9.245,314 B2
-60
4.62 Apply Decompression Algorithm
Apply Inverse Look-up Table
Demosaic Green Picture Elements
Apply Noise Reduction to Green Elements
DemoSolic Red drid Blue Picture Elements
Reconstruct Red ond Blue Elements
FIG 12 U.S. Patent Jan. 26, 2016 Sheet 14 of 18 US 9.245,314 B2
/1 A/7
Apply Decompression Algorithm
Demosaic Picture Elements
Reconstruct Red dnd Blue Elements
Apply Inverse Look-up Table
FIG 124
U.S. Patent Jan. 26, 2016 Sheet 16 of 18 US 9.245,314 B2
U.S. Patent Jan. 26, 2016 Sheet 17 of 18 US 9.245,314 B2
|-
s sa
?I6J) U.S. Patent US 9.245,314 B2
9/6/7, US 9,245,314 B2 1. 2 VIDEO CAMERA configured to convert the focused light into a signal of raw image data representing the focused light. The camera can CROSS-REFERENCE TO RELATED also include a memory device and means for compressing and APPLICATIONS recording the raw image data at a frame rate of at least about 23 frames per second. This application is a continuation of U.S. patent applica In accordance with yet another embodiment, a video cam tion Ser. No. 13/464,803, filed on May 4, 2012, entitled era can comprise a portable housing having at least one “VIDEO CAMERA.” which is a continuation of U.S. patent handle configured to allow a user to manipulate the orienta application Ser. No. 12/101,882, filed on Apr. 11, 2008, tion with respect to at least one degree of movement of the entitled “VIDEO CAMERA, which claims benefit under 35 10 housing during a video recording operation of the camera. A U.S.C. S 119(e) to U.S. Provisional Patent Application Nos. lens assembly can comprise at least one lens Supported by the 60/911,196, filed Apr. 11, 2007, and 61/017.406, filed Dec. housing and configured to focus light at a plane disposed 28, 2007. The entire contents of each of the foregoing appli inside the housing. A light sensitive device can be configured cations are hereby incorporated by reference herein. to convert the focused light into raw image data with a hori 15 Zontal resolution of at least 2 k and at a frame rate of at least BACKGROUND about twenty three frames per second. A memory device can also be configured to store video image data. An image pro 1. Field of the Inventions cessing system can be configured to compress and store in the The present inventions are directed to digital cameras, such memory device the raw image data at a compression ratio of as those for capturing still or moving pictures, and more at least six to one and remain Substantially visually lossless, particularly, to digital cameras that compress image data. and at a rate of at least about 23 frames per second. 2. Description of the Related Art Another aspect of at least one of the inventions disclosed Despite the availability of digital video cameras, the pro herein includes the realization that because the human eye is ducers of major motion pictures and some television broad more sensitive to green wavelengths than any other color, cast media continue to rely on film cameras. The film used for 25 green image data based modification of image data output Such provides video editors with very high resolution images from an image sensor can be used to enhance compressibility that can be edited by conventional means. More recently, of the data, yet provide a higher quality video image. One however, such film is often scanned, digitized and digitally Such technique can include subtracting the magnitude of edited. green light detected from the magnitudes of red and/or blue 30 light detected prior to compressing the data. This can convert SUMMARY OF THE INVENTIONS the red and/or blue image data into a more compressible form. For example, in the known processes for converting gamma Although some currently available digital video cameras corrected RGB data to Y'CbCr, the image is “decorrelated’, include high resolution image sensors, and thus output high leaving most of the image data in the Y' (a.k.a. “luma'), and as resolution video, the image processing and compression tech 35 Such, the remaining chroma components are more compress niques used on board Such cameras are too lossy and thus ible. However, the known techniques for converting to the eliminate too much raw image data to be acceptable in the Y'CbCr format cannot be applied directly to Bayer pattern high end portions of the market noted above. An aspect of at data because the individual color data is not spatially corre least one of the embodiments disclosed herein includes the lated and Bayer pattern data includes twice as much green realization that video quality that is acceptable for the higher 40 image data as blue or red image data. The processes of green end portions of the markets noted above. Such as the major image data Subtraction, in accordance with some of the motion picture market, can be satisfied by cameras that can embodiments disclosed herein, can be similar to the Y'CbCr capture and store raw or Substantially raw video data having conversion noted above in that most of the image data is left a resolution of at least about 2 k and at a frame rate of at least in the green image data, leaving the remaining data in a more about 23 frames per second. 45 compressible form. Thus, in accordance with an embodiment, a video camera Further, the process of green image data Subtraction can be can comprise a portable housing, and a lens assembly Sup reversed, preserving all the original raw data. Thus, the result ported by the housing and configured to focus light. A light ing system and method incorporating Such a technique can sensitive device can be configured to convert the focused light provide lossless or visually lossless and enhanced compress into raw image data with a resolution of at least 2 kata frame 50 ibility of Such video image data. rate of at least about twenty-three frames per second. The Thus, in accordance with an embodiment, a video camera camera can also include a memory device and an image can comprise a lens assembly supported by the housing and processing system configured to compress and store in the configured to focus light and a light sensitive device config memory device the raw image data at a compression ratio of ured to convert the focused light into a raw signal of image at least six to one and remain Substantially visually lossless, 55 data representing at least first, second, and third colors of the and at a rate of at least about 23 frames per second. focused light. An image processing module can be configured In accordance with another embodiment, a method of to modify image data of at least one of the first and second recording a motion video with a camera can comprise guiding colors based on the image data of the third color. Additionally, light onto a light sensitive device. The method can also the video camera can include a memory device and a com include converting the light received by the light sensitive 60 pression device configured to compress the image data of the device into raw digital image data at a rate of at least greater first, second, and third colors and to store the compressed than twenty three frames per second, compressing the raw image data on the memory device. digital image data, and recording the raw image data at a rate In accordance with another embodiment, a method of pro of at least about 23 frames per second onto a storage device. cessing an image can be provided. The method can include In accordance with yet another embodiment, a video cam 65 converting an image and into first image data representing a era can comprise a lens assembly Supported by the housing first color, second image data representing a second color, and and configured to focus light and a light sensitive device third image data representing a third color, modifying at least US 9,245,314 B2 3 4 the first image data and the second image databased on the DETAILED DESCRIPTION OF EMBODIMENTS third image data, compressing the third image data and the modified first and second image data, and storing the com FIG. 1 is a schematic diagram of a camera having image pressed data. sensing, processing, and compression modules, described in In accordance with yet another embodiment, a video cam the context of a video camera for moving pictures. The era can comprise a lens assembly Supported by the housing embodiments disclosed herein are described in the context of and configured to focus light. A light sensitive device can be a video camera having a single sensor device with a Bayer configured to convert the focused light into a raw signal of pattern filter because these embodiments have particular util image data representing at least first, second, and third colors ity in this context. However, the embodiments and inventions 10 herein can also be applied to cameras having other types of of the focused light. The camera can also include means for image sensors (e.g., CMY Bayer as well as other non-Bayer modifying image data of at least one of the first and second patterns), other numbers of image sensors, operating on dif colors based on the image data of the third color, a memory ferent image format types, and being configured for still device, and a compression device configured to compress the and/or moving pictures. Thus, it is to be understood that the image data of the first, second, and third colors and to store the 15 embodiments disclosed herein are exemplary but nonlimiting compressed image data on the memory device. embodiments, and thus, the inventions disclosed herein are not limited to the disclosed exemplary embodiments. BRIEF DESCRIPTION OF THE DRAWINGS With continued reference to FIG. 1, a camera 10 can include a body or housing 12 configured to Support a system FIG. 1 is a block diagram illustrating a system that can 14 configured to detect, process, and optionally store and/or include hardware and/or can be configured to perform meth replay video image data. For example, the system 14 can ods for processing video image data in accordance with an include optics hardware 16, an image sensor 18, an image embodiment. processing module 20, a compression module 22, and a stor FIG. 2 is an optional embodiment of a housing for the age device 24. Optionally, the camera 10 can also include a camera schematically illustrated in FIG. 1. 25 monitor module 26, a playback module 28, and a display 30. FIG. 3 is a schematic layout of an image sensor having a FIG. 2 illustrates a nonlimiting exemplary embodiment of Bayer Pattern Filter that can be used with the system illus the camera 10. As shown in FIG. 2, the optics hardware 16 can trated in FIG. 1. be supported by the housing 12 in a manner that leaves it FIG. 4 is a schematic block diagram of animage processing exposed at its outer Surface. In some embodiments, the sys module that can be used in the system illustrated in FIG. 1. 30 tem 14 is supported within the housing 12. For example, the FIG. 5 is a schematic layout of the green image data from image sensor 18, image processing module 20, and the com the green sensor cells of the image sensor of FIG. 3. pression module 22 can be housed within the housing 12. The FIG. 6 is a schematic layout of the remaining green image storage device 24 can be mounted in the housing 12. Addi data of FIG.5 after an optional process of deleting some of the tionally, in Some embodiments, the storage device 24 can be original green image data. 35 mounted to an exterior of the housing 12 and connected to the FIG. 7 is a schematic layout of the red, blue, and green remaining portions of the system 14 through any type of image data of FIG. 5 organized for processing in the image known connector or cable. Additionally, the storage device 24 processing module of FIG. 1. can be connected to the housing 12 with a flexible cable, thus FIG. 8 is a flowchart illustrating an image data transforma allowing the storage device 24 to be moved somewhat inde tion technique that can be used with the system illustrated in 40 pendently from the housing 12. For example, with Such a FIG 1. flexible cable connection, the storage device 24 can be worn FIG. 8A is a flowchart illustrating a modification of the on a belt of a user, allowing the total weight of the housing 12 image data transformation technique of FIG. 8 that can also to be reduced. Further, in Some embodiments, the housing can be used with the system illustrated in FIG. 1. include one or more storage devices 24 inside and mounted to FIG. 9 is a schematic layout of blue image data resulting 45 its exterior. Additionally, the housing 12 can also support the from an image transformation process of FIG. 8. monitor module 26, and playback module 28. Additionally, in FIG. 10 is a schematic layout of red image data resulting some embodiments, the display 30 can be configured to be from an image transformation process of FIG. 8. mounted to an exterior of the housing 12. FIG. 11 illustrates an exemplary optional transform that The optics hardware 16 can be in the form of a lens system can be applied to the image data for gamma correction. 50 having at least one lens configured to focus an incoming FIG. 12 is a flowchart of a control routine that can be used image onto the image sensor 18. The optics hardware 16, with the system of FIG. 1 to decompress and demosaic image optionally, can be in the form of a multi-lens system providing data. variable Zoom, aperture, and focus. Additionally, the optics FIG. 12A is a flowchart illustrating a modification of the hardware 16 can be in the form of a lens socket supported by control routine of FIG. 12 that can also be used with the 55 the housing 12 and configured to receive a plurality of differ system illustrated in FIG. 1. ent types of lens systems for example, but without limitation, FIG. 13 is a schematic layout of green image data having the optics hardware 16 include a socket configured to receive been decompressed and demosaiced according to the flow various sizes of lens systems including a 50-100 millimeter chart of FIG. 12. (F2.8)Zoom lens, an 18-50 millimeter (F2.8) Zoom lens, a 300 FIG. 14 is a schematic layout of half of the original green 60 millimeter (F2.8) lens, 15 millimeter (F2.8) lens, 25 millime image data from FIG. 13, having been decompressed and ter (F1.9) lens, 35 millimeter (F1.9) lens, 50 millimeter (F1.9) demosaiced according to the flowchart of FIG. 12. lens, 85 millimeter (F1.9) lens, and/or any other lens. As FIG. 15 is a schematic layout of blue image data having noted above, the optics hardware 16 can be configured such been decompressed according to the flowchart of FIG. 12. that despite which lens is attached thereto, images can be FIG.16 is a schematic layout of blue image data of FIG. 15 65 focused upon a light-sensitive Surface of the image sensor 18. having been demosaiced according to the flowchart of FIG. The image sensor 18 can be any type of video sensing 12. device, including, for example, but without limitation, CCD, US 9,245,314 B2 5 6 CMOS, vertically-stacked CMOS devices such as the This is merely one example of the pattern of green image data Foveon R sensor, or a multi-sensor array using a prism to that can be deleted. Other patterns and other amounts of green divide light between the sensors. In some embodiments, the image data can also be deleted. image sensor 18 can include a CMOS device having about 12 In some alternatives, the camera 10 can be configured to million photocells. However, other size sensors can also be used. In some configurations, camera 10 can be configured to delete /2 of the green image data after the red and blue image output video at “2 k” (e.g., 2048x1152 pixels), “4 k” (e.g., data has been transformed based on the green image data. 4,096x2.540 pixels), “4.5 k” horizontal resolution or greater This optional technique is described below following the resolutions. As used herein, in the terms expressed in the description of the Subtraction of green image data values from format of xk (such as 2 k and 4 k noted above), the “x' the other color image data. quantity refers to the approximate horizontal resolution. As 10 Optionally, the image processing module 20 can be con such, “4 k” resolution corresponds to about 4000 or more figured to selectively delete green image data. For example, horizontal pixels and “2 k” corresponds to about 2000 or more the image processing module 20 can include a deletion analy pixels. Using currently commercially available hardware, the sis module (not shown) configured to selectively determine sensor can be as Small as about 0.5 inches (8 mm), but it can which green image data to delete. For example, such a dele be about 1.0 inches, or larger. Additionally, the image sensor 15 18 can be configured to provide variable resolution by selec tion module can be configured to determine if deleting a tively outputting only a predetermined portion of the sensor pattern of rows from the green image data would result in 18. For example, the sensor 18 and/or the image processing aliasing artifacts, such as Moiré lines, or other visually per module can be configured to allow a user to identify the ceptible artifacts. The deletion module can be further config resolution of the image data output. ured to choose a pattern of green image data to delete that The camera 10 can also be configured to downsample and would present less risk of creating Such artifacts. For subsequently process the output of the sensor 18 to yield example, the deletion module can be configured to choose a video output at 2 K, 1080p, 720p, or any other resolution. For green image data deletion pattern of alternating vertical col example, the image data from the sensor 18 can be “win umns if it determines that the image captured by the image dowed, thereby reducing the size of the output image and 25 sensor 18 includes an image feature characterized by a plu allowing for higher readout speeds. However, other size sen rality of parallel horizontal lines. This deletion pattern can sors can also be used. Additionally, the camera 10 can be reduce or eliminate artifacts, such as Moiré lines, that might configured to upsample the output of the sensor 18 to yield have resulted from a deletion pattern of alternating lines of Video output at higher resolutions. image data parallel to the horizontal lines detected in the With reference to FIGS. 1 and 3, in some embodiments, the 30 image. sensor 18 can include a Bayer pattern filter. As such, the However, this merely one exemplary, non-limiting sensor 18, by way of its chipset (not shown) outputs data example of the types of image features and deletion patterns representing magnitudes of red, green, or blue light detected that can be used by the deletion module. The deletion module by individual photocells of the image sensor 18. FIG.3 sche can also be configured to detect other image features and to matically illustrates the Bayer pattern output of the sensor 18. 35 use other image data deletion patterns, such as for example, In some embodiments, for example, as shown in FIG. 3, the but without limitation, deletion of alternating rows, alternat Bayer pattern filter has twice as many green elements as the ing diagonal lines, or other patterns. Additionally, the deletion number of red elements and the number of blue elements. The module can be configured to delete portions of the other chipset of the image sensor 18 can be used to read the charge image data, Such as the red and blue image data, or other on each element of the image sensor and thus output a stream 40 image data depending on the type of sensor used. of values in the well-known RGB format output. Additionally, the camera 10 can be configured to insert a With continued reference to FIG. 4, the image processing data field into the image data indicating what image data has module 20 optionally can be configured to format the data been deleted. For example, but without limitation, the camera stream from the image sensor 18 in any known manner. In 10 can be configured to insert a data field into the beginning Some embodiments, the image processing module 20 can be 45 of any video clip stored into the storage device 24, indicating configured to separate the green, red, and blue image data into what data has been deleted in each of the “frames' of the three or four separate data compilations. For example, the Video clip. In some embodiments, the camera can be config image processing module 20 can be configured to separate the ured to insert a data field into each frame captured by the red data into one red data element, the blue data into one blue sensor 18, indicating what image data has been deleted. For data element, and the green data into one green data element. 50 example, in some embodiments, where the image processing For example, with reference to FIG. 4, the image processing module 20 is configured to delete /2 of the green image data module 20 can include a red data processing module 32, a in one deletion pattern, the data field can be as Small as a blue data image processing module 34, and a first green image single bit data field, indicating whether or not image data has data processing module 36. been deleted. Since the image processing module 20 is con As noted above, however, the Bayer pattern data illustrated 55 figured to delete data in only one pattern, a single bit is in FIG. 3, has twice as many green pixels as the other two sufficient to indicate what data has been deleted. colors. FIG. 5 illustrates a data component with the blue and In some embodiments, as noted above, the image process red data removed, leaving only the original green image data. ing module 20 can be configured to selectively delete image In some embodiments, the camera 10 can be configured to data in more than one pattern. Thus, the image data deletion delete or omit some of the green image data. For example, in 60 field can be larger, including a sufficient number of values to Some embodiments, the image processing module 20 can be provide an indication of which of the plurality of different configured to delete /2 of the green image data so that the total image data deletion patterns was used. This data field can be amount ofgreen image data is the same as the amounts of blue used by downstream components and or processes to deter and red image data. For example, FIG. 6 illustrates the mine to which spatial positions the remaining image data remaining data after the image processing module 20 deletes 65 corresponds. /2 of the green image data. In the illustrated embodiment of In some embodiments, the image processing module can FIG. 6, the rows n-3, n-1, n+1, and n+3 have been deleted. be configured to retain all of the raw green image data, e.g., US 9,245,314 B2 7 8 the data shown in FIG. 5. In such embodiments, the image better compression. The amount of compression will depend, processing module can include one or more green image data at least in part, on the entropy of the original information in processing modules. the image. As noted above, in known Bayer pattern filters, there are In some embodiments, the magnitudes Subtracted from a twice as many green elements as the number of red elements red or blue pixel can be the magnitude of the value output and the number of blue elements. In other words, the red from a green pixel adjacent to the Subject red or blue pixel. elements comprise 25% of the total Bayer pattern array, the Further, in some embodiments, the green magnitude Sub blue elements corresponded 25% of the Bayer pattern array tracted from the red or blue elements can be derived from an and the green elements comprise 50% of the elements of the average of the Surrounding green elements. Such techniques Bayer patternarray. Thus, in some embodiments, where all of 10 are described in greater detail below. However, other tech the green image data is retained, the image processing module niques can also be used. 20 can include a second green data image processing module Optionally, the image processing module 20 can also be 38. As such, the first green data image processing module 36 configured to selectively subtract green image data from the can process half of the green elements and the second green other colors. For example, the image processing module 20 image data processing module 38 can process the remaining 15 can be configured to determine if subtracting green image green elements. However, the present inventions can be used data from a portion of the image data of either of the other in conjunction with other types of patterns. Such as for colors would provide better compressibility or not. In this example, but without limitation, CMY and RGBW. mode, the image processing module 20 can be configured to FIG. 7 includes schematic illustrations of the red, blue and insert flags into the image data indicating what portions of the two green data components processed by modules 32, 34, 36. image data has been modified (by e.g., green image data and 38 (FIG. 4). This can provide further advantages because subtraction) and which portions have not been so modified. the size and configuration of each of these modules can be With Such flags, a downstream demosaicing/reconstruction about the same since they are handling about the same amount component can selectively add green image values back into of data. Additionally, the image processing module 20 can be 25 the image data of the other colors, based on the status of such selectively switched between modes in which is processes all data flags. of the green image data (by using both modules 36 and 38) Optionally, image processing module 20 can also include a and modes where /2 of the green image data is deleted (in further data reduction module (not shown) configured to which it utilizes only one of modules 36 and 38). However, round values of the red and blue data. For example, if, after the other configurations can also be used. 30 Subtraction of green magnitudes, the red or blue data is near Zero (e.g., within one or two on an 8-bit scale ranging from Additionally, in some embodiments, the image processing 0-255 or higher magnitudes for a higher resolution system). module 20 can include other modules and/or can be config For example, the sensor 18 can be a 12-bit sensor outputting ured to perform other processes, such as, for example, but red, blue, and green data on a scale of 0-4095. Any rounding without limitation, gamma correction processes, noise filter 35 or filtering of the data performed the rounding module can be ing processes, etc. adjusted to achieve the desired effect. For example, rounding Additionally, in Some embodiments, the image processing can be performed to a lesser extent if it is desired to have module 20 can be configured to Subtract a value of a green lossless output and to a greater extent if some loss or lossy element from a value of a blue element and/or red element. As output is acceptable. Some rounding can be performed and Such, in some embodiments, when certain colors are detected 40 still result in a visually lossless output. For example, on a 8-bit by the image sensor 18, the corresponding red or blue element scale, red or blue data having absolute value of up to 2 or 3 can can be reduced to Zero. For example, in many photographs, be rounded to 0 and still provide a visually lossless output. there can be large areas of black, white, or gray, or a color Additionally, on a 12-bit scale, red or blue data having an shifted from gray toward the red or blue colors. Thus, if the absolute value of up to 10 to 20 can be rounded to 0 and still corresponding pixels of the image sensor 18 have sensed an 45 provide visually lossless output. area of gray, the magnitude of the green, red, and blue, would Additionally, the magnitudes of values that can be rounded be about equal. Thus, if the green value is subtracted from the to Zero, or rounded to other values, and still provide a visually red and blue values, the red and blue values will drop to Zero lossless output depends on the configuration of the system, or near Zero. Thus, in a Subsequent compression process, including the optics hardware 16, the image sensor 18, the there will be more Zeros generated in pixels that sense a black, 50 resolution of the image sensor, the color resolution (bit) of the white, or gray area and thus the resulting data will be more image sensor 18, the types of filtering, anti-aliasing tech compressible. Additionally, the Subtraction of green from one niques or other techniques performed by the image process or both of the other colors can make the resulting image data ing module 20, the compression techniques performed by the more compressible for other reasons. compression module 22, and/or other parameters or charac Such a technique can help achieve a higher effective com 55 teristics of the camera 10. pression ratio and yet remain visually lossless due to its As noted above, in Some embodiments, the camera 10 can relationship to the entropy of the original image data. For be configured to delete /2 of the green image data after the red example, the entropy of an image is related to the amount of and blue image data has been transformed based on the green randomness in the image. The Subtraction of image data of image data. For example, but without limitation, the proces one color, for example, from image data of the other colors 60 sor module 20 can be configured to delete /2 of the green can reduce the randomness, and thus reduce the entropy of the image data after the average of the magnitudes of the Sur image data of those colors, thereby allowing the data to be rounding green data values have been Subtracted from the red compressed at higher compression ratios with less loss. Typi and blue data values. This reduction in the green data can cally, an image is not a collection of random color values. reduce throughput requirements on the associated hardware. Rather, there is often a certain degree of correlation between 65 Additionally, the remaining green image data can be used to Surrounding picture elements. Thus, Such a Subtraction tech reconstruct the red and blue image data, described in greater nique can use the correlation of picture elements to achieve detail below with reference to FIGS. 14 and 16. US 9,245,314 B2 9 10 As noted above, the camera10 can also include a compres about 30 minutes of video at 12 megapixel resolution, 12-bit sion module 22. The compression module 22 can be in the color resolution, and at 60 frames per second. However, the form of a separate chip or it can be implemented with soft storage device 24 can have any size. ware and another processor. For example, the compression In some embodiments, the storage device 24 can be module 22 can be in the form of a commercially available 5 mounted on an exterior of the housing 12. Further, in some compression chip that performs a compression technique in embodiments, the storage device 24 can be connected to the accordance with the JPEG 2000 standard, or other compres other components of the system 14 through standard commu sion techniques. nication ports, including, for example, but without limitation, The compression module can be configured to performany IEEE 1394, USB 2.0, IDE, SATA, etc. Further, in some type of compression process on the data from the image 10 embodiments, the storage device 24 can comprise a plurality processing module 20. In some embodiments, the compres of hard drives operating under a RAID protocol. However, sion module 22 performs a compression technique that takes any type of storage device can be used. advantage of the techniques performed by the image process With continued reference to FIG. 1, as noted above, in ing module 20. For example, as noted above, the image pro Some embodiments, the system can include a monitor module cessing module 20 can be configured to reduce the magnitude 15 26 and a display device 30 configured to allow a user to view of the values of the red and blue data by subtracting the Video images captured by the image sensor 18 during opera magnitudes of green image data, thereby resulting in a greater tion. In some embodiments, the image processing module 20 number of Zero values, as well as other effects. Additionally, can include a Subsampling system configured to output the image processing module 20 can perform a manipulation reduced resolution image data to the monitor module 26. For of raw data that uses the entropy of the image data. Thus, the example, such a Subsampling system can be configured to compression technique performed by the compression mod output video image data to support 2 K, 1080p, 720p, or any ule 22 can be of a type that benefits from the presence of larger other resolution. In some embodiments, filters used for demo strings of Zeros to reduce the size of the compressed data saicing can be adapted to also perform downsampling filter output therefrom. ing, such that downsampling and filtering can be performed at Further, the compression module 22 can be configured to 25 the same time. The monitor module 26 can be configured to compress the image data from the image processing module performany type of demosaicing process to the data from the 20 to result in a visually lossless output. For example, firstly, image processing module 20. Thereafter, the monitor module the compression module can be configured to apply any 26 can output a demosaiced image data to the display 30. known compression technique. Such as, but without limita The display 30 can be any type of monitoring device. For tion, JPEG 2000, Motion.JPEG, any DCT based codec, any 30 example, but without limitation, the display 30 can be a codec designed for compressing RGB image data, H.264, four-inch LCD panel supported by the housing 12. For MPEG4, Huffman, or other techniques. example, in some embodiments, the display 30 can be con Depending on the type of compression technique used, the nected to an infinitely adjustable mount configured to allow various parameters of the compression technique can be setto the display 30 to be adjusted to any position relative to the provide a visually lossless output. For example, many of the 35 housing 12 so that a user can view the display 30 at any angle compression techniques noted above can be adjusted to dif relative to the housing 12. In some embodiments, the display ferent compression rates, wherein when decompressed, the 30 can be connected to the monitor module through any type resulting image is better quality for lower compression rates of video cables such as, for example, an RGB orYCC format and lower quality for higher compression rates. Thus, the video cable. compression module can be configured to compress the 40 Optionally, the playback module 28 can be configured to image data in away that provides a visually lossless output, or receive data from the storage device 24, decompressed and can be configured to allow a user to adjust various parameters demosaic the image data and then output the image data to the to obtain a visually lossless output. For example, the com display 30. In some embodiments, the monitor module 26 and pression module 22 can be configured to compress the image the playback module 28 can be connected to the display data at a compression ratio of about 6:1, 7:1, 8:1 or greater. In 45 through an intermediary display controller (not shown). As Some embodiments, the compression module 22 can be con Such, the display 30 can be connected with a single connector figured to compress the image data to a ratio of 12:1 or higher. to the display controller. The display controller can be con Additionally, the compression module 22 can be config figured to transfer data from either the monitor module 26 or ured to allow a user to adjust the compression ratio achieved the playback module 28 to the display 30. by the compression module 22. For example, the camera 10 50 FIG. 8 includes a flowchart 50 illustrating the processing of can include a user interface that allows a user to input com image data by the camera 10. In some embodiments, the mands that cause the compression module 22 to change the flowchart 50 can represent a control routine stored in a compression ratio. Thus, in Some embodiments, the camera memory device. Such as the storage device 24, or another 10 can provide for variable compression. storage device (not shown) within the camera 10. Addition As used herein, the term “visually lossless” is intended to 55 ally, a central processing unit (CPU) (not shown) can be include output that, when compared side by side with original configured to execute the control routine. The below descrip (never compressed) image data on the same display device, tion of the methods corresponding to the flow chart 50 are one of ordinary skill in the art would not be able to determine described in the context of the processing of a single frame of which image is the original with a reasonable degree of accu Video image data. Thus, the techniques can be applied to the racy, based only on a visual inspection of the images. 60 processing of a single still image. These processes can also be With continued reference to FIG. 1, the camera10 can also applied to the processing of continuous video, e.g., frame include a storage device 24. The storage device can be in the rates of greater than 12, as well as frame rates of 20, 23.976, form of any type of digital storage. Such as, for example, but 24, 30, 60, and 120, or other framerates between these frame without limitation, hard disks, flash memory, or any other rates or greater. type of memory device. In some embodiments, the size of the 65 With continued reference to FIG. 8, control routine can storage device 24 can be sufficiently large to store image data begin at operation block 52. In the operation block 52, the from the compression module 22 corresponding to at least camera 10 can obtain sensor data. For example, with refer US 9,245,314 B2 11 12 ence to FIG. 1, the image sensor 18, which can include a different processes can be applied to the different colors of Bayer Sensor and chipset, can output image data. image data. However, these are merely exemplary curves that For example, but without limitation, with reference to FIG. can be used to process the image data, or curves or transforms 3, the image sensor can comprise a CMOS device having a can also be used. Additionally, these processing techniques Bayer pattern filter on its light receiving surface. Thus, the can be applied using mathematical functions such as those focused image from the optics hardware 16 is focused on the noted above, or with Look Up Tables (LUTs). Additionally, Bayer pattern filter on the CMOS device of the image sensor different processes, techniques, or transforms can be used for 18. FIG. 3 illustrates an example of the Bayer pattern created different types of image data, different ISO settings used by the arrangement of Bayer pattern filter on the CMOS during recording of the image data, temperature (which can device. 10 affect noise levels), etc. In FIG. 3, column m is the fourth column from the left edge After the operation block 54, the flowchart 50 can move to of the Bayer pattern and row n is the fourth row from the top an operation block 56. In the operation block 56, the red and of the pattern. The remaining columns and rows are labeled blue picture elements can be transformed. For example, as relative to column m and row n. However, this layout is noted above, green image data can be subtracted from each of merely chosen arbitrarily for purposes of illustration, and 15 the blue and red image data components. In some embodi does not limitany of the embodiments or inventions disclosed ments, a red or blue image data value can be transformed by herein. Subtracting a green image data value of at least one of the As noted above, known Bayer pattern filters often include green picture elements adjacent to the red or blue picture twice as many green elements as blue and red elements. In the element. In some embodiments, an average value of the data pattern of FIG. 5, blue elements only appear in rows n-3, n-1, values of a plurality of adjacent green picture elements can be n+1, and n+3. Red elements only appear in rows n-2, n, n+2, subtracted from the red or blue image data value. For and n+4. However, green elements appear in all rows and example, but without limitation, average values of 2, 3, 4, or columns, interspersed with the red and blue elements. more green image data values can be calculated and Sub Thus, in the operation block 52, the red, blue, and green tracted from red or blue picture elements in the vicinity of the image data output from the image sensor 18 can be received 25 green picture elements. by the image processing module 20 and organized into sepa For example, but without limitation, with reference to FIG. rate color data components, such as those illustrated in FIG.7. 3, the raw output for the red element R-22 is surrounded As shown in FIG. 7, and as described above with reference to by four green picture elements G-2 s. G-1-2. G. s. 2. FIG. 4, the image processing module 20 can separate the red, and G-2-1. Thus, the red element R-22 can be trans blue, and green image data into four separate components. 30 formed by subtracting the average of the values of the sur FIG. 7 illustrates two green components (Green 1 and Green rounding green element as follows: 2), a blue component, and a red component. However, this is merely one exemplary way of processing image data from the +1. image sensor 18. Additionally, as noted above, the image Similarly, the blue elements can be transformed in a similar processing module 20, optionally, can arbitrarily or selec 35 manner by Subtracting the average of the Surrounding green tively delete /2 of the green image data. elements as follows: After the operation block 52, the flowchart 50 can move on to operation block 54. In the operation block 54, the image data can be further processed. For example, optionally, any G-1)/4 (2) one or all of the resulting data (e.g., green 1, green 2, the blue 40 FIG. 9 illustrates a resulting blue data component where image data from FIG.9, and the red image data from FIG.10) the original blue raw data B, is transformed, the new can be further processed. value labeled as B', (only one value in the component is For example, the image data can be pre-emphasized or filled in and the same technique can be used for all the blue processed in other ways. In some embodiments, the image elements). Similarly, FIG. 10 illustrates the red data compo data can be processed to be more (mathematically) non-lin 45 nent having been transformed in which the transformed red ear. Some compression algorithms benefit from performing element R-22 is identified as R-2-2. In this state, the Such a linearization on the picture elements prior to compres image data can still be considered “raw' data. For example, Sion. However, other techniques can also be used. For the mathematical process performed on the data are entirely example, the image data can be processed with a linear curve, reversible such that all of the original values can be obtained which provides essentially no emphasis. 50 by reversing those processes. In some embodiments, the operation block 54 can process With continued reference to FIG. 8, after the operation the image data using curve defined by the function y=x-0.5. In block 56, the flowchart 50 can move on to an operation block Some embodiments, this curve can be used where the image 58. In the operation block 58, the resulting data, which is raw data was, for example but without limitation, floating point or can be substantially raw, can be further compressed to data in the normalized 0-1 range. In other embodiments, for 55 using any known compression algorithm. For example, the example, where the image data is 12-bit data, the image can compression module 22 (FIG. 1) can be configured to per be processed with the curvey=(X/4095)0.5. Additionally, the form Such a compression algorithm. After compression, the image data can be processed with other curves, such as y=(X+ compressed raw data can be stored in the storage device 24 c) g where 0.01