USOO8654842B2 (12) United States Patent (10) Patent No.: US 8,654,842 B2 Ganesh et al. (45) Date of Patent: Feb. 18, 2014 (54) ACCELERATED VIDEO ENCODING 6,044,408 A 3/2000 Engstrom et al. 6,101,276 A 8, 2000 Adiletta et al. 6,128,026 A 10/2000 Brothers, III (75) Inventors: SNESN RWSRS 6,252,905 B1 6/2001 Pokrinchak et al. onald J. Yunsil, Kirkland, (US); 6,434,196 B1* 8/2002 Sethuraman et al. ... 375/240. 12 Gary J. Sullivan, Redmond, WA (US); 6,721,359 B1 4/2004 Bist et al. Glenn F. Evans, Kirkland, WA (US); (Continued) Shyam Sadhwani, Bellevue, WA (US); Stephen J. Estrop, Carnation, WA (US) FOREIGN PATENT DOCUMENTS (73) Assignee: Microsoft Corporation, Redmond, WA EP 1347,650 9, 2003 (US) WO WOO2O7446 A2 1, 2001 WO WOO219095 3, 2002 (*) Notice: Subject to any disclaimer, the term of this OTHER PUBLICATIONS patent is extended or adjusted under 35 U.S.C. 154(b) by 1651 days. Rossi, “Multicore Signal Processing Platform with Heterogeneous Configurable Hardware Accelerators', 2013, IEEE, p. 1-14.* (21) Appl. No.: 11/673,423 (Continued) (22) Filed: Feb. 9, 2007 Primary Examiner — Taghi Arani (65) Prior Publication Data Assistant Examiner — Gregory Lane (74) Attorney, Agent, or Firm — Carole A Boelitz: Raghu US 2007/O2O1562 A1 Aug. 30, 2007 Chinagudabha; Micky Minhas Related U.S. Application Data (57) ABSTRACT (63) Continuation-in-part of application No. 1 1/276,336, A video encoding acceleration service to increase one or filed on Feb. 24, 2006. more of the speed and quality of video encoding is described. The service acts as an intermediary between an arbitrary (51) Int. Cl. Video encoder computer program application and arbitrary H04N 7/2 (2006.01) video acceleration hardware. The service receives one or (52) U.S. Cl. more queries from the video encoder to identify implemen USPC ............ 375/240.12; 375/240.22; 375/240.25; tation specifics of the video acceleration hardware. The ser 382/232 vice interfaces with the video acceleration hardware to obtain (58) Field of Classification Search the implementation specifics. The service communicates the USPC ........... 375/240.22, 240.12, 240.16; 382/232, implementation specifics to the video encoder. The imple 382/239 mentation specifics enable the video encoder to: (a) deter See application file for complete search history. mine whether one or more of speed and quality of Software encoding operations associated with the video encoder can be (56) References Cited increased with implementation of a pipeline of one or more Supported encoding pipeline configurations and capabilities, U.S. PATENT DOCUMENTS and (b) implement the pipeline by interfacing with the ser 5,805,228 A * 9/1998 Proctor et al. ........... 375,240.22 vice. 5,926,226 A 7, 1999 Proctor et al. 5.990,958 A 11/1999 Bheda et al. 20 Claims, 7 Drawing Sheets Assis - i3 E".is ------ 8 programodules 108 IrpiiWidea Wides Processifieds 112 Data - Siedadsskxus E.g., a third-party Appliastian of 122 Arbitrary Architecture. Etc. 18 Aceclarated Wida Encoding 8erwkca 18 ther processing Modula (E.g., decoder, Flitars, rendering, .) 2. itserogram Module 14 .G., operating system, device drivers, Ete Programata i Erded Wis site 128 star Progren Data E.g., decoded source Wideo data tooder objects), seriss, Reaponses, Acces?ation Hardwars implementation Specifics. Selected Widericklig Pipeline Configuration. Allocated Buforts, search Protes, Metadata, Etc. 24: US 8,654,842 B2 Page 2 (56) References Cited “Sage TV: Why does SageTV “look” better than it's competition? (picture quality)'. http://htpcnew.com/main/php?id=interview 1. U.S. PATENT DOCUMENTS Feb. 23, 2006, 3 pages. “Why MetaVR Uses Microsoft DirectX'. http://www.metavir.com/ 6,748,019 B1 6, 2004 Lin et al. technology/directX.html, Feb. 23, 2006, 3 pages. 7,634,776 B2 * 12/2009 Parameswaran et al. ..... T18, 105 “Windows Media; Enabling DirectX Video Acceleration'. http:// 2002fOO65952 A1 5, 2002 Sullivan et al. msdn.microsoft.com/library/default.asp?url=/library/en-us/ 2002/0114394 A1* 8, 2002 Ma ........................... 375,24O16 wmform95/htm/enablingdirectXvideoacceleration.asp, Feb. 23, 2003/0052909 A1 3/2003 Mo et al. 2006, 3 pages. 2003/0118103 A1* 6/2003 Guevorkian et al. ..... 375,240.12 Barjatya, “Block Matching Algorithms for Motion Estimation'. 2003. O156643 A1 8/2003 Song Digital Image Processing in Natural Sciences and Medicine, No. 2003/0195998 A1 10/2003 Estrop 2004/02284.05 A1 11, 2004 Yu et al. class ece 6620, Apr. 26, 2004, pp. 1-6. 2004/0268.397 A1* 12/2004 Dunbar et al. .................. 725/88 Extended European Search Report mailed Jul. 5, 2012 for European 2005.0024363 A1 2/2005 Estrop patent application No. 0775.1404.0, 16 pages. 2005.0025241 A1 2/2005 Sullivan et al. Harrand, et al., “A Single-Chip CIF 30-Hz, H261, H263, and H263+ 2005, OO63461 A1 3/2005 Lee et al. Video Encoder/Decoder with Embedded Display Controller', IEEE 2005/00781-19 A1 4/2005 Chiaruzzi et al. Journal of Solid-State Circuits, IEEE Service Center, Piscataway, NJ, 2005/0094730 A1 5/2005 Chang et al. USA, vol. 34, No. 11, Nov. 1, 1999. 2005, 0123057 A1* 6/2005 MacInnis et al. ........ 375,240.25 Kelly, et al., “Fast Image Interpolation for Motion Estimation using 2006.0056708 A1* 3, 2006 Shen et al. .. 382,232 Graphics Hardware', Real-Time Imaging VIII, Jan. 20-22, 2004, San 2006/0197765 A1 9, 2006 Dutta et al. ................... 345,506 Jose, CA, USA, Proceedings of Electronic Imaging, Science and Technology, Jan. 20, 2004, pp. 184-194. OTHER PUBLICATIONS Korean Office Action mailed Apr. 24, 2012 for Korean patent appli cationNo. 10-2008-7020009, a counterpart foreign application of US “CinePlayer DVD Decoder Pack for Windows XP; Summary”, http:// patent No. 7,929,599, 3 pages. www.Sonic.com/products/Consumer/CinePlayer/Technology/accel Pastrnak, et al., “Hierarchical QoS concept for Multiprocessor Sys eration.aspx. 14 pages. tem-on-chip” Workshop on Resource Management for Media Pro http://www.hisdigital.com/html/9000series brochure.pdf. cessing in Networked Embedded Systems, Mar. 31, 2005, pp. 139 http://msdn.microsoft.com/library/default.asp?url=/library/en-us/ 142. graphics/hh?.graphics/dxvaguide d531ae3a-1132-44e9-9931 Sullivan, et al., “Windows Platform Design Notes: Designing dObdd76e676fxm.asp. Hardward for the Microsoft Family of Operating Systems— http://www.nforcershq.com/article-print-891.html. Microsoft DirectXVA: Video Acceleration API/DDI' Jan. 23, 2001, “OC Workbench', http://www.ocworkbench.com/hardware/ pp. 1-88, retrieved from the internet at http://download.microsoft. Leadtekwinfast'620gts/winfast2.htm, 2 pages. comfclownload? 1/6/1/161ba512-40e2-4cc9-843a-923143f.3456cf. “Sage TV: Why does SageTV “look” better than it's competition? Yang, et al., “Search Speed and Power Driven Integraged Software (picture quality)'. http://htpcnew.com/main.php?id=interview1, 3 and Hardware Optimizations for Motion Estimation Algorithms'. pageS. 2004 IEEE International Conference on Multimedia and Expo, Jun. “Why MetaVR Uses Microsoft DirectX'. http://www.metavir.com/ 27-30, 2004, Taipei, Taiwan, IEEOperations Center, Piscataway, NJ, technology/directX.html, 3 pages. vol. 1, pp. 707-710. “Windows Media; Enabling DirectX Video Acceleration'. http:// JP Office Action mailed Dec. 25, 2012 for JP patent application No. msdn.microsoft.com/library/default.asp?url=/library/en-us/ 2008-556422 a counterpart foreign application of US patent No. Wmform95/htm/enablingdirectXvideoacceleration.asp, 3 pages. 7,929,599, 8 pages. “CinePlayer DVD Decoder Pack for Windows XP; Summary”, http:// Shen et al., “Accelerate Video Decoding With Generic GPU”, IEEE www.Sonic.com/products/Consumer/CinePlayer/Technology/accel Transaction on Circuits and Systems for Video Technology, IEEE, eration.aspx. Feb. 23, 2006, 14 pages. May 2005, vol. 15, No. 5, pp. 685-pp. 693. Excalibur RADEON, http://www.hisdigital.com/html/9000series Cheng et al., “Fast Blocking Matching Algorithms for Motion Esti brochure.pdf downloaded at least as early as Jun. 15, 2005. mation', 1996 IEEE International Conference on Acoustics, Speech, http://msdn.microsoft.com/library/default.asp?url=/library/en-us/ and Signal Processings Proceedsings, Alanta, May 7-10, 1996, pp. graphics/hh?.graphics/dxvaguide d531ae3a-1132-44e9-9931 2311-2314. dObdd76e676fxm.asp downloaded at least as early as Jun. 15, 2005. European Office Action mailed May 25, 2013 for European Patent http://www.nforcershq.com/article-print-891.html downloaded at Application No. 07751404.0, a counterpart foreign application of US least as early as Jun. 15, 2005. Patent No. 7,929.599, 7 pages. “OC Workbench', http://www.ocworkbench.com/hardware/ Leadtek?winfast'620gts/winfast2.htm, Feb. 23, 2006, 2 pages. * cited by examiner U.S. Patent Feb. 18, 2014 Sheet 1 of 7 US 8,654,842 B2 Host Computing Device 102 Host Acceleration Processor(s) Hardware Input Video Data Wided Encode Module 122 (E.g., a Third-Party Application of Arbitrary Architecture, Etc.) 116 Accelerated Video Encoding Service 118 WA APIs Other Processing Modules 128 (E.g., Decoder, Filters, Rendering, Etc.) 120 Other Program Modules 114 (E.G., Operating System, Device Drivers, Etc.) Encoded Wided Data 126 Other Program Data (E.G., Decoded Source Video Data, Encoder Object(s), Queries, Responses, Acceleration Hardware implementation Specifics, Selected Video Encoding Pipeline Configuration, Allocated Buffer(s), Search Profiles, Metadata, Etc.) 124 U.S. Patent Feb. 18, 2014 Sheet 2 of 7 US 8,654,842 B2 Host Raw Wideo Accelerator Source Picture Memory \ Store -1 e s er m 2O6 208 Mode ecision, (Dotted paths may or may MW Selection not be present in design) & Rate Cotro 1 Coded - Picture Memory Store 1. - ..... --- Trasform & Cantization y Cuantizer riversion & 28 Transform in Welsior Entropy Code Wideo estinatio U.S. Patent Feb. 18, 2014 Sheet 3 of 7 US 8,654,842 B2 Evaluate Supported Pipeline Configuration(s) and Receive Encoded or Decoded implementation Specifics To Source Video Data ldentify Operations That Could Benefit from Hardware Acceleration, Make Speed/Quality Decisions, Minimize Dataflow Transitions, and for Etc.