(12) United States Patent (10) Patent No.: US 8,929,443 B2 Crinon Et Al
Total Page:16
File Type:pdf, Size:1020Kb
USOO8929.443B2 (12) United States Patent (10) Patent No.: US 8,929,443 B2 Crinon et al. (45) Date of Patent: Jan. 6, 2015 (54) RECOVERING FROM DROPPED FRAMES IN 6,289,054 B1 9/2001 Rhee REAL-TIME TRANSMISSION OF VIDEO 6,530,055 B1* 3/2003 Fukunaga ..................... T14f746 6,874,118 B1 3/2005 Pirzadeh et al. OVER PNETWORKS 7.684,430 B2 * 3/2010 Gaur et al. .................... 370,447 2002/0159525 A1 10/2002 Jeong (75) Inventors: Regis J. Crinon, Camas, WA (US); 2003/0097631 A1 5/2003 Cheung Walid Ali, Sammamish, WA (US); 2004/0027991 A1* 2/2004 Jang et al. ..................... 370,230 Nelson Philip Pinto, Redmond, WA 2005/0036546 A1* 2/2005 Rey et al. ................. 375,240.12 (US) s s 2006,0005101 A1 1/2006 Li et al. 2006, OO13318 A1 1/2006 Webb et al. 2008/0100694 A1* 5/2008 Barkley et al. ............. 348, 14.08 (73) Assignee: Microsoft Corporation, Redmond, WA 2008/01 15185 A1 5/2008 Qiu et al. US (US) OTHER PUBLICATIONS (*) Notice: Subject to any disclaimer, the term of this & 8 -- patent is extended or adjusted under 35 Rhee, et al., “Error Recovery for Interactive Video Transmission over U.S.C. 154(b) by 1354 days. the Internet, retrieved at (shttp://ieeexplore.ieee.org/stampfstamp. jsp?arnumber=00848254>>, IEEE Journal on Selected Areas in (21) Appl. No.: 12/350,975 Communications, vol. 18, No. 6, Jun. 2000, p. 1033-1049. (22) Filed: Jan. 9, 2009 (Continued) (65) Prior Publication Data Primary Examiner — Hassan Phillips f 1 Assistant Examiner — Mark A Mais US 2010/0177776A1 Jul. 15, 2010 (74) Attorney, Agent, or Firm — Andrew Smith; Jim Ross; (51) Int. Cl. Micky Minhas H04N 7/2 (2006.01) H04N 9/03 (2014.01) (57) ABSTRACT H04L 2/64 (2006.01) Technologies for recovering from dropped frames in the real H04N 9/02 (2014.01) time transmission of video over an IP network are provided. A (52) U.S. Cl. Video streaming module receives a notification from a receiv CPC ....... H04N 7/26015 (2013.01); H04N 7/26021 ing module that a data packet has been lost. The video stream (2013.01); H04L 12/6418 (2013.01) ing module determines, based on the type of video frame USPC ................................. 375/240.12; 375/240.27 conveyed in the lost packet and the timing of the lost packet in (58) Field of Classification Search relation to the sequence of video frames transmitted to the None receiving module, whether or not a replacement video frame See application file for complete search history. should be sent to the receiving module. If the video streaming module determines a replacement video frame is warranted, (56) References Cited then the video streaming module instructs a video encoding U.S. PATENT DOCUMENTS module to generate a replacement video frame and then trans mits the replacement video frame to the receiving module. 5,319,648 A * 6, 1994 Bux et al. ...................... T14f748 5,321,397 A 6, 1994 Stone 20 Claims, 8 Drawing Sheets RECOWER FROMDROPRED FRAMES IN WIDESTREAM 42 RECEIVENTIFICATION OF LOSTPACKET INCLUDINGPACKETSECUENCENUMBER 44 DETERMNE FRAMETYPE OFFRAME CONVEYED IN LOSTPACKET 405 IS YES THEDROPPEDFRAMEA B-FRAME NO 408 DETERMINESEQUENCENUMBER OFLAST PACKET OF MOSTRECENTSP-FRAME OR-FRAME(nLastkeySea) YES GENERATEAND SENDNEW SP-FRAME NO 44 IS THEDROPREDFRAMEA YES PFRAMER 46 GENERATEANOSENONEW FRAME US 8,929.443 B2 Page 2 (56) References Cited industrial controls.eetchina.com/PDF/2007JUNICCOL 2007JUNO1 IMDS DASE TA 02. OTHER PUBLICATIONS pdf?SOURCES=DOWNLOAD>>, White Paper, Aug. 2006, p. 1-16. Chen, et al., "Error-Resilient Real-Time Video Communications over Richardson, Iain. “H.264 and MPEG-4 Video Compression'. IP-Based LANS ', retrieved at <http://arboree.ntu.edu.tw/ retrieved at <http://www.sauronz.com/manuales/H.264. And. ~mschen/paperpS/ma icme()1.pdf>>, p. 4. MPEG-4.Video. Compression.pdf>>, p. 1-306. Golston, et al., “Video Compression: SystemTrade-Offs with H.264. VC-1 and other Advanced CODECs ', retrieved at http://www. k cited. by examiner U.S. Patent US 8,929.443 B2 008 U.S. Patent US 8,929.443 B2 aEEE:?aRae &][O]&&-*, E]SIEIOEJEJ&D-JSS.Elºj-1?0 ? yle\zic018;808 008 Z08 U.S. Patent US 8,929.443 B2 008 U.S. Patent Jan. 6, 2015 Sheet 7 of 8 US 8,929.443 B2 RECOVER FROMDROPPED FRAMES INVIDEO STREAM 400 RECEIVE NOTIFICATION OF LOSTPACKET INCLUDING PACKET SEQUENCE NUMBER (nLostSeq) 404 DETERMINE FRAMETYPE OF FRAME CONVEYED IN LOSTPACKET 406 IS THE DROPPED FRAMEA YES B-FRAME? DETERMINE SEQUENCE NUMBER OF LAST PACKET OF MOST RECENTSP-FRAME OR-FRAME (nLastkeySeq) 412 IS GENERATE AND SEND NEW nLostSeq > nLastkeySeq? SP-FRAME 414 IS THE DROPPED FRAMEA P-FRAME? GENERATE AND SEND NEW -FRAME Figure 4 US 8,929,443 B2 1. 2 RECOVERING FROMIDROPPED FRAMES IN FIG. 2 is a block diagram showing an example of a play REAL-TIME TRANSMISSION OF VIDEO back sequence of video frame and the dependencies between OVER PNETWORKS the video frame types, according to embodiments presented herein; BACKGROUND FIGS. 3A-3D are block diagrams showing examples of Real-time transmission of video over the Internet involves sequences of video frames transmitted and received in the the transmission of a sequence of pictures or frames from a real-time transmission of video over an IP network, according Source computer to one or more destination computers. The to embodiments described herein; sequence of frames is often generated by a video encoder FIG. 4 is a flow diagram showing methods for recovering module, according to one of any number of video compres 10 from dropped frames in the real-time transmission of video, sion and encoding standards. Each frame is divided into pack according to embodiments described herein; and ets and the packets are transmitted or “streamed by the FIG. 5 is a block diagram showing illustrative computer Source computer to the destination computers over an enter hardware and Software architectures for a computing system prise-wide IP network or the Internet. capable of implementing aspects of the embodiments pre Because of the nature of Internet communications, it is 15 sented herein. possible that one or more of the packets in the video stream may be lost during transmission, or may arrive at the desti DETAILED DESCRIPTION nation computer corrupted. Traditional techniques for the streaming of video over the Internet may account for the loss The following detailed description is directed to technolo of a packet in the video steam by retransmitting the individual gies for recovering from dropped frames during the real-time packet to the destination computer, or including error correc transmission of video over an IP network. Utilizing the tech tion information in the transmission, as in forward error cor nologies described herein, a video streaming module can be rection (“FEC’). These techniques may reduce efficiency of implemented that increases the efficiency of bandwidth utili available bandwidth utilization, however, since they often Zation by reducing the number of re-transmissions of lost or result in the unnecessary re-transmission of data or redundant 25 corrupt data packets in the video stream. data transmitted in the video stream. While the subject matter described herein is presented in It is with respect to these considerations and others that the the general context of program modules that execute in con disclosure made herein is presented. junction with the execution of an operating system and appli cation programs on a computer system, those skilled in the art SUMMARY 30 will recognize that other implementations may be performed in combination with other types of program modules. Gener Technologies are described herein for recovering from ally, program modules include routines, programs, compo dropped frames in the real-time transmission of video over an nents, data structures, and other types of structures that per IP network. According to this description, a video streaming form particular tasks or implement particular abstract data module receives a notification from a receiving module that a 35 types. Moreover, those skilled in the art will appreciate that data packet has been lost. The video streaming module deter the subject matter described herein may be practiced with mines, based on the type of video frame conveyed in the lost other computer system configurations, including hand-held packet and the timing of the lost packet in relation to the devices, multiprocessor systems, microprocessor-based or sequence of video frames transmitted to the receiving mod programmable consumer electronics, minicomputers, main ule, whether or not a replacement video frame should be sent 40 frame computers, and the like. to the receiving module. If the video streaming module deter In the following detailed description, references are made mines that a replacement video frame is warranted, then the to the accompanying drawings that form a part hereof, and Video streaming module instructs a video encoding module to that show, by way of illustration, specific embodiments or generate a replacement video frame and then transmits the examples. In the accompanying drawings, like numerals rep replacement video frame to the receiving module. 45 resent like elements through the several figures. It should be appreciated that the above-described subject FIG. 1 shows an illustrative operating environment 100 matter may be implemented as a computer-controlled appa including several Software components for transmitting a ratus, a computer process, a computing System, or as an Video stream to a receiver across a network, according to article of manufacture Such as a computer-readable medium. embodiments provided herein. The environment 100 includes These and various other features will be apparent from a 50 a source computer 102, which may be a server computer, reading of the following Detailed Description and a review of personal computer (“PC”), a network appliance, or a special the associated drawings.