CONSIDERATIONS FOR A REAL TIME BROADBAND DELIVERED INTERACTIVE GAMING SERVICE
Charles H. Jablonski VP Operations, OnLive Inc
Abstract Introduction The march of enabling technology and the Real Time graphics intensive applications, continued improvement in broadband such has “first person shooter games” have connectivity and availability have enabled presented challenges in being server/cloud many services and products to be delivered to based because of the inherent roundtrip end consumers and enterprises as various latencies. compressed data and media files. The success of services such as video on demand and This paper will describe an approach that “ultra” high speed broadband are well known provides the ability not only to deliver real and their effects have caused consumer tastes time games but also that is able to deliver any and expectation to evolve accordingly. graphics intensive application over existing broadband infrastructure. In few short years linear packaged media has evolved from a shrink wrap retail point of The approaches taken to provide the sale product to a anywhere, anytime, almost necessary low latency compression, server any device availability. At times the business and data center architecture, integrating into issues have been far more daunting that the exiting carrier and ISP data infrastructures as technological, but one undisputed fact is that well as the impacts of “last mile” ISP (DSL the consumer continues to be more and Cable) considerations as well as the demanding and the expectations of required performance of the date delivery, availability, selection, schedule and packet loss and jitter specifications of the independence increases. infrastructure. Further the same technologies that have An approach using existing PC/Mac client enabled these distribution advances have lead for delivery of the service to end consumers to a far greater availability of consumer will be also be described as well as a small unilateral and multilateral consumptions stand alone low cost (<$30) terminal adapter platforms (PCs, Net books, Tablets, Portable that connects a consumer television directly to Devices) and the ability to imbed the a internet connection to enable game play. functionality into broader consumer platforms, and inexpensive stand alone The system allows a variety game content, devices. media and enterprise application to also be provisioned and delivered over the same One of the last frontiers remaining is true infrastructure. real time interactive consumer experiences in the home, using inexpensive and available end devices, that are provisions by centralized . “cloud based” systems. In these applications, such as gaming, the consumer experience
2010 Spring Technical Forum Proceedings - Page 62 cannot be fulfilled unless the challenges of where the initial developer and published latency, quality and availability using cloud receiving none of that incremental revenue. based systems and broadband networks are Some high end titles are resold and used 4-5 met. times (as evidenced by registrations to the publishers). OnLive is one such system, and it is believed the first among many that will be This “quantized” method of sale also taking advantage of the broadband makes incremental releases, episodic, infrastructure to widely deliver these upgrades extremely problematical in the interactive real time applications. games market. Although there is a strong online community and infrastructure its broad Business Issues and Consumer Expectations use is limited to small add-ons, consumer social features test and chat. What few online As with any good technological sales mechanisms in existence they either innovation, there must be an underlying suffer from extremely long and large business reason to support the development downloads, or they are very limited in their and implementation and the implementation catalog of current first release titles. must be compelling enough for all parties to embrace it. Science experiments, although Typically software publishers receive less intellectually fulfilling, must have the support than 50% of the MSRP at the retailer point of and resources of a real business problem to sale. Clearly there is room for improvement in evolve into broad and accepted use. the economics of the existing system. Now, when one examines the exiting consumer The gaming industry, although a large experience for high end interactive games, mature business (>$50B annually worldwide), that picture has room for improvement as the distribution of the end product to well. consumers has been dominated by retail point of sale. Even though the economics of game Typically a consumer has to spend $500 development resemble other media (high (in early cycle) on a proprietary closed development costs, intensive consumer gaming console, that by definition and promotion and advertising, short selling practice will require replacement in 3-5 years, season and short initial release life) the actual purchase games at retail, sometimes having to end distribution is the last unfulfilled digital wait in line or just wait for availability of a delivery for modern media. hot game, spend additional monies to participate in a social network for that Typically many scores of millions are console. The other choice is to purchase an spend developing a game, a few score spent in extremely high end PC (>$5000) and suffer advertising its release, and then the selling the same issues acquiring the games and season is typically a few weeks before the software. holidays. This is all delivered through shrink wrap product that is sold through retail Naturally with retail purchase model the distribution channels. These channels suffer end consumer may not have immediate access from various impeded costs (duplication, to new “hot” games, and be in the position on distribution, platform fees, retailer margins) as not knowing the game experience until it has well as theft, returns and “spoilage”. Also been purchased brought home, installed and since the end product is in the consumers used. hands piracy is a constant battle, and >30% of the games are resold through the used market
2010 Spring Technical Forum Proceedings - Page 63 As with the publisher side of the equation, availability of new release games from this systems is ripe for a disruptive change various publishers. and consumers continue to expect to enjoy Up until now, the key limitation in existing real time availability and experience of all technologies has been the latency of real time other digital media, an online, cloud based video compression systems. These existing gaming system that addresses these business technologies (JPEG, MPEG et.al.) have either and consumer needs can be a commercial required bandwidth only available on large success. corporate LANs, or required many milliseconds to compress real time images. The Consumer Experience (Typically game play requires <100ms of response time to be effective in competitive In order for a server based, cloud served game play) system to be successful; it must meet or exceed the existing experience expectations The OnLive Platform for current consoles and PCs. What does that incorporate? The following is a brief description of the OnLive platform: One the image quality must be the same. That requires the ability to render, in real When the user performs an action on a time, high definition, high frame rate images. computer or TV connected to OnLive (e.g. The response time for the system, using presses a button on controller or moves a readily available controllers, must be fast mouse) that action is sent up through the enough not to limit the game play because of internet to an OnLive data center and routed lag or latency, a large amount of amount to a server that is running the game the user is devices must be available and supported, be playing (or the application the user is using— deployable on existing PCs or Macs, it must since the interactive demands for video games be extendable to a large screen display device are generally higher, remote video game (i.e. TV), be enabled and installed simply over operations will be primarily described in the exiting broadband infrastructure, afford social following paragraphs, but these discussions infrastructure and features and have wide are entirely applicable to remote application operations).
OnLive Platform
2010 Spring Technical Forum Proceedings - Page 64
The game computes the next video frame Mac, a cell phone, or a cable TV set-top box, based on that action, then a proprietary chip introduces a certain amount of latency (i.e. compresses the video from the server very lag) from the point you perform an action and quickly, and the user’s PC, Mac or OnLive you see the result of that action on the screen. MicroConsole™ decompresses the video and Sometimes the lag is very noticeable (e.g. on displays the new frame of video on the user’s some TV set-top boxes it takes over a second computer display or TV set. The entire round to move a selection box in a program guide). trip, from the point the button is pressed to the Sometimes it isn’t noticeable (e.g. if you have point the display or TV is updated is so fast a well-designed game running on fast that, perceptually, it appears that the screen is hardware, and pressing the fire button results updated instantly and that the game is actually in what appears to an instantaneous display on running locally. your screen of the your gun firing).
When the user performs an action on a But, it’s important to note that, even when computer or TV connected to OnLive (e.g. your brain perceives game response to be presses a button on controller or moves a “instantaneous”, there is always a certain mouse) that action is sent up through the amount of latency from the point you perform internet to an OnLive data center and routed an action and your display shows the result of to a server that is running the game the user is that action. There are several reasons for this. playing (or the application the user is using— To start with, when you press a button, it since the interactive demands for video games takes a certain amount of time for that button are generally higher, remote video game press to be transmitted to the computer or operations will be primarily described in the game console (it may be less than a following paragraphs, but these discussions millisecond (ms) with a wired controller or as are entirely applicable to remote application much as 10-20 ms when some wireless operations). The game computes the next controllers are used, or if several are in use at video frame based on that action, then a once). Next, the game needs time to process proprietary chip compresses the video from the button press. Games typically run between the server very quickly, and the user’s PC, 30 and 60 frames per second (fps), so that Mac or OnLive MicroConsole™ means they only generate a new frame every decompresses the video and displays the new 1/30th to 1/60th of a second (33ms to 17ms). frame of video on the user’s computer display (Further, when games are generating complex or TV set. The entire round trip, from the scenes, sometimes they take longer.) So, even point the button is pressed to the point the if the game responds right away to a button display or TV is updated is so fast that, action, it may not generate a frame for 17- perceptually, it appears that the screen is 33ms or more that reflects the result of the updated instantly and that the game is actually action. And, then finally, there is a certain running locally. amount of time from the point the game completes generating the frame until the The key challenge in any cloud system is frame appears on your display. Depending on to minimize and mitigate the issue of the game, the graphics hardware, and the perceived latency to the end user. particular monitor you are using, there may be almost no delay, to several frame times of Latency Perception delay. And, if your game is an online game, there typically will be some delay to send a Every interactive computer system that is message reflecting your action through the used, whether it is a game console, a PC, a internet to other game players, and the game
2010 Spring Technical Forum Proceedings - Page 65 may (or may not) delay the action occurring to the player’s display. And, of course, in your game so as to match your screen OnLive has to do this at rate of 60fps with action to that of screen action of players who HDTV resolution video over a consumer are playing the game remotely. So, in internet connection, running through summary, even when you are running a game consumer internet gear in the home. on a local machine there is always latency. Over Cable and DSL connections, OnLive The question is simply how much latency. is able to achieve this if the user’s home is So, while there certainly are more subtleties to within about 1000 miles of the OnLive data the perception of latency, as a general rule of center. So, through OnLive, a user who is thumb, if a player sees a fast-action game 1000 miles away from a data center can play a respond within 80ms of an action, not only video game running on a server in the data will the player perceive the game as center with the perception (and the game play responding instantaneously, but the player’s score) as if the game is running locally. performance will just as good as if the latency was shorter. OnLive’s Latency Calculations And, as a result, 80ms is the “latency budget” needed to meet for the OnLive The simplified diagram below shows the system to be practical. That is to say, OnLive latencies encountered after a user’s action in has up to 80ms to: send a controller action the home makes it way to an OnLive data from the player’s home through the internet to center, which then generates a new frame of an OnLive data center, route the message to the video game and sends it back to the user’s the OnLive server running the game, have the home for display. Single-headed arrows show game calculate the next frame and output the latencies measured in a single direction. video, compress the video, route the Double-headed arrows show latencies compressed video out of the data center, send measured roundtrip. the compressed video to the player’s home through the internet, decompress the video on the players computer or and output the video
OnLive System Latency
2010 Spring Technical Forum Proceedings - Page 66 There latency numbers shown here are the video compressor is carefully managed to numbers that OnLive has seen in practice, not exceed the data rate of the user’s internet given the way the OnLive system was connection because if it did, that might result architected and optimized, and reflect what in queuing of packets (incurring latency) or has been measured after using OnLive in dropped packets. Since the user’s home data various locations over the years. If you add rate is constantly changing, the OnLive up all of the worst-case numbers, it shows service is constantly monitoring the available the latency can be as high as 80ms. That said, data rate, and constantly adapting the video it is highly unlikely that every segment will compression (and if necessary, dropping the be worst case so the total latency will likely video resolution) to stay below the available be much less (and indeed, that is what we see data rate. in practice). One common misconception about home broadband connections are that the latency is ISP latency directly tied to data rate (i.e. the effective connection speed) and/or data throughput Potentially, the largest source of latency is (i.e. the data rate available to a particular the “last mile” latency through the user’s user). Latency is actually largely independent Internet Service Provider (ISP). This latency of data rate, so long as the data throughput can be mitigated (or exacerbated) by the demands are less than the capacity of the design and implementation of an ISP’s broadband connection. network. Typical wired consumer networks in the US incur 10-25ms of latency in the last OnLive video decompression latency mile, based on OnLive’s measurements. Wireless cellular networks typically incur Once the compressed video data and other much higher last mile latency, potentially data is received by the OnLive client (i.e. the over 150-200ms, although certain planned OnLive application running as a plug-in or 4G network technologies are expected to standalone in your PC or Mac, or the OnLive decrease latency. MicroConsole attached to your TV), then it is decompressed. The time needed for Within the internet, assuming a relatively decompression depends on the performance direct route can be obtained, latency is of your PC or Mac (CPU and frame buffer largely proportional to distance, and the bandwidth…no GPU is needed), and may roughly 22ms worst case round-trip latency vary from about 1 to 8ms. If your computer’s is based on about 1000 miles of distance CPU and/or memory bus is tied up doing (taking into account the speed of light another processing-intensive task or if you through fiber, plus the typical delays OnLive have an extremely low performance has seen due to switching and routing computer, OnLive may find it is unable to through the internet. decompress video at full screen resolution. If so, then it will scale down the video window Consequently, OnLive will be locating its accordingly. But, we have found most data centers such that the distance to most of computers made in the last few years work the US population is less than 1000 miles fine up to their screen resolutions so long as The compressed video, along with other data they are not tied down running some other required by the OnLive client to keep it intense application at the same time. In any tightly sync’d with the OnLive service, is case, even if you are in a processing- then sent through the internet back to the constrained situation, OnLive will select a user’s home. Notably, the data generated by
2010 Spring Technical Forum Proceedings - Page 67 video frame size which will maintain low latency.
OnLive round-trip latency
As mentioned before, while there is a certain amount of latency variability in each leg of the journey, it is rare that a given user will end up in a worst-case scenario with each leg. Consequently, what we typically see in practice are latencies on the order of 40 to 60ms. Sometimes we see latencies that are higher and sometimes we see latencies that are shorter. And, we expect latencies to OnLive Data Rates continue to decline as “last mile” infrastructure is upgraded, both for wired and The 1280 x 720 resolution currently wireless networks. provided, provides adequate visual experience, and with a peak data rate at 5Mps Video Quality allows the service to be widely available to consumers through existing broadband As previously discussed video quality infrastructure. is paramount to successful consumer experience. Not only does the image quality Consumer Infrastructure have to be acceptable, the rendered frame rate must also be high to provide the gaming As described earlier the OnLive system experience has been engineered to be deployable on a wide variety of PCs and Macs using a downloadable software client. As for the large screen device, OnLive has developed its own consumer TV adapter, the MicroConsole™.
2010 Spring Technical Forum Proceedings - Page 68 MicroConsole TV Adapter
The OnLive MicroConsole™, which will There are USB connectors to allow be generally available later this year, has integration of standard controllers and been developed to allow simple integration devices, wireless radios for headsets, and an into a home consumer display. The device optical audio connector. The cost of this has an standard NIC connector for access to device is minimal to allow wide deployment the broadband service, an HDMI adapter to and availability to the consumer. connect directly to the large screen device to provide high image quality as well as multichannel audio. (A high quality audio experience is another “must have” for a successful consumer gaming experience.)
2010 Spring Technical Forum Proceedings - Page 69
OnLive Game Service Portal
Consumer and Social Features Further this system has a continuous As with any current consumer offering, ease recording buffer that allows the immediate of use and simplicity of navigation is capture of the preceding game play as a paramount. Social features must also be “brag clip” that can captured stores and available to support competitive game play. downloaded for use in other social media and social networks. In addition to providing access to games, OnLive includes a multitude of social As this system is completely cloud based, features such as social groups (friends) text all user information is safely stored, include and voice chat. One of the unique offerings state of game play, so the user can pause, log in the OnLive service is the feature of brag out and resume from the existing place in the clips and spectating. As all the games game and maintain all their statistics, sessions are running in OnLive datacenters, rankings and data from their previous the games sessions are available and can be session. This information can be accessed on seen by other OnLive users in real time. the OnLive system anywhere. This allows a Using multicast distribution any of these user to log off and resume their play across streams can be distributed to any OnLive town or across the country. This allows user so your friends can watch, comment and game playing and spectating by individuals cheer your game play. throughout the system.
2010 Spring Technical Forum Proceedings - Page 70 Product (Games) Availability In March of this year OnLive announced it will launch it service in June 2010. The OnLive system and its servers utilize standard PC games with minimal Other Applications and Enterprise Solutions modifications. A SDK is freely available for any game publisher or author to provision their As the OnLive platform has been initially games for the OL platform. As virtually all targeted for the consumer gaming experience, games are developed on PC development other applications such as media distribution systems virtually all the currently available and and playback are easily incorporated and blend developed games can be easily provisioned and well with the cost and performance targets of provided for the platform. the consumer platforms. Further this technology (low latency, high OnLive has secured long term arrangements quality, cloud based) that enable high end with virtually every game publisher that allows gaming applications to be provisioned over their products to be provided to OnLive at broadband networks to inexpensive widely launch in June of 2010. These agreements available PC and Mac platforms also enables provide for the same day and date release of cloud based “SAS” type implementation of titles that will be available at retail stores. high end, expensive, piracy prone design, graphics, engineering and architectural Because of the nature of the OnLive applications. These applications being cloud systems publishers can now produce and based can leverage off the OnLive distribute not only primary releases, but infrastructure, and reduce the end users cost of provide episodes, mini release and features for high end PCs and support. Work continues by their games after the initial release. The OnLive in this area. OnLive system supports purchase, rental, subscription and demo distribution of product. Summary Try before you buy now becomes a reality for OnLive has developed and implemented the the consumer. technology that enables cloud based real time interactive gaming. This service can be widely OnLive Status available via exiting broadband connections, and has been optimized for minimal latency, OnLive has been in development for several high quality over these networks. years. It was publicly announced in March 2009 at the Games Developer Conference, and Further the system is easily integrated into beta began shortly after that. The Beta trials the consumer environment and provides unique have been extensive and provided valuable features and availability of games, products insight and detailed information on the OnLive and social interaction. The business issues, Technology, service and network performance. games availability and features all mesh well All the Beta participants have been helpful and and represent an marked improvement for both the ISP and carriers have been extremely the consumer and the games providers. supportive in the efforts to bring this service to OnLive is among the first of such services market. This information has enabled OnLive that will become widely available taking to improve its underlying technology and advantage of broadband and will help drive the systems. uptake and demand of the consumer for improved bandwidth and quality of broadband. It also enabled the improved of the real time The system allows has unique advantages for adaptive aspects of the system and provide business and enterprise applications. valuable feedback to the carriers, games providers and ISPs.
2010 Spring Technical Forum Proceedings - Page 71