Yinzcam: Experiences with In-Venue Mobile Video and Replays Nathan D
Total Page:16
File Type:pdf, Size:1020Kb
YinzCam: Experiences with In-Venue Mobile Video and Replays Nathan D. Mickulicz, Priya Narasimhan, and Rajeev Gandhi, YinzCam, Inc. and Carnegie Mellon University https://www.usenix.org/conference/lisa13/technical-sessions/papers/mickulicz This paper is included in the Proceedings of the 27th Large Installation System Administration Conference (LISA ’13). November 3–8, 2013 • Washington, D.C., USA ISBN 978-1-931971-05-8 Open access to the Proceedings of the 27th Large Installation System Administration Conference (LISA ’13) is sponsored by USENIX. YinzCam: Experiences with In-Venue Mobile Video and Replays Nathan D. Mickulicz, Priya Narasimhan, Rajeev Gandhi YinzCam, Inc. / Carnegie Mellon University Pittsburgh, PA [email protected], [email protected], [email protected] [email protected], [email protected], [email protected] Abstract YinzCam allows sport fans inside NFL/NHL/NBA venues to enjoy replays and live-camera angles from different per- spectives, on their smartphones. We describe the evolu- tion of the system infrastructure, starting from the initial in- stallation in 2010 at one venue, to its use across a dozen venues today. We address the challenges of scaling the system through a combination of techniques, including dis- tributed monitoring, remote administration, and automated replay-generation. In particular, we take an in-depth look at our unique automated replay-generation, including the dash- board, the remote management, the remote administration, and the resulting efficiency, using data from a 2013 NBA Playoffs game. Figure 1: Fan experience at the original Mellon Arena deployment for YinzCam, May 2010, showing three dif- ferent smartphones displaying three different live cam- 1 Introduction era angles for a Pittsburgh Penguins home game. Fans inside sporting venues are often far removed from the action. They have to rely on the large stadium video-boards broadcast rights; all of the other content is available to fans to see close-ups of the action or be fortunate enough to be outside the stadium. YinzCam’s primary infrastructure is close to the field. YinzCam started as a Carnegie Mellon currently hosted on Amazon Web Services (AWS) [1] and research project in 2008, with the goal of providing live supports over 7 million downloads of the official iOS, An- streaming camera angles and instant to hockey fans on their droid, BlackBerry, and Windows Phone mobile apps of 40+ Wi-Fi-enabled smartphones inside the Pittsburgh Penguins’ NHL/NFL/NBA sports teams and venues within the United venue, Mellon Arena [5]. The original concept consisted States. Twelve of these venues additionally support Yinz- of the Penguins Mobile app that fans could download and Cam’s live video (including the NFL RedZone Channel at use on their smartphones, exclusively over the in-arena Wi- NFL stadiums) and instant replay technology over both Wi- Fi network, in order to receive the unique in-stadium video Fi and cellular distributed antenna systems (DAS) [6]. content. Figure 1 shows the user experience at our initial The production of live video and instant replays requires installation for the Pittsburgh Penguins. the orchestration of both computerized systems and human While YinzCam started with an in-stadium-only mobile efforts. In the video-processing plane, we utilize hundreds experience, once the research project moved to commercial- of virtual and physical machines that communicate over var- ization, the resulting company, YinzCam, Inc. [21], decided ious wireless and wired networks. These systems include au- to expand its focus beyond the in-venue (small) market to dio and video encoders, stream routers, web servers, media encompass the out-of-venue (large) market. servers, and databases. In the management plane, we utilize YinzCam is currently a cloud-hosted mobile-video ser- remote-access technologies, automated-monitoring systems, vice that provides sports fans with real-time scores, news, and a game-day-operations team. We group these compo- photos, statistics, live radio, streaming video, etc., on their nents into three major systems: the content-generation sys- mobile devices anytime, anywhere, along with live video tem (CGS), the mobile-wireless network (MWN), and the and replays from different camera angles when these fans management-operations system (MOS). are inside sports venues. The game-time live video and in- Our first in-venue installation was located at Mellon stant replays are effectively gated to the stadium because of Arena, the home of the Pittsburgh Penguins until 2010. In 1 USENIX Association 27th Large Installation System Administration Conference 133 this installation, the CGS, MWN, and MOS were all co- and provides insight into the system we use today. Finally, located in a small room that was adjacent to the arena’s video section 6 discusses the lessons we learned throughout this production studio. Our equipment included a video encoder, process. three rack-mounted servers, and a terminal for the operator. As we grew from one team in 2010 to five teams in 2011, 2 The YinzCam System we struggled to scale the initial design of our system to mul- YinzCam develops the official mobile apps for more than tiple installations with thousands of users. The load on our 40 professional sports teams across four leagues (the NHL, systems became too much for a single server or encoder to NFL, NBA, and MLS). We provide these apps for over 7 handle. Teams wanted to use more than the maximum of million users across iPhone, iPad, Android, BlackBerry, and four camera angles that our initial designs allowed. New Windows Phone 7/8. YinzCam-powered apps have access to technologies like 4G LTE [17] and WiMAX [19] made cel- a wide variety of sports-related content such as news, photos, lular networks a viable option for the MWN, but our sys- videos, podcasts, schedules, rosters, stats, and scores. This tem was designed only for Wi-Fi. Replay cutting was a content is updated in real time as teams publish new statistics slow, labor-intensive, and error-prone process that frequently and media. frustrated our system operators. Adding to the frustration, our operators were immersed in a high-pressure, game-time User Experience. YinzCam apps also support unique fea- environment filled with distractions. Travel costs piled up tures not available in other sports apps, such as exclusive while we shuttled operators back and forth across the coun- live-video streaming and on-demand instant-replays. From try for every sporting event. There was a critical need for a their seats or anywhere in the building, fans can use Yinz- cost-effective, low-latency scalable automated solution that Cam apps to watch live game action from multiple camera addressed these challenges. angles on their smartphone. In addition, fans can watch replays of game events from the same set of camera an- Contributions. The overarching goal of this paper is to gles. Some venues also provide fans with streams of pop- describe how we evolved our initial system, with all of its ular sports-related TV-channels, such as the NFL RedZone limitations, into the scalable and efficient system that we use channel [13]. today to support thousands of sports fans across venues in To use our in-venue features, the user first configures his the United States. Instead of simply scrapping our existing or her smartphone to connect to the in-venue Wi-Fi network design and starting over, we slowly evolved our system by (this step is not required for DAS networks). Next, after addressing each of the aforementioned challenges individu- opening the app, the user navigates to a special in-venue sec- ally. In this paper, we discuss candidly the weaknesses in tion of the app. As shown in Figure 2, the user can then our original system architecture, the changes we introduced choose to watch live camera angles or browse through sev- to make our system work efficiently and cost-effectively at eral replays of game action. Both live videos and instant scale, and the lessons we learned during this transformation. replays are offered from multiple different camera angles. Concretely, our contributions in this paper are: Only one camera angle may be viewed at a time. A case-study of our migration to the cloud, including a These features are only accessible to fans while inside the • discussion of how we chose which components to mi- team’s home venue. Once inside and connected to an in- grate and how the cloud helped us scale; venue network, fans are able to access a special in-venue A description and evaluation of our automated-replay- section of the mobile app. This section initially presents fans • cutting system, including a discussion of how we im- with a list of game highlights, as well as options for view- proved its efficiency, reliability, and maintainability ing additional replays or live streams (see Figure 2). Once a through automation, decoupling of subsystems, and fan chooses a live stream or instant replay, he or she is pre- configuration management; sented with a selection of camera angles available for that A description of our remote-management infrastruc- item. Upon choosing an angle, the app begins playback of • ture, including our automated-monitoring systems and the corresponding video. operations dashboards. Although only a part of a large app that contains a wealth of other features and content, our in-venue live streaming To the best of our knowledge, there exists no other auto- and instant replays have proven to be the most technically- mated mobile-replay generation and management system for sophisticated (and technically-challenging) of all of the ser- sports venues. vices we provide. We separate our system into a content The rest of this paper is organized as follows. Section 2 plane and a management plane, with some components ex- describes our mobile applications in detail and also gives a isting in both planes.