ALLALL IN IN THE THE GAMEGAME

To TV sports fans, the yellow first- down line appears on stadium grass as if by magic, thanks to the wizardry of a group of former defense engineers SPECTRUM IEEE By Tekla S. Perry • November 2003

Stan Honey, the force behind the creation of Sportvision’s yellow football line, poses on a miniature football field in the company’s laboratory. The yellow line on the live video feed is generated to appear as if painted on the turf behind Honey. 31 TELEVISION

t is “one of the most significant innovations to football cover- car vehicle navigation; others had stayed age since instant replay,” pronounces Fred Gaudelli, executive at SRI. With the defense industry slump- ing, Honey has little trouble arousing producer of football for the ABC TV network. And yet it interest in his project to break new stemmed from a failure in hockey broadcasts that was rejected ground in TV sports broadcasting. The goal is simple. Make the televised by die-hard fans as a garish gimmick that defaced their game. image of a glow so it’s eas- I ier for the viewer to spot, and, when it’s If you’ve watched a pro football broad- Faster than a speeding puck going really fast, put a tail on it showing cast in the United States lately, you’ve Flash back to 1994. IEEE Member Stan its path. Murdoch and company hope the seen what Gaudelli is talking about: an Honey [preceding page] is executive vice system can overcome the main complaint eerily realistic bright yellow line created president of technology for Rupert Mur- about televised hockey: the trouble that on the playing field that shows you exactly doch’s vast media and entertainment casual viewers have in following the fast- how far the offense has to carry the ball to empire, News Corp. (Sydney, Australia). moving puck on TV screens. get a first down. “It makes it easier to It’s June, and Honey is in a meeting with Honey lays out his proposal. To sum watch the game, and that is what our job , then president of News Corp.’s up: it’s never been done before, it will is all about,” Gaudelli tells IEEE Spectrum. (now chairman and CEO of Fox undoubtedly be hard, but it’s doable. That yellow line has become such a Sports Television Group). Honey is rhap- In other words, it’s “just the perfect staple in U.S. football that no self-respect- sodizing about the possibilities of virtual project.” He quickly gathers a team of ing network would think of televising a billboards, which would let technicians 10 and enlists help from Vista Research game without it. It even won an Emmy insert any graphics at will into the images LLC (New York City), a group of award, for technical innovation. The tiny of actual billboards in a stadium. Hill rejects defense engineers, and Shoreline Stu- company that pioneered the technology, the idea (since implemented by Sportvision dios Inc. (Vancouver, B.C., Canada), a Sportvision Inc., with offices in Mountain and others), but suddenly asks, “Could you spinoff of Silicon Graphics Inc. (Moun- View, Calif. (headquartered in Chicago), track and highlight a hockey puck?” tain View, Calif.). The project becomes now covers up to 300 games a year with Honey responds, “I tracked things a “a taste of Camelot” for the engineers 18 crews. The state-of-the-art workstations lot harder than a hockey puck for the mil- involved, Honey says. the crews use, along with sensors and itary, David, but you couldn’t afford it.” To track the hockey puck, a number of other hardware and software, solve a bar- “Just how much would it cost?” parameters have to be fed into a computer rage of fiendishly difficult image-process- Hill asks. system and updated continually. First, the ing problems in a fraction of a second. “It would take two years to develop and system has to know exactly where the And some casual viewers have no idea cost about $2 million,” is the reply. broadcast cameras are focused. It also has that the yellow line they see on the field is “You don’t understand the econom- to have some idea of how each camera a computer graphics figment no more ics of sports,” Hill tells Honey. “Write lens distorts the image; different brands of “real” than the weather maps that seem to a memo.” lenses vary. It then has to figure out which swirl behind television meteorologists. A few days later, back in his office camera’s feed is being displayed to viewers The story of the yellow-line system, near San Francisco, Honey gets a call at any moment. which is called 1st & Ten, is a classically cir- from Murdoch, who gets right to the Meanwhile, the system also has to cuitous one that begins in the world of point. “David says you can track and know exactly where the puck is and how military technology. It goes into the hockey highlight a hockey puck, and you can fast it is traveling, and then it has to cre- arena, where an early version of Sport- get it done by the 1996 January All-Star ate a graphic based on that data and over- vision’s tracking technology turned the game, and it’d only cost $2 million,” lay it onto the video image 60 times a sec- puck into a flashy orb that streaked across Murdoch says. “That is now your high- ond. All these things have to be exactly the ice with a fiery tail. It moves onto the est priority. If anybody asks you about synchronized, as the cameras are zoom- football field. And most recently, it has the money, tell them to call me.” ing and panning and the puck is traveling branched out into car racing sponsored by With just 18 months to go before the at up to 160 km an hour. NASCAR (for National Association of game, Honey immediately starts assem- Making it work requires putting Stock Car Automotive Racing, based in bling his team, relying heavily on engi- infrared transmitters in each hockey Daytona Beach, Fla.), where broadcasters neers he’d worked with at SRI Interna- puck. Calibrating the system proves dif- PHOTOGRAPHS:ROBERTHOUSER November 2003

• conjure up graphics to give detailed race tional in Menlo Park, Calif., in the early ficult. It is finally accomplished by data in real time. A future version of the 1980s. The group had developed an over- drilling holes in the ice and filling them technology will even take NASCAR data the-horizon radar, underwater sensors, with blue dye to enable the system to and channel it to homes, where it will con- and an ultraprecise radio-positioning sys- accurately calculate locations on the ice. trol the movements of virtual cars in inter- tem for the military. Some of them had “It took some convincing to be allowed to IEEE SPECTRUM active video games so couch potatoes can gone on with Honey when he founded do that,” recalls Marvin White, now 32 test their racing skills against the pros. Etak Inc., the company that pioneered in- Sportvision’s chief technology officer. Technically, the project is a success, and it clear that it was to provide information, all times and figure out where in the it comes in on budget and on time. not entertainment. “It’s possible to imag- image to do your drawing—a number of The technology works fine but man- ine,” Honey says, “that the diehard fans factors make the seemingly simple task of ages to offend even hockey fans with its would have seen that as an advantage.” drawing a line actually more difficult. lack of subtlety. Fox Sports Television For one, the line has to be drawn as if Group (Los Angeles) chooses to highlight Off the ice, onto the field it were under the players, not as an over- the puck with a large, bright, fuzzy blue Meanwhile, unwilling to disband his lay. Also, the distortion of the television spot. A red rocket trail appears, painted crack team, Honey needs another proj- lenses becomes more critical—if the right over the players, when the puck is ect—fast. An idea for a fancy new puck trail is a little off, it’s no big deal, but traveling at high velocity. “It was comic- telestrator for commentator John Mad- if the yellow line is curved incorrectly, it strip-like, Flash Gordon,” Honey says. den’s play diagrams goes nowhere, but is immediately apparent next to the real Called “FoxTrax,” it makes its debut at the once again, Hill has another idea. “Why white lines on the field. That lens distor- 1996 All-Star Game with a series of televi- don’t you just do the first-down line?” he tion changes constantly as the cameras sion commercials and a huge fanfare. asks Honey. “It’s clean, it’s simple, and zoom in and out. Complicating all of this Stacks of newspaper articles debate its pros it’s important.” (The telestrator as is the fact that, unlike hockey rinks, which are flat, football fields are not— they have a crown down the middle to A custom-made color key enables the yellow allow drainage. line to appear only over But the biggest problem of all is the the green grass, not color “keying.” over the green jerseys. Color keying is done all the time in broadcast television and movies. The classic example is the weather forecaster in front of a blue screen; the image in the blue screen is later replaced by a video image of a weather map, and it looks as if the weather forecaster is in front of the map. This type of keying is simple; the processor simply replaces any blue pixel with the second image. If a pixel is not blue, it doesn’t replace it. When this tech- nology is being used, actors simply do not wear anything blue or parts of them would seem to disappear. But a football field is not blue; it is many shades of green, and the exact color and cons, the “Late Show with David Let- described by Honey is, nevertheless, of the green changes every time a cloud terman” spoofs it, and hockey ratings jump developed later and introduced in 2002.) passes over the field. Some fields have to their highest levels ever. This time, though, Murdoch does not patches of brown dirt as well. Some play- But serious hockey fans hate it. After bring out the checkbook. Honey and his ers also have green uniforms, or brown- three years, pro hockey broadcasts in the team, along with two News Corp. execu- ish ones, or uniforms that become United States switch networks, and the tives, Jerry Gepner and Bill Squadron, stained with grass or mud. As a worst- system dies a quiet death. “There are two spin out a new company, Sportvision, tak- case scenario, the engineers consider the ways for a product to fail,” Rick Cavallaro, ing rights to use all the patents and other San Francisco 49ers playing the Green Sportvision’s vice president of product intellectual property they have from the Bay Packers at sunset, after a recent rain: development, tells Spectrum. “One is for hockey puck project. In exchange, News the image will have multiple colors of it not to work; the other is for people to Corp. gets 10 percent of the company. brown dirt, because there is dirt sitting in SPECTRUM IEEE say it works great and we hate it.” the sun, dirt in the shade, dry dirt, and In hindsight, Honey says, the system The color map conundrum wet dirt. It will have multiple shades of would have been better accepted had it Drawing a simple first-down line has got green grass. And it will have the 49ers been subtler. Better to have placed the to be much simpler than continually wearing brown pants and the Packers in • image of a gray disk under the puck— tracking a puck bouncing around and their green shirts [see photo, above]. November 2003 and underneath the players—and shown traveling at up to 160 km an hour, right? Even with all that green and brown to the puck’s track only when it was moving Wrong. While some elements of the first- deal with, the system would have to make faster than a preset threshold, he says. down line problem are similar to some in sure that the yellow line is never drawn That track could have been a clean black the hockey puck problem—you have to over a player. Determining the key that line, drawn as if by a drafting tool, making know where the cameras are pointed at would define what is field (to be drawn 33 TELEVISION

upon) and what is player (not to be drawn of grass seem to peek through, making creates the lion’s share of virtual first- upon) is going to be a challenge. the line look like yellow chalk on the grass. down lines, others, like PVI Virtual Cavallaro, the project leader, fears that Viewers are puzzled: is the line really on Media Services LLC, in Lawrenceville, “in some cases, there just wouldn’t be the field or not? And that mystery initially N.J., have gotten into the act. enough pure color distinction,” resulting creates a lot of interest. in some yellow line appearing incongru- Today, Sportvision crews crisscross the Next up, NASCAR ously on a player’s shirt, say. country during football . A crew Sportvision’s latest conquest is NASCAR So the first thing the team does is de- of two typically arrives the day before the racing, a sport in which Sportvision relies velop a more sophisticated method of color game, along with a small rack of com- on a Global Positioning System (GPS) keying than had ever been done before, puters, about the size of a dormitory receiver in each racing car to send loca- one that can be redefined quickly as light- refrigerator. They first measure the slope tion information to a computer system ing changes. The group produces a huge of the field using a laser surveying sys- that, during a two-second broadcast delay, table of color definitions in multiple for- mats, the most common of which is RGB, specifying colors by their red, green, and blue components. An operator can use whichever set of color definitions works best in a particular situation. Proving that a sophisticated enough color key can be developed takes a month; then the team Sportvision’s technology creates moves on to solving its other problems. virtual dashboards in this screen Each television camera, it turns out, shot from an NBC broadcast and electronically measures the zoom and one from InDemand [far right], a pay-per-view service. Also shown focus position of its lens. This information are the race car’s number and is, amazingly, not used for much of any- position on the track and the thing until Sportvision taps into it to make name of the driver. sure the yellow line is correctly laid out. But zoom and focus information is not enough; the team also has to add sensor rings under the camera that measure its pan and tilt and the attitude of the tripod. In the latest version, the solution is a controller mounted on the tripod head that collects all this data for each frame of video, synchronized by the camera’s ver- tem, fit the rings on each camera, and generates graphics and other data related tical sync signal and then modulated onto calibrate zoom positions. An hour before to each car’s movement and position on an audio frequency before being sent the game starts, the operator chooses the the course, including its , brake sta- back to the production area via a micro- color key, based on the colors of the field tus (on or off), and tachometer reading. phone channel. in both shade and sun and those of the The technology was first used in a live With the project about halfway players’ uniforms. During the game itself, race in 2001. through, the team decides to go with yel- every time a new first down is made, an The imaging problem was similar to low, after considering and rejecting operator looks at the screen and clicks that in football, because the television orange, blue, and red. “Yellow isn’t the either on the ball or on the chain gang; camera lenses have long zoom ranges. most obvious choice,” Cavallaro admits. this positions the line. This problem had been solved before. But “Orange would make more sense because Meanwhile, sensors on three main at its inception, the NASCAR project was the chain gang [the officials who mark and cameras continuously send camera posi- faced with a conundrum. It was clear that measure first downs] has orange flags. But tion information down an audio chan- to get accurate positioning of each car on orange looks bad and yellow looks good.” nel. While the video feed is delayed a the track, a GPS receiver was needed in Finally, it’s all done. fraction of a second, the computers con- each car. But the conventional wisdom of The technology is first embraced by sider all the data, determine whether the the time was that GPS could never work November 2003

• ESPN, which introduces it in the fall of line should be drawn in that image, and, in a racing environment because of the 1998 and receives an Emmy for technical if it should, consult the color key to amount of electrical noise generated innovation. The introduction is done with- determine which pixels should be inside each car; the extreme multipath out fanfare, without teaser advertisements. switched to yellow—and repeats this interference due to all the metal nearby as It just appears one Sunday in September, process 60 times a second. a car raced around a track; the metal fence IEEE SPECTRUM with the intensity of the line turned down Viewers almost immediately accepted that leans out over the track, blocking, in 34 far below what it typically is today; blades the technology. Today, while Sportvision effect, half the sky; and the fact that the GPS receivers would have to provide power each car’s GPS and telemetry “When the time comes for people at accurate positioning within centimeters package. During the 2001 Watkins Glen home to make decisions about how to cus- instead of the more typical meters. (N.Y.) International Race, one battery tomize their viewing experience, they will The solution was to give the GPS pack overheated and started a fire in the need data from the sporting event. And we receivers a head start in figuring out their battery enclosure, emitting smoke into will have that data,” Honey says. The com- position by restricting the possibilities to the car. (The fire was extinguished with- pany already has been granted over 20 a three-dimensional ribbon of track, out injury, though the car did not finish.) patents related to gathering data at an measured in advance to within a few cen- Engineers determined that the bat- event and sending it downstream to timeters of accuracy. teries chosen were not stable under high devices in the home. Designing the telemetry, the method vibration, and reported this to NASCAR. But that future may take a while to by which the GPS information would be They then designed and manufactured arrive, and Sportvision isn’t holding its sent back to the computer system from replacement alkaline battery packs that breath in anticipation. The company, founded during the dot-com boom, was initially pressured by investors to become an Internet com- pany instead of focusing on standard broadcast television, which was then per- ceived as a technology dinosaur. But Sportvision resisted, arguing that TV broadcasters have real money. Then, with its trucks and sensors at every sporting event, the company would be well positioned to dominate interactive Internet sports if and when that time came. Today, Sportvision is a solid, if still small, privately held business. It projects that it will be profitable on an operating basis in 2004. Honey meanwhile in October stepped down as company president to be the nav- igator of a sailing crew attempting to set the record for the fastest passage around the world. He remains on the board. “We haven’t made a bazillion doing the car, was also a challenge. Previous were much larger and heavier but less this,” Honey told Spectrum, “but it’s NASCAR telemetry systems relied on a flammable, and had those ready to install the most fun we’ve ever had. It’s not transmission control protocol/Internet in time for the next weekend’s race. like when we were building military protocol (TCP/IP)–like approach of NASCAR, which had been mandating systems. Everybody gets to see what acknowledge/retry. But the Sportvision the use of Sportvision’s technology, gave we’re doing, and we’re doing some- system didn’t have time for that; it had to drivers a chance to opt out; none did. thing that is both technically hard and get the information for all the cars, com- is in good cheer.” • pute the graphics, and display them in less Internet in the future than the two seconds the live broadcast Today, the Sportvision crew is expanding To Probe Further was being delayed. So the engineers the NASCAR tracking system to interac- For information about Sportvision products designed a customized telemetry system tive television and video games. In these and career opportunities, see http://www. that used spread-spectrum modulation of games, an actual race is virtually recon- sportvision.com. the signals, and assigned each car two pre- structed with slightly smaller vehicles and As a navigator of racing sailboats, Stan SPECTRUM IEEE determined time slots for transmission so a home player’s car is inserted, allowing Honey currently holds both the transat- there would be no signal collisions. the gamer to race against the actual driv- lantic and transpacific records. See the site After the design of the NASCAR sys- ers. The company has introduced an Inter- athttp://www.pegasus.com/Transpac- tem, RACEf/x, was complete, the technol- net application called PitCommand, in philippe-kahnlightsurf-071503c.htm. • ogy proven, and the NASCAR authorities which paid subscribers can watch a virtu- For more details on the controversy over November 2003 pleased enough with it to mandate its instal- al aerial view of the race and view the track the hockey puck system when it was first lation in all cars, there was a problem that and data for any car they select, run instant introduced, see http://www.pub.umich.edu/ nearly stopped the technology in its tracks. replays at will, and change perspective. daily/1996/apr/04-08-96/sports/hockey. RACEf/x was designed to use lith- Ultimately, the Internet is where the white.column.html and http://slam.canoe.ca/ ium–manganese dioxide batteries to company sees its future. HockeyNHL97Playoffs/jun2_strachan.html. 35