Augmenting Live Broadcast Sports with 3D Tracking

Information

Rick Cavallaro

Sportvision, Inc., Mountain View, CA, USA

Maria Hybinette ∗

Computer Science Department, University of Georgia, Athens, GA, USA

Marvin White

Sportvision, Inc., Mountain View, CA, USA

Tucker Balch

School of Interactive Computing, Georgia Institute of Technology, Atlanta, GA, USA

Abstract

In this article we describe a family of technologies designed to enhance live television sports broadcasts. Our algorithms provide a means for tracking the game object (the ball in baseball, the puck in hockey); instrumenting broadcast cameras and creating informative graphical visualizations regarding the object’s trajectory. The graphics are embedded in the live image in real time in a photo realistic manner. The demands of live network broad- casting require that many of our effects operate on live video streams at frame-rate (60 Hz) with latencies of less than a second. Some effects can be composed off-line, but must be

Preprint submitted to IEEE MultiMedia Magazine complete in a matter of seconds. The final image sequence must also be compelling, ap- pealing and informative to the critical human eye. Components of our approach include:

(1) Instrumention and calibration of broadcast cameras so that the geometry of their field of view is known at all times, (2) Tracking items of interest such as balls and pucks and in some cases participants (e.g., cars in auto racing), (3) Creation and transformation of graphical enhancements such as illustrative object highlights (e.g., a glowing puck trail) or advertisements to the same field of view as the broadcast cameras and (4) Techniques for combining these elements into informative on-air images. We describe each of these technologies and illustrate their use in network broadcast of hockey, baseball, football and auto racing. We focus on vision based tracking of game objects.

Key words: Computer Vision, Sporting Events

1 Introduction and Background

We have developed a number of techniques that provide an augmented experience for viewers of network (and cable, e.g., ESPN) broadcast sporting events. The final televised video contains enhancements that assist the viewer in understanding the game and the strategies of the players and teams. Some effects enable advertise- ments and logos to be integrated into the on-air video in subtle ways that do not distract the viewer. Example visuals include (1) graphical aids that help the audi- ence easily locate rapidly moving objects or players across the ice in hockey or the ball in football and baseball; (2) information such as the speed or predicted trajec- tories of interesting objects such as a golf ball, golf club or a baseball or bat; (3)

∗ Corresponding author. Address: University of Georgia Email address: [email protected] (Maria Hybinette). URL: http://www.cs.uga.edu/˜maria (Maria Hybinette).

2 highlights of particular locations within the arena, such as in soccer, marking the offside line – a line that moves dynamically depending on the game play.

Fig. 1. In the K-Zone system multiple technologies, including tracking, camera instrumen- tation and video compositing come together at once to provide a virtual representation of the strike zone over home plate and a occasionally a trail on the ball.

A cornerstone of our approach is a careful calibration of broadcast cameras (those that gather the images that are sent to the viewing public) so that the spatial cov- erage of the image is known precisely [Honey and et al (1999); Gloudemans and et al (2003)]. A separate but equally important aspect of the approach is the sensor apparatus and processing that provides a means for locating and tracking important objects in the sporting event, for instance, the locations of the puck and players in a hockey game [Honey and et al (1996)].

The raw video images and their augmentations (the graphics) are merged using a technique similar to the chroma-key process [White and et al (2005)]. However, our compositing process is more capable than standard chroma-keying because our approach allows translucent (or opaque) graphical elements to be inserted between foreground and background objects. A novelty is that our approach allows for spec- ifying multiple ranges of colors to be keyed, and some specifically not keyed. This enables the addition of graphics such as corporate logos or other graphical details

3 such as a virtual off-sides line in soccer to the ground on a field in such a way that the graphics appear to have been painted on the grass.

This work is driven by the demands of network television broadcasting and is ac- cordingly focused on implementations that run in real time and that are econom- ically feasible and practical. Nevertheless, many of these ideas can be applied in other arenas such as augmented reality [Azuma and et al (2001)]. We will describe several applications of the approach; some incorporate computer vision to track ob- jects while others use non-vision sensors but serve to illustrate important aspects of our general approach.

In the following sections we describe the evolution of our tracking system and broadcast effects from 1996 to 2006. Our initial system was successfully deployed during the 1996 NHL all-star game, enhanced televised live hockey with a visual marker (a bluish glow) over the puck in the broadcast image [Cavallaro (1997);

Honey and et al (1996, 1999)]. Since that time, we have applied our technolo- gies to broadcast football, baseball [Cavallaro and et al (2001a,b)], soccer, skating,

Olympic sports and NASCAR racing events [Cavallaro and et al (2003)]. In this paper we focus on individual tracking technologies and the core instrumentation and compositing techniques shared by the various implementations.

2 Overview: Adding Virtual Elements to a Live Broadcast

Most of our effects depend on a real-time integration of graphical elements together with raw video of the scene to yield a final compelling broadcast image. Before we discuss methods for integrating dynamically tracked elements into a broadcast image, it is instructive to consider the steps necessary for integrating static graphical

4 A&?*8 A&?*8&% E4$'+/";3!F&"+ B*C3D/%&$

)'$*%#*(+ ,6'$-*37&83 :-*;& )'$*%#*(+ B*C3D/%&$ 94&'*+$' ,*-&'*

<$'&;$'5"% >*" )*#=;'$5"% !"#$%&'( 0/?+ @$$- 2*++&3 2*++& ,$-45+*+/$" ,$-4$(/+$'

!11&#+3:-*;&

.'*46/#*? .&$-&+'/# !11&#+ !?&-&"+( 0'*"(1$'-*+/$" :-*;&

!"F/'$"-&"+3 2$%&?

Fig. 2. An overview of the video and data flow from the field of play (left) to final compos- ited image (right). elements such a logo, or a virtual line marker with raw video.

Figure 2 provides a high-level overview of the flow of video and data elements through our system to insert a virtual graphical element (in this case a corporate logo) into a live football scene. For the effect to appear realistic, the graphic must be firmly “stuck” in place in the environment (e.g., the ground). Achieving this component of realism proved to be one of the most significant challenges for our technology. It requires accurate and precise calibration of broadcast cameras, cor- rect modeling of the environment and fast computer graphics transformations and compositing.

5 For now, consider the sequence of steps depicted along the bottom of the diagram. A corporate logo is transformed so that it appears in proper perspective in the scene.

The transformation step is accomplished using a model of the environment that describes the location of the camera and the relative locations of and orientations of virtual graphical objects to be included. Encoders on the camera are used to calculate the current field of view of camera and perform the appropriate clipping and perspective transformation on the graphical elements.

We must also compute a matte that describes at each pixel whether foreground or background elements are present. The matte is used by the compositor to determine an appropriate mixture of the effect image and raw video for the final image.

2.1 Adding Dynamic Objects to the Image

Given the infrastructure described above it is fairly straightforward to add dynamic elements. The locations of the graphical elements can be determined using any of a number of tracking algorithms as appropriate to the sport.

As an example, we have recently developed a system for tracking the locations of all the competitors in a motor racing event. Each vehicle is equipped with a GPS receiver, an IMU, and other measurement devices. The output of each device is transmitted to a central processor that uses the information to estimate to position of each vehicle [Cavallaro and et al (2003)].

Given the position information and the pose of the broadcast cameras we are able to superimpose relevant graphics that indicate the positions of particular vehicles in the scene (see Figure 3).

6 Fig. 3. Objects that are tracked dynamically can be annotated in the live video. In this ex- ample Indy cars are tracked using GPS technologies, then the positions of different vehicles are indicated with pointers.

2.2 Camera Calibration

Fig. 4. The broadcast camera is instrumented with precise shaft encoders that report it’s orientation and zoom.

Yards lines, puck comet trails and logos must be added to the video at the proper locations and with the right perspective to look realistic. To determine proper place- ment, the field of view (FOV) of the broadcast camera is computed from data col- lected from sensors on the cameras. Others have also addressed this problem and propose to exploit redundant information for auto calibration [Welch and Bishop

7 (1997); Hoff and Azuma (2000)].

To determine the FOV of the video image, the tripod heads broadcast cameras are

fitted with optical shaft encoders to report pan and tilt (a trip head is shown in Fig- ure 4). The encoders output 10,000 true counts per revolution (we obtain 40,000 counts per revolution by reading the phase of the two optical tracks in the encoders).

The angular resolution translates to 0.6 inches at 300 feet. Zoom settings are deter- mined by converting an analog voltage emitted by the camera electronics to digital using an analog-to-digital converter.

The broadcast cameras are registered to their environment before the game by pointing them at several reference locations on the field (e.g., the 50 yard line in football or home plate in baseball) and annotating their location in the image using an input device such as mouse click. This registration process enables us to know the precise location and orientation of each broadcast camera.

2.3 Video Compositor

Many industries such as broadcasting, videography and film making include tech- niques for combining images that have been recorded in one environment with those images recorded in another. For example, weather forecasters are depicted in front of an augmented animated weather map while in reality they are in front of a blue screen; or actors are filmed safely in a studio while they appear to the audience on the South Pole or climbing Mount Everest.

Chroma keying is a standard technique that replaces the original blue screened background with that of the special effect, for instance, a more interesting geo- graphical location, giving the illusion of an actor submerged in a more intriguing

8 environment. The resulting image is called the composite image. An extension to chroma keying, typically called linear keying (or digital compositing) allows the in- clusion of transparent objects by proportionally displacing the pixels in one image

(foreground) with corresponding pixels of another image containing the augmented objects (background). The specific proportion that a pixel is displaced is determined by a by an alpha signal. The alpha signal determines the “opacity” of the foreground pixel to the corresponding background pixel. An alpha signal of ! = 0 indicates that the foreground pixel is transparent while an alpha signal of ! = 1 indicates that it is opaque. The values of the alpha signal for all the pixels in an image is given by a grey scale raster image and is typically referred to as the matte. Limitations of traditional techniques include that they only key in colors in a single contiguous range of colors (e.g., blues) and they are computationally intensive.

Our technique improves on standard chroma key techniques and allows the addition of virtual objects in the scene, for example a virtual first down line to a television broadcast of American football, or augmenting video with virtual advertisements.

The system allows the specification of multiple colors also allows the specification of colors that should not be blended (i.e. for foreground objects). Our approach gains efficiency by considering only those pixels that the prospective graphics effect will affect.

Pixel blending is determined by a combination of factors. The color map indicates which colors can be blended and by how much. The color map also which colors should not be blended. The green related to grass, for example, the white relating to chalk and brown relating to dirt can be modified while colors in the players’ uniforms should not.

The compositing system is depicted in Figure 5.

9 !"#$"%&

'(()*+ !"#$"%&'"(

!!

"#$#%&'()&*(+(

Fig. 5. The compositor.

The compositor considers four inputs: The program video or foreground images, the effect or background, an optional input !M or control ! which can be used to modify how the effect is blended into the program video without taking into effect the color in the program video. It can be used to prevent blending the video signal based on the color map. A high level view is shown in Figure 5. The effect typically uses designated transparent colors to provide a geometric keying effect so that the program video is only altered by non-transparent colors.

In the figure the program video contains players (unmodifiable) indicated by white circles, chalk indicated by black diagonally striped background. The effect includes a horizontal stripe embedded by a transparent background indicating a horizontal graphic to be added to the video image and the transparent background indicates areas in the video that are unaffected by the augmented graphics. The composite includes both the horizontal line inserted “behind the players.”

The compositor computes the composite pixels (C) by considering the foreground pixel (FG), the background pixel (BG), !M and !FG by first computing ! and next

10 the final composite value for the composite pixel as follows:

!FG 2 (0.5 !M) if 0.0 !M 0.5 ! =  ∗ ∗ − ≤ ≤ (1)  2 (! 0.5) if 0.5 < ! 1.0 ∗ M − M ≤   C =(1 !) FG+ ! BG (2) − ∗ ∗ Performance can be gained by avoiding accessing the color map when 0.5 < ! M ≤ 1.0. Note that when !M is 0.5 then ! becomes 0 and the background pixel (the effect) is ignored and only passes the foreground pixel to the composited image and when !M is 1 the background (see 2). !M = 0 indicates that blending is determined to the value of !FG. !M is used to force either the background or the foreground pixel to be used or bias the blending.

The selection of colors to include or exclude are made using a GUI with a pointing device to indicate the blending value !FG for inclusion or exclusion. The selection is then processed and inclusion and exclusion filters are generated.

3 Vision-Based Baseball Tracking

We have developed techniques for tracking a baseball in two phases of play: 1) from pitcher to plane of home base, and 2) after a hit, from the batter to a predicted or actual impact with the ground. We will focus initially on the pitch phase.

3.1 Problem Definition: The Pitch

Our objective is to determine a parametric equation describing the motion of the ball from the pitcher to the front plane of the 3-dimensional strike zone. Once we

11 know the equation and parameters of motion, the track of the ball can be included in broadcast graphics.

An important issue therefore is the form of the equation describing the motion of the ball and the relevant parameters. In his book The Physics of Baseball, Robert

Adair describes a number of factors that affect a baseball in flight, including: initial position, initial velocity, spin rate and axis, temperature, humidity and so on [Adair

(2002)]. In our work, we approximate Adair’s equations by assuming the ball is acted upon by a constant force A throughout it’s flight.

S

A G

Fig. 6. The forces on a baseball in flight, viewed from the pitcher’s mound: G is the force of gravity; S is the force resulting from the spin of the ball and its interaction with the atmosphere; A is the net acceleration we solve for in our model. The deceleration due to drag is normal to the plane of the figure.

The net constant force, A, includes the force of gravity G, the force resulting from spin S, and deceleration due to drag (Figure 6). Some of these forces are not really constant. For instance, the force of drag is proportional to the square of velocity, which decreases over time; and the rate of spin, which also decreases over time.

Furthermore, the force arising from spin will always be perpendicular to the flight path, which is curving. And of course drag will be in line with the curving flight

12 path. However, we have found that assuming a constant acceleration yields good results. It’s worth bearing in mind that we don’t extrapolate our equations forward to find the point of impact. We use our knowledge of ball position throughout the pitch to approximate the parameters we seek.

We define the global reference frame as follows: The origin is the point of home plate nearest the catcher; +y is in the direction from the plate to the pitcher; +z is up; +x is to the pitchers left. We construct equations for x, y, and z in terms of time t as follows:

1 x(t)=x + v t + a t2 (3) 0 0x 2 x 1 y(t)=y + v t + a t2 (4) 0 0y 2 y 1 z(t)=z + v t + a t2 (5) 0 0z 2 z where x0,y0,z0 is the initial position of the ball at release; v0x,v0y,v0z is the initial velocity; and ax,ay,az is the constant acceleration. Our objective then is to find the values for these nine parameters for each pitch.

Our approach is to track the ball in screen coordinates from each of two cameras:

One above and behind home plate (“High Home”) and one above first base (“High

First”). The location of these cameras is illustrated in Figure 7. Using the screen coordinates of the ball at several time instants we are able to infer the parameters of the pitch. Example images from these cameras are provided in Figure 7.

3.2 Solving the Equations of Motion

The position, orientation and focal length of each camera is known. Lens distortion is modeled as well. For the moment, suppose the parameters of motion of the ball

13 Px(t)

Py(t)

Fig. 7. Upper Left: Location of the two cameras used to track the ball during the pitch: High

Home and High First. Upper Right: The lines of position observed by the two cameras for a pitch. Lower Left and Right: Example images from the High Home and High First cameras after the pitcher has released the ball. The Lower right image shows The X and Y position of the ball at time t in CCD coordinates from the High First camera described in the next section. are also known. We could then predict, at any time t, where the ball should appear in the field of view of each sensor camera. We use Px(t) to indicate the X pixel position and Py(t) to indicate the Y pixel position of the ball in the image (see

Figure 7). Px(t) can be estimated in a linear equation as:

Px(t)=C1x0 +C2y0 +C3z0+ (6)

C4v0x +C5v0y +C6v0z+ (7)

C7ax +C8ay +C9az (8)

14 C1 through C9 depend on the time and on the pose of the camera. However, for any particular t they are constants. C1,C2, and C3 are independent of time; They are computed from the camera pose and focal length parameters and represent a projection from the global frame to the camera’s frame. C4,C5, and C6 depend on

2 t and the camera pose. C7 through C9 depend on the camera pose and t . A similar equation describes the Y position of the ball from the camera’s view:

Py(t)=C10x0 +C11y0 +C12z0+ (9)

C13v0x +C14v0y +C15v0z+ (10)

C16ax +C17ay +C18az (11)

The images captured at each camera are processed independently; so we have a location Px(t),Py(t) for the ball for each camera in screen coordinates at each instant in time. We locate the ball in the images from each camera independently (i.e. the cameras are not synchronized). In a later section we describe how the ball is found in an image.

When an image is processed to find the ball we know t (the time the image was captured) and we find Px(t) and Py(t) by locating the ball in the image. The con- stants C1 through C18 are computed immediately using the camera pose parameters, t, and t2. Accordingly, from each image we have two equations; each equation with

10 known values (e.g., Px(t) and C1 through C9) and 9 unknowns.

Video fields (images) are captured at 60 Hz, and the ball is in flight for about 0.5s, so we collect data for 60 equations from each camera for each pitch, or 120 equa- tions in all. The equations are solved for the 9 unknowns using standard Singular

Value Decomposition (SVD) techniques. Because there is some ambiguity in the determination of the location of the ball in each image, the SVD solution is approx- imate, with a residual. A large residual indicates a poorly tracked pitch, in which

15 case the data may be discarded for that pitch.

3.3 The Tracking Cameras

We use Bosch LTC0495 Day/Night cameras in black and white mode. The cameras

1 include an auto iris function that we use to maintain a shutter speed of 1000 or faster. The cameras include an optional IR filter. We ensure that the filter is always engaged. The cameras gather interlaced fields at 60 Hz which are in turn processed by our software.

The software continuously searches the image for a moving ball in each camera image. We take advantage a number of constraints in order to simplify the process of finding and tracking the ball. First, we limit the search in the region of each image where there are no other moving objects. The stretch of grass between the pitcher and home plate is normally devoid of players or other moving objects. Next, we use image differencing to find candidate blobs that might represent the ball. If the blob is not round and light colored, it is discarded. Once an initial candidate blob is found we look for a subsequent blob in a location towards home plate that corresponds with a speed of 60 to 110 miles per hour (nearly all pitches are in this range of speeds). If the track does not meet these requirements it is discarded.

The result is that nearly every pitch is tracked, and very few non-pitches are mis- identified as pitches.

16 Fig. 8. Validating the system. A pitching machine launches baseballs at a foam target.

3.4 Validation, Results and Use

To initially validate the system, we placed a foam board behind home plate at a local ball field. We positioned a motorized pitching machine on the pitcher’s mound and launched 20 pitches at the target. Upon impact the balls left distinct pivots in the foam board (see Figure 8. The Y location of the board was known, and we used our tracking system to predict the X and Z location of each pitch at that plane. 18 of the pitches hit the board (two missed and were not considered). The locations of the pivots were measured carefully and compared with the predicted locations. Average error was less than 1 inch. Since that time we see noise with a Standard Deviation of 0.5 inch and a random bias that is diminishing with continued development to about 0.5 inch.

Figure 9 shows how the trajectory of a pitch can be reconstructed. Several example images from televised broadcasts are shown in Figure 10. The system has been em- ployed in hundreds of Major League Baseball games for national broadcast, includ- ing the World Series in 2005. Video examples are available at www.sportvision.com.

Public response has been positive [Hybinette (2010)].

17 Fig. 9. Here we show the trajectories of three pitches from a Major League Baseball game.

For each pitch the position of the ball is indicated at 1 millisecond intervals. The left image shows a top down view with the pitcher on the right and home plate on the left. The right image provides a view of the pitches from the strike zone. Notice how the pitches seem to

“break” dramatically at the end of their travel.

Fig. 10. Several examples of the tracking system in use on broadcast television.

3.5 After the Ball is Hit

We have also developed a method for estimating the trajectory of the ball after a hit.

This phase of play provides a separate challenge because any fixed camera would not have a sufficient field of view to track the ball in all the locations in a stadium.

18 In a manner similar to the techniques used for pitch tracking, we assume the ball undergoes a constant acceleration. We further simplify the problem by assuming a

fixed location (home plate) for the start of the trajectory.

We use two cameras pointed towards the home plate with fixed pan, tilt focus and zoom. Additional cameras are mounted on tripods having servo motors that allow computer control of the pan and tilt. Initially, the two fixed cameras capture video of the action at home plate. After it is hit we search for the ball in the video frame using image differencing. In a manner similar to that described above for pitch tracking, we compute an initial velocity vector for the ball.

The ball’s path can change as it travels along the initially computed trajectory due to wind, spin, and so on. Thus additional cameras are used to update the path as the ball travels. Typically we place two cameras at opposite sides of the baseball field, approximately halfway between home plate and the home run fence. A computer uses the previously determined trajectories to predict when the future path of the ball will be within the view of the outfield cameras. We predict both the time and the location along the ball’s path and send signals to the servo motors of the cameras so they point to the predicted location at the appropriate time. The outfield cameras do not move when actively capturing video.

For each outfield camera, a processor determines lines of position as discussed previously and the three dimensional location of the ball at multiple times is de- termined. Next, the system determines whether more data can be captured (i.e. if the ball will be in view). If so the system loops back to predict future locations and moves the cameras if necessary. Otherwise the end of balls path is determined.

The end of the path determines the location at which the ball’s path would intersect the ground (field level), assuming that the ball’s path is not interrupted by a fence,

19 a person, a wall, stadium seating or other obstructions. This predicted position is compared to the location of home plate to determine the distance traveled which later can be used in the broadcast.

The system provides a mechanism to display the path and the predicted path of a moving object during broadcast. For example after the system determines the initial path of a baseball the path can be displayed. Graphics show the location where the ball will land. As the system updates the path of the ball the displayed path and the displayed graphics at the end of the path can be updated. For example if the batter hits a fly ball to the outfield, while the outfielder is attempting to catch the ball the television display can show the expected path of the ball and the location in the outfield where ball will land.

4 Vision-based Hockey Puck Tracking

Although the FoxTrax hockey puck tracking effect was our first on-air system, we introduce it last because it is in some ways more complex than our later tracking systems [Cavallaro (1997)]. In particular, the environment in which the puck is tracked is much more challenging than is the case for baseball tracking: Compare an NHL rink with the relative calm of the grass between the pitcher’s mound and home plate. In addition to the players moving rapidly around, the puck is subject to change direction at any moment.

All these factors combine to make it much more difficult to find the puck and track it in video images. Our approach to this problem was to create a new puck that could be found easily. We embed bright Infra-Red LEDs inside the puck along with a circuit designed to pulse them brightly at approximately 30 Hz. See Figure 11.

20 Fig. 11. The FoxTrax hockey puck. Left: The circuitry that is embedded in the pucks. Right:

Several finished pucks ready for play.

10 calibrated sensor cameras are placed in the ceiling, pointed at the ice. The sensor cameras are fitted with IR notch filters (to pass IR and block other frequencies) to enhance the signal to noise ratio of the flashing IR LEDs when viewed from above.

All of the sensor cameras are synchronized with each other, and the puck to ensure images are taken at just the right instant.

The location of the puck is determined using geometry in much the same way the baseball is tracked as described above. Given the location of the puck and the known pose of each broadcast camera we can compose a complete image with a glowing effect for the puck (Figure 12).

Fig. 12. Several images showing the on-air effect, the “glowing hockey puck.”

21 5 Future Work

We have described several techniques to improve the experience for viewers watch- ing sports ranging from baseball and hockey to NASCAR and Olympic Sports.

Initially the hockey puck received mixed reviews from sports critics (e.g., [Mason

(2002)]), but follow on technologies have been quite popular and have resulted in the award of several Emmys; for example for NASCAR’s Trackpass, Football’s 1st and Ten yellow line and Baseball’s KZone. Our focus is to continue to provide the viewer with interesting information while not being intrusive. We also intend to provide interactive web technologies for the sports viewer (as others are also developing, e.g., [Rees et al. (1998); Rafey et al. (2001)]).

We are exploring other techniques for improving vision based tracking such as region based mechanisms like [Andrade et al. (2005)] and using hybrid sensors in poor lighting conditions such as lasers [Feldman et al. (2006)] for robustness and precision.

References

Adair, R. K., May 2002. The Physics of Baseball, 3rd Edition. Harper Perennial.

Harper Collins.

Andrade, E. L., Woods, J. C., Khan, E., Ghanbari, M., 2005. Region-based anal-

ysis and retrieval for tracking of semantic objects and provision of augmented

information in interactive sport scenes. IEEE Transactions on Multimedia 7 (6),

1084–1096.

Azuma, R., et al, 2001. Recent advances in augmented reality. IEEE Computer

Graphics and Applications 21 (6), 34–47.

22 Cavallaro, R., 1997. The FoxTrax hockey puck tracking system. IEEE Computer

Graphics and Applications 17 (2), 6–12.

Cavallaro, R. H., et al, October 2001a. System for determining the end of a path for

a moving object. United States Patent 6,304,665 B1.

Cavallaro, R. H., et al, September 2001b. System for determining the speed and/or

timing of an object. United States Patent 6,292,130 B1.

Cavallaro, R. H., et al, December 2003. Locating an object using gps with addi-

tional data. United States Patent 6,657,584.

Hybinette, M., 2010. Personal Communication.

Feldman, A., Adams, S., Balch, T., 2006. Real-time tracking of multiple targets

using multiple laser scanners (in submission).

Gloudemans, J. R., et al, July 2003. System for enhancing a video presentation of

a live event. United States Patent 6,597,406 B2.

Hoff, B., Azuma, R., 2000. Autocalibration of an electronic compass in an out-

door augmented reality system. In: IEEE and ACM International Symposium on

Augmented Reality (ISAR-2000).

Honey, S. K., et al, October 1996. Electromagnetic transmitting hockey puck.

United States Patent 5,564,698.

Honey, S. K., et al, June 1999. System for enhancing the television presentation of

an object at a sporting event. United States Patent 5,912,700.

Mason, D. S., 2002. “Get the puck outta here!” Media transnationalism and Cana-

dian identity. Journal of sport and social issues 26 (2), 140–167.

Rafey, R. A., Gibbs, S., Hoch, M., Gong, H. L. V., Wang, S., 2001. Enabling custom

enhancements in digital sports broadcasts. In: Web3D ’01: Proceedings of the

sixth international conference on 3D Web technology. ACM Press, New York,

NY, USA, pp. 101–107.

Rees, D., Agbinya, J. I., Stone, N., Chen, F., Seneviratne, S., de Burgh, M., Burch,

23 A., 1998. CLICK-IT: Interactive television highlighter for sports action replay.

In: 14th International Conference on Pattern Recognition (ICPR’98). pp. 1484–

1487.

Welch, G., Bishop, G., 1997. SCAAT: Incremental tracking with incomplete infor-

mation. In: SIGGRAPH ’97: Proceedings of the 24th annual conference on Com-

puter graphics and interactive techniques. ACM Press/Addison-Wesley Publish-

ing Co., New York, NY, USA, pp. 333–344.

White, M. S., et al, June 2005. Video compositor. United States Patent 6,909,438

B1.

24