Europäisches Patentamt (19) European Patent Office Office européen des brevets (11) EP 1 037 167 A2 (12) EUROPEAN PATENT APPLICATION (43) Date of publication: (51) Int. Cl.7: G06T 15/20 20.09.2000 Bulletin 2000/38 (21) Application number: 00301953.6 (22) Date of filing: 09.03.2000 (84) Designated Contracting States: (72) Inventor: Deering, Michael F. AT BE CH CY DE DK ES FI FR GB GR IE IT LI LU Los Altos, California 94024 (US) MC NL PT SE Designated Extension States: (74) Representative: AL LT LV MK RO SI Harris, Ian Richard D. Young & Co., (30) Priority: 15.03.1999 US 268397 21 New Fetter Lane London EC4A 1DA (GB) (71) Applicant: SUN MICROSYSTEMS, INC. Palo Alto, California 94303 (US) (54) System and method for generating and playback of three-dimensional movies (57) A system and method for generating and play- ing back three-dimensional (3D) movies are disclosed. The system is capable of partially rendering frames without relying upon exact viewpoint information. The partially rendered frames may be rendered to the extent possible without performing viewpoint-dependent proc- esses, and then compressed and stored to a carrier media To reduce the amount of data to be stored, the viewer's possible viewpoints may be restricted (e.g., by defining a viewpoint-limiting volume or region). The resulting partially-rendered geometry data may be com- pressed using geometry compression. During playback, the compressed frames are read as a stream, and decompressed. Any final viewpoint-dependent render- ing operations may then be performed (e.g., some light- ing calculations and atmospheric effects, some fogging, specular highlighting; and reflections). A sensor such as a head-tracker may provide real-time viewpoint informa- tion that may be used by the playback system. After ren- dering, the frames are rasterized and then displayed in stereo. EP 1 037 167 A2 Printed by Xerox (UK) Business Services 2.16.7 (HRS)/3.6 12EP 1 037 167 A2 Description to display a 3D football game that viewers can view from almost any angle they desire (e.g., from a bird's eye BACKGROUND OF THE INVENTION view directly overhead or from the side of the fifty yard line). To change their viewpoint, viewers simply move 1. Field of the Invention 5around the coffee table. The head-tracking sensors will determine where the viewer is and display different [0001] This invention relates generally to the field of component images accordingly. When used in connec- computer graphics and, more particularly, to three- tion with a head-mounted display, head tracking allows dimensional (3D) movies. panoramic scenes that "wrap-around" viewers without 10 the need for large projection screens that physically sur- 2. Description of the Related Art round the viewer. Such head-tracked stereo systems produce a look-around holographic feel qualitatively dif- [0002] Traditional motion pictures provide the audi- ferent from older fixed image displays. ence with what may called a "two-dimensional" experi- [0007] However, these new systems have yet to see ence. Most motion pictures are projected on flat screens 15 wide-spread acceptance as movie playback devices. with the audience observing the movie from afar. Aside One reason is that these systems typically rely upon re- from improvements in sound playback and larger rendering each scene every time the viewer changes film/screen formats, the movie going experience has not their viewpoint. For example, as the viewer's head changed significantly for a number of years. changes position (or orientation, in some systems), the [0003] Some films have been produced in visual 20 scene being displayed is re-rendered from scratch using stereo, e.g., with the audience wearing red-blue or the new viewpoint To produce a realistic computer-gen- polarized glasses. Similarly, head-mounted displays erated image, a tremendous number of calculations have also allowed viewers to see visual stereo images must be performed. Even with current advances in and films. But visual stereo has not seen widespread microprocessors, most systems still fall significantly success relative to standard motion pictures. This may 25 short of being able to render detailed realistic scenes in be because the stereo-effect is typically limited to a few real-time. Given the high frame rates need for smooth special effects where a few objects appear to have movie-like animation, it may be many years before dimension or "leap out" at the audience. It is likely that processing power approaches the levels necessary to the effects that simple visual stereo provides does not completely render complex images (e.g., like those in outweigh the added cost of production and the hassle of 30 the movie Toy Story) a frame rates high enough to rival wearing glasses or head-mounted displays to view the current motion pictures. image. [0008] Thus, an efficient system and method for [0004] However, recent advances in computer generating and playing back realistic 3D movies with graphics and display technologies have hinted at the viewer-changeable viewpoints in real-time is desired. In possibly of a more realistic experience for movie goers. 35 particular, a system and method capable of reducing New sensing technologies such as head-tracking may the number of calculations to be performed for real-time increase realism from simple visual stereo to a truly playback of 3D movies is desired. three-dimensional (3D) viewing experience. [0005] As used herein, the term "visual stereo" SUMMARY OF THE INVENTION refers to the process of generating two images (i.e., one 40 for the viewer's left eye and another for the viewer's right [0009] Particular and preferred aspects of the eye). These images may be referred to herein as stereo invention are set out in the accompanying independent component images. In contrast to simple visual stereo, and dependent claims. Features of the dependent 3D images and movies go one step further by dynami- claims may be combined with those of the independent cally changing the viewpoint (ideally on a frame-by- 45 claims as appropriate and in combinations other than frame basis) for each of the two stereo component those explicitly set out in the claims. images. In this way, an objected viewed in 3D will not [0010] The problems outlined above may at least in only appear to leap out of the screen, but objects behind part be solved by a system capable of partially render- it will become obscured or un-obscured as the viewer ing frames without relying upon exact viewpoint infor- changes their viewpoint to different positions around the 50 mation. The partially rendered frames may be rendered object. In other words, a traditional stereo image of a to the extent possible without performing viewpoint- globe will appear to have depth, but a tree-dimensional dependent processes, and then compressed and stored image of the globe will allow the viewer to move around to a carrier medium. To reduce the amount of data to be the globe to see more than one side of it. stored, the viewer's possible viewpoints may be [0006] Thus, head tracking sensors allow graphics 55 restricted (e.g., by defining a viewpoint-limiting volume systems to recalculate the component images accord- or region). ing to where the viewer is in relation to the display [0011] During playback, the compressed frames device. Head tracking may allow a coffee table display are read as a stream, decompressed, and then raster- 2 34EP 1 037 167 A2 ized for display. As part of the rasterization process, the viewpoint-independent) calculations, such as some viewer's viewpoint may be determined and used to per- lighting calculations (e.g., viewpoint-independent reflec- form viewpoint-dependent effects such as some lighting tions, viewpoint-independent shadows, viewpoint-inde- and atmospheric effects (e.g., fogging, specular high- pendent depth of field effects, viewpoint-independent lighting, and reflections). 5bump mapping, and viewpoint-independent displace- [0012] As used herein the term "real-time" refers to ment mapping) and tessellating the three-dimensional performing an action, task, or process rapidly enough graphics data into micropolygons to meet one or more so that the user or viewer will not be substantially dis- predetermined criteria. These predetermined criteria tracted by the amount of time taken to perform the task. may include smoothness and micropolygon size in As used herein, the term "including" means "including, 10 world space. In other embodiments they may also but not limited to." include estimated worst-case micropolygon size in [0013] Movies may be thought of as linear story tell- screen space. This estimated worst-case micropolygon ing. Thus, in 3D movies the viewer may be like Scrooge size in screen space may be estimated using the dis- (i.e., dragged along by ghosts of Christmas past, tance from a closest permitted viewpoint to each micro- present, and future). The viewer can see everything in 15 polygon and or the worst-case angle from all permitted partial or full three dimensions, but generally may not viewpoints to the micropolygon. interact with anything (in some embodiments interaction [0018] In addition to geometry compression, in may be allowed). Furthermore, the viewer can only go some embodiments the micropolygons may be culled to the places where the ghost goes. Note that this cov- by deleting micropolygons that are not visible from any ers not only pre-scripted linear story telling, but also 20 valid viewpoint (i.e., from any viewpoint that satisfies the most forms of remote viewing (e.g., live sporting specified viewpoint limitations. The compressed frames events). are configured to be rasterized in real-time by applying [0014] Looked at another way, current general pur- viewpoint-dependent lighting effects.