My Personal History in the Early Explorations of Computer Graphics

Visual Comput (2005) 21: 961–978 DOI 10.1007/s00371-005-0360-9 ORIGINAL ARTICLE

Gary Demos My personal history in the early explorations of computer graphics

Published online: 22 November 2005 Abstract Gary Demos discovered tions, where they produced effects © Springer-Verlag 2005 computer graphics while hearing for the movies Last Starﬁghter and a computer-generated ﬁlm presen- “2010”, which were both released in tation at CalTech in 1970 by John 1984. Digital Productions used the Whitney Sr. Gary then began working Cray XMP computer, together with under the direction of Ivan Sutherland the Digital Film Printer that they had in Utah to develop early computer developed at Information Interna- graphics hardware and software. In tional. Following a hostile takeover 1974 Gary and John Whitney Jr. by Omibus of Digital Productions in started the “Motion Picture Project” 1986, Whitney/Demos Productions at Information International to pro- was formed, using a Thinking Ma- duce computer generated simulated chine parallel computer. This paper scenes for movies (Futureworld, describes the technical challenges Looker, and Tron) and commercials. and achievements of this early visual These early computer-generated computing. visuals were quite challenging given the level of software and hardware Keywords CGI · Computer graph- · · G. Demos technology available in the 1970’s. In ics Scene simulation Visual Los Angeles, CA, USA 1981 Gary and John left Information effects [email protected] International to form Digital Produc-

John Whitney Sr. 1970–1971 to Fourier Optics”, which gave me an general understanding of lasers, lenses, and electro-optical components. All When I was a college student in 1970 at CalTech, I was of these basic technical foundations were utilized in my fascinated by computers, and the mathematics that they subsequent work. enabled. John Whitney Sr. as visiting teacher presented It should be noted that computer science was not an ac- amazing 16 mm movies of dots and lines, moving in credited degree subject in 1971, for either undergraduates time with music, created by computer. I was particularly or graduate students. Computers were, at that time, con- amazed by the rich colors and deep blacks that he created sidered a tool in support of other scientific disciplines, and using Kodachrome movie film. His enthusiasm for experi- not in-and-of-themselves a discipline. mental filmmaking with computers was contagious, and I began to explore the intricate visual patterns that could be created by computer. CalTech provided me with an excellent undergraduate Evans and Sutherland 1972–1974 education in physics, mathematics, and electrical engineering. Although I did not finish my senior year, I particu- As a consultant to an educational movie (by IBM) about larly enjoyed a senior/graduate class entitled “Introduction computers, I became aware of the work of Ivan Suther- 962 G. Demos land and Dave Evans (as professors), and of Ed Catmul, the tester circuit to Mostek, after which all the chips we and Fred Parke (as graduate students) at the University of received were good. Utah. They were exploring “shaded surfaces with hidden I was quite surprised that there was a great deal of surfaces removed” where the physics of light interacting interest in “painting” into the frame buffer with the pen with surfaces was being crudely modeled with a few hun- and tablet. I felt this was a trivial use of the expensive dred polygons. I was fascinated by the potential of this system, since all that was involved was a loop to put work, and I contacted Ivan Sutherland to see if he was the current color in a memory location matching the pos- interested in applying this computer technology to movie ition of the tablet pen. However, a great deal of excite- making. I found that he shared this same interest, and I ment was created by this simple use of the frame buffer, joined him (as an apprentice) at the Evans and Suther- since apparently the use of random-access in the frame land Computer Graphics Company in 1973, and worked buffer’s memory was very new and interesting to every- on several essential computer graphics technologies under one. There was also excitement about the ability to cycle his direction. These included a large two-pen data table for and update the four color tables (now called lookup ta- creating three-dimensional polygonal surface data, Ivan’s bles, or LUTs) driving the DACs (now integrated with own “hidden surface” algorithm, and the development of LUTs as RAMDACs). Again, I felt this was a trivial use the first DRAM-based frame-buffer for color image dis- of the lookup tables (making pseudo-color animations), play. which I had considered the basis of color spaces for get- The University of Utah became our first customer for ting the most out of the limited 8-bit-per-pixel frame the frame-buffer system, which consisted of 32-boards, buffer memory. In the subsequent several years, however, cost $80 000, providing only 512 × 512 × 8-bit display. Alvy Ray Smith developed and published more broadly However, some features of this system were forward look- useful mappings for the DAC lookup tables. After Ed ing, including hardware zoom and pan, color lookup tables Catmul and Alvy Ray Smith graduated from the Uni- before the DAC’s, and multiple realtime digital video in- versity of Utah the went to the New York Institute of put and output ports. Technology where they ordered a 24-bit-per-pixel three- Another fascinating aspect of this project was the de- wide version of our frame buffer, providing them with the velopment of a DEC PDP-11 DR11-B test interface for first full-color-capable random-access frame-buffer sys- testing the Mostek 4kbit Drams, since the memory chip tem. This system became a central tool for their subse- industry had not yet established its own testing proced- quent work. ures. Of the first eight chips delivered to us, we found that Another interesting part of the frame buffer develop- seven were bad. Of the next eight chips, three were bad. ment was learning about RGB color reference CRT video At that point we gave our testing software (all ones, all ze- monitors. We produced the video 0.4 “gamma” in ana- ros, wandering ones and zeros, and random numbers) and log using a diode ladder after the DAC. Further, we found

Conrac monitors that used the original NTSC color phos- of random numbers. I authored a paper to share with phors, having a substantially better red and green color my colleagues on this work, although it was never pub- purity than the SMPTE-C phosphor colors that were be- lished. coming ubiquitous at that time. The quality difference Another activity was the continued development of between an RGB signal to the monitor vs. an NTSC- Ivan Sutherland’s recursive hidden surface algorithm. I coded color signal (which was significantly degraded) also worked on the Rand Corporation’s DEC PDP-11 comput- amazed me. ers in the very early morning hours each day. I discovered It should be noted that the University of Utah at this that recursive subdivision was very sensitive to precision time was unique in having a computer graphics gradu- problems, and I presented these problems to Ivan, which I ate degree program. Many of the graduate students went felt were fundamental to this type of algorithm. Although on to become key figures in the computer graphics in- we made many successful images using Ivan’s recursive dustry. For example, Ed Catmul and Alvy Ray Smith algorithm, I was of the opinion that the algorithm was not founded Pixar (Ed is currently the president of Pixar). sufficiently stable to support moving image production. Ed showed the first “picture-texture” mapping technique, Through this work with Ivan, however, I gained substan- using the parametric values of bicubic patches to lookup tial expertise in synthetic image rendering technology, and pixels in the texture map. Also, Jim Clark founded Sil- I developed an understanding of the subtleties of hidden icon Graphics, which for many years was the largest surface algorithms. computer graphics workstation company. I was privileged In 1974, there was little infrastructure for “Venture enough to be invited by Dave Evans and Ivan Suther- Capital”, and the Picture/Design group was not successful land to attend portions of Ed Catmul’s and Jim Clark’s in finding a source of funding. Ivan went to Rand Corpo- Ph.D. thesis presentations. There were many others of ration, and then later to CalTech as a professor. special note at the University of Utah, including Gary Watkins, Bui Tuong Phong, Fred Parke, Frank Crow, Martin Newall (the creator of the “teapot”), and Richard Riesenfeld. Hank Christiansen, as a visitng professor from Information International 1974 to 1981 BYI, used the Watkins algorithm together with his “color John Whitney Jr. and I decided to continue our pursuit fringing” method to color the stresses in deforming ob- of the potential of computer simulated images. John had jects. Hank also created deformation techniques for three- worked with Information International Inc. (also called dimensional polygonal objects using “displacement vec- “Triple-I”) on the movie “WestWorld” in 1973 with tors”. author/director Michael Crichton. Information Interna- In 1975, Jim Blinn also entered the Utah graduate pro- tional was the world leader in digital film scanning and gram, and immediately began using the frame buffer in recording technology for microfilms, and they were very the manner that I had anticipated, as a tool for computer interested in moving toward color scanning, and record- graphics algorithms. ing, for still images, motion pictures, and color halftone printing. Information International’s board of directors included Picture/Design Group 1974 founder Ed Fredkin, a technology entrepreneur from MIT, president Al Fenaughty, and noted fathers of artificial in- telligence Marvin Minsky from MIT and John McCarthy In 1974, Ivan and I joined up with Glen Fleck and from Stanford (the inventor of the LISP computer lan- John Whitney Jr. to work on starting the “Picture/Design guage). After our presentation to the board, we were given Group”. Glen Fleck had been a designer with Charles and approval to gradually build new computer graphics tech- Ray Eames, very well-known educational film and ex- nology using their DEC PDP-10 and PDP-15 (in the III-15 hibit designers. Our first project was a test for Carl Sagan’s version) computers. “Cosmos” project, which at that time was going to be a feature film at Wolper Productions. For this project, I immediately discovered that the random number libraries FutureWorld available on the DEC computers showed unnatural radial patterns when using more than a few thousand stars in John Whitney Jr began marketing for us, and we soon galaxy simulations. I therefore developed my own ran- found that Michael Crichton was planning a sequel to dom number generator from a combination of available “WestWorld” called “FutureWorld” for summer release in algorithms, after studying the literature on the subject. 1976, to star Peter Fonda, Yul Brenner, and Blythe Danner. No single algorithm was random and pattern-free with I also discovered that there were some excellent com- hundreds of thousands of stars. I found that combina- puter engineers and mathematicians working at Infor- tions of algorithms, some multiplicative, some additive, as mation International, including Malcolm McMillan and well as others that I developed, provided the best source Karol Brandt, who soon joined our “movie project”. Other 964 G. Demos

Fig. 3. Peter Fonda’s 4000-polygon head database from FutureWorld, copyright 1976 Information International in-house computer experts (many from MIT) also pitched increased computational time with the square of the num- in to help us get started. Of particular note was Dan ber of scanlines (with the area of the image), whereas Cameron, a digital designer who had been a lead engineer scanline algorithms were only a little slower than linear for many of Information International’s digital scanning ones as a function of the number of scanlines. How- and film recording systems. We also interviewed design- ever, we added Phong-shaded specular hilights, which ers, and acquired Art Durinski who recently graduated required area-based computations for the highlight re- from UCLA’s design school. gions. Further, when using transparency, the rendering For “FutureWorld” John Whitney Jr., Dan Cameron, time increased with the average depth complexity (the and I worked out a three-camera photography system, average number of layers visible in depth prior to the using pin-registered Mitchell 35 mm cameras. We took background or prior to, and including, an opaque surface). simultaneous frames from 0 degrees, 45 degrees, and 90 I felt that transparency was a required attribute of render- degrees of Peter Fonda in both full-body and upper-torso, ing realistic scenes. We used transparency for not only although we only used the head data from the upper glass and windows, but for clouds, ground-fog, smoke, torso. We made up Peter in all white, and we placed flames, and other essential simulation effects. I also in- black fiduciary dots and crosses all over his face and sisted on the need for multi-thousand-line resolution. It body, to provide multi-camera registration points. Large was nearly a decade later before anyone’s renderers, other 30” prints were made from the images. Mal McMillan than my own, were able to exceed one thousand lines for wrote software that extended Ivan’s two-pen idea to undo movie production. I was certainly proud that we produced lens perspective and distortions, thus recreating the three- nearly a minute of computer simulation for FutureWorld at dimensional form of Peter Fonda from multiple views. 2300-line resolution in 1976. Mal and I also worked together to implement Ivan’s idea In addition to the computer simulation, I also created of “skinning” to create polygons in ribbons as we filled the a digital composite for the “Samurai warrior” scene. This detailed surface. The final data for Peter’s head was about scene used a locked-off shot of an empty decompression 4000 polygons. chamber combined with an action shot from the same Meanwhile, I worked with Frank Crow to develop camera location. The scene had three Samurai warriors a scanline-based hidden surface and lighting renderer, materializing inside the chamber. Using the difference be- starting with the work of Bouknight. The Bouknight al- tween the empty and actor scenes, I isolated the warriors gorithm had the advantage that it could support trans- for processing. It was necessary to use a garbage matte parency, unlike the Watkins algorithm. I believed that as well, in order to eliminate the shadows from the dif- scanline-based algorithms were needed, since I was de- ference. I created an algorithm that sampled the warrior’s signing the renderer for 2300 scanlines (3012 horizontally pixels and created close-packed triangles the same color as by 2300 vertically). Area-based algorithms, which were the underlying pixel. The triangles began large and grad- the other main line of research in rendering algorithms, ually decreased in size until they approached single-pixel 966 G. Demos resolution, completing the materialization. I believe this Working with Frank Crow, we developed a mechan- was the first digital composite used in a movie. I used ism for “co-planar polygons”, which allowed surface nor- 3012 × 2300 resolution for this scene. mals to follow tiny edge contours. This allowed fine object edges to kick off highlights, greatly helping to create Close Encounters test a realistic object appearance. Many real object edges are naturally rimmed with highlights. In 1977, we worked with Vilmous Zigmond, Steven Spiel- We also felt it was beneficial to build an “affect-effect” berg, Julia Philips, Michael Philips, and Doug Trumble simulator to create halation, star filter, and glow effects on tests for a scene in “Close Encounters of the Third from bright highlights on the simulated objects. I wrote Kind”. We tested the concept of free camera movement a program in the film recorder that drew hundreds of scene tracking, wherein “witness points” at known loca- thousands of tiny faint vectors away from every bright tions were visible in the scene. Using 4” spherical light pixel, using my pattern-free random numbers to control globes at known positions on the set, the camera was the direction and brightness of each vector line (computer moved freely on a crane in tests for the scene where the generated lines were called “vectors” at that time). This space ships enter the arena. There was particular interest method faithfully simulated the camera and lens optical in the agile “cuboids”, which were simulated white cubes effects generated by bright highlights when photographic that zipped around the people in the scene in free flight. real scenes. Mal McMillan developed an automated scan tracking At the 1977 Siggraph in SanJose, I brought a 35 mm system using the PFR, where the “center of gravity of projector to the filmshow audience of a hundred people. I the density” of the witness points was used to track their believe that this was the first time that many people had two-dimensional locations in the frame. Mal’s tracking al- seen photorealism and high resolution in computer simu- gorithm was relatively fast, needing only a few seconds lations. I also believe that the audience was surprised by per frame to track the points. From this data, Mal de- the glow and halation star effects being simulated on the veloped an inversion/relaxation algorithm, combined with highlights of simulated objects. Many attendees said that data smoothing, to determine the location and orienta- this was an “aha” for them, showing that simulated mov- tion of the camera within the scene for every frame. This ing objects could appear completely real. method was better than camera mount tracking, since the About this time, I began working with Jim Blinn, who minute gate weave of the film could be tracked precisely was finishing his Ph.D. at the University of Utah. Jim de- using witness points. veloped a scanline-based bicubic-patch algorithm, which Such systems are now also called “3D tracking” or “3D had computational performance that scaled linearly with match move” systems. These systems allow very high pre- vertical resolution, except when area-oriented processing cision tracking of three-dimensional scenes, so that com- was applied. However, our use of texture mapping, bump puter generated images can be placed “into” the scene mapping, and reflection mapping in conjunction with the with perfect registration. It is a form of three-dimensional bicubic patches nearly always yielded area-scaling per- compositing. formance. This high computational requirement limited We then simulated objects that would appear to be our use of bicubic patches to short sequences when using within the scene in a careful three-dimensional sense, with bicubic patches, and our normal processing used the much emphasis on the “cuboids”. This was also the first time more computationally-efficient polygon rendering. that I worked on glow, flare, and halation effects, in order We created about 100,000 lines of code for the ren- to make the cuboids appear to glow and change colors in derer, which was a mixture of Fortran and assembler code. a fog-shrouded environment. We called this renderer “Tranew”, meaning new trans- At this early time, we were not able to gain enough of parency algorithm. The resulting algorithm was efficient the production team’s confidence to get the contract for ef- at 2300 scanlines, and provided two key lights, ambient fects for the movie (which went to Doug Trumble using light, backlight, transparency, bicubic patch support, tex- motion control and models). However, we gained signifi- ture mapping (as initially demonstrated by Ed Catmul), cant confidence in our ability to place simulated objects and bump texture (from Jim Blinn) using a height field to within real scenes with a free-moving camera. perturb surface normals. We also added reflection mapping, using the surface normals to lookup reflection im- Short logo productions ages in an environment map. Our own Mal McMillan developed mathematical tools for creating spherical and After FutureWorld, we started exploring short several- cylindrical reflection environments from photographs, and second productions, given that several minutes per frame from multi-image computer renderings. limited our production capacity. Television logo’s seemed One notable project at this time were the “Pyramid to be a good fit for our early capacity, with the plan Films” logo, with a transparent pyramid refracting a rain- to gradually work up to 30-second commercial produc- bow from sunrise on the ocean. Mal developed a “tro- tion. choidal” bump map, which we animated to create the My personal history in the early explorations of computer graphics 967

Fig. 4. I.I.I. museum, copyright 1979 Information International moving waves. Another notable project was the KCET the clock in order to ensure that the processing needs were productions (PBS in Los Angeles) logo, with a postcard met for our various production projects. opening up into a night time flyover of Hollywood. The At that time, a megabyte of memory was the size of entire night street scene was compressed into the flat card, a refrigerator, and a 40Gbyte disk was the size of a wash- and expanded as the card flipped up to the camera, so that ing machine. One way that we were able to achieve ac- the camera could fly into the city. ceptable performance was the use of memory-to-memory transfers between computers, known as DMA (direct memory access). We also built a variety of custom digi- Our computer systems tal systems, using fast (but small) 4kbit SRAM memories and specialized ALU (arithmetic logic unit) and dedicated By today’s computer standards, our computational equip- multiplier chips. The computer industry was very con- ment was extremely crude. The system software was fused about the proper size of digital computing “words”, functional but provided only basic support. However, we and we had a hodgepodge of 8-bit, 9-bit, 16-bit, 18-bit, worked hard to make the best of our computers, which 32-bit, and 36-bit computers. This benefited us to some were considered powerful at the time. We used the PDP-10 degree, since 9-bit, 18-bit, and 36-bit precision provided computer, which occupied most of a room. We later were better computational precision for computer pixel com- able to use a prototype computer known as the “Super putations. In practice we were completely intermixed be- Foonly F1” developed in conjunction with recent Stanford tween power-of-two and multiple-of-9-bit word and byte graduates Phil Petit, Dave Poole, and Jack Holloway. This sizes. I attempted to use very little floating point, since it ambitious computer had 5000 ECL chips, and used twenty was quite slow at that time. I wrote numerous assembly 100-Amp 5V switching power supplies. Unfortunately, the code routines supporting accelerated 9-bit, 18-bit, 36-bit, Foonly was only marginally reliable, and would fail about and even 72-bit computations. I found that we needed once a day. Because of this, we needed to watch it around 72-bit integer precision (as opposed to 36-bit integer) for 968 G. Demos depth computations, in order to provide for accurate inter- film was quite grainy by today’s standards, but was rela- sections between nearly planar objects in large panoramic tively stable, allowing reasonably precise system calibra- environments. The shadow algorithm that Frank Crow de- tion. In order to provide the most image area for the extra veloped for me depended upon accurate shadow umbra generation required when scanning, we used a VistaVision plane intersection with objects when creating the shadow Acme-style Richardson movement. This movement pulled boundaries. 4-perforations on the film, so norm 4-perf 35 mm as well as 8-perf VistaVision could be used for both scanning and The digital film printer recording. It was typical to scan VistaVision, and record 4- perf. A custom camera housing was developed for us by Beginning with the FutureWorld project, we used the I.I.I. Doug Fries, supporting the large optical path behind the programmable film reader, or PFR, as a base system, to film gate for the scanning dichroic filters, condensing op- which we added color film scanning and recording capa- tics, and photomultipliers. bilities. We adapted available lenses and color filters for The PFR used a programmable raster that we repro- the scanner and recorder cameras. The PFR used a 5” duced for the DFP, allowing scans to rotate, keystone, and diameter cathode ray tube (CRT). resize. In this way, the optical resizing portion of scan- Using this system, we performed a mock digital film ning was handled directly in the scan. Further, anamorphic printer test, where we scanned images shot by LucasFilm scanning was directly applied when needed. For record- Industrial Light and Magic (ILM), and performed a sim- ing, the raster was usually applied in a relative uniform ple reproduction. Richard Edlund, then at ILM, supervised manner (usually square pixels), with the pixel samples be- our test. We discovered that we lost substantial resolution ing computed with the anamorphic squeeze taken into ac- and tonal detail in performing this copy. However, we be- count. Thus, there was no need for anamorphic optics, and lieved that we knew how to improve upon the mock printer we were thus able to handle widescreen movies directly, to create a digital film printer (the DFP) that would per- including mixing flat VistaVision elements with anamor- form well as a digital replacement for optical film print- phic computer simulated images to create an anamorphic ers. 35 mm widescreen result. At about this time, we added Mark Grossman and Tom For film recording, a 12-bit to 12-bit hardware lookup McMahon to join Karol Brandt in our hardware team, and table provided a known transformation to yield specific we added Bill Dungan, Craig Reynolds, and Larry Malone densities on print, and corresponding specific densities to our software team. on a mid-light calibrated print. In this way, the mapping The DFP design used a 7” diameter CRT, with a 1 of light, as computed by the renderer, as well as dens- mil spot, allowing us to achieve about 5000 pixel reso- ity from the scanner, could be processed to yield deter- lution. The spot itself was shaped somewhere between ministic calibrated results. The blending of texture maps a Gaussian and a cylindrical shape (uniform spot), which from the scanner, in density units, and pixels from the made it optimal for recording and scanning resolution. rendering algorithm, in light proportions, became a key Another key design element was an entirely new opti- system requirement. The use of film density units was not cal system, designed with Pacific Optical. In order to sufficient. capture the scanned light behind the film gate, the light The speed performance of the DFP was 600 ms per needed to be collimated to be nearly parallel, or telecen- pixel for scanning, and 200 ms per pixel for each color tric going through the film plane. The collimated light (red, green, and blue) for recording, although two or was split into red, green, and blue with red/cyan and three passes over the red was usually required. For our blue/green dichroic filters, with trimmer filters at each 3012 × 2304 resolution, a scan only took four seconds, photomultiplier. A unique feature of the DFP was a set and a frame recording only required about eight seconds. of fibre-optic light pipes surrounding the lens that were This was substantially faster than we could process the gathered to a fourth photomultiplier, which then provided data in our large general-purpose mainframe computers. an independent look at the CRT spot. This effectively Some of the optical mechanical units, or OMUs, were ca- gave a measurement of the light coming from the CRT pable of both scanning and recording, with a change of for each pixel spot. Using this reference path measure- camera head. The recorder head used a color filter wheel ment, the ratio of the light leaving the CRT to the light between the CRT and the film, whereas the scanner had gathered by each photomultiplier, then yielded a direct the color splitting and sensing system behind the film measurement of the density at each pixel on the film. This plane. color density measurement was processed using analog We therefore built the “DFP pipeline” consisting of logarithmic amplifiers, subtracting the reference path, and 3-D cross-color lookup tables and interpolators for each then analog-to-digital (AtoD) converted using an 11-bit of the three scanners, a 3-D cross-color table for color- AtoD. difference matte compositing (such as blue-screen), and The color negative film, which was ubiquitous at the a general purpose computational unit for general pro- time, was the newly introduced EK5247 from Kodak. This cessing. The general purpose unit, using microcode, My personal history in the early explorations of computer graphics 969

Fig. 5. Cover of Computer magazine, November 1978 (note recursive texture mapping), copyright 1978 Information Inter- national could emulate the PDP-10 instruction set that we used spline-based techniques for expressing and controlling in the Foonly computer. The DFP pipeline also con- motion during animation. Using these tools, Mal cre- tained a new 1000-line frame buffer with an Ikegami color ated a butterfly animation and a worm animation that CRT monitor. All of the pipeline, including the frame played realtime on the graphics screen. The realism buffer, was built using the Intel 4kx1 SRAM, which was of the animation movement gave us confidence that small but fast. We built a memory board having 72 such we could begin to explore three-dimensional character chips, and used the board at many places throughout the animation. pipeline. Another area of emphasis at this time was increasing Although we began the development of the DFP in memory and computational efficiency so that we could 1978, it several years to complete all of the portions of create high scene complexity. In 1978, we did a test for the system. Some portions, such as the frame buffer, be- LucasFilm of the Star War’s “X-Wing” ship. Art Durin- came operational in 1979. The film scanners and recorders ski encoded a 15 000 polygon wing on our two-cursor data became operational in 1981, being delayed primarily due table. Using symmetry with four instances of the wing and to long development times on the optical portions of the one fuselage, a 75 000 polygon X-Wing fighter was cre- system. ated. Mal developed 3-D motion algorithms for creating a “peal off” of the X-Wing fighters, combined with a sweep- Further explorations at I.I.I. ing three-dimensional camera move. The resulting motion, when rendered, created a dramatic space scene. For mo- One area that we thought would have substantial po- tion, we created lower detail versions of the X-Wing for tential was three-dimensional character animation. Mal long shots, and used the higher detail version for medium McMillan wrote realtime animation algorithms, and an and close shots. We showed these results to LucasFilm interpreted language for controlling them based upon in the summer of 1978. In the summer of 1979, we were “Castle”. I wrote math macros such as “slow in” and given permission to publicly show the X-Wing, and it “slow out” for the otherwise assembler-only III-15 18- became the cover of the August 1979 “Computer” mag- bit-word minicomputers. Mal developed 3-space path azine, concurrent with Siggraph in Chicago. This image, 970 G. Demos

Fig. 6. Database built from actress Susan Dey for the “Cindy” head model for the movie Looker. Art Durinski and Larry Malone are drawing polygonal reference lines on Susan Dey. Copyright 1980 Information International combined with early spacecraft motion tests, created sub- raster. The natural raster size for the Computer magazine stantial interest that year in the potential of high scene cover at 8” × 10” was 4500 × 5600 creating a 133-line complexity, high resolution, and photorealistic lighting. screen for Y, C, M, and K at 30 degree angles. Using the The X-Wing needed to have a dirty “used future” look “chromalin” color halftone printing preview systems, we consistent with the Star Wars movies. For this, we ap- made numerous experimental color separation tests un- plied a “color-per-vertex” technique using dirty brown til we were satisfied that we could create direct halftones tones combined with normal smooth shading (and copla- of any of our color images. As it turned out, we never nar polygons for edges). We called this database of dirt the needed this capability again, since our use of 4 × 5” “dirtabase”. At about this time, John Whitney Jr. coined Ektachrome and negative film at 6144 × 4096 resolution the term “Digital Scene Simulation” to describe our ren- was preferred in future publications as the color image dered synthetic image process. master. Because I.I.I. was working on color halftone recording In 1979, we proposed simulated space ships for the onto full-paper-size film, I wrote a conversion and filter- movie “Meteor”. However, at that time, the producers ing program to create the halftone dots directly from the thought of computer-generated images as being line im- My personal history in the early explorations of computer graphics 971

Fig. 7. I.I.I. Juggler “Adam Powers” with Ken Rosenthal in motion-capture suit, and vector version, copyright 1980 Information Interna- tional ages. We therefore produced 3-D line images representing of colors and the treatment of objects near the edges monitor footage for Meteor. However, it was clear to us of the frame. By experiment, we were able to pro- that photoreal space-ship simulation would be possible in duce amazing three-dimensional stereoscopic simulated the near future. scenes. I worked on 3-D stereoscopic tests in 1980 with Mur- ray Lerner, which in 1982 became dual-70 mm Magic Looker Journeys for the Kodak Pavilion for the opening of Disney’s Epcot theme-park in Florida. Eight minutes In 1980, Michael Crichton wrote the script for the movie of stereoscopic computer graphics were produced as Looker, based upon John Whitney Jr’s conviction that optical elements for that show (which utilized exten- photo-real simulated people would be practical in the sive 65 mm opticals). We found, through testing, that next ten to twenty years. The movie Looker was about 3-D stereoscopic production benefited from the inher- using simulated actors in commercials. Six minutes of ently infinite depth of field of computer simulation. effects were produced, including three-dimensional con- We also found that care was needed in the choice struction of actress Susan Dey’s head and body. We also 972 G. Demos

Fig. 8. Coke cans (note lettering detail on top of can, as well as reflections), copyright 1985 Digital Productions My personal history in the early explorations of computer graphics 973 created line-based and text-based scenes matched with based technology he called ASAS, the actor/scriptor an- the shaded images, using “hold takes” where a reference imation system. Craig used ASAS to create a broadly- mark 100-frames prior to the first frame allowed a sin- capable front end to control the Tranew renderer. All of the gle negative to be run through multiple cameras. This scenes produced by I.I.I. for Tron used ASAS. created first-generation images with a composite of mul- The producers of Tron decided to spread the work tiple image sources. We also produced the anamorphic between multiple facilities. The other facilities included widescreen “Looker hypnotizing gun effect”. Digital Effects (the “bit”), Abel (line-drawing opening), The face and body simulation used a four-mirror setup, and Magi (light cycles). III scenes were the major final which allowed a single front movie camera to see from sections of the movie including the MCP (master control above, from the left, and from the right. This allowed program) face, solar sailor on the sea of simulation. a single frame to provide all of the views necessary to cre- The DFP film recorder was used to make VistaVision ate a three-dimensional body and face surface, using Mal elements for the 65 mm film. McMillan’s updated perspective and distortion removal John Whitney Jr. and I left I.I.I. in the Spring of 1981, software. We used the same white makeup and fiduciary while Tron was just starting production. marks, as well as drawing lines for polygonal boundaries Tron was released summer 1982. directly onto the face and body.

The juggler Digital Productions 1981–1986 Having this multi-mirror setup allowed us to take moving At the time we began at I.I.I. there was little awareness images of Ken Rosenthal juggling and doing a back ﬂip. of the potential of computer simulated images. However, Using the data from these multi-views in each 35 mm ﬁlm by 1981, we had demonstrated enough of the potential frame of motion, Mal McMillan was able to create a data- that we found some industry investment interest in the po- smoothing and 3-D reconstruction system providing one tential. A graphics company of the time named Ramtec, of the earliest examples of 3-D simulated performance an- which made frame buffers, invested in John Whitney Jr. imation. and myself to allow us to form Digital Productions. We named the juggler Adam Powers and gave him a tuxedo and top hat for his juggling magician’s performance. The Cray computer

Considering the future I decided to use the Cray computer for rendering. The Cray used 64-bit floating point exclusively, had 16Mbytes John Whitney Jr. began commenting in magazine inter- main memory, and 64 Mbytes IO subsystem memory (two views that he believed that the degree of photorealism 35 mm frames, one still frame). As before, the disks were in both appearance and motion was becoming a matter like washing machines (12 of them at 600 Mbytes each, of creative choice, and not of technical limitation. John for 7 Gbytes total). The Cray’s special performance came talked about recreating performances of famous actors from 64-long “vectors”, of 64-bits each, organized to ef- from the past. I speculated that long and medium shot ficiently pipeline computations. Contrary to the common photorealism for actors, animals, and alien creatures were wisdom of the time, which said that three-dimension and becoming practical. I also speculated that crowds of such three-color computations were not suited to vector pro- simulated characters of creatures would become an im- cessing, I found that vectors could be used to gain optimal portant cinematic tool. John talked with Michael Crichton Cray performance on computer simulated rendering. about how interesting and effective it might be to simu- Since I was in a hurry to write our renderer and late dinosaurs in a movie. We were able to see much of the demonstrate some basic capability, we started with Hank potential of the computer simulation medium at this time. Christiansen’s “Movie.BYU” Watkins algorithm. We were quickly making images on our Ramtek frame buffers. Just Tron as quickly, my team replaced the heart of the Watkins algorithm with a Bouknight-style algorithm to support Beginning in 1979, John Whitney Jr. began working with transparency, and we added highlights and shadows. We Steven Lisberger and Bill Kroyer on story development knew that Frank Crow’s umbra-based shadows were com- and scene planning for “Tron”. putationally expensive with high scene complexity, so we Because of the Juggler, Disney could see that 3-D began working with low-detail shadow objects, that fit simulation was becoming viable as a new medium in and precisely into high detail space ships. of itself. Our initial emphasis was on high scene complexity Craig Reynolds, who I hired as a recent graudate and computational efficiency, since we were preparing for from MIT’s architecture/machine group, created the LISP- large scale simulated special effects production. 974 G. Demos

Larry Yaeger, Craig Upson, Fred Bradford, Jim Hardy, real spaceship scenes. Working with Gary Adelson and Steve Williams, Larry Luther, Ron Moskowski, Mitch Eddie Denault as producers, and Nick Castle as director, Wade, David Ruhoff, Andy Davidson, MaryAnn Morris, we began production on 250 scenes and 22 minutes of full- and Emily Nagel formed our initial software develop- screen special effects. ment team. We soon added Beth Leitner, Kevin Rafferty, We created complex objects, such as the 250,000 poly- Jack Green, Kathy Prestera, Michael Kory, and others to gon “Gunstar” ship, 50,000 polygons for the “deck fight- our “encoding” team, and Jim Rygiel, Brad Degraf, and ers”, and 150,000 polygons for the “mother ship”. others to our “technical director” team. We used the Evans We chose 2560 × 2048 resolution, since the Ramtek and Sutherland PS300 line-drawing perspective displays frame buffers were 1280 × 1024 (one quarter the area). and large dual-cursor Talos tables for encoding, and the Our computations on the Cray produced simulated pix- IMI graphics perspective display system for building mo- els with 64-bit floating-point accuracy. We truncated and tion. fixed them to 12-bits (for each of R, G, and B) for film recording (and for scanning), and dithered the values to Last Starfighter 8-bits (per color component) for display. In late 1982, after the completion of Tron and Magic We began working with Miguel Tejada-Flores and Ron Journeys at III, we were able to acquire the DFP OMU’s Cobb on production planning for a script by Jonathan for our film recording and scanning. Jim Rapley and Larry Betuel called “The Last Starfighter”. At this time, video Sinclair joined us from I.I.I., and they worked with David games were becoming popular. The Starfighter script fea- Ruhoff and Bob Allison to build an interface between the tured video games as a path to recruit pilots for real star- Cray high-speed channel (100 Mbytes/second) and the ships. Given that we were just starting Digital Produc- OMU interface. Phil Chen became our expert on ensur- tions, we felt that the initial delivery of crude video game ing that the DFP and the film Lab (MGM) remained in graphics would naturally evolve into the complex photo- perfect daily calibration. Mal McMillan joined our soft- My personal history in the early explorations of computer graphics 975 ware team and Art Durinski joined the encoding team. (At For reflection maps, we favored the use of the cube (six the same time, Larry Malone, Tom McMahon, and Craig faces), so that we could render six 90-degree width and Reynolds went from I.I.I. to form the Symbolics graph- height views from a given location, and then use the re- ics division, further refining the LISP-based approach to sulting six images as a reflection environment from that computer scene simulation.) location. Ron Cobb quickly learned the art of creating three- Unlike previous texture mapping, which required dimensional objects in the computer, developing intricate bicubic patches or quadric surfaces, I devised a simple space ship designs for the movie. Ron Cobb was also method of creating and interpolating parametric (u,v) fascinated by the motion choreography potential of the re- values to allow texture mapping over arbitrary topology altime line-preview in the IMI displays. Ron Cobb worked polygonal objects. This required much less computa- closely with the software and technical directing team of tion than bi-cubic patches, and allowed our renderer Brad Degraf, Larry Luther, and Ron Moskowski. to perform picture and transparency texture very effi- The Last Starfighter was released in the summer of ciently. 1984. This was only two years after the release of Tron, I added numerous light sources to the renderer with yet represented a large step forward in scene complexity programmable falloff to simulate point sources, line and detail. sources, and area sources. Further, we could place multiple lights within the scene with specific angular radiation The renderer patterns. Combined with shadows, these simulated lights allowed us to create rich scenes. With Larry Yaeger and myself leading the team, we In addition to normal light with color, rolloff function, built a high efficiency renderer having one million lines and angle, we also built specialized unreal lamps called of code, about 1/4 of which was the Cray assembler. “dark lamps”. These would pull light out of a scene, rather We used “overlays” in the linker to allow us to switch than adding light to it. This was useful for augmenting between various modules within the limited memory. the automatic shadow and lighting algorithms, in order to The Cray XMP had two processors sharing 16 Mbytes remove light from areas. Instead of luminance and lumi- of memory, so we needed to keep the renderer below nosity, we referred to these dark lights as having “gloomi- 8 Mbytes to be able to fit two into memory simultan- nance” and “gloominosity”. eously. Our overlays included the fluid-dynamics particle system, the compositor, the fractal renderer, and 2010 the main full-featured renderer. Walter Gish wrote the fractal renderer overlay that we used for moonscapes Upon the completion of Last Starfighter we were con- and planetscapes in Starfighter, together with the com- tracted to create the Jupiter planet scenes for 2010 (the se- positor and the main renderer for the ships in the fore- quel to 2001) for director Peter Hyams. 2010 was another ground. 70 mm production, and we worked with Richard Edlund, The main full-featured renderer was capable of ren- who had left ILM to form Boss Films. Richard provided dering entire complex scenes during a single scan down us with a 65 mm camera, which we outfitted on the DFP the image raster. The compositor, however, allowed lay- recorder OMU. The optics and filters for VistaVision and ered computing to optimize rendering time using multiple 4-perf were optimized for fast recording. However, the rendering layers, composited together. We stored a trans- 65 mm off-the-shelf El-Nikor lens and color filters lost parency channel with each layer for front-to-back com- several stops, requiring about two minutes per frame in- positing, but we preferred back-to-front compositing, to stead of about ten seconds. avoid needing to store the transparency channel. However, The key to the creation of the swirling clouds of the main all-in-one renderer supported transparency tex- Jupiter was the creation of a Navier-Stokes fluid dynam- ture maps in addition to picture texture, bump texture, ics particle system by Larry Yaeger and Craig Upson. and reflection maps. The transparency texture allowed us A flow field was encoded using wind directions taken to perform composites directly in the renderer, without from Voyager spacecraft images of Jupiter from JPL. Ron using the compositing overlay. For example, the explo- Gless made a painting with much higher detail and color sions in the Last Starfighter were all texture pictures with fidelity using the JPL images as a guide. We scanned transparency maps that defined the opacity/transparency a three-wide-vertical-panel image of the Jupiter “snake- of each explosion. We often had dozens of explosions skin” painting, which we scanned as three frames in Vis- during the battle scenes between the Gunstar ship and taVision on the DFP. Using a frame-blending overlap, the deck fighters. Compositing required clear definition of we created a single snakeskin at 6 k × 2.5 k resolution to depth precedence, whereas the use of transparency texture be wrapped as a cylinder onto the Jupiter sphere. Using and picture texture in a scene automatically processed the the fluid-flow-particle overlay, the texture image would depth of each explosion with respect to other ships and be “advected between” each frame by interpolating the explosions. force field vectors and applying the resulting force to five 976 G. Demos million colored particles. The particles were then filled Mary Ann Morris and Lynn Benner also developed into a pixel raster to create a texture map for the next a phonetically-drive lip-position system for driving dialog frame. from syllables. Their system used a multi-displacement- Since Jupiter was a polygonal object, the dent and col- vector capability that I built into the renderer, which lapse was handled by normal displacement vectors applied allowed blend proportions of any number of end pos- to the vertices. Since the texture was mapped onto the itions, or of displacement vectors (or both). We called polygons, the flowing fluid deformed with the sphere. We this “multi-level interpolation”. We felt that we were very produced about five minutes of “digital Jupiter” scenes in close to a facile system for controlling speech, as well as 65 mm for 2010. for controlling other facial features such as smiling, eye 2010 was released in December 1984, making it the expressions, eyebrow expressions, etc. second major motion picture effects project that we fin- John Whitney Jr. was particularly interested in this ished in 1984. character animation development, and he convinced Mick John Whitney Jr. and I were recognized by the Jagger to create a music video using simulated characters Academy of Motion Picture Arts and Sciences in March for his song “Hard Woman”. Within the lower budget per of 1985 with a scientific and engineering award for the frame of music video, John, Bill Kroyer, and Chris Baily “photo-realistic simulation of motion picture photography were able to create expressive animated tube figures in by means of computer generated images, 1984”. a simulated day-of-the-dead festive world. The primary in- put was film that we shot of a person performing moves for Commercials and logos the animated characters. The person, as usual, was covered with fiduciary witness marks. The frames of film were Our next emphasis was on the production of commer- projected onto the 2-cursor data table, and were projected cials and logos in order to keep our substantial staff and directly onto the line-drawing animation screen. computing capacity busy. We produced logos for major Even with these sources of animation motion, much of companies, such as the AT&T sphere, which ran for fif- the production used direct animation control from the dial teen years on network television. The AT&T Logo was boxes and parameters that could be selected for smooth produced working with designer Saul Bass. motion to the desired postures. We also produced high-budget commercials for the Pontiac Fiero, the Chevy Astro-Van and Mercedez Benz, Compositing with the DFP all featuring photo-realistic cars. The Astro-Van scene not only used simulated vans, but we also photographed Of additional interest was the use of green-screen com- people leaving a real van, and scanned the frames in Vis- positing using the DFP. I suggested that we use green taVision on the DFP for creating composites. instead of blue, since the blue record of film is very noisy. For STP oil treatment, we produced a visual simulation I worked with Michelle Feraud to create a clean matting of a transparent working car engine, including a camera fly algorithm for partially-covered pixels, which we used to through of the oil path within the engine. put moving images of Mick into the simulated scenes of the video. Michael Whitney supervised the green-screen Character animation photography of Mick Jagger. Bill Villarreal also began making hardware improvements to the OMUs to support Working with Mal, and following on Mal’s pioneering this work. 3-D animation work, Larry Luther, Ron Moskowski, Al- We did a test with Peter Anderson and Peter Ellenshaw lan Peach, Brad Degraf, Lynn Benner, MaryAnn Morris, of Disney of our digital blue-screen compositing for the Stephan Fangmaier, Mike Ullner, Yun Chen Sung, Shelly elephant movie “Baby”. They provided very helpful ad- Lake, Bill Kroyer, and Chris Baily developed new 3-D an- vice on how to improve our composites. As we refined imation tools, and helped to further improve the renderer. the compositing algorithms, we began to gradually include We began to see expressive character animation, which we the use of blue-screen and green-screen composites in our used in tests and commercials. I was my belief at the time productions. that 2-D animation would eventually be replaced by 3-D We also recreated Mal’s witness point tracking using character animation. in-frame-buffer-memory tracking, instead of tracking in One of our projects was to connect the IMI to a “Wal- the scanner. Mal’s new version of the smoothing and do” hand-operated sensor provided by Jim Henson. Brad relaxation/inversion algorithms performed flawlessly. We Degraf and Larry Luther tested the idea of directly con- also began to gradually include witness-point tracking trolling the 3-D line-drawn character in the computer by “match-moves” in our productions, to excellent effect. use of two such hand-operated Waldo controls. This was For still frames, in addition to 4 × 5and8× 10 Ek- quite facile, and we decided that we could develop an trachrome and negative, we began using 35 mm 8-perf effective 3-D animation system based solely upon such Kodachrome 25 at 5120 × 3072 resolution. Kodachrome controls. 25 has a wide color gamut and a deep black, although the My personal history in the early explorations of computer graphics 977 inter-layer interactions are large. Using our 3-D cross- I chose a Thinking Machine CM-2 with 16,384 pro- color lookup table, we were able to invert the inter-layer cessors in a SIMD (single instruction, multiple data) con- interactions so that we could use Kodachrome like any figuration. Having had success with the 64-long vector other color film, creating calibrated matches to our motion registers of the Cray XMP, I felt that the massively-parallel picture film negative. architecture would be effective. Of particular interest was our discovery that 11-bits are We found that some operations, such as the sorts re- needed for wide-range image media such as Kodachrome. quired for hidden surface and transparency computations Since our pixels were inherently noise-free using the Cray were only moderately efficient. However, lighting compu- 64-bit floating point, we could truncate accurately to 9, tations, and other mathematically-intensive computations 10, 11, and 12-bits. We found that contour bands, showing for every pixel were extremely efficient. both steps and hue changes, were visible below 11 bits on We also teamed up with Symbolics, to utilize software smooth surfaces of objects such as the Gunstar ship. The and hardware developed by our former team members contour bands were barely visible at 11-bits, and became from I.I.I. I very much enjoyed programming the Symbol- invisible at 12-bits. ics and Thinking Machine computers in LISP. In the years A decade later, I received the Motion Picture Academy since them, I have often reflected on the crude nature of “Scientific and Engineering Award” recognition in 1995 the “C” and “C++” languages in comparison to the rich along with Dan Cameron for digital scanning technology, LISP environment that I had available with the Symbolics. shared with David DiFrancesco and Gary Starkweather of As a pilot showcase project, Craig Reynolds of Sym- Lucasfilm/Pixar, and with Scott Squires of ILM. bolics and Philippe Bergeron, MaryAnn Morris and others I also received the 1996 Motion Picture Academy in our group worked together to create the animated short Technical Achievement Award recognition for “digital “Stanley and Stella”. Craig showcased his latest ASAS- compositing systems” along with David Ruhoff, Michelle like automated characters in the birds in this short, which Feraud, and Dan Cameron. he called “boids”, which was short for “bird-oids”. Our technical team also consisted of Karl Sims, Karol Brandt, Larry Sinclair, Michael Whitney, MaryAnn Mor- ris, Mitch Wade, Jim Hardy, Craig Burkhardt, Mal McMil- Whitney/Demos Productions 1986–1988 lan, Peter Walford, Yun Chen Sung, and David Ruhoff. We did election graphics in 1988 for CBS, using the After Digital Productions was taken over by Omnibus in new D1 digital video tape systems from Sony, and disk a hostile takeover in 1986, John Whitney Jr. and I formed arrays from Abekas. Although component digital video Whitney/Demos Productions. was a significant improvement over previous NTSC ana-

Fig. 10. Gary Demos with his 1280 × 720/72 fps camera on the cover of Television Broadcast magazine, January 2000 978 G. Demos log systems, I still found it difficult to work at the very the digital video systems, to attempt to work out archi- reduced resolution of standard video. tectures for high definition video systems for the future. John Whitney Jr. worked with Michael Bedard to de- I eventually isolated digital image compression as a key velop tests for three-dimensional animation of his “Sitting ingredient, and I have focused many of my efforts on al- Ducks”, although we were not able to get this produc- gorithms to retain high quality with efficient compression, tion launched at the time (but we came close to launching doing this work between 1988 and 2002 in DemoGraFX, a television series with CBS). and since 2002 on my own. My most recent work in- In 1988, John Whitney Jr redirected and reorganized volves many layers as well as extended dynamic range the company to become US Animation, and I left to form (well above reference white). For more information about DemoGraFX. my recent compression work see my 2004 SMPTE paper entitled “High quality, wide dynamic range, compression PDI and Pixar system”. I apologize to those who worked with me on these Although I had nothing to do with Pixar or Pacific Data various early computer image simulation efforts, who Images (PDI), both companies persevered through the fol- I forgot to mention here. I worked with so many tal- lowing years until computing power became sufficient for ented people over this time span that I probably forgot their digital animation requirements. PDI became Dream- to mention a few key names. However, everyone who works animation. Both companies followed the model that contributed to these efforts has my gratitude, for help- I felt was going to be successful, which was the produc- ing to create an exciting time where everything was tion of movies in their entirety (not just effects sequences new, and every week we saw something that was being for movies). I also felt that 3-D animation had substantial done for the first time. It is not possible to have more potential as a new story telling medium, and I believe that than one era of first times for computer simulated im- this is now broadly recognized. Disney, Rhythm and Hues, ages. I was very privileged to be able to help create, Sony Pictures ImageWorks, and others have also produced and to experience together with others who also created epic works in 3-D character animated movies. along side me, many new things, which revealed to all High-definition and compression of us a wondrous potential for a truly new medium. The computer became, in that era, a tool for creativity, in- In DemoGraFX, I took much of what I learned in high stead of a being exclusively a tool for engineering and resolution imaging, and the weaknesses that I found in mathematics.

GARY DEMOS was fascinated by the hardware and software challenges of computer graphics, beginning in the 1970’s. Gary pursued this interest through Information International (1974–1981), Digital Productions (1981–1986), and Whitney/Demos Productions (1986–1988). In 1988 Gary formed DemoGraFX, which became involved in technology research for advanced television systems and digital cin- ema, as well as consulting and contracting for computer companies and visual effects companies Gary began to specialize in research relative to high performance cameras and digital compression based upon the discrete cosine transform. In 2003 DemoGraFX was sold to Dolby Labs. Gary is presently working solo in the development of new wavelet-based and optimal-ﬁlter-based moving image compression technology for high bit-depth and high dynamic range.