Continuously Adjustable Pulfrich Spectacles Ken Jacobsα, Binghamton University, Binghamton NY Ron Karpfβ, ADDIS Incorporated, Corvallis OR

ABSTRACT

A number of Pulfrich 3-D movies and TV shows have been produced, but the standard implementation has inherent drawbacks. The movie and TV industries have correctly concluded that the standard Pulfrich 3-D implementation is not a useful 3-D technique. Continuously Adjustable Pulfrich Spectacles (CAPS) is a new implementation of the Pulfrich effect that allows any scene containing movement in a standard 2-D movie, which are most scenes, to be optionally viewed in 3-D using inexpensive viewing specs. Recent scientific results in the fields of human perception, optoelectronics, video compression and video format conversion are translated into a new implementation of Pulfrich 3- D. CAPS uses these results to continuously adjust to the movie so that the viewing spectacles always conform to the optical density that optimizes the Pulfrich stereoscopic illusion. CAPS instantly provides 3-D immersion to any moving scene in any 2-D movie. Without the glasses, the movie will appear as a normal 2-D image. CAPS work on any viewing device, and with any distribution medium. CAPS is appropriate for viewing Internet streamed movies in 3-D. Keywords: Pulfrich effect, Translational research, 3Deeps Action Specs, Continuously Adjustable Pulfrich Spectacles

1. INTRODUCTION 3-D cinema is now of great interest, but production and display costs are high. Continuously Adjustable Pulfrich Spectacles (CAPS), also called 3Deeps Action Specs or just 3Deeps, seizes upon scientific advances in understanding human perception, in optoelectronics, digital video compression, and digital format conversion resulting in a new approach to 3-D movies. To any standard 2-D film, the 3-D provided by 3Deeps approaches the quality of specially produced 3-D movies, but without the added cost. 3Deeps is 3-D produced by the spectacles alone. A movie viewed through 3Deeps spectacles will continually go into and out of 3-D. Those scenes lacking discernable movement will appear planar, or in 2-D. But for those scenes containing movement, 3Deeps spectacles convert motion parallax into stereopsis. Properly made 3Deeps Action Specs sit unobtrusively before a viewer’s eyes with lenses that are clear, only appropriately darkening for those scenes 3Deeps can convert to a realistic sense of 3-Dimensional depth. An appropriate analogy may be found with a live football broadcast. During the huddle when there is little movement, 3Deeps will show planar or 2-D. As the team breaks and enters play formation, 3Deeps will show in realistic depth. With the snap of the ball, 3Deeps converts the action on the screen into vivid 3-D. 3Deeps’ technology is best suited to generating realistic 3D for action scenes. This “limitation” is offset by the following benefits: • For every scene with “action,” 3Deeps can provide stereopsis comparable to that of dual-image 3-D movies. This is achieved without any addition to production or exhibition costs, as there is no requirement for special 3-D cameras, 3- D digital formats, and 3-D projectors, etc. In other words, 3Deeps provides 3-D without any cost burden to the existing video infrastructure used to film or view current 2-D movies. • With 3Deeps, the broadcast medium is irrelevant. Movies sourced from DVDs, digital video projectors, streaming Internet video, cable, etc can all be viewed in 3-D. Further, there is no restriction as to the device on which 3D images can be viewed. 3Deeps works with movie screens, notebooks, TVs, computer monitors, cell phones, etc. • As a single-image system, 3Deeps simultaneously can be viewed in both 2-D and 3-D. Two viewers can sit side-by- side watching a movie. The viewer with 3Deeps spectacles sees 3-D; the viewer without 3Deeps spectacles sees 2-D. Since 3Deeps always have one clear lens, it blocks substantially less light than other 3-D systems.

α [email protected] . Ken’s email address is a reference to his nervous system performances using double analysis-projector set- up for deriving 3-D from standard 2-D film. β [email protected]

Stereoscopic Displays and Applications XXII, edited by Andrew J. Woods, Nicolas S. Holliman, Neil A. Dodgson, Proceedings of SPIE-IS&T Electronic Imaging, SPIE Vol. 7863, 78630O © 2011 SPIE-IS&T · CCC code: 0277-786X/11/$18 · doi: 10.1117/12.870750 SPIE-IS&T/ Vol. 7863 78630O-1

• Since 3Deeps does not transmit dual-images, it utilizes the full resolution capabilities of the 2D video transmission. • Finally, 3Deeps employs a totally different mechanism by which dual-images are presented to the viewer’s eye-brain that eliminates the causes of eye fatigue and discomfort present in other 3-D systems. The structure of this paper is as follows. The paper first provides a background of the standard implementation of Pulfrich 3-D movies and TV. Then the paper briefly reviews the science explaining the Pulfrich stereoscopic effect, and proposes a new implementation of Pulfrich spectacles called 3Deeps. 3Deeps displays every scene in a 2-D movie that contains any motion as 3-D. 3Deeps achieves this effect by continuously adjusting the lenses so that they always conform to the optical density that optimizes the Pulfrich stereoscopic illusion. The paper then briefly reviews the proposed optoelectronic lenses that are used to construct the lenses of 3Deeps Action Specs. Optimal 3Deeps synchronization of motion in a movie with 3Deeps Action Specs requires calculation of a “Characteristic Motion Vector” (CMV). The CMV characterizes the direction and speed of lateral motion in a scene. The paper continues by describing how the CMV is preferably calculated directly from the change information between frames already used to compress and decompress digital movies. Another means to calculate the CMV is from the change information between frames that is already used for up-conversion and de-interlacing of digital movies. Finally, the paper proposes an implementation of 3Deeps so that 2-D movies streamed over the Internet can be viewed in 3-D. A prototype 3Deeps system will be shown at the Conference Demo session, with a demonstration of 2-D movies streaming WiFi from YouTube and showing in 3-D on an Apple iPad device.

2. BACKGROUND In the 1982 seminal text “Foundations of the Stereoscopic Cinema” in talking about independent filmmakers that work in the stereoscopic medium, the author Lenny Lipton remarks “Ken Jacobs has . . . exploited the Pulfrich phenomenon by equipping audience members with neutral-density monocles1.” The Pulfrich phenomenon was first identified about 90 years ago, and his since been used for 3-D movies, but only sparingly. As will be seen, CAPS or 3Deeps is a new implementation of the Pulfrich phenomenon that greatly expands its usefulness as a technique for 3-D movies. 3-D motion picture stereoscopy requires that each eye see a separate, slightly offset view of a scene. One method to achieve this relies upon the Pulfrich effect, triggered on viewing a scene containing motion through glasses with one lens neutral-dark and the other lens clear. With Pulfrich 3-D the darker lens is to the side that foreground motion is directed. Pulfrich 3-D has been used for relatively few commercial projects. The standard implementation of Pulfrich Spectaclesχ imposes such severe limitations on the film or video that the industry has correctly concluded that it is not a useful technique for general 3-D video. The problem of viewing a movie through a Pulfrich monocle is clear. When a movie is viewed through a Pulfrich monocle, as screen motion changes direction and speed, screen parallax is continually switching between positive and negative with changing depth magnitude. Since the Pulfrich monocle has a fixed lens, the viewer must continually move the dark neutral lens before the correct eye to maintain correct screen parallax orientation. Specially made Pulfrich movies control for these problems by severely limiting screen motion. Pulfrich movies tend to have constant lateral speed, only in a single direction, with static lighting conditions. Not surprisingly, then Pulfrich 3-D was used for live telecasts of the Rose Bowl parade. 3Deeps addresses these problems by using electronically controllable lenses that constantly adjust the orientation and neutral dark shade so the correct parallax is always presented to the viewer. Pulfrich 3-D has been used in production of such movies as the 1971 feature length movie “I, Monster” starring Christopher Lee; for the 1997 second season finale of the network TV sitcom “Third Rock From The Sun”; for episodes of the Discovery Channel’s “Shark Week” in 2000; for the 1993 Doctor Who charity special “Dimensions in Time”; an episode of “Power Rangers”; segments of the animated programs “The Bots Master” and “Space Strikers”; Howard Stern’s “Howard Stern's Butt Bongo Fiesta” in 1992; the Nintendo Entertainment system videogame “Org-3D”; and “Lost Dimension in 3-D” for the Super Nintendo gaming system. The feature length 3-D movie “Future Fighters”, scheduled for release in 2011, uses Pulfrich 3-D. Since it is the 3Deeps spectacles alone that account for the 3-D effect, Pulfrich movies display every scene containing lateral motion in 3-D, and movies viewed through 3Deeps do not require any special preparation or formatting. Movies

χ A simple means to construct Pulfrich Spectacles is by removing one lens from neutral gray sunglasses.

SPIE-IS&T/ Vol. 7863 78630O-2

more than 100 years old and from the silent era can be seen in 3-D. “I've shown two 1903 movies off DVD - Skyscrapers of New York (boat goes down river shooting at docks and buildings in background going by) and Georgetown Loop (shooting out of train) and had the audience view with Pulfrich glasses . . . and 3D effects are amazing. "2 2.1 The Pulfrich Effectδ Of the many clues the mind uses to infer depth, stereopsis or simultaneous images from two eyes set 2 ½ inches apart, on average, are the most important. Every system other than 3Deeps is a dual-image system. Dual-image systems record two images with separate camera lenses. 3Deeps needs only the single image most commonly used to record motion to create a second accompanying image with which to triangulate depth. It is natural to assume that the eyes continuously send visual stimulation to the brain – but that is not the case. While the more than 100 million visual receptors in each eye are ‘continuously’ stimulated, the retinas trigger at ‘discrete’ time intervals sending visual information to the brain. The triggering is faster with brighter retinal illumination and slower for darker retinal illumination. 3Deeps relies on the Pulfrich phenomenon3 in which a darkened neutral density filter is introduced before one eye – delaying its visual message. This creates parallel perspectives for the mind to construct depth from even the slightest movement of objects, but particularly from the motion most common and natural to cinematography - the lateral shift in a sequence of frames. A person with normal vision will see in effect, past and present; the actual image on-screen, and the image that had been seen on-screen a moment before. These somewhat contradictory images will be resolved in the mind as if they were two simultaneous perspectives, by the same method that the mind normally provides the appearance of depth from two 2-D perspectives. 2.2 The Pulfrich Effect And Negative Parallax The geometry of 3Deeps is identical to every other method of 3-D. Figure 1 4 ε, “Pulfrich Negative Parallax” shows how 3-D is produced using Pulfrich effect.

Figure 1: Pulfrich Negative Parallax

δ Carl Pulfrich for whom the stereo phenomenon is named described the effect in 1922. He was a German astronomer and a famous designer of stereoscopic instruments, including rangefinders for naval gunnery, stereocomparators, and the “blink microscope” for detecting movements of celestial bodies. Pulfrich was blind in one eye and never able to see the 3-D effect he so aptly described. ε Figures 1, 4 and 5 use Figure 3.3 from Lenny Lipton’s reference as a base, on which is overlaid explanatory text and graphics. Lenny Lipton’s Figure 3.3 explains negative parallax. We have added text and graphics to explain the Pulfrich phenomenon.

SPIE-IS&T/ Vol. 7863 78630O-3

When a focused object in the foreground has a lateral motion vector from left-to-right, and the darker neutral density lens is placed before the right eye, the object appears closer to the audience or with negative screen parallax. That is, the left eye-brain will see an instant image, and the right eye-brain will see a lagging image. When these two images are fused by the eye-brain, the result is stereopsis, with the focused image appearing in front of the plane of the screen. This is the means by which 3Deeps converts 2-D planar movies into 3-D stereoscopic movies. Pulfrich converts screen parallax into stereopsis, preserving all of a movie’s visual depth cues, including screen parallax. In Lenny Lipton’s text, he remarked “. . . the only desirable asymmetries in a stereoscopic system of photography and projection are the asymmetries of horizontal parallax.5” Rather than use film techniques to provide a dual frame image, 3Deeps uses a controlled illumination asymmetry to provoke the eye-brain to form a second asymmetric image. By creating this illumination asymmetry, Pulfrich provokes motion parallax into stereopsis. By controlling the difference (delta) in the illumination asymmetry, 3Deeps controls the extent of the negative parallax. In the following Sections we explain how control over the difference (delta) in illumination asymmetry controls the extent of negative parallax. 2.3 Retinal Reaction Time The eyes’ retinal triggering mechanism has been well studied and documented in the scientific field of visual perception. Some of the earliest and most important work is published in peer-reviewed journals such as the American Journal of Psychology and the American Journal of Optometry with research in part supported by grants from the National Institutes of Health 6 7 8 9.

Figure 2: Retinal Reaction Time Figure 2 shows the retinal reaction time under various lighting conditions. As an example, with light approximating that of a “Grey sky at noon” the retinal reaction time is about 100 msec, while for light approximating a “Night sky with a full moon”, the retinal reaction time is about 300 msec. By differentially controlling the retinal reaction time of each eye, the illusion of depth can be created from sequential frames of a video. If one eye views a movie through a clear lens in a room with ambient light that has the brightness of a “Grey sky at noon”’ and the other eye views the movie through a neutral dark lens simulating light of a “Night sky with a full moon” the difference in Retinal Reaction time between the two eyes is 200 milliseconds. For this example, on a standard HiDef 1080p 100Hz LCD TV, the brain will perceive images that are offset by 20 frames of video.

SPIE-IS&T/ Vol. 7863 78630O-4

2.4 Controlling the Pulfrich Effect Using the Retinal Reaction Time curve, a response surface can be calculated showing optimal darkening of the motion directed lens. 3Deeps uses an objective function to set the optical density (OD) of the shaded eye that is based on the retinal illumination of the clear or unshaded eye, and the motion vector of a focal object (in screen traverse time). For instance, a reasonable objective function is to set the OD of the shaded 3Deeps lens by selecting an OD for the motion-directed eye so the eye-brain will perceive the lagging image at a distance from the instant image equal to the average interocular. For such an objective function, the dual 3Deeps images from corresponding left- and right-image points subtends the same angle to the viewer’s eyes as with normal stereopsis. The viewer perceives the 2-D with normal depth. In general, the 3Deeps objective function should be chosen to achieve orthoscopy in viewing – a depth- enhanced viewing that appears exactly as it would appear in real-life at the time of photography. This is achievable since unlike dual-image stereoscopic systems that project 2 images, 3Deeps is customized for every viewer. This dynamic quality of 3Deeps needs to be clearly understood. 3Deeps continuously adjusts for every frame of the movie, and to the lighting conditions. For two viewers sitting side-by-side, but with slightly different lighting conditions, 3Deeps lenses will adjust the lenses differently for each viewer. While the CMV (Characteristic Motion Vector) in the movie is identical for all viewers, lighting conditions are not. 3Deeps is implemented by transmitting the CMV to the lenses, allowing the spectacles to determine illumination, and then 3Deeps customizes the difference (delta) in lens OD to achieve orthoscopy for each viewer.

Figure 3: 3Deeps Lens Settings Figure 3 provides the optimal retinal illumination through the dark lens (z-axis in units of Luminance) as a function of retinal illumination through the clear lens (x-axis in units of Luminance translated into their approximate lighting condition), and speed of motion of a focal object between frames of the motion picture (y-axis in units of seconds for a

SPIE-IS&T/ Vol. 7863 78630O-5

focal edge to completely traverse the viewing screen). Standard Pulfrich viewing spectacles (with a fixed clear lens and a fixed dark lens) can only achieve optimal Pulfrich effect within a limited area labeled ‘A’. A 3Deeps prototypeφ system that we have developed uses electronically controllable LCD lenses, and currently can achieve optimal Pulfrich effect in the restricted limited area labeled ‘B’. Outside those areas, the depth effect is confusing or non-existent. The response surface of Figure 3 suggests a new approach to 3-D. Continuously adjustable 3Deeps specs use lenses that continuously synchronize to each frame of the video with real-time signalization so that 3Deeps Action Specs always take the optimum state, retarding the leading frame, to maximize the Pulfrich illusion on the entire response surface bounded by ‘C’ in Figure 3. For such advanced 3Deeps, all scenes containing motion in any video would appear in distinct 3-D. This includes scenes with slow to fast motion in either direction, and over a range of lighting conditions from bright light to moderately dark. So long as there is screen movement, 3Deeps can provide the two offset images appropriate for viewing a scene in convincing depth, but identifying the two optimal images is a complex computer-intensive image-processing problem. Of several possible theoretical solutions, the requisite image processing has already been extensively implemented as solutions to unrelated problems in the fields of digital video compression and digital video format conversion. By leveraging those solutions, every instance of screen motion can synchronize precisely to 3Deeps Action Specs in real- time to appear with pronounced 3-dimensionality. 2.5 Calculating Screen Parallax – Two Examples Two examples are provided showing how 3Deeps Action Specs calculates screen parallax on a continuous frame-by- frame basis. The first example uses the 3Deeps prototype, while the second clip would use a next generation 3Deeps to fully display screen parallax. Screen parallax using 3Deeps is dependent on the illumination asymmetry between the lenses of the spectacles. Faster screen motion requires less illumination asymmetry; slower screen motion requires a larger illumination asymmetry. The 3Deeps prototype is severely limited and can only show a very constrained illumination asymmetry. For this reason the movie clips chosen for the demo are scenes with fast lateral motion.

Figure 4: Calculating Screen Parallax for NC Fair Clip

φ The prototype system uses shutter glass 3-D spectacles. Shutter glasses use rapidly switchable LCD glass that, even when clear, block more than 65% of the available light.

SPIE-IS&T/ Vol. 7863 78630O-6

Figure 4 shows how screen parallax is calculated for the demo “North Carolina Fair Clipψ” enabling this 2-D clip to be viewed in 3-D when the viewer is wearing 3Deeps Action Specs. The “North Carolina Fair Clip” has fast foreground lateral motion with a central focal object traversing the screen from right-to-left in about 2 seconds. The clip was filmed a good distance from the fair ride – both the base and the top of the ride are visible. For the central focal object to move 2 ½ inches in object space requires that retinal illumination of the motion-directed eye be set so the lagging image is perceived 1 to 2 frames from the instant image. At 30 fps that is a difference of no more than 66 msec. The neutral gray lenses of the limited 3Deeps prototype block substantial light with the lighter (clear) lens blocking about 75% of the light. Through this lens, the retina of the eye has a luminance of about –1.5 on the Retinal Reaction Time curve of Figure 2 corresponding to a retinal delay of about 250 msec. Since the direction of screen motion in this film clip is right-to- left, this ‘clear’ lens is configured to control retinal illumination in front of the right-eye. We set the darker lens in front of the motion directed left-eye to have a retinal delay that is 66 msec slower than the right-eye, or a total retinal delay of 316 msec (about –2.5). With this illumination asymmetry, the eye-brain perceives a lagging image 2 frames before the instant image. This produces negative screen parallax and stereopsis when viewing the 2-D “North Carolina Fair Clip”.

Figure 5: Calculating Screen Parallax – Clip from the Avatar Figure 5 shows how screen parallax is calculated for a clip from the 2-D version of the Action movie “Avatarτ”. 3Deeps enables this 2-D clip to be viewed in 3-D when the viewer is wearing 3Deeps Action Specs, and produces substantially the same dual images as in the 3-D movie. This clip has slower foreground lateral motion than the previous clip, requiring a greater illumination asymmetry. For this clip, a central focal object traverses the screen from left-to-right in about 5 seconds with respect to the background. For the central focal object to move 2 ½ inches in object space requires that retinal illumination of the motion-directed eye be set so the lagging image is perceived about 5 frames from the instant image. At 30 fps that is a difference of 165 msec. This illumination difference is unobtainable with our present prototype. However, it is achievable with better optoelectronic lenses. Next generation 3Deeps Action Specs will have electronically controllable lenses that can achieve any neutral gray optical density from 10% blockage (clear lens) up to 90% blockage (dark lens.) Using these next generation 3Deeps Action Specs with this clip, the clear lens could have a retinal illumination of about +2 on the Retinal Reaction Time curve of Figure 2, corresponding to a retinal delay of about 100 msec. Since the direction of screen motion in this film

ψ The North Carolina Fair Clip was produced by Todd E. Gaul (http://www.photophile.com) as a demonstration of the Pulfrich Effect. The video is posted at YouTube.com - http://www.youtube.com/watch?v=1mnWI_u_zBg. τ Avatar is a 2009 epic science fiction film written and directed by James Cameron

SPIE-IS&T/ Vol. 7863 78630O-7

clip is left-to-right, this ‘clear’ lens is configured to control retinal illumination in front of the left-eye. We set the darker lens in front of the motion directed right-eye to have a retinal delay that is 165 msec slower than the left-eye, or a total retinal delay of 265 msec. With this illumination asymmetry, the eye-brain perceives a lagging image 5 frames before the instant image. This will produce negative screen parallax and stereopsis when viewing this clip from the movie “Avatar”. Viewing of this clip through the current limited 3Deeps Action Specs does not produce sufficient screen parallax for othoscopy. Next generation 3Deeps Action Specs are being developed that have a much wider range of lens ODs allowing a corresponding increased difference (delta) between the eyes retinal delay. With such next generation 3Deeps, this clip will show with lifelike depth when viewed through 3Deeps.

3. 3DEEPS SPECTACLES USING OPTOELECTRONIC LENSES Optoelectronics is an enabling technology for 3Deeps Action Specs. Electronics materials respond reliably and predictably to the application of a voltage potential, and materials that change color and light transmittance in response to a voltage potential have long been known. The scientific field of optoelectronics has studied and catalogued materials that respond to the application of an electronic potential by changing its optical density. Electrochromic material10 γ can be used to fabricate the 3Deeps, though other optoelectronic materials could similarly be used. Electrochromics devices have been used for self-darkening rear-view mirrors, windows to reduce buildings’ energy costs, controllable windows in newer model Boeing planes, and motorcycle helmets. Infrared electrochromics have been tested for USAF jet pilot helmets. Electrochromic sunglassesη have electronically controlled lenses, with the right and left lenses assuming the same darkened state. 3Deeps spectacles can be built using such optoelectronic materials with a microprocessor controlling variable darkening for each individual lens. For every other 3-D spectacle systems it is sufficient to interpose lenses or a barrier between the eyes and the viewing screen. However, 3Deeps depends on controlling retinal illumination, that includes not only light energy emitting from the picture, but also other light sources such as ambient light. 3Deeps spectacle design must account for all sources of retinal illumination. There is a useful analogy between the development of shutter-glasses and 3Deeps. The Eclipse method of mechanical shutters (1920s) “. . .used motorized shutters in spectacle frames rotating in synchronization with the projector shutter11.” In the mid-1970s the mechanical shutter method for 3-D stereoscopy was updated using recent developments in electronics. Then the ‘ghosting’ problem was solved resulting in the CrystalEyes shutter glasses (The StereoGraphics Corporation) enabling a new and better electronic commercially viable shuttering approach to 3-D movies. The shuttering technology – still in use today – has been researched, studied, improved and commercialized for more than 30 years. Analogously, 3Deeps Action Specs can use optoelectronic lenses to implement a new and better electronic version of the Pulfrich ‘monocle’.

4. SYNCHRONIZATION OF 3DEEPS ACTION SPECS WITH THE MOVIE Video compression, de-interlacing, and up-conversion are enabling technologies for 3Deeps. Synchronization of 3Deeps Spectacles with the movie requires calculation of a “Characteristic Motion Vector” (CMV). The CMV characterizes the direction and speed of lateral motion in every frame of a movie. It is calculated from frame-by-frame change information, and is stored as a single motion vector that summarizes the major direction and speed of motion in each frame of video. It is a single vector – having both direction of lateral movement and scalar speed.

γ Electrochromism is the phenomenon displayed by some chemicals of reversibly changing color when an electric potential is applied. Electrochromism has a history dating back to the nineteenth century and there are thousands of chemical systems that have already been identified as electrochromic. Other materials that reversibly change color when an electric potential is applied include suspended particle devices, polymer dispersed liquid crystal devices, and SmartGlass http://smartglass.com . η Several companies have already developed and marketed electrochromic sunglasses. In 1993 Nikon produced Selspeed, the world's first electrochromic sunglasses, with lenses that changed in response to brightness. Reportedly, in the film “Mission Impossible” Tom Cruise wore one of only three pairs of Nikon Selspeed electrochromic sunglasses still available in Europe. Other companies have also created electrochromic sunglasses, but in each case have failed commercially.

SPIE-IS&T/ Vol. 7863 78630O-8

The direction of the CMV is then used to set the orientation of the 3Deeps lenses – right-lens dark and left-lens clear when the direction of motion is from left-to-right; left-lens dark and right-lens clear when the direction of motion is from right-to-left; and, right-lens and left-lens clear when there is no motion in the scene. The scalar portion of the vector is used to optimally set the optical density of the darkened lens to optimize the Pulfrich stereoscopic effect and achieve orthoscopy. For the last 60 years video engineers have worked on the problems of de-interlacing, up-conversion, and digital video compression/decompression. The intensive image processing algorithms that are key to the solution of these digital cinema problems also process change information between frames, and store and use motion vectors. These motion vectors are necessary and sufficient to also calculate the 3Deeps CMV. 3Deeps reuses these well-researched comprehensive image-processing solutions to digital cinema problems that are already implemented by the industry to calculate the CMV and in real-time synchronize the 3Deeps Action Specs with any 2-D movie.

Figure 6: Calculation of the 3Deeps Characteristic Motion Vector (CMV) Figure 6 summarizes the impressive body of image processing solutions that enable 3Deeps on-the-fly, real-time synchronization between any movie and 3Deeps spectacles. Uncompressed, a 2-hour TV Broadcast quality video is about 194 Gbytes, or 42 DVD12. Without the computationally intensive image processing developed by the industry over the last 50-years and used for up-conversion, de-interlacing, and video compression, digital movies would not be possible. 3Deeps reuses this image processing to calculate the 3Deeps Characteristic Motion Vector that summarizes lateral motion between frames of a movie. The CMV is then used to optimize the optical densities of the 3Deeps spectacles. Depending upon the nature of the digital file, different algorithms may be employed to calculate the CMV. For instance, for an MPEG-4 digital file that contains good coding of ‘sprites’, it may be sufficient to calculate the CMV using the motion vectors associated with such sprites. In general however, three methods for developing on-the-fly, real- time synchronization between a digital movie and 3Deeps Action Specs can be used. 4.1 Software-based 3Deeps Synchronization 3Deeps synchronization between a video and 3Deeps can be developed directly from information already calculated and contained in a compressed digital video file. With burgeoning demand for motion video, bandwidth is a limited and precious commodity. To minimize bandwidth, advances in the field of digital video compression are commonly utilized. Since the background for a scene changes little between frames and that same background will redundantly appear in temporally related video frames, digital video compression relies on identifying, quantifying and recording ‘temporal redundancies’ or those areas of a frame of digitized video that are identical from frame-to-frame.

SPIE-IS&T/ Vol. 7863 78630O-9

Consider the widely used MPEG13 digital compression techniques, a standard for recording and distributing digital video. In MPEG compression, a frame of video is organized as 8x8-pixel macroblocks. MPEG records not just background macro-blocks that do not move, but the motion of macro-blocks as they move between frames, with as much as 64 pixel horizontal and vertical offset or displacement between frames. These motion vectors are encoded in the compressed video bit-stream as (x,y) or (horizontal, vertical) pairs of integers taking values from –65 to +64 indicating movement of the associated macro-block left or right, or up and down. These calculated motion vectors can in turn be used by 3Deeps to then calculate the speed and direction of motion that characterize each frame of video in a motion picture. That is, the 3Deeps CMV is just a summary statistic calculated from the motion vectors. This explains why no expensive video processing is required to produce the 3Deeps synchronization signals – only inexpensive calculations from the already calculated motion vectors in the MPEG video stream. Method 1: The 3Deeps synchronization signals are calculated when a video is compressed. The synchronization are then stored in the compressed video, and used for real-time control of the continuously adjustable 3Deeps spectacles. Method 2: 3Deeps synchronization signals are calculated from the motion vectors in the compressed video file in real- time when the video is decompressed by the video CODEC, and used for real-time control of the continuously adjustable 3Deeps spectacles. 4.2 Hardware-based 3Deeps Synchronization Real-time synchronization between a video and 3Deeps can be developed directly from information calculated by a Digital Video Format Conversion Chip. To maximize picture quality, motion vectors must be calculated in real-time by the video format conversion chips that are in every Digital TV and Digital Projector. These motion vectors are sufficient to calculate the 3Deeps synchronization signals. While the calculation of these motion vectors is a complex and expensive process, use of these already-calculated motion vectors to synchronize 3Deeps to the motion video in real-time is simple, straightforward, and inexpensive. Two of the problems addressed by the field of digital video format conversion chips are de-interlacing and up- conversion. Consider the following simplified example. The movie “The Wizard Of Oz” was filmed at 24 frames per second, but when shown on Cable TV, it will be viewed by customers with older interlaced TV using the NTSC format, as well as with newer HiDef digital LCD TV that may have refresh rates of 100 Hz. Where do all those extra video frames for a 100Hz non-interlaced TV come from, without distorting and introducing motion artifacts? In our example, when the “Wizard Of Oz” is broadcast for general viewing, the TV may be required to generate 3 new frames of video for each single frame of broadcast video. Creating these extra frames in real-time is achieved using a digital technique known as ‘Motion Compensation’. Motion compensation is implemented by tracking motion between frames for small blocks of pixels, requiring complex real-time image processing to construct the new frames of videoι. The motion vectors calculated to implement “Motion Compensation” can in turn be used by 3Deeps to calculate the speed and direction of motion that characterize each frame of video in a motion picture. That is, the 3Deeps CMV is just a summary statistic calculated from the “Motion Compensation” motion vectors. 3Deeps will calculate the 3Deeps synchronization control using the motion vectors that are already calculated by the video format conversion chip. No expensive image processing are required to produce the 3Deeps synchronization signals – only inexpensive calculations from the motion vectors that are now calculated and stored by the video format conversion chips. Method 3: 3Deeps synchronization signals are calculated in real-time when a video is reformatted for viewing from calculations already performed by the video formation conversion chip, and then used for real-time control the continuously adjustable 3Deeps spectacles.

ι For most of us, the most powerful computer in our home is the video format conversion chip in their digital TV – not their PC or cell-phone. An LCD digital TV that sells retail for $1000 may contain a single video format conversion chip that costs $100 wholesale. It has been estimated that the Phillips Melzonic Motion compensation chip first produced more than 10 years ago, had a processing power about 100 times as great as that of the then current Intel Pentium chip.

SPIE-IS&T/ Vol. 7863 78630O-10

5. CONTROL SIGNALIZATION FOR 3DEEPS SPECTACLES Once the 3Deeps synchronization signals are calculated in real-time they are then used to control 3Deeps Action Specs. Continuous control is provided on a frame-by-frame basis so the 3Deeps Action Specs are perfectly synchronized to the movie to always take the optical density state that maximizes the Pulfrich effect. Even if 3Deeps Action Specs are fabricated from a material that changes state slowly, they are optimal in the sense that they are ‘continuously adjusting’ to the optimal state. Communication of the synchronization signals can be via any of the well-known means. All that is required is broadcast of a low baud (~ 200 bytes/second) 3Deeps sync signals. Radio frequency (RF), infrared, bluetooth, or other wireless means may be used. Alternatively, a directly wired protocol may be implemented. Other means of communication may be used such as steganographic techniques to embed inaudible sync sounds in the audio, or invisible sync markings in the video stream of the movie.

6. AN APPLICATION - 3DEEPS FOR INTERNET STREAMING VIDEOS A 3Deeps system for Internet streaming video can be built by processing compressed digital video files. When a digital video is uploaded to a video streaming service, those services immediately compress the video using extensive image- processing software on powerful high-end computers. The image-processing algorithms compress the video according to a minimax criterion to minimize file size subject to video picture quality constraints. In one of the many possible implementations of 3Deeps, the optimal synchronization signals to control. 3Deeps Action Specs can be calculated during the video decompression on the viewer’s device from the temporal redundancy motion information coded into the compressed video. This is the previously described Method 2, and is shown in more detail in Figure 7. Figure 7 shows a movie first being uploaded to the video distribution service. The movie may have been recorded on any type of video camera from a low-resolution cell-phone camera used for casual filming to a super high-resolution digital camera used to film high budget movies.

Figure 7: 3Deeps Action Specs For Internet Streaming Video After the video is uploaded over the Internet to a video distribution service, it is then processed and stored for distribution in a compressed video format such as FLV or MPEG. Powerful image-processing compression algorithms are used during the format Processing, and motion vectors representing the “video redundancies” are stored in the

SPIE-IS&T/ Vol. 7863 78630O-11

compressed video format version of the file in Video Storage. When the video is accessed for viewing, the Video Server streams the video to the viewing device. As the media player or applet decompresses the file for playback, the video CODEC calculates the 3Deeps Synchronization on a frame-by-frame bases. Continuous control signalization of the 3Deeps Action Specs can be via any means of wired or wireless means using. With this straightforward implementation, all scenes of a movie containing motion streamed over the Internet can be viewed in 3-D while wearing 3Deeps Action Specs. If the movies are viewed without spectacles, then they appear in 2- D. No changes to the movie, means of distribution, video formats, or viewing monitors are required.

7. ADVANTAGES AND LIMITATIONS OF 3DEEPS ACTION SPECS 7.1 Advantages 3Deeps is a single-image system, not burdened by the additional costs of dual-image that require the special 3-D cameras, 3-D digital formats, and 3-D viewing venues characteristic of dual-image systems. Advantages include: • 3Deeps works within the existing video infrastructure for 2-D movies. • 3Deeps is consistent with simultaneous 3-D and 2-D viewing. • Every scene in a movie with motion can be viewed in 3-D. 3Deeps achieves stereoscopy comparable to other methods of 3-D stereoscopy whenever lateral motion is present. • No additional costs to produce a movie for viewing with 3Deeps. • Only slightly increased costs for viewing via 3Deeps (for low-cost purchase of the Specs). • 3Deeps works over any medium – DVD, broadcast, digital video projector, streaming video. • 3Deeps works on any device – movie screen, TV, computer monitor, cell phone. • 3Deeps utilizes the full resolution qualities of video transmission • Since no muscular effort is required to fuse dual images for stereoscopy, traditional eye fatigue problems are not an issue. Since 3Deeps is a single-image system, none of the projection, transmission and format issues for dual-image stereoscopy are relevant. Focusing on dual images is no longer subject to Accommodation/Convergence, or Divergence of the eye muscles. Since 3Deeps employs a totally different mechanism for presenting dual-image to the eye-brain, most of the discomfort issues associated with dual-image stereoscopic systems are avoided It has been reported that even “Stereoblind people, who can't fuse random-dot stereograms (i.e. MagicEye pictures), still perceive the apparent motion of the Pulfrich pendulum. The authors conclude that stereoblind people retain some residual binocular mechanism for depth perception.”14 7.2 Limitations In 3Deeps, filter darkening is to the side towards which foreground objects move. Static images will still show as 2D. This is actually fortunate, as the fixed image of a face or a conversation against an unmoving background, recorded by an unmoving camera, is a situation where 3-D could be distracting. There is information that is best presented flat, as with talk, charts, and exposition scenes. It is when an action scene begins that 3-D’s vividness is appreciated, as it transforms flat-screen activity into a depth that one can feel part of. Further, there’s a refreshed awareness that comes from moving from 2-D to 3-D. It has been noted that “Even small differences between left and right image illumination strain the eye.”15. This has not been found to be a problem with the prototype 3Deeps. The problem may be alleviated since 3Deeps will go in and out of 3-D space depending on the measure of lateral motion in a scene. When no motion is present, both lenses of the 3Deeps spectacles have the same OD, and the viewer is no longer subjected to an illumination asymmetry.

SPIE-IS&T/ Vol. 7863 78630O-12

8. 3DEEPS ACTION SPECS PROTOTYPE A α-level 3Deeps Action Spec prototype system has been developed using shutter glasses with a 3Deeps controller, and can be viewed at the Conference Demo session. The prototype system is depicted in Figure 8. The prototype runs on an iPadϕ device on clips stored at YouTube.comκ streaming over WiFi. The 3Deeps spectacles/controller attaches to the audio-out earphone port. Alternatively, the prototype will also operate on an attendee’s device (e.g. iPhone, iPad, Google Android, laptop computer, etc) that has an audio-out microphone port and WiFi access. A test library of video clips have been specially prepared for 3Deeps viewing by replacing one channel of the stereo audio with 3Deeps sync control. Viewers wearing 3Deeps spectacles view the video in 3-D, while viewers without spectacles view the video in 2-D.

Figure 8: 3Deeps Action Specs Prototype System In the Demo, we use tones in one of the two stereo audio channels to transmit preset 3Deeps synchronization, and the second of the two stereo audio channels for the audio of the movie. This is only demonstrative for the prototype and conference Demo. Commercial 3Deeps systems would calculate 3Deeps synchronization and control 3Deeps spectacles in real-time. Real-time synchronization of 3Deeps with the movie has not yet been implemented. However, it should be readily apparent how this is done. The motion picture and television industries have been working on the problems of up- compression, de-interlacing, digital compression, digital image processing, digital stabilization and auto-focus for the last 6 decades. That body of work is highly complex and calculates, stores and uses ‘motion vectors’. The ingenious image processing solutions are already integrated into the operation of numerous consumer devices. As we have previously explained, the ‘motion vectors’ resulting from any of those solutions can be simply 'reused' to optimally sync motion to 3Deeps Action Specs. Additional work is on-going to implement real-time synchronization and will be available in a future β-level prototype.

9. CONCLUSIONS Inexpensive Continuously Adjustable Pulfrich Spectacles (CAPS or 3Deeps Actions Specs) can be fabricated with fast switching neutral density optoelectronic lenses, wireless communications using lightweight RF (Radio Frequency) signals transmitting no more than 200 feet, and low-power low-end control microprocessors. The lens optical densities that maximize the 3-D Pulfrich stereoscopic effect can be calculated on a frame-by-frame basis. The optimal 3Deeps synchronization can be pre-calculated and stored in the digital video, or performed in real-time. 3Deeps provides 3-D immersion to any viewing device and with any mode of digital video distribution including bandwidth constrained streaming Internet video. 3Deeps allows simultaneous 2-D viewing along with spectacles-enabled 3-D viewing.

10. ACKNOWLEDGEMENTS We would like to thank Mark Seiler, and Ray Zone for their assistance.

ϕ IPad - Apple iPad, Apple Inc, Apple Inc., Cupertino, CA 95014 with 9.7-inch (diagonal) LED-backlit display at 1024-by-768-pixel resolution at 132 pixels per inch (ppi) with 1GHz Apple A4 Processor. κ Youtube.com, San Bruno, CA 94066.

SPIE-IS&T/ Vol. 7863 78630O-13

11. REFERENCES [1] Lipton, L., [Foundations of the Stereoscopic Cinema], Van Nostrand Reinhold Company Inc., 52 (1982) [2] Ray Zone, 3D Center for Art and Photography, Email, June 11, 2010 [3] http://en.wikipedia.org/wiki/Pulfrich_effect [4] ibid 1, Pg 94. [5] Ibid 1, Pg 190 [6] Lit, A., ”The Magnitude Of The Pulfrich Stereophenomenon as A Function Of Binocular Differences Of Intensity At Various Levels Of Illuminations”, American Journal of Psychology, 62, 159-181 (1949). [7] Lit, A., “Magnitude of the Pulfrich Stereophenomenon as a Function of Target Thickness”, Journal of the Optical Society of America, Vol 50, No 4,pp 321-327 (April 1960). [8] Lit, A., Hyman, A., “The Magnitude Of The Pulfrich stereophenomenon as a function of distance of observation”, American J of Optom., 1-17, Monograph No 122 (November 1951). [9] Lit, A., “The Magnitude Of The Pulfrich Stereophenomenon as a Function of Target Velocity”, Journal of Experimental Psychology, Vol 59, No 3, 165-175 (1960). [10] Monk, P., Mortimer, R., Rosseinsky, D., [Electrochromism and Electrochromic Devices], Cambridge University Press, 2007. [11] ibid. 1, Pg 69. [12] Iian Richardson, [Video Codec Design], John Wiley and Sons, 2002 [13] http://www.mpeg.org/ MPEG – Motion Picture Experts Group [14] http://newfoundlandnews.blogspot.com/2006/10/pulfrichs-pendulum.html [15] ibid. 1, Pg 44.

SPIE-IS&T/ Vol. 7863 78630O-14