Brigham Young University BYU ScholarsArchive
Theses and Dissertations
2020-08-26
Improvements in Optical Trap Displays
R. Wesley Rogers Brigham Young University
Follow this and additional works at: https://scholarsarchive.byu.edu/etd
Part of the Engineering Commons
BYU ScholarsArchive Citation Rogers, R. Wesley, "Improvements in Optical Trap Displays" (2020). Theses and Dissertations. 8686. https://scholarsarchive.byu.edu/etd/8686
This Thesis is brought to you for free and open access by BYU ScholarsArchive. It has been accepted for inclusion in Theses and Dissertations by an authorized administrator of BYU ScholarsArchive. For more information, please contact [email protected], [email protected]. Improvements in Optical Trap Displays
R. Wesley Rogers
A thesis submitted to the faculty of Brigham Young University in partial fulfillment of the requirements for the degree of
Master of Science
Daniel E. Smalley, Chair Stephen Shultz Ryan Camacho
Department of Electrical and Computer Engineering
Brigham Young University
Copyright © 2020 R. Wesley Rogers
All Rights Reserved ABSTRACT
Improvements in Optical Trap Displays
R. Wesley Rogers Department of Electrical and Computer Engineering, BYU Master of Science
This thesis improves on the design of the Optical Trap Display (OTD), presented in 2018 [1]. Contributions include: real time animation; single beam, multiparticle suspension, point primitive anisotropic scattering, and virtual image approximation. First, real time animation was demonstrated on the OTD for the first time in full color at up to 30Hz refresh. Second, multi- particle systems allow for scaling of the display by a multiplicative factor, potentially up to orders of magnitude greater than the first OTD. Third, anisotropic scattering of point primitives was shown for individual suspended particles and multiple simultaneously suspended particles. Fourth, virtual images have been previously considered impossible in volumetric displays but by using perspective projections we have shown in simulation and experiment for the first time that an effect similar to a virtual image can be created.
Keywords: optical trap, photophoretic trap, 3D display, volumetric display ACKNOWLEDGMENTS
I gratefully acknowledge the patient tutelage of Dr. Daniel Smalley throughout my graduate program. I appreciate the support of my family, as well as the consistent, loving support of my wife Madisyn, without whose encouragement and support I would have never finished a graduate degree.
I would also like to thank the generous support of the National Science Foundation (NSF) who have helped fund a portion of this work.
TABLE OF CONTENTS
LIST OF FIGURES ...... vii
CHAPTER 1: INTRODUCTION...... 1
1.1 Advancing Volumetric Displays ...... 1 1.2 Background ...... 2 1.2.1 Volumetric Displays Thus Far ...... 2
1.2.2 Previous Work on the Optical Trap Display (OTD) ...... 4
1.2.3 Visual Cues ...... 4
1.2.4 Photophoresis ...... 12
1.3 Overview of the Text ...... 12
CHAPTER 2: IMPROVING PHOTOPHORETIC TRAP VOLUMETRIC DISPLAYS ... 13
2.1 Abstract ...... 13 2.2 Introduction ...... 14 2.3 Trapping ...... 15 2.4 Scanning ...... 17 2.5 Scaling ...... 17 2.6 Robustness ...... 20 2.7 Safety ...... 21 2.8 Occlusion ...... 22 2.9 Applications ...... 27
iv 2.9.1. Surveillance ...... 28
2.9.2 Medicine ...... 28
2.9.3 Corporeal AI Agents ...... 28
2.10 Conclusion...... 30 2.11 References ...... 30
CHAPTER 3: SIMULATING VIRTUAL IMAGES IN OPTICAL TRAP DISPLAYS ...... 34
3.1 Abstract ...... 34 3.2 Introduction ...... 34 3.3 Theory ...... 37 3.3.1 Optical Trap Displays ...... 37
3.3.2 Perspective Projection ...... 37
3.4 Experiment ...... 38 3.5 Results ...... 39 3.6 Analysis...... 41 3.7 Conclusion ...... 43 3.8 References ...... 44
CHAPTER 4: CONCLUSION AND FUTURE WORK ...... 45
4.1 Conclusion: ...... 45 4.2 Future Work: ...... 46
Bibliography ...... 49
v References Included in Chapters 1 and 4: ...... 49 References Included in Chapter 2: ...... 50 References included in Chapter 3: ...... 53
Appendix A: Generating Perspective Projection in MATLAB...... 55
Appendix B: Blender Simulation of Virtual Image ...... 85
vi LIST OF FIGURES
Figure 1- 1. Reproduction of the figure in [3]. The three families of 3D displays are shown with a basic breakdown of strengths and weaknesses...... 3
Figure 1- 2. Reproduction of the figure in [16]. Table showing primary visual cues ranked by influence in three different distance zones (personal, action, and vista) relative the viewer...... 5
Figure 1- 3. The Blue block on the left is shown occluding the green block on the left. The right blue and green blocks show the effect of transparency on occlusion...... 7
Figure 1- 4 Relative density is shown here with a series of identical objects placed in a grid, all equally spaced but appearing more dense in the view as they recede into the distance...... 8
Figure 1-5 The green and red cubes pictured are the same size but placed at different distances to the camera. This shows relative size, the red cube looks smaller because it is farther away. This also shows height in the visual field, the red block appears at a different height in the view even though it is at the same height from the plane as the green cube...... 8
Figure 1- 6. Aerial perspective can be seen in this image. The mountains in the distance appear bluer than they are in reality and are blurred compared to objects closer to the viewer...... 9
Figure 1- 7. Vergence is part of the ocular motor visual cues and is shown here as a pair of eyes looking at objects various distances from viewer. The angle of the eye positions is shown with the white lines running from the eyes to objects...... 10
Figure 1- 8. Motion perspective shown using sequential pictures from left to right as the viewer moves past objects. The view of the objects changes as the relative position between the viewer and the objects changes...... 10
Figure 1- 9. Accommodation is the change of the focus of the eye. As focus changes from the green cube in the front to the yellow cube in the back, the objects not at the focus blur ...... 11
Figure 1- 10. Binocular disparity is the difference between the image each of your eyes, here we have an image on the left and right taken from spatially separated positions...... 11
Figure 2-11 (a) Photo of single-color, single particle, vector, video rate image 1 cm tall, circa 2016. (b) Photo of three-color, single-particle, line raster not video rate image, 1 cm tall, circa 2018. (c) Conceptual image of three-color, multiple-particle, volume raster image, video rate 10 cm tall...... 15
Figure 2-12.Multi-particle scaling. (a) Single-particle display with complex pathing and simple illumination. (b) Multiple-particle display with a simple path and complex illumination. (c) Lab result showing a single-particle system (image courtesy of Joel Rasmussen). (d) Lab result showing multiple particles in a linear array from a single laser source. (e) Concept showing a planar array of suspended particles rastering a volume image, video rate refresh, large scanning volume...... 19
vii Figure 2-13. Occlusion. (a) Anisotropy makes it possible to eclipse or occlude objects. (b) Setup for observing scatter from multiple angles simultaneously. (c) Isotropic scatter. (d) Anisotropic scatter. (e) Particle exhibiting isotropic scatter; this particle has relatively uniform scatter over 4pi steradians. (f) Particle exhibiting anisotropic scatter. (g) Two particles, one above the other, demonstrating alternating brightness moving from front to back [27]...... 25
Figure 2-14.Interactive applications. (a) Composite of photos from first OTD animation (see Visualization 2). (b) Satellite surveillance concept. (c) Guided catheterization concept. (d) Corporeal AI agent “holonurse” concept...... 27
Figure 3-15 OTD Display and Simulated Virtual Images Concept a. Optical Trap Display (OTD) b. 3D Vector, long exposure, image drawn by OTD c. Flat, rastered, long-exposure image drawn by OTD (content from Big Buck Bunny) d. Simulated virtual image concept with flat moving/rotating plane at the back of a draw volume filled with real images/objects such as 3D OTD images or 3D printed objects...... 36
Figure 3-16 Experiment Setup a. An OTD display projects a flat moon image at the back of a draw volume that contains a 3D printed house. The image is updated at persistence of vision frame rates (12 frames per second) using the perspective projection base based on expected camera location. b. A close-up of the house position, moon position, and perceived moon position in 3D space...... 39
Figure 3- 17 Experiment results a-c. Parallax for particle at z=0 (in front of the house) d-f. Simulation result, parallax for particle at z=0, with perspective projection. g-i. Experiment result, parallax for particle at z=0, with perspective projection The parallax is consistent with a particle at z=8 mm (behind the house). For full video see supplemental Document and Visualization 2...... 40
Figure B- 18. Left. Zoomed in image from God’s eye view of Chapter 3 visualization 4. Looking at the moon in the center of this image we can see that the moon is not occluded by the window frame properly. Right. Same frame from the camera used to generate the perspective projection...... 86
Figure B- 19. Showing the material node structure for the plane displaying the “virtual” images...... 87
viii
CHAPTER 1: INTRODUCTION
1.1 Advancing Volumetric Displays
Although significant advancements and work have been accomplished in the area of 3D volumetric displays, the perfect volumetric display has not been demonstrated to date. The goal of future volumetric displays should be fulfilling all visual cues in a large variety of circumstances (indoors, outdoors, multi-viewer, etc.). Many impressive display technologies can fulfill a portion of these requirements, but the future of the 3D volumetric display is to control all visual cues in every circumstance. Blundell [1] points out a number of shortcomings in the volumetric display community to date such as: “major limitations on image space visibility (e.g. a single ‘window’ onto image space), limited and non-scalable image space dimensions, variations in voxel attributes, low fill factor, low brightness of voxels, and high density of materials used for image space formation (static volume).” [1] These limitations in the displays are problematic to achieving the gold standard of volumetric displays as they limit the displays ability to fulfill one or more of the visual cues that the human brain interprets to understand the world around us.
This thesis improves on the design of the Optical Trap Display (OTD), presented by Dr.
Daniel Smalley [2] bringing it closer to the perfect volumetric display. Improvements include: real time animation, multiple suspended particles using a single trapping beam, anisotropic scattering of point primitives, and approximating virtual images. First, real time animation allows for the display of video content on the OTD for the first time. Second, multi-particle
1 systems allow for scaling of the display by a multiplicative factor, potentially up to orders of magnitude greater than the first OTD. This allows significantly larger displays to be created which will improve key performance metrics such as accuracy recreating visual cues. Third, anisotropic scattering of point primitives shows that based on particle morphology and dynamics the scattering can change in meaningful ways. Fourth, virtual images have been previously considered impossible in volumetric displays, but by using perspective projections we have shown for the first time that an effect similar to a virtual image can be created. Introducing virtual images allows for an increase in display size without an equivalent increase in display hardware size thus increasing potential OTD applications.
1.2 Background
The following background sections serve to provide the reader with the minimum information necessary to understand the following chapters. Discussion includes: previous work in volumetric displays, a brief introduction to visual cues, and a brief introduction to photophoresis. Each of these topics are extensive and warrant a full investigation for the interested reader. An in-depth study can be found for each topic in the associated references.
1.2.1 Volumetric Displays Thus Far
The Volumetric display is one type of 3D display. The volumetric display, or point display family, is defined as “the display’s scatterers or emitters are co-located with the actual image points”. This family of displays comes with advantages and disadvantages compared to the ray and wave display families, see figure 1. A more in-depth discussion can be found from
Smalley [3].
2 Figure 1- 1. Reproduction of the figure in [3]. The three families of 3D displays are shown with a basic breakdown of strengths and weaknesses.
Within the family of volumetric displays there are different approaches to creating the
volumetric image points. Each of these approaches deserves an in-depth discussion that can be
found in the associated references. Approaches include: Static Volume (semi-transparent
medium [4, 5], induced micro disturbance [6, 7], Swept Volume (rotating [8] or translating [9]),
and Free space (induced plasma [10], holovect [11], and of course the OTD [2], holodust [12],
fog display [13, 14]). Each approach offers different advantages, but none have achieved the
3 ultimate gold standard of performance in volumetric displays which is total control over all visual cues in all circumstances.
1.2.2 Previous Work on the Optical Trap Display (OTD)
The previous work on OTD technology not included in this thesis is primarily the work from my group under the direction of Dr. Daniel Smalley published in 2018 [2]. This work laid the foundation for the photophoretic trap volumetric display and introduced the concept of an optical trap display (OTD). Capabilities of the display at the time of the publication included display of long exposure rastered images, display of long exposure complex vectored images, persistence of vision (POV) simple images, both long exposure POV images in 2D and 3D. The
OTD display at that time was capable of the pictorial cues (occlusion only from a predetermined fixed viewing point because the images generated by the OTD are not self-occluding and no live image updating (animation) was demonstrated at that time), accommodation and vergence for real image points, binocular stereopsis for real image points, and motion parallax for real image points. We note here the limitation to real image points only as this will be the focus of the developments discussed in chapter 3.
As discussed in the previous section, the “finish line” for development in volumetric displays is total control over all visual cues in all circumstances. The next section will outline a basic understanding of the major visual cues to help explain this goal.
1.2.3 Visual Cues
The ability to understand the future of volumetric displays relies on an understanding of the visual cues the volumetric seek to fulfill. Below I will briefly describe the major visual cues with some examples that may be familiar to most readers. The visual cues are the true measure of
4 how effective a display is at creating the perception of 3D. Controlling all of the visual cues completely would produce a visual result that the human viewer would not be able to differentiate from reality. Each visual cue can have different challenges associated with a particular display technology and we will address some of these challenges for OTD displays in chapter 2. Each of these visual cues affect the human visual perception of the world, but depending on distance from the viewer, some visual cues have a greater impact on visual perception than others (see in figure 1).
Figure 1- 2. Reproduction of the figure in [16]. Table showing primary visual cues ranked by influence in three different distance zones (personal, action, and vista) relative the viewer.
We see in Figure 2 that the visual cues can be complicated by the consideration of distance from the viewer to the visual information. Cutting and Vishton break down the general trends of influence into three distinct distance zones from the viewer. Personal space is generally considered to be up to 2 meters from the viewer. Action space is considered to start at the edge of the personal space and extend to 30 meters. Vista space is considered to start at the edge of
5 action space and extend to infinity. Understanding the level of influence in different spatial zones in relation to virtual images will be discussed further in chapter 3.
In the following chapters we will often refer to the pictorial visual cues as a set for simplicity. The pictorial cues include occlusion, relative size, relative density, height in the visual field, and aerial perspective. These cues derive their collective name from the ability to control each cue using only two-dimensional display technologies such as a painting on a canvas or a conventional 2D television. The following paragraphs briefly explain each of the visual cues.
Occlusion or interposition can be described as the covering of one point by another. This cue is the most influential in all three of the spatial zones. Gneiting says, “As one object blocks the view of another, the mind automatically recognizes that the now invisible object is behind the visible one. To prove this, look out your window. The window is closer than the scene that can be seen through it. It follows that the frame of the window blocks, or occludes, your view of the outside world. If this cue is removed, the result is a scene where every object is always visible, creating a world that appears to be filled with semi-transparent, ghost objects.“ [15] Given that this cue is the most influential in all three spatial zones, it is of particular importance to control because a conflict between occlusion and a different visual cue will create a noticeable conflict of cues leading to possible viewer discomfort or fatigue. Chapter 2 section 7 further discusses occlusion in OTD displays.
6 Figure 1- 3. The Blue block on the left is shown occluding the green block on the left. The right blue and green blocks show the effect of transparency on occlusion.
Relative size refers to the size of objects dependent on position. If you were to take two identical items, teapots for example, and place them at different positions in the visual field of an observer, the more distant object would appear smaller than the identical item at a closer distance to the viewer.
Height in the visual field or height in the picture plane refers to the angle of elevation off the optical axis of the observer. This is easiest to picture when looking out over a long distance in relation to the horizon. Objects farther away on the ground plane will be closer to the horizon line than objects on the same plane closer to the viewer.
7 Figure 1-5 The green and red cubes pictured are the same size but placed at different distances to the camera. This shows relative size, the red cube looks smaller because it is farther away. This also shows height in the visual field, the red block appears at a different height in the view even though it is at the same height from the plane as the green cube. Relative density refers to the projected retinal density of a cluster of objects or textures,
whose placement is stochastically regular, as they recede into the distance [16].
Figure 1- 4 Relative density is shown here with a series of identical objects placed in a grid, all equally spaced but appearing more dense in the view as they recede into the distance.
8 Aerial perspective or atmospheric perspective all refer to the color and clarity shift associated with distance from the viewer. This effect is present at close distances but so minor that it is difficult to detect. At large distances the effect is sufficient to see a blue shift in color due to Rayleigh scattering and gaussian blurring from diffraction of light through the air.
Figure 1- 6. Aerial perspective can be seen in this image. The mountains in the distance appear bluer than they are in reality and are blurred compared to objects closer to the viewer.
Motion perspective or motion parallax is the change in appearance and position in the visual field of an object based on movement of the viewer relative to the object. For example, this occurs any time you walk around an object for example. The approach in chapter 3 will discuss this cue in greater detail.
9 Figure 1- 8. Motion perspective shown using sequential pictures from left to right as the viewer moves past objects. The view of the objects changes as the relative position between the viewer and the objects changes. Convergence or vergence refers to the ocular motion necessary to keep an object in the center of view of each eye. The most extreme example of this is when a person points each eye at extreme angles toward the nose, commonly referred to as cross eyed. Cross eyed is the extreme example of what our body naturally does as we focus on objects closer to us. As the object comes closer the eye will naturally point inward to maintain visual.
Figure 1- 7. Vergence is part of the ocular motor visual cues and is shown here as a pair of eyes looking at objects various distances from viewer. The angle of the eye positions is shown with the white lines running from the eyes to objects. Accommodation refers to the change in the optical power of the human eye when focusing at different depth planes. If you have ever placed something too close to your eye and strained to focus on the item, that straining feeling is the ciliary muscle of your eyes attempting
10 to flex the lens to focus on the item. The eyes do this for many distances, but it is most evident in
closer regions due to the strain.
Figure 1- 9. Accommodation is the change of the focus of the eye. As focus changes from the green cube in the front to the yellow cube in the back, the objects not at the focus blur
Binocular disparity, stereopsis, and diplopia refer to the difference in optical information
received by each eye based on the natural spatial separation of the human eyes. This can be
easily seen by placing your hand touching your nose and closing one eye at a time. One eye will
see the front of your hand while the other eye can see the back of your hand. When you have
both eyes open the brain does a good job of naturally stitching these different views into a single
information stream.
Figure 1- 10. Binocular disparity is the difference between the image each of your eyes, here we have an image on the left and right taken from spatially separated positions.
11 1.2.4 Photophoresis
The photophoretic trap is based on the process of photophoresis, whereby a particle can be suspended in gas or liquid when illuminated by a sufficiently strong light source. The photophoretic force can be defined as,