Automatic Generation of a Realistic Looking Single Image Stereogram Using Stereo Vision

Home , Autostereogram, Stereoscopy

2016 23rd International Conference on Pattern Recognition (ICPR) Cancún Center, Cancún, México, December 4-8, 2016 Automatic Generation of A Realistic Looking Single Image Stereogram using Stereo Vision

Minh Nguyen School of Engineering, Computer and Mathematical Sciences Auckland University of Technology Auckland, New Zealand Email: [email protected]

Abstract—Stereogram or autostereogram, frequently known as Magic Eye picture, is a two-dimensional (2D) image. Hidden inside each stereogram image is a floating object which appears in three-dimensional. We present a fully featured, web-based, online system that produces on-the-fly stereograms (or autostereograms) from a pair of side-by-side images. The system consists of three steps: (i) two side-by-side near stereo images are processed to reconstruct a disparity map, (ii) multiple regions of left or right images are automatically selected for building stereogram, (iii) base distance is estimated for the best human perception, and (iv) reconstruction of stereogram. Stereogram is built with a variety of transparency options, thus, multiple regions of images can be presented in the final output and display of transparent 3D layers in stereogram is possible. Moreover, the system is portable and can be used in smartphones with an Internet connection. Image Fig. 1. A random-dot stereogram that encodes a 3D profile of a shark [8]. pairs can be acquired directly on the phone and stereogram is reconstructed and displayed on the phone screen. These novel features place the system ahead of current alternatives and allows A. Motivation a wide variety of users to experience stereo reconstruction and stereogram generation in a quick and easy manner. Additionally, Beside anaglyphs, stereograms are also frequently seen the system could serve as a platform for online based visual on the Internet related to the keywords “3D images or perception studies. “stereoscopic images. An example of stereograms can be seen in Figure 1, which looks similar to a wallpaper where I.INTRODUCTION random coloured patterns [9] are repeated. Its name originally Autostereograms or single-image stereograms were invented comes from “Stereogram, which is referred to an image by Sir David Brewster around 1840 as an improvement over pair which could be viewed by using a stereoscope. In fact, the kaleidoscope, the apparatus which allowed representing this picture encodes a 3D scene of a shark, which can be 3D shapes [1]. Modern days’ stereograms for computers were “seen with proper viewing technique (no filter glasses are first introduced by Tyler et al. in 1990 [2]. Tyler managed to required). The appearance of this stereogram is established demonstrate the ability to generate depth illusions by using by encoding three-dimensional shark image into distortions image patterns consisting of a single image random dot stere- of symmetrical patterns which are decoded by misleading the ogram. The proposed 2D image contains an unlimited range eye convergences. To achieve the 3D illusion, a viewer can in depth and can create 3D in-depth profiles both above and focus on a point in front of or behind the image surface. The behind the image plane. Stereograms quickly gained interest repeating patterns will thus be moved sideways, and the brain from the public in the 90’s through the patented application attempts to fuse them together by focusing on certain pairs “Magic Eye” [3] as the only accessible 3D images in the at different parallax angles. The image patterns, thus, appear early days of computing. Applications of stereograms range to float at different depths correspondingly. The performance from public-driven media support (3D books or videos [4]) to for in-depth perception was claimed to be similar or even watermarking [5], vision therapy [6], and cristallography [7] to more advantageous than attempting to view side-by-side stereo name a few. Many 3D design tools offer to generate stereogram pairs [7]. stills or videos from depth map and texture map of a scene There have been attempts to use stereograms of random- or to use a 3D object as a start up point. There is, however, dot images [10] for efficient watermarking [5] Thanks to its no available publication or software which allows a user to distinctive appearance and low cost to produce, stereograms create a stereogram still or video from a pair of stereo images are relatively popular for entertainment for many years. On acquired by a user. the current Internet, there exists a significant number of 978-1-5090-0357-0/15/$31.00 c 2015 IEEE similar online applications: [11], [12], [13], [14]. Most of them

978-1-5090-4846-5/16/$31.00 ©2016 IEEE 3815 enable users to quickly and interactively build stereograms by choosing from a number of given depth-maps and patterns. Despite a great number of similar products, to the best of our knowledge, none of them utilise stereo vision technique to generate stereograms, i.e. by using the depth and pattern information recovered from a pair of left and right stereo images. Accordingly there is little if no communication on the use of auto-stereogram to embed both depth and texture information for a given stereo-pair.

B. Benefits of the Proposed System We propose a scheme to generate stereogram quickly from a pair of real-life stereo images which are captured by user cameras or cell phones. Compared to the aforementioned similar solutions, this proposed application presents some Fig. 2. Our proposed stereogram creation system contains four image advantages below: processing steps. • Instead of choosing from a selection of given depth models, the system creates stereogram from a real-life II.BACKGROUNDSAND RELATED WORKS pair of photos acquired by any conventional single sensor camera. A. History and literature reviews • Repeated colour texture are selected from the source images, hence, the look of stereogram is related to the There are two ways to diverge the eye viewing directions: captured scene. wall-eyed and cross-eyed, which also correspond to two ways • Repeated patterns are automatically chosen from var- to generate a stereogram. Wall-eyed viewing method requires ious areas of the picture to obtain stereograms with the eyes to adopt a relatively parallel angle, we can naturally diverse looks (as seen in Fig. 5(a) and Fig. 5(b)). A few achieve it by looking into the far distance. Cross-eyed viewing, stereograms can be overlaid on each other to make the on the other hand, requires a relatively convergent angle, such combined picture contain parts chosen regions (as seen as looking at an object very close to the eyes. Some may find in Fig. 5(c)). one easier than the other technique; indeed, more stereograms • Our application also lets user re-generate depth from any are constructed to be wall-eye-viewed. With the wrong viewing stereogram. Those who have difficulty seeing stereogram technique, the same 3D scene can still be seen but with the from the first place can reveal what is hidden. depth reversed. For many years, stereograms have been popularly used for C. Overall Process to Create Stereogram artistic expression and in children’s books (often called Magic Overall, the proposed process contains four steps as demon- Pictures or Magic Eyes). Additionally, Katrusiak et al. have strated in diagram 2: used stereograms as a portable methodology to display 3D crystal structures on 2D image [7]. The method facilitates 1) Stereo Matching is used to extract depth information a clear presentation of crystal elements, which may help to from side-by-side images with a known disparity range. solve three-dimensional geometrical and structural problems This process outputs a disparity map that characterises in demonstration and education. the observed 3D surface. Different forms of stereoscopy were discussed above to 2) Texture selection picks multiple regions in the reference show that stereoscopy is far more extensive than is commonly image (left or right) to be repeated within the stere- realised. There were some associations created to promote the ogram. study and collection of stereoscopic products, for instance, 3) Base-distance selection is applied to adjust the display the Stereoscopic Society1 founded in 1893, and the National of stereogram so that it can be easily viewed by different Stereoscopic Association2 founded in 1974. Also, there were people (degree of eye convergent) and on different many books and magazines dedicated solely to 3D stereo- platforms (printed papers, computer screens, cell-phone scopic imaging, e.g. the Stereo World magazine3, which is screens). widely published six times per year. Many may be found in an 4) Stereogram creation is about the reconstruction from the enormous collection belonging to Harold A. Layer [15], who above parameters. spent twenty years collecting stereoscopic books published The paper is continued with another four subsequent sec- from the year 1853 to 1986. tions. Sec. II revises the background of stereogram and other related works. Sec. III discusses on our system design and 1www.stereoscopicsociety.org.uk implementation. It also presents a number of results and 2www.stereoworld.org Sec. IV concludes the paper. 3www.stereoworld.org/stereoWorld.php

3816 ∗ ∗ B. Three dimension perception from Stereogram In this equation dP = pPL − pPR => dP ∼ dP ; therefore: d ∼ −d∗ . This means there is a direct relationship between Single image stereograms are two-dimensional images that P P stereo vision disparity d∗ and stereogram d . We can use the have the unique property that when viewed in a specific P P result of stereo matching to create stereogram, which will be way, a visual illusion of a three-dimensional (3D) scene is described further in Section III. perceived. Stereograms have been found to be a useful tool for studying stereoscopic vision in humans [16], as they provide a C. Other related works straightforward way to manipulate particular depth cues and to On the current Internet, there exists a significant number insert conflicts between these cues. The fact that they require of similar online applications. Most of them enable users to no extra hardware devices for viewing is an added benefit. quickly and interactively build stereograms by choosing from As examples, Likova et al. used stereograms to study the a number of given depth-maps and patterns. For instance, dynamic aspects of stereo vision in humans [17] and to explore the web-tool named Easy Stereogram builder lets the user stereo-vision deficiency in humans, Skrandies [18] analysed select a shape mask and a texture image of their choice to how random-dot stereograms, related to stereograms, evoked generate a new stereogram [11]. Stereocreator [12] is another activity in cortical neurones that were sensitive to binocular website where the user can enter letters and symbols hidden disparity. in a stereogram. Rune Vision [19] is a slightly more efficient solution to the user customizability, it allows the user to sketch a depth pattern by hand. Similarly to another modern looked web-site in [13] which also allows free-hand painting of depth-map using Macromedia Flash. Besides the above non- profit solutions, other commercial applications are introduced in http://stereogram.qarchive.org/, namely Stereographic Suite, Easy Stereo, Big 3D, 3DMiracle, 3DMiracle, 3DMonster, 3DMonster [14]. Despite the large number of similar products, to the best of our knowledge, none of them utilises stereo vision technique to generate stereograms, i.e. by using the depth and pattern information recovered from a pair of left and right stereo images. Katrusiak et al. [7] use stereograms to display crystal structures on a 2D image. Katrusiak Claimed that the performance Fig. 3. Diagram of human 3D perception which is obtained from viewing a for in-depth perception was similar or even more advantageous stereogram than of stereo pairs. Depth perception is produced when an ob- servers two eyes focus and converge on an object at somewhat Figure 3 demonstrates the background theory of stereogram. different angles. There are also some websites solely dedicated Shown in the figures are two points: B is a point on the to auto-stereogram display and collection such as Stereogram background and P is a position of a point in front of the Gallery [20]. background. Human eyes have a base distance beyes, they look III.DESIGN AND IMPLEMENTATION at the stereogram; however, they do not focus on the picture surface. Then, a 3D point appears in our brain if the eyes A. Depth Extraction determine two projected correspondence points (which may It is a known that to perceive a 3D scene visually, two look similar in shape or colour). For instance, points pBL and views of the same scene need to be obtained, with a small pBR helps to specify position of B, points pPL and pPR helps horizontal difference in these viewing positions. This process to specify position of P in space. Theoretically, the maximum generates disparity map from a pair of near stereo images. A distance dP = pPR − pPL between two correspondence stereo-like view of a static scene can also be produced with points pL, pR on stereogram can not be greater than beyes: one single conventional camera by imaging the scene from two beyes ≤ dP ; at beyes = bP , point P is at infinity. From the different viewpoints at two different moments. This acquisition figure, we have: allows creating a stereo pair with flexible baseline distances; thus, stereo pairs of much closer and farther scenes can be ZPA dP ZPA × beyes achieved Uncalibrated Rectification is done via an estimation = => dP = (1) ZPE beyes ZPE of a Fundamental Matrix from a set of correspondences between two images. The Fundamental Matrix was introduced ∗ The distance dP is different from disparity dP in a stereo by Luong and Faugeras [21], which is a 3×3 matrix combining vision where the smaller disparity, the farther the point; but the column 3-vectors pl and pr of homogeneous coordinates there is a relationship. How do they relate? In stereo vision, of the corresponding 2D points in the left and right images, we have: respectively as follow: beyes ZPE = f (2) > ∗ pr F pl = 0 (3) dP

3817 If F is defined, an image pair can be re-sampled to conform to the standard stereo geometry. The rectification is achieved by moving the actual epipoles to the horizontal position at infinity. Therefore, in this system, we start with the Lucas- Kanade optical flow [22] in a pyramid, which matches good feature to track (GFT) points [23], namely the KLT method. It returns many correspondences within both the images. After (a) A view of stereogram where (b) A view of stereogram where left region of the image is repeated right region of the image is re- that RANSAC [24] is used to estimate the Fundamental matrix peated from a set of correspondences containing outliers using the 8-point algorithm [25]. Image rectification generates horizontally aligned epipolar stereo pair with the knowledge of the fundamental matrix. Two iterations of guidance Dynamic Programming Stereo (BM+DP)4 is used to extract disparity depth from rectified images. The algorithm is evaluated with the Middlebury 2001 and 2003 datasets [26] and resulted with 10.1% of mismatches (89.9% accuracy).

Fig. 5. Parallel-eyed stereogram created from the Middlebury Teddy dataset.

relate to one pixel in the disparity map, they are the corre- spondent points. Our stereogram generation contains a number of steps as described below:

1) Create an empty image Istereogram with its width and height are deﬁned as:

Wstereogram = Wreference + Wpattern Fig. 4. A real-life stereo pair of a sofa (at Ponsonby, Auckland) processed by our BM+DP algorithm. Hstereogram = Hreference

B. Pattern Selection 2) Draw the chosen image pattern on Istereogram, assume 0 0 C. Stereogram creation its horizontal image area is between x and x + Wpattern. This image pattern is kept constant and used Conceptually, to view a stereogram, both eyes should focus to created all other missing image area in Istereogram. on a point above or behind the screen. The viewing directions 3) Autostereogram is made for every scanline, from left to are converged, the repeating texture patterns are horizontally right to the right of the selected pattern, and from right translated and merged on top of each other. If identical image to left to the left of the selected pattern. patterns are well overlaid (they look identical), our brain 4) From left to right, the stereogram pixels are ﬁlled by the identiﬁes that it is a true correspondence and assign to it following code: an imaginary position in 3D. The 3D depth of any other point is dependent on the relative disparity of correspondences f o r ( y = 0 ; y < DepthImage−>height; y++) { for (x = startX; x < s t er e o gr a m −>width ; x ++) { compared to the previously determined points. i f ( x < DepthImage−>width ){ From the above idea, we can reconstruct a stereogram depth = get2D(DepthImage, y, x); by adding pixels around a selected pattern to represent all } xToDraw = x + PATTERN WIDTH − d ep t h ; points of the disparity map. A disparity map is normalised xToGet = x ; from [minIntensity, maxIntensity] to [0, 255]. Thus, the i f ( xToDraw>=s t a r t X +PATTERN WIDTH && disparity is minimised with g(x, y) = 0 = black maximised xToDraw < s t er e o gr a m −>width ){ set2D(stereogram , y, xToDraw, with g(x, y) = 255 = white. set2D(stereogram , y, xToGet)); Knowing that in the stereogram, two pixels with identical } colour act as virtual conjugate pixel projections in 3D and } } 4http://www.ivs.auckland.ac.nz/quick stereo/index.php

3818 5) From right to left, the stereogram pixels are ﬁlled by the following code: f o r ( y = 0 ; y < originalDepth −>height; y++) { for (x = startX; x >= 0 ; x−−) { d ep t h = get2D(originalDepth , y, x); int xToDraw = x + depth; i n t xToGet = x + PATTERN WIDTH; set2D(stereogram , y, xToDraw, get2D(stereogram , y, xToGet)); } } Fig. 7. A stereogram of a god statue (Hanoi, Vietnam) 6) At this point, a single image stereogram is made. Following the above pseudo codes, we can obtain quickly a stereogram from a pair of near stereo images. For instance, Fig. 6, 7, 8 are three other examples of stereograms made from real-life images generated by this proposed method.

D. Depth Extraction For various reasons, many viewers still struggle to see the 3D illusion of stereograms. Therefore, we provide an extra solution for the situation - a computational depth extraction. Our system accepts inputs as stereogram images, reconstruct depth from them, render the 3D proﬁles using general computer graphic engines, and display on a browser. To reconstruct 3D information from any arbitrary auto-stereograms, we need to know which type (cross- Fig. 8. A stereogram of a ﬂower garden (Auckland, New Zealand) eyed/paralled-eyed) the stereograms are. In fact, there was a similar research by Kimmel, who has proposed a technique to |m| In other words, we crop a 2 pixel slide from each side of reconstruct a depth map from random-dot auto-stereograms the stereogram to form a pair of left/right stereo images and using minimal surface area based correlation technique [27]. supply them to the general automatic stereo matching process.

IV. CONCLUSION We have presented a process to create on the ﬂy, person- alised single image stereogram. Different from most similar tools, which require users to choose or manually draw a depth map and texture, our system uses stereo vision techniques to build dynamically stereograms from user provided real- life photos acquired by a standard digital camera. Not only does the system have entertainment value, but it could also be used as an easily accessible platform for human perceptual experiments conducted online.

REFERENCES [1] D. Brewster, The Kaleidoscope, its history, theory and construction, John Murray, Albemarle Street, 1858. [2] D. F. McAllister, Stereo and 3D Display Technologies, Encyclopedia of Fig. 6. A stereogram of a local ﬂower (Sapa, Vietnam) Imaging Science and Technology (2006) 1327–1344. [3] T. Baccei, R. Salitsky, Random dot stereogram and method for making Our solution on another hand bases directly on our current the same (12 1994). stereo vision system; here, depth-map is achieved by choosing URL http://www.patentlens.net/patentlens/patent/US 5371627/ [4] G. Levine, G. Priester, Hidden Treasures: 3-D Stereograms, Sterling a correct disparity searching range that are not including zero Publishing Co Inc, 2008. and applying stereo matching on it. The disparity range is [5] I. Lee, J. Min, H. Lee, S. Min, S. Kim, A scaling parameter optimization determined as: of watermarking using autostereograms as random dot images, in: Proceedings of the IEEE International Conference on Visual Information Engineering, 2003. VIE 2003, University of Surrey, Guildford, Italy, [m, n] where m > 0 & n > m for parallel-eyed 2003, pp. 314–316. drange = [6] M. Laxler, J. Cohen, L. Press, An expanded guide to the keystone [n, m] where m < 0 & n < m for cross-eyed stereogram cards, Journal of Behavioral Optometry 2 (3) (1991) 59– (4) 66.

3819 [7] A. Katrusiak, Crystallographic autostereograms, Journal of Molecular Graphics and Modelling 19 (3) (2001) 363–367. [8] J. Y. Bouguet, Encyclopedia of Autostereogram - From the website, http: //www.statemaster.com/encyclopedia/Autostereogram, visited on Sep 25, 2012. [9] P. Boyer, Random-dot stereograms, Stereo World 17 (1990) 30–33. [10] H. Thimbleby, S. Inglis, I. Witten, Displaying 3d images: algorithms for single-image random-dot stereograms, IEEE Computer 27 (1994) 38–48. [11] EasyStereogramBuilder.com, Easy stereogram builder - From the website, www.easystereogrambuilder.com/3d-stereogram-maker.aspx, visited on Ju,l 2011. [12] K. Kondo, Online Stereogram Generator - From the website, www. eyetricks.com/stereograms/onlinetools/stereocreator.htm, visited on Jul 20, 2011. [13] Flash-gear.com, Flash Gear Stereo - From the website, www.flash-gear. com/stereo, visited on Jul 20, 20112. [14] QArchive.org, Stereogram creation packages - From the website, stereogram.qarchive.org, visited on Jul 20, 2011. [15] H. A. Layer, The literature of stereoscopy: 1853-1986, Stereo World 14 (1) (1987) 44–45. [16] K. Minev, L. Likova, Autostereograms as a research tool in stereoscopic vision: Interactions between some cues in perception of motion-in- depth, in: Procceedings of European Conference on Visual Perception 28 (ECVP99), Trieste, Italy, 1999. [17] L. T. Likova, C. W. Tyler, Spatiotemporal relationships in a dynamic scene: stereomotion induction and suppression, Journal of Vision 3 (4). arXiv:http://www.journalofvision.org/content/3/4/5.full.pdf+html, doi:10.1167/3.4.5. URL http://www.journalofvision.org/content/3/4/5.abstract [18] W. Skrandies, Assessment of depth perception using psychophysical thresholds and stereoscopically evoked brain activity, Documenta Oph- thalmologica 119 (2009) 209–216, 10.1007/s10633-009-9202-9. URL http://dx.doi.org/10.1007/s10633-009-9202-9 [19] R. S. Johansen, Stereogram Gallery - From the website, http://runevision. com/graphics/stereo/stereogallery, visited on May 9, 2011. [20] D. Bessmertniy, The World of Hidden 3D Stereograms - From the website, http://www.hidden-3d.com, visited on May 9, 2011. [21] Q. T. Luong, O. D. Faugeras, The fundamental matrix: Theory, algorithms, and stability analysis, International Journal of Computer Vision 17 (1996) 43–75. [22] B. D. Lucas, T. Kanade, An iterative image registration technique with an application to stereo vision, in: Proceedings of the International joint conference on Artificial intelligence, Vancouver, Canada, 1981, pp. 674– 679. [23] J. Shi, C. Tomasi, Good features to track, in: Proceedings of the Com- puter Society Conference on Computer Vision and Pattern Recognition, New York, USA, 1993, pp. 593–600. [24] M. A. Fischler, R. C. Bolles, Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography, Communications of the ACM 24 (6) (1981) 381–395. [25] R. I. Hartley, In defense of the eight-point algorithm, IEEE Transactions on Pattern Analysis and Machine Intelligence 19 (6) (1997) 580–593. [26] D. Scharstein, R. Szeliski, A taxonomy and evaluation of dense two- frame stereo correspondence algorithms, International journal of computer vision 47 (1) (2002) 7–42. [27] R. Kimmel, 3D shape reconstruction from autostereograms and stereo, Journal of Visual Communication and Image Representation 13 (1) (2002) 324–333.

3820