SCALABLE SOFTWARE CONTROL OF MILLION-ELEMENT CYBER-PHYSICAL SYSTEMS USING A GRAPHICS PROCESSING UNIT by VELJKO KRUNIC B.S., University of Colorado, 2000 M.S., University of Colorado, 2003 A thesis submitted to the Faculty of the Graduate School of the University of Colorado in partial fulfillment of the requirement for the degree of Doctor of Philosophy Department of Computer Science 2010 Copyright © 2010 Veljko Krunic, All Rights Reserved. See Appendix B for License on this work. This thesis entitled: Scalable Software Control of Million-Element Cyber-Physical Systems Using a Graphics Processing Unit written by Veljko Krunic has been approved for the Department of Computer Science _____________________________________________ Associate Professor Richard Han, Advisor _____________________________________________ Professor Clayton H. Lewis _____________________________________________ Assistant Professor Adjunct Willem Schreuder _____________________________________________ Professor Yung-Cheng Lee _____________________________________________ Randolph Ware, PhD Date__________________ The final copy of this thesis has been examined by the signatories, and we Find that both the content and the form meet acceptable presentation standards Of scholarly work in the above mentioned discipline. ! Krunic, Veljko (Ph.D., Computer Science) Scalable Software Control of Million-Element Cyber-Physical Systems Using a Graphics Processing Unit Thesis directed by Associate Professor Richard Han Abstract Cyber-Physical Systems consisting of hundreds of thousands of elements are emerging, with even bigger systems likely to emerge in the immediate future. However, in order for emerging and reasonably anticipated systems to be practical, the software control of million-element Cyber-Physical Systems needs to be addressed. This PhD thesis describes the software control algorithms necessary for the realization of million-element Cyber-Physical Systems. This work will show that Graphics Processing Unit (GPU) based control of such Cyber-Physical Systems provides significant benefits, both in the form of fast control of large numbers of elements, as well as in terms of providing a viable and scalable option by using inexpensive, off-the-shelf hardware. GPU control will be shown to be particularly well suited for the combination of the virtual environment with the manipulation of the physical shape of the environment in which the user resides. The main contributions of this PhD thesis consist of novel algorithms that utilize existing off-the-shelf GPUs to control the Constrained Motion Cyber-Physical Systems comprised of multi million element systems, and demonstrate the feasibility and scalability of such control algorithms. It will be shown how both control and coordination of the elements can be achieved, while at the same time accounting for the physical limitations of the Cyber-Physical System elements. The approach presented here ! iii results in the ability to control the position of the actuation elements in Cyber-Physical Systems, as well as additional physical attributes of the system such as temperature, perceived elasticity of the actuating elements, slipperiness of the ground in large scale systems, etc. We describe how to further extend our approach to deal with existing Cyber-Physical Systems like catoms/Claytronics [1], [2], CirculaFloor [3] and MEMS- based tactile devices, as well as describe an approach to addressing the physical safety of the user in large scale Cyber-Physical Systems. ! iv Za moju baku, koja vise nije sa nama. I za ostatak moje porodice, svugde u Svetu. Vasoj ljubavi i podrsci dugujem sve svoje uspehe. Dedicated to my grandma who is no longer with us. And to the rest of my family all over the world. Your love and support made my success possible. ! Acknowledgments I was lucky to benefit from the expertise, help, insight and patience of many people while I worked on this PhD thesis. I would like to thank the following people, without whom this work wouldnʼt be possible in its present form: 1. I would like to thank my wife Marija Krunic for love and understanding during the long process of the obtaining my PhD and the much more important process of living and enjoying life. 2. I am grateful for a lot of invaluable feedback from Professor Clayton Lewis. I found that in multiple situations where I needed help in scoping and understanding broader implications of my doctoral work, his suggestions, help and experience helped me find the right track. He suggested that further uses of the software control algorithms originally developed for PRE should be investigated in other areas, and that helped me expand the scope of this work from its initial PRE/Holodeck focus to what it is today. Furthermore, he provided many invaluable suggestions in the areas of the HCI and psychophysics research, that relate to the assistive uses of this technology, and he also provided many other suggestions about the PhD studies process. Finally, he challenged me to consider further work in the area once this thesis has been completed. 3. Professor Willem Schreuder provided help and expertise in many areas of this work, and I strongly benefited from his insight, experience and knowledge. He was the first one to point out similarities between the approach used for the ! vi calculation of the moxel position and shadow buffer algorithm. He suggested that multiple output channels (moxel displacement, heat, etc) should be investigated in relation to PRE and proposed use of the OpenGL render buffer. He also advised me to consider feedback from the physical systems in my algorithms as well as to investigate mechanism(s) for directly accessing data that are already on the graphics card with a combination of OpenCL and OpenGL. He further suggested that I should consider overlapping surfaces when combining catoms and moxels, proposed limiting the surface of the rendering to only the area immediately around the user in combining moxels and CirculaFloor, proposed that “halfway up” failed moxels could be addressed by the raising of other moxels and proposed “relaxing” moxels below the user in the case that user is falling as a safety measure. Furthermore, he provided many suggestions related to both the style of this document as well as the substance of work described, and he challenged me to think about potential future research areas. I am grateful for the experience and insight he shared with me during multiple brainstorming sessions we had. His help was crucial in identifying areas for possible improvement and shaping this work to its current form. 4. Professor Yung-Cheng Lee acted as a liaison in working with his students involved with building a physical assistive device [4]. His help was crucial in steering me toward MEMS devices and applying my work to this area. He had many suggestions in the area of the physical characteristics of MEMS devices. Furthermore, he provided the insight that the ability to perceive the texture of ! vii the image will be very useful even for people who are not visually disabled (e.g. ability to touch the display and feel the texture of the physical object on the image) and pointed me to TeslaTouch [5]. Finally, he suggested that I should consider adopting my algorithms to account for power consumption of the mechanical system. 5. Professor Steven Oullette helped me on short notice with the review and interpretation of the data collected during the performance test. That helped me understand the implications of the data, not only from the perspective of the benchmark, but also from the perspective of the statistical process control. He helped me understand the nature of the data (including the fact that we are dealing with more than one distribution) and he also pointed out what type of statistics are appropriate to calculate from the data we have, as well as what type of graphs will be the most appropriate representation for those data. Furthermore, his looking at the data from the perspective of the statistical process control helped me understand why these are more appropriate forms of data representation to use. I found his help invaluable and appreciate his time and willingness to help me on the short notice given. 6. Along with my advisor, Professor Richard Han, I had published original papers describing PRE [6], [7]. That work provided an outline of the corresponding algorithms that are expanded on in this document. He provided many suggestions in relation to the organization and presentation of material in this document, suggested writing an overview of the problem domain and ! viii contributions that is now in Chapter 4. He also pointed to median filter being related to one of the steps in how this work addresses moxels stuck in down position. Finally he provided help in grammatical and organizational correction of Abstract and Chapter 1 of this document and many other suggestions related to the style of this document. 7. Randolph Ware, PhD provided many suggestions relating to the practical applications of the PRE/Holodeck environment and challenged me to better understand broader implications of the use cases that I investigate in this PhD thesis. He pointed me towards carbon nanotubes as an area in which my work could be applicable and provided other suggestions relating to this document. 8. I would like to thank to Professor Kevin Shaub for consultation regarding Shapiro-Wilk exponential test. 9. Professor Hiroo Iwata pointed me to the CirculaFloor Project after learning about PRE/Holodeck and responded to questions