DOCSLIB.ORG

A 3D Camera-Based System Concept for Safe and Intuitive Use of a Surgical Robot System

Total Page：16

File Type：pdf, Size：1020Kb

Download full-text PDF Read full-text
A 3D Camera-Based System Concept for Safe and Intuitive Use of a Surgical Robot System A 3D camera-based system concept for safe and intuitive use of a surgical robot system zur Erlangung des akademischen Grades eines Doktors der Ingenieurwissenschaften von der KIT-Fakultat¨ fur¨ Informatik des Karlsruher Instituts fur¨ Technologie (KIT) genehmigte Dissertation von Philip Matthias Nicolai aus Heidelberg Tag der mundlichen¨ Prufung:¨ 9. Juni 2016 Erster Gutachter: Prof. Dr.-Ing. Dr. h.c. Heinz Worn¨ Zweiter Gutachter: Prof. Paolo Fiorini, PhD Credits This thesis would not have been possible without the support of numerous people. First, I’d like to thank my supervisor, Prof. Dr.-Ing. Dr. h.c. Heinz Worn,¨ for pro- viding me with the opportunity to work in the exciting and fast-moving research field of surgical robot systems and 3D camera systems. My sincere thanks also goes to Prof. Paolo Fiorini, PhD, who led the European research project Patient Safety in Robotic Surgery (SAFROS), during which I had the chance to develop many concepts of this thesis, and who graciously agreed to be the second reviewer of my thesis. A further big “Thank you” goes to Dr. Jorg¨ Raczkowsky, leader of the medical research group MeGI at IAR-IPR, for his support, discussions and for his trust that allowed me to freely explore research topics. In parallel with this thesis, the OP:Sense system has been created as a collabo- ration with multiple colleagues from both medical research groups at IAR-IPR. OP:Sense aims to integrate the results from different projects and theses, and I could not have finished this thesis in its final scope without relying on many parts of OP:Sense contributed by my colleagues. I’d like to thank my (ex-)colleagues for the great working atmosphere and collaboration both on scientific endeavours and otherwise: Tim Beyl, Thorsten Brennecke, Julien Mintenbeck, Holger Monnich,¨ Luzie Schreiter, Yaokun Zhang, Andreas Bihlmaier and Jessica Hutzl. Special thanks is reserved for Mirko Kunze for his general humour throughout the day and for meticulously proof-reading my thesis draft even in the final stages of his own thesis. Most important, I’d like to thank Simone Rupp for supporting me throughout the course of this thesis, even if at times we only saw each other awake when I came home from work at the same time as she had to get up early. The final thanks is reserved for my mother Helga Nicolai. She selflessly supported me throughout my whole life and worked hard to open up all possibilities to me, but always respected my decisions and never pressured me to follow any specific path in life she might or might not have wished for. This thesis is dedicated to her. Abstract Within the last decades, surgical robot systems have been integrated operation rooms worldwide. However, in current robotic procedures, the surgical personnel has to devote a significant part of attention in order to ensure and monitor seamless functioning of the robot system. To overcome this limitation, this thesis explores the feasibility of developing a system for safe and intuitive use of surgical robots, based on state-of-the-art range imaging cameras and newly developed algorithms. A novel concept for an Operating Room (OR) monitoring system is proposed that can perceive the environment of a surgical robot using multiple 3D cameras and detect potentially harmful situations between the robot, its surroundings and the persons in its vicinity, i.e. the OR personnel and the patient. Such a system is realized in a generic way in order to be applicable to different surgical robot systems. It is optimized for a low spatial footprint for not interfering with the OR personnel and their actions in already crowded ORs. Furthermore, the system provides intuitive feedback to the OR personnel whenever safety-critical events are detected, without drawing on their attention otherwise. The realized system was extensively evaluated using the OP:Sense surgical re- search platform. Based on the proposed approach of establishing a virtual safety zone around each robot arm, the system was shown to reliably detect and therefore avoid impending collisions, without requiring information about the trajectory of the robot. To ensure the applicability of use within the operating room, the effects of sterile draping on range imaging cameras were analyzed. A filtering method was put forward to eliminate these effects within the realized ToF camera system, allowing for successful detection of impending collisions even for draped robots. The results indicate that a 3D-camera-based supervision system can effectively contribute to the safety of use of surgical robot systems in the OR, allowing the OR personnel to completely focus on their medical tasks. The proposed methods contribute to scene supervision for human-robot cooperation and show the feasibility of the approach. Zusammenfassung Motivation Seit die ersten kommerziell erhaltlichen¨ Chirurgierobotersysteme Mitte der Neun- ziger Jahre klinisch eingesetzt wurden, haben sich sowohl der Einsatz derartiger Systeme als auch die Vielfalt der abgedeckten Operationen vervielfacht. Im Jahr 2014 wurden alleine mit dem am Markt fuhrenden¨ da Vinci System uber¨ 500 000 Operationen durchgefuhrt.¨ Insbesondere im Bereich der minimal-invasiven Chirurgie hat der Einsatz von Chirurgierobotern jedoch neben den medizinischen Aspekten noch weitere Kon- sequenzen: Der Chirurg fuhrt¨ die Operation von außerhalb des sterilen Bereichs per Telemanipulation des Robotersystems durch, was direkte Auswirkungen auf die Arbeitsablaufe¨ und Rollenverteilung im Operationssaal hat. Da aktuelle und in der Entwicklung befindliche chirurgische Robotersysteme großtenteils¨ keine Sensorik enthalten, mit der sie ihre Umgebung erfassen konnten,¨ muss das OP- Personal sowohl die korrekte Funktionsweise als auch die Kollisionsfreiheit der Robotersysteme fortlaufend uberwachen.¨ Ziel dieser Arbeit ist daher die Entwicklung und Umsetzung eines 3D-Kamera- basierten Uberwachungssystems¨ fur¨ den Operationssaal, das die sichere Anwen- dung chirurgischer Robotersysteme uberwacht¨ und mittels raumlicher¨ erweiterter Realitat¨ (AR) Informationen z.B. zum Roboterzustand intuitiv fur¨ das OP-Personal erfassbar macht. Hierdurch soll das OP-Personal unterstutzt¨ und entlastet werden, um ihm die uneingeschrankte¨ Konzentration auf die medizinischen Aspekte der Operation zu ermoglichen.¨ Methoden Im Mittelpunkt dieser Arbeit steht die Konzeption und Realisierung eines Uber-¨ wachungssystems bestehend aus mehreren, raumlich¨ verteilten 3D Kameras, das die unmittelbare Umgebung der Operationsliege in Echtzeit als Punktwolke erfasst und in Bezug auf verschiedene Sicherheitsaspekte analysiert. Um jeden Roboter- arm wird eine virtuelle, dynamische Sicherheitshulle¨ gebildet, mittels derer die erfasste Szene segmentiert wird. Bei Verletzungen der virtuellen Sicherheitshulle¨ wird zunachst¨ eine raumliche¨ Analyse durchgefuhrt,¨ um die Ursache der Verlet- zung zu ermitteln. Falls diese im Eindringen eines externen Objektes oder eines vii Menschen besteht, werden je nach Zustand des Robotersystems verschiedene Reaktionen ausgefuhrt,¨ um beispielsweise Kollisionen zwischen Roboter und Mensch zu vermeiden. Grundlage fur¨ dieses Konzept ist der entwickelte Shape Cropping Algorithmus, mittels dessen die sicherheitshullenbasierte¨ Segmentierung der erfassten Szene umgesetzt wird. Weiterhin ermoglicht¨ dieser Algorithmus die Detektion des Robotersystems in der Szene, um den korrekten Aufbau des Robotersystems in Ubereinstimmung¨ mit einer praoperativen¨ Planung zu uberpr¨ ufen.¨ Das entwickelte Kamerasystem gliedert sich in zwei separate Teilsysteme, deren Kameras die Szene anhand unterschiedlicher Technologien dreidimensional er- fassen. Das erste Teilsystem besteht aus sieben Time-of-Flight (ToF) Kameras aus dem industriellen Bereich, das zweite Teilsystem aus vier Kinect v1 Kameras. Aufgrund der unterschiedlichen Vor- und Nachteile der verwendeten Kamera- systeme bezuglich¨ Auflosung,¨ Latenz und externer Kontrolle der Kameraparameter werden die erfassten Informationen in ein zweistufiges Szenenmodell integriert. Die erste Ebene des Szenenmodells beinhaltet niedrig aufgeloste¨ Punktwolken, die von den Time-of-Flight-Kameras mit einer geringen Latenz erfasst werden und uber¨ keine semantischen Informationen verfugen.¨ Diese Daten bilden die Grundlage fur¨ sicherheitsrelevante Funktionen wie die Erkennung potentieller Kollisionen. Die zweite Ebene beinhaltet hoher¨ aufgeloste¨ raumliche¨ Daten inklu- sive semantischer Informationen, insbesondere der Erkennung von Menschen in der Szene. Diese Daten sind jedoch erst mit einer großeren¨ Latenz verfugbar.¨ Zur Uberbr¨ uckung¨ dieser semantischen Lucke¨ wurde ein Algorithmus entwick- elt, der die Vorwartspropagation¨ von semantischen Annotationen ermoglicht.¨ Dieser erlaubt es, die auf der zweiten Ebene des Szenenmodells vorhandenen semantischen Annotationen fortlaufend vorwarts¨ zu berechnen, so dass sie auf der ersten Ebene des Szenemmodells direkt zur Verfugung¨ stehen. Somit kann beispielsweise bei der Detektion einer bevorstehenden Kollision des Roboters anhand der vorwarts-berechneten¨ semantischen Information erkannt werden, ob der Roboter mit einem Menschen oder der Umgebung kollidieren wurde,¨ so dass unterschiedliche Reaktionen ausgelost¨ werden konnen.¨ Um Informationen zum Zustand des Robotersystems oder andere Hinweise intui- tiv fur¨ das OP-Team verfugbar¨ zu machen, wurde ein entsprechendes Konzept ent- worfen und umgesetzt, das auf projektorbasierter raumlicher¨ erweiterter Realitat¨ basiert. Dies ermoglicht¨ (i) die Visualisierung
Load more

Recommended publications
  • A Quality Controllable Multi-View Object Reconstruction Method for 3D Imaging Systems
    J. Vis. Commun. Image R. 21 (2010) 427–441 Contents lists available at ScienceDirect J. Vis. Commun. Image R. journal homepage: www.elsevier.com/locate/jvci A quality controllable multi-view object reconstruction method for 3D imaging systems Wen-Chao Chen a, Hong-Long Chou b, Zen Chen a,* a Department of Computer Science, National Chiao Tung University, 1001 University Road, Hsinchu 300, Taiwan b Altek Corporation, Science-Based Industrial Park, Hsinchu, Taiwan article info abstract Article history: This paper addresses a novel multi-view visual hull mesh reconstruction for 3D imaging with a system Received 2 July 2009 quality control capability. There are numerous 3D imaging methods including multi-view stereo algo- Accepted 22 February 2010 rithms and various visual hull/octree reconstruction methods known as modeling from silhouettes. Available online 21 April 2010 The octree based reconstruction methods are conceptually simple to implement, while encountering a conflict between model accuracy and memory size. Since the tree depth is discrete, the system perfor- Keywords: mance measures (in terms of accuracy, memory size, and computation time) are generally varying rapidly 3D imaging system with the pre-specified tree depth. This jumping system performance is not suitable for practical applica- Modeling from silhouettes tions; a desirable 3D reconstruction method must have a finer control over the system performance. The Octree model XOR projection error proposed method aims at the visual quality control along with better management of memory size and System performance computation time. Furthermore, dynamic object modeling is made possible by the new method. Also, Dynamic modeling progressive transmission of the reconstructed model from coarse to fine is provided.
    [Show full text]
  • Combining Computer Vision and Knowledge Acquisition to Provide Real-Time Activity Recognition for Multiple Persons Within Immersive Environments
    Combining Computer Vision and Knowledge Acquisition to provide Real-Time Activity Recognition for Multiple Persons within Immersive Environments Anuraag Sridhar Schoool of Computer Science and Engineering University of New South Wales, Sydney, Australia A thesis submitted in fulfilment of the requirements for the degree of PhilosophiæDoctor (PhD) 2012 b c d Abstract In recent years, vision is gaining increasing importance as a method of human- computer interaction. Vision techniques are becoming popular especially within immersive environment systems, which rely on innovation and novelty, and in which the large spatial area benefits from the unintrusive operation of visual sensors. However, despite the vast amount of research on vision based analysis and immersive environments, there is a considerable gap in coupling the two fields. In particular, a vision system that can provide recognition of the activ- ities of multiple persons within the environment in real time would be highly beneficial in providing data not just for real-virtual interaction, but also for so- ciology and psychology research. This thesis presents novel solutions for two important vision tasks that sup- port the ultimate goal of performing activity recognition of multiple persons within an immersive environment. Although the work within this thesis has been tested in a specific immersive environment, namely the Advanced Visual- isation and Interaction Environment, the components and frameworks can be easily carried over to other immersive systems. The first task is the real-time tracking of multiple persons as they navigate within the environment. Numer- ous low-level algorithms, which leverage the spatial positioning of the cameras, are combined in an innovative manner to provide a high-level, extensible frame- work that provides robust tracking of up to 10 persons within the immersive environment.
    [Show full text]
  • Incorporating Dynamic Real Objects Into Immersive Virtual Environments
    Incorporating Dynamic Real Objects into Immersive Virtual Environments Mary Whitton, Benjamin Lok Samir Naik Frederick P. Brooks Jr. University of North Carolina at Charlotte Disney Corporation University of North Carolina at Chapel Hill [email protected] [email protected] [whitton, brooks]@cs.unc.edu Abstract In the assembly verification example, the ideal VE system would have the participant fully convinced he was actually We present algorithms that enable virtual objects to interact performing a task [Sutherland 1965]. Parts and tools would have with and respond to virtual representations, avatars, of real mass, feel real, and handle properly with appropriate visual and objects. These techniques allow dynamic real objects, such as the haptic feedback. The user would interact with virtual objects as if user, tools, and parts, to be visually and physically incorporated he were actually doing the task, and virtual objects would respond into the virtual environment (VE). The system uses image-based to the user’s actions. object reconstruction and a volume query mechanism to detect Both assembly and servicing are hands-on tasks and the collisions and to determine plausible collision responses between principal drawback of virtual models — that there is nothing there virtual objects and the avatars. This allows our system to provide to feel, to give manual affordances, and to constrain motions — is the user natural interactions with the VE. a serious one for these applications. Using a six degree-of- We have begun a collaboration with NASA Langley Research freedom wand to simulate a wrench, for example, is far from Center to apply the hybrid environment system to a satellite natural or realistic, perhaps too far to be useful.
    [Show full text]
  • Multicamera Real-Time 3D Modeling for Telepresence And
    Multicamera Real-Time 3D Modeling for Telepresence and Remote Collaboration Benjamin Petit, Jean-Denis Lesage, Clément Menier, Jérémie Allard, Jean-Sébastien Franco, Bruno Raffin, Edmond Boyer, François Faure To cite this version: Benjamin Petit, Jean-Denis Lesage, Clément Menier, Jérémie Allard, Jean-Sébastien Franco, et al.. Multicamera Real-Time 3D Modeling for Telepresence and Remote Collaboration. International Jour- nal of Digital Multimedia Broadcasting, Hindawi, 2010, Advances in 3DTV: Theory and Practice, 2010, Article ID 247108, 12 p. 10.1155/2010/247108. inria-00436467v2 HAL Id: inria-00436467 https://hal.inria.fr/inria-00436467v2 Submitted on 6 Sep 2010 (v2), last revised 18 Apr 2012 (v3) HAL is a multi-disciplinary open access L’archive ouverte pluridisciplinaire HAL, est archive for the deposit and dissemination of sci- destinée au dépôt et à la diffusion de documents entific research documents, whether they are pub- scientifiques de niveau recherche, publiés ou non, lished or not. The documents may come from émanant des établissements d’enseignement et de teaching and research institutions in France or recherche français ou étrangers, des laboratoires abroad, or from public or private research centers. publics ou privés. Hindawi Publishing Corporation International Journal of Digital Multimedia Broadcasting Volume 2010, Article ID 247108, 12 pages doi:10.1155/2010/247108 Research Article Multicamera Real-Time 3D Modeling for Telepresence and Remote Collaboration Benjamin Petit,1 Jean-Denis Lesage,2 Clement´ Menier,3 Jer´
    [Show full text]
  • Rendering Methods for Augmented Reality
    Rendering Methods for Augmented Reality Dissertation der Fakultat¨ fur¨ Informations- und Kognitionswissenschaften der Eberhard-Karls-Universitat¨ Tubingen¨ zur Erlangung des Grades eines Doktors der Naturwissenschaften (Dr. rer. nat.) vorgelegt von Dipl.-Inform. Jan T. Fischer aus Mainz Tubingen¨ 2006 Tag der mundlichen¨ Qualifikation: 14.06.2006 Dekan: Prof. Dr. rer. soc. Michael Diehl 1. Berichterstatter: Prof. Dr.-Ing. Dr.-Ing. E.h. Wolfgang Straßer 2. Berichterstatter: Prof. Dr. rer. nat. Bernd Frohlich¨ (Bauhaus-Universitat¨ Weimar) 3. Berichterstatter: PD Dr. rer. nat. Dirk Bartz Erklarung¨ Hiermit erklare¨ ich, dass ich die Arbeit selbstandig¨ und nur mit den angegebenen Hilfsmitteln angefertigt habe und dass alle Stellen, die im Wortlaut oder dem Sinne nach anderen Werken entnommen sind, durch Angaben der Quellen als Entlehnung kenntlich gemacht worden sind. Tubingen,¨ Februar 2006 Jan Fischer iv Zusammenfassung In den letzten Jahren hat sich die Erweiterte Realitat¨ (englisch: Augmented Reality)zuei- ner vielversprechenden und schnell an Bedeutung gewinnenden Anwendung der Computer- graphik entwickelt. Augmented-Reality-Systeme kombinieren vom Computer erzeugte gra- phische Darstellungen mit der Ansicht der realen Welt. Mehrere zentrale Problemstellungen lassen sich im Bereich der Augmented Reality identifizieren. Dabei handelt es sich um die Entwicklung spezieller Anzeigegerate,¨ das Kamera-Tracking, den Systementwurf, die Benut- zerinteraktion und Darstellungsverfahren. Wahrend¨ sich der Großteil vorhergehender Arbei- ten mit
    [Show full text]
  • Multi-Layer Skeleton Fitting for Online Human Motion Capture
    Multi-Layer Skeleton Fitting for Online Human Motion Capture Christian Theobalt Marcus Magnor Pascal Schuler¨ Hans-Peter Seidel Max-Planck-Institut fur¨ Informatik Stuhlsatzenhausweg 85, Saarbruck¨ en, Germany ¡ ¡ ¢ theobalt ¡ magnor schueler hpseidel @mpi-sb.mpg.de Abstract has also been considered. Optical flow and prob- abilistic body part models were used to fit a hier- This paper describes a new approach to fitting a archical skeleton to walking sequences [2]. None kinematic model to human motion data which is of the above approaches runs in real-time or comes a component of a marker-free optical human mo- close to interactive performance, however. If real- tion capture system. Efficient vision-based fea- time performance is to be achieved, comparably ture tracking and volume reconstruction by shape- simple models, such as probabilistic region repre- from-silhouette are applied to raw image data ob- sentations and probabilistic filters for tracking [24], tained from several synchronized cameras in real- or the combination of feature tracking and dynamic time. The combination of both sources of informa- appearance models [8] are used. Unfortunately, tion enables the application of a new method for fit- these approaches fail to support sophisticated body ting a sophisticated multi-layer humanoid skeleton. models. We present results with real video data that demon- New methods for the acquisition and efficient strate that our system runs at 1-2 fps. rendering of volumetric scene representations ob- tained from multiple camera views, known as shape 1 Introduction from silhouette or the visual hull [12, 20, 18, 4], have been presented. Recent research shows that it The field of human motion capture is an example for is possible to acquire and render polyhedral visual the coalescence of computer vision and computer hulls in real-time [15].
    [Show full text]
  • Accelerated Real-Time Reconstruction of 3D Deformable Objects from Multi-View Video Channels
    Accelerated Real-Time Reconstruction of 3D Deformable Objects from Multi-View Video Channels Holger Graf1, Leon Hazke1, Svenja Kahn1, Cornelius Malerczyk2 1 Fraunhofer Institute for Computer Graphics Research IGD, Fraunhoferstr. 5, 64283 Darmstadt, Germany {Holger.Graf, Leon.Hazke, Svenja.Kahn}@igd.fraunhofer.de 2 University of Applied Science Giessen-Friedberg, Faculty of Mathematics, Natural Sciences and Computer Sciences Wilhelm-Leuschner-Strasse 13, 61169 Friedberg, Germany {Cornelius.Malerczyk}@mnd.fh-friedberg.de Abstract. In this paper we present a new framework for an accelerated 3D reconstruction of deformable objects within a multi-view setup. It is based on a new memory management and an enhanced algorithm pipeline of the well known Image-Based Visual Hull (IBVH) algorithm that enables efficient and fast reconstruction results and opens up new perspectives for the scalability of time consuming computations within larger camera environments. As a result, a significant increase of frame rates for the volumetric reconstruction of deformable objects can be achieved using an optimized CUDA-based implementation on NVIDIA's Fermi-GPUs. Keywords: Image based 3D reconstruction, GPU, CUDA, Real-Time reconstruction, Image-Based Visual Hull. 1 Introduction In this paper we present an efficient high resolution Image Based Visual Hull (IBVH) algorithm that entirely runs in real-time on a single consumer graphics card. The target application is a real-time motion capture system with a full body 3D reconstruction. The topic of 3D scene reconstruction based on multiple images has been investigated during the last twenty years and produced numerous results in the area of computer graphics and computer vision. Especially, real-time 3D reconstruction of target objects within a GPU environment has become one of the hot issues nowadays.
    [Show full text]
  • Marker-Less Real Time 3D Modeling for Virtual Reality Jérémie Allard, Edmond Boyer, Jean-Sébastien Franco, Clément Ménier, Bruno Raffin
    Marker-less Real Time 3D Modeling for Virtual Reality Jérémie Allard, Edmond Boyer, Jean-Sébastien Franco, Clément Ménier, Bruno Raffin To cite this version: Jérémie Allard, Edmond Boyer, Jean-Sébastien Franco, Clément Ménier, Bruno Raffin. Marker-less Real Time 3D Modeling for Virtual Reality. Immersive Projection Technology, May 2004, Ames, United States. inria-00349066 HAL Id: inria-00349066 https://hal.inria.fr/inria-00349066 Submitted on 23 Dec 2008 HAL is a multi-disciplinary open access L’archive ouverte pluridisciplinaire HAL, est archive for the deposit and dissemination of sci- destinée au dépôt et à la diffusion de documents entific research documents, whether they are pub- scientifiques de niveau recherche, publiés ou non, lished or not. The documents may come from émanant des établissements d’enseignement et de teaching and research institutions in France or recherche français ou étrangers, des laboratoires abroad, or from public or private research centers. publics ou privés. Marker-less Real Time 3D Modeling for Virtual Reality Jérémie Allard Edmond Boyer Jean-Sébastien Franco Clément Ménier Bruno Raffin fi[email protected] Laboratoire Gravir, Laboratoire ID INRIA Rhône-Alpes 655 avenue de l’Europe, 38334 Saint Ismier, France Abstract equipments (and a tight body suit). In immersive environ- ments, most of the time, only a few points are tracked, the Today, immersive environments mainly use a few 3D posi- user’s head and an interaction device, a wand for example. tions given by trackers to align the real and virtual worlds. Tracking only a few points shorten calibration procedures In this paper we present an alternative approach using com- and ease passing the equipment from one user to the other.
    [Show full text]
  • Download File
    SUPPORTING MULTI-USER INTERACTION IN CO-LOCATED AND REMOTE AUGMENTED REALITY BY IMPROVING REFERENCE PERFORMANCE AND DECREASING PHYSICAL INTERFERENCE Ohan Oda Submitted in partial fulfillment of the requirements for the degree of Doctor of Philosophy in the Graduate School of Arts and Sciences COLUMBIA UNIVERSITY 2016 © 2016 Ohan Oda All rights reserved ABSTRACT Supporting Multi-User Interaction in Co-Located and Remote Augmented Reality by Improving Reference Performance and Decreasing Physical Interference Ohan Oda One of the most fundamental components of our daily lives is social interaction, ranging from simple activities, such as purchasing a donut in a bakery on the way to work, to complex ones, such as instructing a remote colleague how to repair a broken automobile. While we inter- act with others, various challenges may arise, such as miscommunication or physical interference. In a bakery, a clerk may misunderstand the donut at which a customer was pointing due to the uncertainty of their finger direction. In a repair task, a technician may remove the wrong bolt and accidentally hit another user while replacing broken parts due to unclear instructions and lack of attention while communicating with a remote advisor. This dissertation explores techniques for supporting multi-user 3D interaction in aug- mented reality in a way that addresses these challenges. Augmented Reality (AR) refers to inter- actively overlaying geometrically registered virtual media on the real world. In particular, we address how an AR system can use overlaid graphics to assist users in referencing local objects accurately and remote objects efficiently, and prevent co-located users from physically interfer- ing with each other.
    [Show full text]
  • A Multi-GPU Accelerated Virtual-Reality Interaction Simulation Framework
    RESEARCH ARTICLE A multi-GPU accelerated virtual-reality interaction simulation framework 1 1 2 3 Xuqiang ShaoID *, Weifeng Xu , Lina Lin , Fengquan Zhang 1 School of Control and Computer Engineering, North China Electric Power University, Baoding, Hebei, China, 2 92524 troops, Ningbo, Zhejiang, China, 3 School of Computer Science, North China University of Technology, Beijing, China * [email protected] a1111111111 a1111111111 a1111111111 Abstract a1111111111 a1111111111 In this paper, we put forward a real-time multiple GPUs (multi-GPU) accelerated virtual-real- ity interaction simulation framework where the reconstructed objects from camera images interact with virtual deformable objects. Firstly, based on an extended voxel-based visual hull (VbVH) algorithm, we design an image-based 3D reconstruction platform for real objects. Then, an improved hybrid deformation model, which couples the geometry con- OPEN ACCESS strained fast lattice shape matching method (FLSM) and total Lagrangian explicit dynamics Citation: Shao X, Xu W, Lin L, Zhang F (2019) A (TLED) algorithm, is proposed to achieve efficient and stable simulation of the virtual multi-GPU accelerated virtual-reality interaction simulation framework. PLoS ONE 14(4): objects' elastic deformations. Finally, one-way virtual-reality interactions including soft tis- e0214852. https://doi.org/10.1371/journal. sues' virtual cutting with bleeding effects are successfully simulated. Moreover, with the pur- pone.0214852 pose of significantly improving the computational efficiency of each time step, we propose Editor: Huijia Li, Central University of Finanace and an entire multi-GPU implementation method of the framework using compute unified device Economics, CHINA architecture (CUDA). The experiment results demonstrate that our multi-GPU accelerated Received: September 12, 2018 virtual-reality interaction framework achieves real-time performance under the moderate Accepted: March 10, 2019 calculation scale, which is a new effective 3D interaction technique for virtual reality applications.
    [Show full text]
  • Novel View Telepresence with High-Scalability Using Multi-Casted Omni-Directional Videos
    NOVEL VIEW TELEPRESENCE WITH HIGH-SCALABILITY USING MULTI-CASTED OMNI-DIRECTIONAL VIDEOS Tomoya Ishikawa, Kazumasa Yamazawa and Naokazu Yokoya Graduate School of Information Science, Nara Institute of Science and Technology, Ikoma, Nara, Japan Keywords: Telepresence, Novel view generation, Multi-cast, Network, Image-based rendering. Abstract: The advent of high-speed network and high performance PCs has prompted research on networked telepres- ence, which allows a user to see virtualized real scenes in remote places. View-dependent representation, which provides a user with arbitrary view images using an HMD or an immersive display, is especially effec- tive in creating a rich telepresence. The goal of our work is to realize a networked novel view telepresence system which enables multiple users to control the viewpoint and view-direction independently by virtual- izing real dynamic environments. In this paper, we describe a novel view generation method from multiple omni-directional images captured at different positions. We mainly describe our prototype system with high- scalability which enables multiple users to use the system simultaneously and some experiments with the system. The novel view telepresence system constructs a virtualized environment from real live videos. The live videos are transferred to multiple users by using multi-cast protocol without increasing network traffic. The system synthesizes a view image for each user with a varying viewpoint and view-direction measured by a magnetic sensor attached to an HMD and presents the generated view on the HMD. Our system can generate the user’s view image in real-time by giving correspondences among omni-directional images and estimating camera intrinsic and extrinsic parameters in advance.
    [Show full text]
  • Online Model Reconstruction for Interactive Virtual Environments Benjamin Lok University of North Carolina at Chapel Hill Department of Computer Science
    Online Model Reconstruction for Interactive Virtual Environments Benjamin Lok University of North Carolina at Chapel Hill Department of Computer Science ABSTRACT system to render a visually faithful avatar. Calculating exact models is difficult and not required for many real-time We present a system for generating real-time 3D reconstructions applications. A useful approximation is the visual hull. A shape of the user and other real objects in an immersive virtual from silhouette concept, the visual hull is the tightest model that environment (IVE) for visualization and interaction. For can be obtained by examining only object silhouettes [5]. example, when parts of the user's body are in his field of view, our system allows him to see a visually faithful graphical Our system uses graphics hardware to accelerate examining a representation of himself, an avatar. In addition, the user can volume for the visual hull. By using the framebuffer to compute grab real objects, and then see and interact with those objects in results in a massively parallel manner, the system can generate the IVE. Our system bypasses an explicit 3D modeling stage, and reconstructions of real scene objects from arbitrary views in real- does not use additional tracking sensors or prior object time. The system discretizes the 3D visual hull problem into a set knowledge, nor do we generate dense 3D representations of of 2D problems that can be solved by the substantial yet objects using computer vision techniques. We use a set of specialized computational power of graphics hardware. The outside-looking-in cameras and a novel visual hull technique that resulting dynamic representations are used for displaying visually leverages the tremendous recent advances in graphics hardware faithful avatars and objects to the user, and as elements for performance and capabilities.
    [Show full text]