VR Technology TNM086
VR Software Principles
Synopsis
Advanced Scene graphs
X3D OpenSceneGraph Event systems Multi screen output and synchronization VR software systems
Motivation for Scene Graphs
OpenGL
Basic primitives – no real (complex) objects Linear structure No visible interrelations Complex scenes in Virtual Reality
Dynamic and interrelated objects Materials, properties and effects (shadows) Interactions
Scene Graph Principles
Directed Acyclic Graph (DAG)
Hierarchy of nodes (tree) Reflects hierarchy of objects Traversed for processing Essentials
Nodes (group or leaf) Edges Nodes
Encapsulate behaviour E g group, geometry, properties
Where are these graphs?
In graphics engines and exchange formats
OpenSceneGraph (OSG) VulkanSceneGraph (VSG) (OGRE, Java 3D, THREE.js, etc) VRML and X3D – exchange formats In Game Engines
Unreal Engine, Unity 3D, Frostbite, etc... These use ”flat” scene graphs Most systems implement their own structure Different priorities – performance, reusability, dynamics, etc. VRML/X3D
Transforms are groups
Implicit separation Shape nodes
Selected properties Geometry child No leaking property
Node Reuse
More than one parent
Allowed by DAG Share child node Less memory use Less code Less to update
Hierarchical Properties
1) Apply new properties 2) Traverse subtree 3) Revert new properties and return
/// Traverse the scenegraph. void MatrixTransform::traverseSG( TraverseInfo &ti ) { ti.pushMatrices( matrix->getValue(), matrix->getValue().inverse() );
X3DGroupingNode::traverseSG( ti ); ti.popMatrices(); }
VRML/X3D Molecule Example
Water Molecule
1 Oxygen 2 Hydrogen
X3D Code
Tree structure
Hierarchy reflected in XML structure XML nodes become scene graph nodes Node children are specified as node contents
E g
E g
X3D Code
C++ API – explicit definitions (or file format .osgt)
OpenGL scenegraph (VulkanSceneGraph coming) Create nodes and specify children Set properties or use default values Import model from file
Put model into scene graph auto *node = load(filename); group->addChild(node); Modifying properties
Manual traversal
Automatic by visitors Geometry and Appearance
Group Group
Shape Geode StateSet
Appearance Geometry Drawable DrawableDrawable ShapeDrawable Material Shape
OpenSceneGraph C++ Code int main(){ osgViewer::Viewer viewer;
osg::PositionAttitudeTransform* moleculeXForm = new osg::PositionAttitudeTransform;
osg::PositionAttitudeTransform* oxygenXForm = new osg::PositionAttitudeTransform; osg::PositionAttitudeTransform* hydrogen1XForm = new osg::PositionAttitudeTransform; osg::PositionAttitudeTransform* hydrogen2XForm = new osg::PositionAttitudeTransform;
osg::Geode* oxygenGeode = new osg::Geode; osg::Geode* hydrogen1Geode = new osg::Geode; osg::Geode* hydrogen2Geode = new osg::Geode;
osg::Sphere O = new osg::Sphere(); osg::ShapeDrawable* shapeO = new osg::ShapeDrawable(O);
osg::Sphere H1 = new osg::Sphere(); osg::ShapeDrawable* shapeH1 = new osg::ShapeDrawable(H1);
osg::Sphere H2 = new osg::Sphere(); osg::ShapeDrawable* shapeH2 = new osg::ShapeDrawable(H2); OpenSceneGraph C++ Code
oxygenGeode->addDrawable(shapeO); hydrogen1Geode->addDrawable(shapeH1); hydrogen2Geode->addDrawable(shapeH2);
oxygenXForm->addChild(oxygenGeode); hydrogen1XForm->addChild(hydrogen1Geode); hydrogen2XForm->addChild(hydrogen2Geode);
moleculeXForm->addChild(oxygenXForm); moleculeXForm->addChild(hydrogen1XForm); moleculeXForm->addChild(hydrogen2XForm);
viewer.setSceneData(moleculeXForm); return viewer.run(); }
OSG Uses Visitor Design Pattern
Visitor operates on nodes in the tree
node->traverse(operator_ptr) Performing operations all of one type or selected nodes Typical operations
Resolution reduction, simplification Intersection check, for interaction Partitioning (e.g. kd-tree) Reordering or optimization
OSG Use Extensive Inheritance
Use of forward declaration ...to avoid including other headers Compiler complains about ”incomplete type” You need to include more headers namespace osg {
class Billboard; class ClearNode; class ClipNode; Use of base type functions ...... that are inherited for convenience Search in base class documentation for functions Sometimes overloaded with other parameters → overload resolution problem
OSG Use Extensive Inheritance
Overload resolution problem
A function has different parameters in different scopes p->function(123) Error: wrong parameters, candidates are: void A::function(void) Use explicit function reference p->Base::function(123) static_cast
Matrices in 3D Graphics
OpenSceneGraph uses R0 x R0 y R0 z 0 R R R 0 Row major storage T = 1 x 1 y 1 z R2 x R2 y R2 z 0 Row vectors ( t x t y t z 1) Explicit pre-/post-mult x '=x M m M v
T =[ R0 x R0 y R0 z 0 R1 x R1 y R1 z 0 R2 x R2 y R2 z 0 t x t y t z 1]
Most others use R0 x R1 x R2 x t x R0 y R1 y R2 y t y T = Column major storage R0 z R1 z R2 z t z ( 0 0 0 1 ) Column vectors t t t t A B C=(C B A ) x '=M v M m x VR Software Systems
Multi Screen Output
Multi-processing
Single processing or threads
Single multi-head graphics card (or SLI) Single process with one memory address space
Clusters
Multiple computers, multiple program instances One graphics card per computer
Separate memory Memory and State Access
State synchronization and locking
Necessary for consistent rendering To avoid tearing Techniques
Threads – these share heap memory Processes – use explicit ”shared memory” space Serialization over network e.g. TCP/IP Low latency is more important than bandwidth One primary, many replica (or ”Image Generators”) (master / slave) primary / secondary
source / replica Synchronization
Types
state synchronization what states to render (poses, view, colours, ...) frame synchronization (swap lock) when to switch to the next rendered frame hardware sync software swap lock over TCP/IP generator lock (gen-lock hardware) when to send the front buffer to the display when to draw left/right image
State Synchronization
Synchronizing independent states
Simulation time, pseudo-random seeds, etc Dependent states can be estimated locally
Unreal Engine 4
Multi-player ”Replication” for state synchronization Replication Graph, Actor Replication, Component Replication nDisplay Actor Replication
With OpenGL: swap_sync_policy = NV swap lock Synchronization Procedure
Primary 0 Replica 1 Replica 2 Replica 3
Update primary states Send primary states Receive primary states Receive primary states Calculate dep. Receive Calculate dep. states Calculate dep. primary states states states Render Calculate dep. Render left eye Render states left eye left eye Render Render Render Render right eye left eye right eye right eye wait for all Render wait for all wait for all right eye wait for all swap buffers swap buffers swap buffers swap buffers
VR Software Systems
Purpose
set up cluster nodes and start the VR software connect to tracking systems, audio, etc. handle per display settings, such as viewpoint handle state serialization and synchonization perform necessary undistortion and filtering
Oculus SDK
C API Software Oculus SDK main Initialize HMD Cross platform Create GL Context Configure Tracking GL initialization Create RBO Configure Rendering by client software Initialize Oculus Start Frame
Start Frame Process Tracker Data Update Scene
Render Scene End Frame End Frame
Close Oculus Process and Display
Close GL Context
VR APIs / Frameworks
SGCT Software SGCT In-house Init OpenGL Init OpenGL Initially simple Pre Sync Pre Sync MinVR Serialize Sync Minimal core De-serialize Post Sync Post Sync Pre Draw Plugin-ins Clear Buffers Clear Buffers Gramods Draw Draw In-house Scene graph Post Draw Lab platform Post Draw Lock Game Engines
Swap Buffers High level Gramods Basics
... Gramods Basics
Get node specific configuration
std::unique_ptr
std::vector
gmNetwork::RunSync *run_sync = nullptr; std::shared_ptr
...
Gramods Basics
Rendering and sync swap
bool alive = true; while (alive) {
for (auto window : windows) window->processEvents(); gmCore::Updateable::updateAll();
for (auto window : windows) { if (!window->isOpen()) continue; window->RendererDispatcher::renderFullPipeline(); }
if (run_sync) run_sync->wait();
alive = false; for (auto window : windows) { if (!window->isOpen()) continue; window->swap(); alive |= true; } } Gramods Basics
Sync data typedef gmNetwork::SyncSData
gmTrack::PoseTracker::PoseSample pose; if (wand->getPose(pose)) { *sync_wand_position = pose.position; *sync_wand_orientation = pose.orientation; } }
run_sync->wait(); // Wait for primary to have sent its values data_sync->update(); // Swap old values for newly synchronized update_states(time); // Use the data to update scenegraph states } OSG Integration
std::shared_ptr
... std::vector
... void OsgRenderer::render(Camera camera, float near, float far) { if (!is_initialized) setup();
GLint vp[4]; glGetIntegerv(GL_VIEWPORT, vp); viewer->getCamera()->setViewport(vp[0], vp[1], vp[2], vp[3]);
Eigen::Matrix4f proj = camera.getProjectionMatrix(near, far); viewer->getCamera()->setProjectionMatrix(osg::Matrix(proj.data()));
Eigen::Matrix4f view = camera.getViewMatrix().matrix(); viewer->getCamera()->setViewMatrix(osg::Matrix(view.data()));
viewer->renderingTraversals(); } Other Advanced Display Systems
Combination of display and tracking
E.g. Cardboard VR and Oculus Rift Hardware and user specific calibration
Interpupillary distance handling Tracking, latency compensation and time warping Optical correction (distortion, aberration, vignetting)
Motion-to-Photon w Time Warping
Warp the final image using fresh tracker data
HMD needs to have extremely low latency @60 fps: less than 2 frames ≈ 33 ms less than 1 frame ≈ 17 ms
Frame i Frame i+1
Tracker Tracker
Pause Warp image
Update states Draw scene Networked Objects Event Systems
Event Systems
Networked Objects
Define data flow Automated run-time processing Software
H3D API Visualization Toolkit (VTK) Voreen (Volume Rendering Engine) Inviwo – Interactive Visualization Workshop
H3D API
Networked scene graph Events in the scene graph
More tasks given to scene graph Program distributed as ”engines” Connecting fields
Sending values Event network
Event Systems
Nodes
Encapsulate a behaviour Fields
Define properties of nodes Contain data: colour, translation, light intensity, etc Sensors
Event sources: Timers, triggers, mouse, etc Engines
Activated by incoming events
Process incoming data and computes new data Event Systems
5 Fields 5 6 2 -9 Contains values, e g SField, MField Copy values, calculate new values Contains nodes, e g SFNode, MFNode
NodeNode Copy pointers, create new objects Node Node Require good reference counters Routing
One field value to many fields Engine may accept many input Lazy evaluation
Sensor Example
MouseSensor
SFVec2f position SFVec2f motion SFBool leftButton
Simple Example
3D mouse sensor
Controls object transform Connect field route from 3D mouse transform route to X-form matrix
Engine Example
PythonScript node
Field type implemented as class (F2I) Field instance defined as class instance (float2int) float2int = F2I()
Event Script Example
Animation
Switch to select ”frame” Floating point TimeSensor Script to convert float to integer
Python Script Code
# Copyright 2006, Karljohan Lundin # from H3D import * from H3DInterface import * input = SFFloat() output = SFInt32() class F2I( TypedField( SFInt32, SFFloat ) ):
def update( self, event ): input_list = self.getRoutesIn() return int(input_list[0].getValue()) float2int = F2I() input.route(float2int) float2int.route(output)
X3D Code
X3D Code
Scene Graph