EMBEDDING SENSOR VISUALIZATION in MARTIAN TERRAIN RECONSTRUCTIONS At

EMBEDDING SENSOR VISUALIZATION IN MARTIAN TERRAIN RECONSTRUCTIONS at

Gerhard Paar1, Gerd Hesina2, Christoph Traxler2, Valérie Ciarletti3,Dirk Plettemeier4, Christoph Statz4, Kathrin Sander1, Bernhard Nauschnegg1 r@Joanneum.

1JOANNEUM RESEARCH, Austria athrin.Sande KK 2VRVis Forschungs GmbH, Austria 3UVSQ (UPSay); UPMC (Sorbonne Univ.); CNRS/INSU; LATMOS-IPSL, France 1 4Technische Universität Dresden, Germany

ASTRA, May 13, 2015 “The camera system's images, combinedContents with ground penetrating radar data collec te d w hile t ravelli ng, w ill a llow PanCam scientists on-ground to define suitable drilling locations.” http://exploration.esa.int/mars/45084-exomars-rover/ WISDOM antennas at r@Joanneum. athrin.Sande K K 2

Credit: ESA

ASTRA, May 13, 2015 Embedded Sensor Visualization in Martian Terrain Motivation for the (visual) combination of PanCam &WISDOM& WISDOM •Theglobal strategy for the selection of scientific sites during the Exomars mission will be based on PanCam as well as on WISDOM data

Scientific Site? Credit: ESA, EXM- at PL-RS-ESE-00002 r@Joanneum.

PanCam athrin.Sande KK 3

ASTRA, May 13, 2015 Contents

Scientific Rationales for Visualization leading to PRo3D

PanCam:3DDataand: 3D Data and Visualization

WISDOM: 3D Data and Visualization

PanCam & WISDOM Data Fusion

PanCam &WISDOM& WISDOM @SAFER@ SAFER Campaign at

Outlook r@Joanneum. athrin.Sande KK 4

ASTRA, May 13, 2015 Contents

Scientific Rationales for Visualization leading to PRo3D

PanCam:3DDataand: 3D Data and Visualization

WISDOM: 3D Data and Visualization

PanCam & WISDOM Data Fusion

PanCam &WISDOM& WISDOM @SAFER@ SAFER Campaign at

Outlook r@Joanneum. athrin.Sande KK 5

ASTRA, May 13, 2015 Scientific Rationales for Visualization - PRo3D

PRo 3D: ItInteracti ve real -time rend erer t o expl ore 3D vi si on prod uct s (GPU based methods, Windows)

Explore 3D reconstructions of planetary surface from various viewpoints to address special requirements from the planetary science community, especially geologists, and thus , to give support for the selection and understanding of scientific sites at r@Joanneum. athrin.Sande KK 6

ASTRA, May 13, 2015 Scientific Rationales for Visualization - PRo3D

PRo 3D: ItInteracti ve real -time rend erer t o expl ore 3D vi si on prod uct s (GPU based methods, Windows)

•Studyypp surface structures from various perspectives, in different scales at r@Joanneum. athrin.Sande KK 7

ASTRA, May 13, 2015 PRo3D: Multi-resolution terrain model Scientific Rationales for Visualization - PRo3D

PRo 3D: ItInteracti ve real -time rend erer t o expl ore 3D vi si on prod uct s (GPU based methods, Windows)

•Studyypp surface structures from various perspectives, in different scales at

• Measure the surface (e.g. dimensions, distances) using a variety of

interactive tools r@Joanneum. athrin.Sande KK 8

PRo3D: Measurement of way length betw. two ASTRA, May 13, 2015 surface points at Victoria Crater Scientific Rationales for Visualization - PRo3D

PRo 3D: ItInteracti ve real -time rend erer t o expl ore 3D vi si on prod uct s (GPU based methods, Windows)

•Studyppy surface structures from various perspectives, in different scales at

• Measure the surface (e.g. dimensions, distances) using a variety of r@Joanneum. interactive tools PRo3D: Annotations of regions and features

• Place geological annotations in 3D space athrin.Sande K K 9

ASTRA, May 13, 2015 Scientific Rationales for Visualization - PRo3D

PRo 3D: ItInteracti ve real -time rend erer t o expl ore 3D vi si on prod uct s (GPU based methods, Windows)

•Studyypp surface structures from various perspectives, in different scales at

• Measure the surface (e.g. dimensions, distances) using a variety of

interactive tools r@Joanneum.

• Place geological annotations in 3D space athrin.Sande

PRo3D: PanCam WISDOM Fusion KK

• Watch sensor values in geospatial context, including in particular 10 subsurface WISDOM data

ASTRA, May 13, 2015 Scientific Rationales for Visualization - PRo3D

PRo 3D: ItInteracti ve real -time rend erer t o expl ore 3D vi si on prod uct s (GPU based methods, Windows)

•Studyypp surface structures from various perspectives, in different scales at

• Measure the surface (e.g. dimensions, distances) using a variety of

interactive tools r@Joanneum.

• Place geological annotations in 3D space athrin.Sande KK

• Watch sensor values in geospatial context, including in particular 11 subsurface WISDOM data

ASTRA, May 13, 2015 Contents

Scientific Rationales for Visualization leading to PRo3D

PanCam:3DDataand: 3D Data and Visualization

WISDOM: 3D Data and Visualization

PanCam & WISDOM Data Fusion

PanCam &WISDOM& WISDOM @SAFER@ SAFER Campaign at

Outlook r@Joanneum. athrin.Sande KK 12

ASTRA, May 13, 2015 PanCam Data ExoMars PanCam • Wide angle multispectral stereoscopic panoramic images • ~34° FOV, 50cm stereo base, 1024x1024 pixels at r@Joanneum. athrin.Sande KK 13

ASTRA, May 13, 2015 PanCam Data Processing

PCPanCam 3D processing Output at r@Joanneum. athrin.Sande KK 14

PanCam 2D Input for PanCam 3D Output 3D processing for visualization ASTRA, May 13, 2015 PanCam Processing Results 3D vision products • Distance maps from the Rover’s perspective •Comppylemented by texture information

Orthographic Image (texture) at r@Joanneum.

Distance Map (3D geometry) athrin.Sande K Distance accuracies K

• Down to 3mm in 2m distance to the Rover 15 • ~20mm in 5m distance , ~70mm in 10m distance

ASTRA, May 13, 2015 PanCam 3D Visualization Data Scheme: OPC

• Ordered Point Cloud Format: Optimal for rendering huge geospatial data sets in PRo3D

• 3D points in square patches with neighborhood implicitly defined by raster

• Different Levels of Detail at (geometry and texture) in a

hierarchical structure r@Joanneum. automatically combined in the

rendered mesh athrin.Sande K K

• Out-of-core technique (dynamic 16 loading from disc) Colour coded patches with different LODs

ASTRA, May 13, 2015 3D PanCam Data in PRo3D at r@Joanneum. athrin.Sande KK

ASTRA,PRo3D: May 13, Panam 2015 OPC in Solid Render Mode PRo3D: Panam OPC in Wireframe Render Mode Contents

Scientific Rationales for Visualization leading to PRo3D

PanCam:3DDataand: 3D Data and Visualization

WISDOM: 3D Data and Visualization

PanCam & WISDOM Data Fusion

PanCam &WISDOM& WISDOM @SAFER@ SAFER Campaign at

Outlook r@Joanneum. athrin.Sande KK 18

ASTRA, May 13, 2015 WISDOM Device: Water Ice Subsurface Deposit Observation on Mars • Ultrawideband ground penetrating RADAR • Centrimetric resolution up to a depth of 3 meters below the ExoMars Rover

Profile example of fine resolution Major mission impacts obtained on Mount Etna at • Scientific knowledge about the Martian subsurface r@Joanneum. • Operational support in

identifying interesting drilling athrin.Sande KK sites and possible hazards to the drilling process 19 Left: electronic unit . Right: two identical polarimetric (transm itting and recei vi ng) ) ant ennas

ASTRA, May 13, 2015 WISDOM Data in PRo3D

Three types of data products 1. Contrast profiles 2. Interfaces / iso-surfaces 3. Objects

PRo3D interface at • All data given in VTK xml format for unstructured meshes r@Joanneum. athrin.Sande KK http://www.vtk.org/doc/nightly/html/classvtkUnstructuredGrid.html#details 20 • Can jointly be imported by PRo3D

ASTRA, May 13, 2015 WISDOM Contrast Profiles in PRo3D • Radar values reflected from different depths are displayed as false colour dots along a vertical line in PRo3D (depth correction considering an estimated effective permittivity)

• Subsurface contrast profiles along the rover path can be combined and visualized as radargrams (also in 3D)

•A mesh can be created between these lines and a texture is derived by

interppgolating the measured values of the dots at r@Joanneum. athrin.Sande K K 21 PRo3D: radargram

ASTRA, May 13, 2015 WISDOM Interfaces and Objects in PRo3D

Interfaces / Iso-surfaces

• Geometryyg and normalized brightness values for a detected interface between two subsurface layers (e.g. for a specific estimated

permittivity contrast) PRo3D: solid rendering • PRo3D: colored triangle grid at Objects

• depicts minimum and maximum surface r@Joanneum. representations of buried scatterers with

associated probabilities athrin.Sande K K

• PRo3D: inner and outer sphere 22 PRo3D: wireframe

ASTRA, May 13, 2015 Contents

Scientific Rationales for Visualization leading to PRo3D

PanCam:3DDataand: 3D Data and Visualization

WISDOM: 3D Data and Visualization

PanCam & WISDOM Data Fusion

PanCam &WISDOM& WISDOM @SAFER@ SAFER Campaign at

Outlook r@Joanneum. athrin.Sande KK 23

ASTRA, May 13, 2015 PanCam & WISDOM Data Fusion

Rover coordinate system – Sensor calibration • During Assembly, Integration and Test (AIT) • Get knowledge about the position and orientation of the sensors with respect to the Rover • Necessary for unified geometric context of

PanCam-provided 3D surface models and at WISDOM scans r@Joanneum. Global coordinate system – GNC

• Get the global position and orientation of the sensor data by knowing the athrin.Sande global position and orientation of the Rover at data capturing time KK 24 • Periodically provided / updated by the Rover Guidance, Navigation and Control system (GNC)

ASTRA, May 13, 2015 PanCam & WISDOM Data Fusion

Global data flow • PanCam and WISDOM data are independently processed and georeferenced, and loaded into PRo3D at r@Joanneum. athrin.Sande KK

ASTRA, May 13, 2015 Contents

Scientific Rationales for Visualization leading to PRo3D

PanCam:3DDataand: 3D Data and Visualization

WISDOM: 3D Data and Visualization

PanCam & WISDOM Data Fusion

PanCam &WISDOM& WISDOM @SAFER@ SAFER Campaign at

Outlook r@Joanneum. athrin.Sande KK 26

ASTRA, May 13, 2015 PanCam & WISDOM @ SAFER Campaign

SAFER C ampa ign

• Mars Rover mission simulation campaign SAFER (Sample Acquisition Field Experiment with a Rover) led by RAL and supported by ESA

• Atacama Desert, Chile in October 2013, for 5 days

Main purpose at • Simulation of actual Rover mission as accurate as possible (looking for interesting places, collecting samples..) r@Joanneum. Instruments athrin.Sande 27 • development models of ExoMars instruments PanCam, WISDOM and KK CLUPI

• on-board of a prototype of the ExoMars Rover chassis model

ASTRA, May 13, 2015 PanCam & WISDOM @ SAFER Campaign

Mission Control • No direct interaction between remote center team and field team possible SAFER PanCam Workflow & Data Sources / Dependencies • Remotely from ROCC at Harwell, UK • Decisions essentially

based on instrument data at collected in Chile Æ high need for visualization r@Joanneum.

Data captured athrin.Sande 28 • WISDOM: >3000 soun dings, dis tance o f approx ima te ly 200m over 6 KK

days 28 • PPCanCam: > 4000 images, ~ 10 s tereo panoramas (par tly RGB)

ASTRA, May 13, 2015 PanCam & WISDOM @ SAFER Campaign Success Stories • The 3D location of the WISDOM antennas was provided for each of the soundings • Visualization of the subsurface in a 3D environment following the Rover path was possible (contrast profiles / radargrams) • Identification of a potential interesting rock interface at 60 cm buried

beneath the surface at • Its presence was confirmed by a drilling simulation performed

afterwards r@Joanneum. athrin.Sande 29 • For some WISDOM sites: also PanCam surface 3D vision products KK available. After localization by means of the Rover GNC data, a fusion 29 in PRo3D was possible.

ASTRA, May 13, 2015 PanCam & WISDOM @ SAFER Campaign at r@Joanneum. athrin.Sande 30 KK 30

ASTRA, May 13, 2015 Contents

Scientific Rationales for Visualization leading to PRo3D

PanCam:3DDataand: 3D Data and Visualization

WISDOM: 3D Data and Visualization

PanCam & WISDOM Data Fusion

PanCam &WISDOM& WISDOM @SAFER@ SAFER Campaign at

Outlook r@Joanneum. athrin.Sande KK 31

ASTRA, May 13, 2015 Outlook WISDOM • Upgraded data processing to provide certainty margins of the measured data for 3D visualization (interfaces & objects) PRo3D • Optimized visual appearance of WISDOM data • Data manipulation (e.g. saving / loading of transformations)

• Emb e dding o f fthfurther sensors’d’ da ta (e.g. Inf rared dS Spect romet er at ISEM)

• Simul at ed PCPanCam views tthttdMtito see the reconstructed Martian r@Joanneum. environment from the Rover’s perspective athrin.Sande • RdiRendering of still images from c hosen v iewpo in ts in phthotoreali litistic KK

quality 32 • Video sequence production from navigation paths

ASTRA, May 13, 2015 Acknowledgement

The research leading to these results has received funding from the European Community's Seventh Framework Programme (FP7/2007-2013) under grant agreement n° 312377 PRoViDE, and ESA Contracts SAFER and PRODEX PanCam 3D Vision. We thank the Team of SciSys for their GNC telemetry data provision during the SAFER Campaign. at r@Joanneum. athrin.Sande KK 33

ASTRA, May 13, 2015