Visualization at Boeing: Past, Present, Future

Engineering, Operations & Technology Information Technology

Visualization:Acquiring Information using Past Visual, Analytics Present, and Future at Boeing Dave Kasik Senior Technical Fellow The Boeing Company Dave Kasik and [email protected] Senesac The Boeing Company June, 2011

BOEING is a trademark of Boeing Management Company. Copyright © 2011 Boeing. All rights reserved. A Bit About Us Engineering, Operations & Technology | Information Technology . Dave: . Involved in comppgputer graphics since 1969 . Boeing Senior Technical Fellow . ACM Distinguished Scientist . Stand-in on starship bridges . Known around Boeing as – A curmudgeon about virtual reality in the immersive, stereo sense – An a dvoca te for augment ed realit y – Leading proponent and expert for broad use of visualization for geometric and non-geometric data . Chri s: . Involved in computer graphics since 1990 . Boeing Senior Architect . Specialty - being able to apply technology to real world problems . Passion is to simplify complex problems Copyright © 2012 Boeing. All rights reserved. 2 Why Are We Here? Engineering, Operations & Technology | Information Technology . Boeing builds astounding aerospace products . We require hugely varied technology, ranging from . Basic physics to . Networking (on-board & conventional) to . Computing (real-time systems & traditional) to . Material science to . Natural language analysis to . Basically, you name it, we have it . Boeing generates terror -bytes of data . Has worked with advanced visual analysis techniques to gain more insight from our data

Copyright © 2012 Boeing. All rights reserved. 3 Outline Engineering, Operations & Technology | Information Technology . Motivation . Past: Give a quick Boeing history of projects that changed computer graphics . Present: Stress current 3D use cases in Boeing . FtFuture: H ow is comput er graphi cs evol living t o address industrial needs?

.Moore’s Law: Transistor capacity d oubl es every 24

months Growth of technology .The human brain ’s volume doubles only every 3 x 107

months 1974 Present .The amount of data stored digitally was estimated at 180 exabytes in 2006. Growth of human .By 2011 there will be at capability least 1800 exabytes (1.8 zetabytes) A long time ago Present

Copyright © 2012 Boeing. All rights reserved. 5 Why Visualization? Engineering, Operations & Technology | Information Technology . Broadest communication channel to the human brain . Acquires 80+% of all data communicated to a human . BiBrain mus t process dttdata to gai iin insi ihtght . What can a user do with interactive visualization alone?

Copyright © 2012 Boeing. All rights reserved. 6 Visual Task Analysis Engineering, Operations & Technology | Information Technology . Find an object in a complex scene . FthfdbjttbttdtdfFocus on the found object to better understand surface characteristics (e.g., smoothness, roughness) . Once the object is found, look at objects in the immediately surrounding vol ume . Visually scan the scene . Observe dynamics in the entire scene (conventionally by animation) . Work with multiple versions of the same set of objects to compare the two sets . Find anomalies and patterns . More detail in DJ Kasik, “Strategies for Consistent Image Ptitii”IEEEMltidiJPartitioning”, IEEE Multimedia, Jan-Mar, 2004 , pp. 32-41

Copyright © 2012 Boeing. All rights reserved. 7 The Past: Boeing and Computer Graphics Engineering, Operations & Technology | Information Technology . Computer graphics . Human model . B-spline surface rendering . Fractals . User interface management systems . Industrial-strength NURBS Algorithms . IGES . Augmented reality . FlyThru/IVT . Voxmap PointShell (collision detection) . Visual analytics . Massive model visualization

Copyright © 2012 Boeing. All rights reserved. 8 Computer Graphics Engineering, Operations & Technology | Information Technology . According to Wikipedia, computer graphics has made computers easier to interact with, and better for understanding and itinterpreti ng many t ypes of fdt data. D evel opment s i n comput er graphics have had a profound impact on many types of media... . The term came from Boeing-Wichita’s Verne Hudson in 1960. Bill Fetter popularized it .

Animation by photography

3D-2D ppglotting

Contour plots

Copyright © 2012 Boeing. All rights reserved. 9 Human Model Engineering, Operations & Technology | Information Technology . Fetter developed BOEMAN, the first computer model of a human body in 1964 . Used a pen plotter to do reach studies for an aircraft carrier

Copyright © 2012 Boeing. All rights reserved. 10 Fractal Animation Engineering, Operations & Technology | Information Technology . Loren Carpenter . Produced, directed film called Vol Libre . Resulted in standing ovation at SIGGRAPH’80

Courtesy: L. Carpenter Copyright © 2012 Boeing. All rights reserved. 11 Augmented Reality Engineering, Operations & Technology | Information Technology . Term coined by Boeing’s Tom Caudell in 1990 . Applications evaluated (Caudell, David Mizell)in) in Boeing proved impractical . Boeing continues to invest in Augmented Reality

Copyright © 2012 Boeing. All rights reserved. 12 Massive Model Visualization Engineering, Operations & Technology | Information Technology . Dave Kasik started investigating ways to visualize entire aerospace products in 2004 in collaboration with organizations around the world . Monograph ‘Real-Time Massive Model Rendering’ (Yoon, e t a l. ) 2008

Copyright © 2012 Boeing. All rights reserved. 13 Observations Engineering, Operations & Technology | Information Technology . Boeing has a long tradition in computer graphics and interactive techniques . Visualization has specific value for . Highly complex products . PihProcessing huge amoun tfdtts of data . Widest path into brain . Complex system integration . Complex non-geometric data exploration and analysis tasks . Boeing pushed the state-of-the-art to understand state-of- the-art products . It takes a long time for state-of-the-art in aerospace and computing technology to become widespread

Copyright © 2012 Boeing. All rights reserved. 14 The Present: Computer Graphics Use Cases Engineering, Operations & Technology | Information Technology . Dozens of use cases, including . Visual anal yti cs for non-geometitric da ta . 3D geometry visualization

. Model load time: instantaneous . For groups, “instantaneous” translates to less than one minute . For an individual, five minutes just seems like an eternity unless the person can effectively time share tasks . Reality: the faster the better . “Flying” time: New transformation matrices that respond to mouse action . Ideally, 16 Hz (the human flicker fusion threshold for video) or faster . Practically, 10 Hz or faster . Graphical selection . Feedback appears in < . 25 sec

. Analysis vs. Analytics . When d eali ng with di git al d at a, anal ysi s i s th e d et ail ed examination of any size and number of data collections . Analytics is the science behind analysis . In practice, analytics requires understanding the . Cognitive processes . Data acquisition . Tools . Techniques & methods . Resu lts commun ica tion pa tterns that let people obtain an optimal or reasonable decision based on existing data.

Copyright © 2012 Boeing. All rights reserved. 17 Gain Insight through Data Analysis Engineering, Operations & Technology | Information Technology . Situation awareness . Command and control . Tracking and visibility . Determine status Well suited for . Causal chain analysis advanced visual . Determine why something happened analytics tools that allow interactive . Hypothesis testing exploration and . Explore possible explanations assessment of . DiDetecting anoma lidlies and correl liations complex data . Prevent event occurrences . Prediction/Forecasting . ItitiddImprove quantities ordered . What-if studies . Explore alternatives . SiiSummarizing . Communicate results

Copyright © 2012 Boeing. All rights reserved. 18 Tools and Techniques Engineering, Operations & Technology | Information Technology . Numerous tools and techniques . Statistical analysis . Text and data mining . Control rooms . Operations centers . Dashboards . Alerting . Fish-bone charts . Matrix analysis . Neural nets . Competing hypotheses . Modeling & simulation . Pareto analysis . … . Visual analytics is an emerging class of analysis tool

Tools tics y tar l lytics AS AS AC alyst UST/ ATCH rlight ffron otfire xcel ebook PSS otime GNOS bleau SPIRE gsaw aa aa SS tt TT SS SS aa nn aa RR pp ee aa MM EE SS OO Ji

TfType of S A T S T G St IN- C An An P- Analysis No

Situation Awareness

Tracking & Visibility

Causal Chain Analysis

Hypothesis Testing

Detecting Anomalies & Correlations

Prediction & Forecasting

What-If Studies

Summarizing Copyright © 2012 Boeing. All rights reserved. 20 Visual Analytics Engineering, Operations & Technology | Information Technology Formal definition: “The science of analytical reasoning facilitated by interactive visual itinterf aces.” Translation: Rapidly explore large, complex datasets to ggggain new business insight using interactive visualization.

Detect the expected and discover the unexpected

References: . Illuminating the Path, nvac.pnl.gov (free download), 2005 . Special Issue: Foundations and Frontiers of Visual Analytics, Information Visualization, vol. 8, no. 4, Winter 2009

Copyright © 2012 Boeing. All rights reserved. 21 Bird Strike Analysis Engineering, Operations & Technology | Information Technology . The threat by the numbers: . Approximately 1 bird strike p er 2, 000 flig hts . Around 20 strikes per day on jet transport category aircraft alone . About 1 in 10 strikes are damaging . The reported costs average $123 million per year . However, 80% of strikes go unreported, and the true cost could be as high as $615 million per year . Other factors: . Increasing traffic . Bigger, quieter engines . Twin-engine configurations . Increasing bird populations

Engineering, Operations & Technology | Information Technology . Look at data to understand impact of repetitive stress injuries, including cost. . 777 Body Structures investigated historical data to understand how it aligned with injury data. . Pilot study done with Excel for 777 Forward Body Structures . . Analysis based on three years of historical production data, and tested the limitations of Excel. . Bir d st rik e pro jec t sh owed pot enti al f or vi sual anal yti cs.

Engineering, Operations & Technology | Information Technology

Address back problems

Product Structure

General Info (e.g., GD&T)

Geometry

Copyright © 2012 Boeing. All rights reserved. 27 High-end vs Low-end Engineering, Operations & Technology | Information Technology Use Cases Design Review/ Training/ Eng. & E&M/ Tracing systems 3D Scans/ Sales & Mktg

Safety & Survivability

Quality

Assy & /Maint Instructions

Part Dim & Tol/ Part Context

PtCtlParts Catalog

Design in Context

1,000 100,000 1,000,000 Number of Parts (log scale)

C-17? or new 747

787

777

P-8A

IVT/SuperViewer F-18 essentially unlimited

Satellite UG VisMockUp Parts in Context IVT/Standard Edition < 100,000 parts

Typical Part Adobe Reader Browsers UG JT2go < 1,000 parts

10 Million 100 Million 1 Billion Polygon-Equivalents (log scale)

Copyright © 2012 Boeing. All rights reserved. 29 Low-end Visualization Engineering, Operations & Technology | Information Technology . A Low End Visualization capability requires solutions to several related problems: . A place to host the visualization data or the ability to generate appropriate data on the fly. . A low cost mechanism with which a supplier or customer can navigate to and retrieve the model they want to view . . Low cost visualization tool(s). . Identification of a suitable data format.

Engineering, Operations & Technology | Information Technology Description: The Parts Information PIMS View of 787 Data Management System (PIMS) is a web based application designed to deliver 787 Dreamliner customized parts information to our customers in 3D, via the web, to anywhere in the world. PIMS is the next generation implementation of the Illustrated Par ts Ca ta log. PIMS will b e used b y ai rli ne Owners, Operators, Maintenance users world wide.

Data Viewer Functions  Source data from V5  3D Navigation  Delivered Anywhere  User Text Search via the WWW.  Integrated text and graphic  Surface Definition Only  Use of Transparency (f(reference par t)ts)  Animation (parts explosion) Implementation Architecture  PIMS will be available on board aircraft  Multiple customer hardware & operating systems  Requires Browser plug-in from Lattice Technology  Maintenance Level Views authored using IVT & Delmia

Description: The 787 program uses 3D Model-Based – Definition (MBD) processes to define their engineering type design information . Suppliers and customers require the distribution, query, display and retrieval of 3D MBD engineering type design data for downstream applications like manufacturing, inspection and maintenance en ginee ring . EID suppo rts th e query a nd ret rie va l of authoritative 3D MBD in CATIA native format, STEP AP203 and Universal 3D (U3D) PDF formats.

Data Viewer Functions  CATIA V5  Geometry/Product structure associativity  Product structure  Retrieve product  Engineering Authority (Exact structure Geometry and Topology)  Transparency/color  GD&T and notes  MtMeasurement  Open standards  Section Cuts Implementation Architecture  Multiple hardware & operating systems  Adobe Reader (plug-in or standalone) for viewing and interrogg,gpation of the data, Right Hemisphere for translation  Multiple browsers  REDARS + CDW for geometry + product structure

Description: The Shop floor 3D Work Instruction system is designed to deliver work instructions , via the web, to any shop Production View floor user. The shop floor user can interactively rotate, dynamically zoom and interrogate geometry and dimensions.

Data Viewer Functions  UGNX, PROE, CATIA  Product structure search for V4, limited CATIA V5 part numbers  PLM/XML (Product  Material (e.g., transparency, structure) material)  Surface Definition , B -  Animation (parts explosion) REP data  Canned views defined by the  Work instructions Manufacturing Engineer  Dimensioning  Access to unique instance Implementation Architecture attributes  Potential to run when disconnected from a network?  Multiple hardware & operating systems  Multiple browsers via JAVA applet, stand-alone application based on UG Vis tools

Copyright © 2012 Boeing. All rights reserved. 33 High-end Visualization Engineering, Operations & Technology | Information Technology . Where performance and capacity are critical . ‘(‘Instant’ load time (less than 1 minute) . Performance to interactively manipulate up to 1 billion polygons at 10Hz or faster . Product structure with 200,000 separate parts and 2 million instances . Selection and feedback in .25 seconds or less

Copyright © 2012 Boeing. All rights reserved. 34 Potential Applications Engineering, Operations & Technology | Information Technology . Design in context . Design reviews/error checking . Tracing systems . Engineering and manufacturing analysis . Mechanisms, weights, manufacturing simulation, etc. . Electro-maggynetic analysis . Safety . Survivability . Complete part context . Qualit y i nspec tion . Assembly instructions . Massive 3D scans and points . Part dimensions and tolerances . Airplane-in-a-box (totally disconnected) . Human model (large model done in ICIDO) . Part catalogs . Training and familiarization . Maintenance instructions . Sales and marketing (as animation) Copyright © 2012 Boeing. All rights reserved. 35 787 Structure Engineering, Operations & Technology | Information Technology

Description: Many 787 groups, including many outside of engineering rely on IVT to look at collections of parts that range from a few to an entire tail number. In addition to looking at the raw geometry, users can for queries based on Enovia attributes to select parts that have certain characteristics.

Data Viewer Functions  CATIA V5  Partial / whole aircraft  IVT .ft data  Product structure from the PDM  Enovia Product Structure  Metadata query  EiEnovia met tdtadata  GtdAttibtGeometry and Attribute Viewing  No PMI  Measurement Implementation Architecture  Transparency  Windows  Stored Views  IVT  AiAnima tdted sequences

Description: Use the ic.ido immersive review workstation and its human modeling software. The immersive workstation features a passive stereo display and interaction system to enhance the virtual 3D effect. The interactive interface is implemented with an ART hand and head tracking system. Specific 787 product usage has focused on visual verification of human and systems interfaces in highly integrated volumes. Additional support has occurred for design reviews, trade studies, and partner collaboration.

Data Viewer Functions  Exports from CATIA V5 and  Kinematic replay IVT in VRML format  Integrated human model  MPT managed, simplified product structure  Collision avoidance  Visual Accuracy  ItInterf erence d dtetecti on  No PMI  Flex properties  Measurement Implementation Architecture  Microsoft Windows running a standalone application  Specialized display hardware (3D passive stereo powerwall)  ART hand and head tracking, motion capture, and haptics for input  ic.ido supplied application software

Description: AOG teams provide critical repair service around the world. Prior to the 787, teams relied on crates of engineering drawings to get 672Z11620-1 --G engineering detail about damaged areas and support the formal repair plan. A combination of EID and IVT/SuperViewer is replacing the crates. The result is standalone laptop with the full digital definition of a specific 78 7 ta il a n d a ll th e EID fil es fo r t hat ta il.

Data Viewer Functions  CATIA V5  Geometry/Product structure associativity  Product structure  Retrieve product  Engineering Authority (Exact structure Geometry and Topology)  Transparency/color  GD&T and notes  Measurement  IVT/SuperViewer database  Section Cuts  HlikbtIVTHyperlink between IVT and EID data Implementation Architecture  Must run disconnected from the network  Adobe Reader (plug-in or standalone) for viewing and interrogation of the data  IVT with a SuperViewer license  Data resides on USB-connected drive and local hard drive

. Computer graphics has been riding waves for 50+ years . Wave 1: CAD/CAM (1962 - ~1990) . Wave 2: Arts and Entertainment (~1985 – present) . Wave 3: Doesn’t seem to have started yet.

Copyright © 2012 Boeing. All rights reserved. 43 Computer Graphics Drivers Engineering, Operations & Technology | Information Technology . Possible technology drivers for a third wave: . Display technology beyond pixels – Graphics started with fast phosphors and lines and moved to colored dots . Primitives that are higher order than geometry – Better to support visualizing non-geometric data . Emphasis on graphics systems – Ultimate solutions involve complete systems (input, output, applications, data)

Copyright © 2012 Boeing. All rights reserved. 46 Environmental Trends We See Engineering, Operations & Technology | Information Technology . Geometric data: . Boeing has designed in 3D since late 1980’s . Assembly/manufacturing has moved to 3D – While 777 authority data 2D drawings, 3D interference and engineering analysis had profound impact on accurate, no-shim build – 787 authority data annotated 3D and formatted as 3Dpdf . Support mixes 2D and 3D – 777 relies on 2D documents – 787 mixes 2D and 3D . Caveats – Design must pay attention to downstream use – Usinggypg 3D data as authority has implications for long-term data retention (()LOTAR) . Non-geometric data: . Data has numerous instances that must be integrated – Easy data access a challenge . Visual vocabulary is still emerging . Text analysis difficult

Copyright © 2012 Boeing. All rights reserved. 47 Tech Trends We See Engineering, Operations & Technology | Information Technology . Immersive virtual reality . Flight simulators are a way of life . Installed a few early CAVEs for training and design reviews – Can’t close business case . Few large screen installations (video walls, etc.) . Most prefer working at desk . Staff moves frequently and is globally dispersed . Stereo and immersive trying to penetrate marketing, manufacturing ergonomics, and modeling & simulation . Boeing focus has been on 3D capabilities that have widespread usage . Internally developed 3D viewer has 15,000 registered users . 787 ‘engineering drawings ’ are 3D; delivered with Adobe Reader. Overall user community will exceed 100,000 . Input method gap across device classes is widening . WIMP vs. multi-touch vs. tangibles vs. voice vs. … . Mobile devices showing up everywhere . Augmented reality and analytics are gaining momentum

Copyright © 2012 Boeing. All rights reserved. 48 Technical Challenges Engineering, Operations & Technology | Information Technology . While cost is coming down, it can always get lower. . Vo lume of d at a j ust k eeps getti ng l arger. . Input techniques across different screen sizes are asymmetric . For small- and mid-sized screens, multi-touch and WIMP have different vocabulary – Affects both software and users – Causes problems with both thick and thin client . For large screens (immersive or not) – Wear cumbersome hardware (e.g., wands, gloves, HMDs) – Graphically select an object or a command – Enter text or numbers – Workaround is for a dedicated operator . Stereo for long sessions . Stereo sensitivity can be an issue

Plasti cs? Interaction!

. Develop a better understanding of analysis using visualization . Significant opportunities to expand visualization technology in industry . Moving advanced technology into practice is a contact sport