DICOM Encapsulated X3D/VRML

Review of WG17 Work Item 1 Output: DICOM Encapsulated X3D/VRML

Allan Noordvyk & Justin Ryan Co-Chairs of WG17: 3D Manufacturing Agenda

• WG17 Mandate & Work Item • Specifics of Proposal • New IOD / SOP • Other related changes • Expected use by community • Strategy for adoption by vendors • Larger WG17 context & future • Open Questions

Working Group 17 3D Manufacturing 2 WG17 & Work Item

Our mandate is to facilitate a way to store/query/retrieve 3D models, intended for 3D manufacturing, as DICOM objects (Work Item 1).

The approved Work Item is focused on (a) leveraging the existing and growing ecosystem of DICOM-capable systems in use in healthcare institutions and (b) leveraging standards already in use in the 3D printing industry.

Working Group 17 3D Manufacturing 3 Specifics of Proposal

New Information Object Definition (IOD): • Encapsulated native 3D file (X3D/VRML) for Creation, Review, Update, and Printing (manufacturing)

New Service Object Pair (SOP): • Encapsulated X3D Storage

Working Group 17 3D Manufacturing 4 Specifics of Proposal (cont’d)

Other extensions to DICOM standard • Contain references to existing IODs from which the model is derived (which may be from multiple studies) • Allow users to review 3D printing models for accuracy/relevance prior to printing • Include a 2D preview representation • Indicate whether the 3D printed model contains protected health information

Working Group 17 3D Manufacturing 5 Specifics of Proposal (cont’d)

Why X3D/VRML in addition to STL (Supp. 205) • STL defines ONLY topology • X3D/VRML can define topology AND color • X3D utilizes XML schema • Human Readability • Extensibility for metadata out of scope for DICOM – Material properties – Machine metadata – Other metadata out of scope • Preferred file for 3D Systems technology • Discovered through WG17 ballot of users and vendors

Working Group 17 3D Manufacturing 6 Specifics of Proposal (cont’d)

Example of potential X3D:

Working Group 17 3D Manufacturing 7 Expected Use

The new IOD/SOP is expected to address these real world use cases:

• Creation • Review • Update • Print

Working Group 17 3D Manufacturing 8 Expected Use (continued)

Use Case 1: Creation PACS/VNA • Medical reconstruction software queries Image manger system • User creates patient-specific 3D model (reconstruction and modeling) • Modeler system creates the new type DICOM object containing the 3D model • Same modeler system or additional system adds important additional features (list of potential features): – Color information – Modeler system metadata – Material metadata – Machine metadata – Other metadata out of DICOM scope PACS/VNA

Working Group 17 3D Manufacturing 9 Expected Use (continued)

PACS/VNA Use Case 1: Creation (cont’d) • User saves 3D model back to the patient’s record in DICOM format as either (a) an addition to an existing study or (b) a new study • The Modeler system stores the new DICOM object in the Image Manager system

PACS/VNA

Working Group 17 3D Manufacturing 10 Expected Use (continued)

Use Case 2: Review • At a later time to Use Case 1, a user PACS/VNA indicates desire to visually review a 3D model , prior to 3D printing Display System • The Display system queries the Image Manager for the DICOM objects of new type • The Display system retrieves the indicated object • The Display system extracts the 3D model from the object and displays it to the user, potentially registered for simultaneous display with source images

– The Display system may also illustrate specific colors as defined by the file

– The Display system may show additional metadata

Working Group 17 3D Manufacturing 11 Expected Use (continued)

Use Case 3: Update PACS/VNA • At a later time to Use Case 1, a user indicates desire to modify a 3D model for a particular patient • The Modeler system queries the Image Manager for the DICOM objects of new type • If necessary, the Modeler system retrieves any source images (s1 to sN) required for this modification to occur • User interacts with the Modeler system to adjust the 3D printable model as desired, such as changes in color information or other data as listed in Use Case 1

Working Group 17 3D Manufacturing 12 Expected Use (continued)

Use Case 3: Update (cont’d) • User saves back to the patient’s record in DICOM format as either (a) an addition to an existing study, or (b) a new study • The Modeler system creates the new type DICOM object containing the new version 3D model, populating all required metadata and including a unique identifier reference to the supplanted earlier 3D print model object • The Modeler system stores the new DICOM object in the Image Manager system PACS/VNA

Working Group 17 3D Manufacturing 13 Expected Use (continued)

Use Case 4: Print PACS/VNA • At a later time to Use Case 1, a user indicates desire to print a 3D model for a particular patient • The Print Manager system queries the Print Manager Image Manager for the DICOM objects of new type belonging to the patient • The Print Manager system retrieves the indicated 3D print model object • The Print Manager access the 3D model information within the object, using this to create non-DICOM print instructions for a specific 3d printer

– Deepening on the print technology, the Print Manager may send color or material specific instructions to print as encoded in the file.

Working Group 17 3D Manufacturing 14 Expected Use (continued)

Use Case 4: Print (cont’d) Print Manager • The Print Manager prompts the user for any necessary additional print parameters (e.g. support, bed placement, material parameters, etc.) • The Print Manager submits the print job to the printer • Optionally, the Print Manager may save an updated 3d print object back to the Image Manager in order to preserve exact print parameters used (per Use Case 3, steps 7+).

PACS/VNA

Working Group 17 3D Manufacturing 15 New Module

• Utilizes Module: Manufacturing 3D Model • Utilizes 3 attributes introduced for STL encapsulation: • Measurement Units Code Sequence • Model Modification • Model Mirroring • Model Usage • Add the following attributes for X3D/VRML: • [Texture map?]

Working Group 17 3D Manufacturing 16 New Module Attribute 1

Texture Map Options – X3D • Overview

X3D – 3D Model url in X3D*

Texture Texture Texture … Map A Map B Map n } image files

*Convention (not specification-drive) for Medical Modeling software to have all files in the same folder.

Working Group 17 3D Manufacturing 17 Larger WG17 Context

File Format Scale Potential Strengths In Use Cases Accomp. Files X3D/VRML m JPG, TIF, PNG, High use for 3D Systems3D TGA, SVG, printers. Enables color (texture map) models. Not utilized by GIF, BMP Stratasys technology. X3D is HTML5 compatible & ISO/IEC- 19775-1 standard. OBJ undef MTL*, JPG, High use for Stratasys 3D printers. TIF, PNG, TGA, Enables color (texture map) models. Not utilized by 3D Systems SVG, GIF, BMP technology. Utilized by some VR/AR/ MR technology. 3MF mm JPG, PNG Best perceived future 3D printing format based on WG17 ballot. 3MF is supported by Seimens, Microsoft, Autodesk, HP, and more.

*MTL is a mandatory accompanying file for an OBJ to have color (texture map).

Working Group 17 3D Manufacturing 18 Larger WG17 Context

Texture Map Options – 3MF • Overview

3MF – 3D Model url to maps in 3MF*

Texture Texture Texture … Map A Map B Map n } image files

*Convention (not specification-drive) for Medical Modeling software to have all files in the same folder.

Working Group 17 3D Manufacturing 19 Larger WG17 Context

Texture Map Options - OBJ • Overview

OBJ – 3D Model

url to MTL in OBJ

MTL – Material Library File url to maps in MTL

Texture Texture Texture … Map A Map B Map n } image files

Working Group 17 3D Manufacturing 20 New Module Attribute 1

Texture Map Options • Strong preference to avoid multi- instanced modeling • Potential for high amount of unnecessary instances (texture maps may be recreated if exporting is not optimized) • Potential transmission issues (sending the X3D without the texture map instances

Working Group 17 3D Manufacturing 21 Texture Map Options

Description Pros Cons A Add a new “encapsulated support • Separates the “main” model • Risks becoming a “grab bag” document sequence” & populate with file (in the encapsulated • Unclear if there is some place to store the model the support files document sequence) from the filename/url, which may be an issue if any of the support files. formats need to be able to refer from a support • Creating a new instance for a file back to the model. changed model, requires changing less of the previous instance, since if support sequence has not changed, that part doesn’t need to be updated. B Add a new “texture image sequence” • Tighter definition of the • Because this is a dedicated sequence for texture akin to the “icon image sequence” attributes to prevent misuse. maps, OBJ format, which has texture maps and • Creating a new instance for a a “library” support file, would need to define changed model, requires another location to store the library file. Other changing less of the previous formats might have other file types and thus instance, since if support might need yet more extensions. sequence has not changed, • Unclear if there is some place to store the model that part doesn’t need to be filename/url, which may be an issue if any of the updated. formats need to be able to refer from a support file back to the model.

Working Group 17 3D Manufacturing 22 Texture Map Options

Description Pros Cons C Make the “encapsulated document” a • No new attributes needed • Required SCU/SCP to implement a zip library zip file and include the model and • No need to worry about file • Potentially more difficult to get at the support files in the zip cross-references since that information (e.g., to display only the model would be handled by file without texture maps or to access only a names inside the zip particular map) • higher risk of total loss in case of corruption • Creating a new instance for a changed model, requires changing all of the encapsulated document sequence.

D Make the “encapsulated document” • No new attributes needed • May be more difficult to seek to the main multipart MIME and include the • No need to worry about file model file in the byte stream model and support files as parts cross-references since that • Creating a new instance for a changed model, would be handled by file requires changing all of the encapsulated names inside the multi-part document sequence. body • Potentially more difficult to get at the information (e.g., to display only the model without texture maps or to access only a particular map)

Working Group 17 3D Manufacturing 23 Strategy for Adoption

• Success depends on both: • Traditional PACS/VNA vendors, • Standards-development Organizations, and • Web3D Consortium is an observing member • 3D print model software vendors

Working Group 17 3D Manufacturing 24 Strategy for Adoption (cont’d)

• Groundwork for success is as follows • Enable traditional PACS/VNA vendors and new 3D print model software vendors to participate in new IOD development • Enable user community to push their vendors to adopt the new IOD via: – User group meetings – Conferences – Society meetings (RSNA, SME, WAM3DP, etc.)

Working Group 17 3D Manufacturing 25 Larger WG17 Context

Work Item requires us to support more than just STL: • STL captures geometry as a triangular mesh, but contains no metadata (e.g. units) or color • G-CODE, VRML/X3D, AMF, 3MF, PLY, OBJ/MTL, VTK, FBX and U3D/PDF • Some have prohibitive licensing Future Encapsulation Prospects • Next proposals: OBJ/MTL and 3MF • contain metadata, color information, enhanced geometrical representations • Different user groups

Working Group 17 3D Manufacturing 26 End of Presentation

Questions?

Working Group 17 3D Manufacturing 27 End of Presentation

Appendix A: Supplement 205

Working Group 17 3D Manufacturing 28 Expected Use

The new IOD/SOP is expected to address these real world use cases:

• Creation • Review • Update • Print

Working Group 17 3D Manufacturing 29 Expected Use (continued)

Use Case 1: Creation PACS/VNA • Medical reconstruction software queries Image manger system • User creates patient-specific 3D model (reconstruction and modeling) • Modeler system creates the new type DICOM object containing the 3D model, populating all required metadata • User saves 3D model back to the patient’s record in DICOM format as either (a) an addition to an existing study or (b) a new study • The Modeler system stores the new DICOM object in the Image Manager system PACS/VNA

Working Group 17 3D Manufacturing 30 Expected Use (continued)

Use Case 2: Review PACS/VNA • At a later time to Use Case 1, a user indicates desire to visually review a Display System 3D model , prior to 3D printing • The Display system queries the Image Manager for the DICOM objects of new type • The Display system retrieves the indicated object • The Display system extracts the 3D model from the object and displays it to the user, potentially registered for simultaneous display with source images

Working Group 17 3D Manufacturing 31 Expected Use (continued)

Working Group 17 3D Manufacturing 32 Expected Use (continued)

Working Group 17 3D Manufacturing 33 Expected Use (continued)

Use Case 4: Print PACS/VNA • At a later time to Use Case 1, a user indicates desire to print a 3D model for a particular patient Print Manager • The Print Manager system queries the Image Manager for the DICOM objects of new type belonging to the patient • The Print Manager system retrieves the indicated 3D print model object • The Print Manager access the 3D model information within the object, using this to create non-DICOM print instructions for a specific 3d printer (e.g. .stl)

Working Group 17 3D Manufacturing 34 Expected Use (continued)

PACS/VNA

Working Group 17 3D Manufacturing 35 New Module

• Introduces Module: Manufacturing 3D Model • Addresses 3D model (3DM) information not present in either current DICOM or STL model data formats • Contains 3 attributes: • Measurement Units Code Sequence • Model Modification • Model Mirroring • Model Usage • We expect this module will be used in future additional 3D manufacturing related IODs Working Group 17 3D Manufacturing 36 New Module Attribute 1

Measurement Units Code Sequence • Specifies how the unit scale of the STL file should be interpreted • Removes ambiguity Allowed Values: • m • cm • mm • um

Working Group 17 3D Manufacturing 37 New Module Attribute 2

Model Modification • Specifies whether modification (beyond Enumerated Values: mirroring) was used to • Yes create the model • No • Specifies whether the object represents the observed patient anatomy in the source data

Working Group 17 3D Manufacturing 38 New Module Attribute 2

Model Mirroring • Specifies whether

mirroring of anatomy Enumerated from the other side of Values: • Yes the patient was used to • No create the model

Working Group 17 3D Manufacturing 39 New Module Attribute 3

Model Usage • Specifies the use for which the manufactured object is initially intended.

Defined Terms: • Education • Diagnosis/Planning • Instrument • Prosthetic/Orthotic • Implant • Quality Control • Simulation

Working Group 17 3D Manufacturing 40 New Module Attribute 3: Terms

Model Usage: Defined Terms

EDUCATION Modeled object is intended for patient or care-giver education / informed consent, or training residents and fellows DIAGNOSIS/PLANNING Modeled object is intended to be used to assist with diagnosis and/or pre-operative procedure planning INSTRUMENT Modeled object is intended to be used as a patient-matched intra-operative tool during a medical procedure PROSTHETIC/ORTHOTIC Modeled object is intended to be used as a fully external prosthetic/orthotic IMPLANT Modeled object is intended to be used as a wholly or partially internal implant QUALITY CONTROL Modeled object is intended to be used for quality control purposes SIMULATION Modeled object is intended to be used for simulation and/or practice of a surgery or other medical procedure.

Working Group 17 3D Manufacturing 41 Other Aspects Addressed

• Patient Confidentiality • Clarified use of existing Burned in Annotation and Recognizable Visual Features attributes as they apply to 3D models • Preview Image • Desired to allow for visual selection of different models in same study without need to load, interpret and render • Pre-rendered image optionally supplied when IOD created • Leveraged existing method - (0088, 0200) Icon Image Sequence – to illustrate pre-rendered preview image

Working Group 17 3D Manufacturing 42 Strategy for Adoption

• Success depends on both: • Traditional PACS/VNA vendors, and • 3D print model software vendors

• Both must adopt the new standard simultaneously

Working Group 17 3D Manufacturing 43 Strategy for Adoption (cont’d)

Working Group 17 3D Manufacturing 44 Larger WG17 Context

Work Item requires us to support more than just STL: • STL captures geometry as a triangular mesh, but contains no metadata (e.g. units) or color • G-CODE, VRML/X3D, AMF, 3MF, PLY, OBJ/MTL, VTK, FBX and U3D/PDF • Some have prohibitive licensing Future Encapsulation Prospects • Ballot ended for next proposals: OBJ/MTL, VRML/X3D, and 3MF • contain metadata, color information, enhanced geometrical representations

Working Group 17 3D Manufacturing 45 Open Question #5

Would inclusion of selected attributes of the implant description module be beneficial for 3D models that are intended to be implants?

Working Group 17 3D Manufacturing 46 Open Question #6

Would there be meaningful benefit from incorporating further implant detail via the use implant defined terms, code sets, or other encoding (e.g. CID 73xx family)? • Authors left out such level of detail in the interest of simplicity. However, a hip assembly can contain 20 parts. It is possible that it would be useful to identify which part is represented by which model. However, existing means of identification (e.g. Document Title) may be sufficient for real-world use.

Working Group 17 3D Manufacturing 47 Open Question #7

Is the existing use of Frame-of-Reference adequate to handle registration in all cases? • Authors believe it does. Also, follows current conventions of 3D modeling software (model uses coordinate system of primary source series).

Working Group 17 3D Manufacturing 48 Open Question #8

The Model Usage attribute does not distinguish between different types of instrument guides. Is there actual value to adding additional granularity (e.g. distinguishing needle, blade, drill guides) sufficient to justify additional complexity? • The additional granularity was initially included, but later omitted by the authors in the interest of simplicity.

Working Group 17 3D Manufacturing 49 Open Question #9

The Model Usage attribute does not distinguish between different levels of permanence for implants. Is there actual value to adding additional granularity (e.g. distinguishing absorbable, removable, and permanent implants) to justify additional complexity? • The additional granularity was initially included, but later omitted by the authors in the interest of simplicity.

Working Group 17 3D Manufacturing 50 Open Question #10

The Module Usage attribute is included to help filter when multiple models are returned a query, routing for handling, etc. Are the values selected sufficient for these use cases?

Working Group 17 3D Manufacturing 51 Open Question #11

Is reference to a small number of representative images sufficient for the goal of supporting linked comparison? • Our options thus appear to be (a) require only a small number of representative images from each source series or (b) define a different attribute that represents a series-level reference.

Working Group 17 3D Manufacturing 52 Open Question #12

Is using only metric units from um to sufficient for to identify any scale in routine use for medical and non- destructive-testing purposes? • We only used metric units for patient safety reasons and to match current clinical practice (which appears to universally use either mm or m). • We created a new context group specifically for 3D printing scale (7xx3). This ensures that other units could be added in the future should a new use case arise.

Working Group 17 3D Manufacturing 53 Open Question #13

Related to the question above. Is a multiplication factor for unit scale also required? • For example, there is no way to specify that the unit scale of the model is 2.54 mm. Instead the creator of the STL would need to divide all coordinates in the file by 2.54 and specify the unit scale as mm.

Working Group 17 3D Manufacturing 54 Open Question #14

What other derivations need to be added to LOINC for 3D print models? Currently only CT and MR derived models can be coded using LOINC references. Also, what is the best way to capture fused multi-modality fusion models in a single LOINC code? • Some individuals are working with 3D printing from ultrasound. We assume it should be added to LOINC.

Working Group 17 3D Manufacturing 55