Mobile Trends and Novel Applications in Digital Pathology

Liron Pantanowitz MD Matthew G. Hanna MD Professor of Pathology & Biomedical Informatics Clinical Instructor/Pathology Informatics Fellow University of Pittsburgh Medical Center University of Pittsburgh Medical Center, Pittsburgh, USA Pittsburgh, USA Notice of Faculty Disclosure

In accordance with ACCME guidelines, any individual in a position to influence and/or control the content of this ASCP CME activity has disclosed all relevant financial relationships within the past 12 months with commercial interests that provide products and/or services related to the content of this CME activity.

The individual below has responded that he/she has no relevant financial relationship(s) with commercial interest(s) to disclose:

Liron Pantanowitz, MD Hamamatsu ConsultantConsultant Fee Hologic ConsultantConsultant Fee Objectives

1. Discuss the current impact of mobile health on digital pathology applications 2. Discuss emerging applications of different wearable technology in pathology 3. Demonstrate several pathology use cases with the HoloLens Handheld Computing Park et al. J Pathol Inform 2012, 3:15 • Handheld computing has been around since the 1980’s. • Millions of smartphone users worldwide use health care apps. • Mobile computers today no longer just include smart phones & tablets. • Wearable technology (tech togs) is supported by ubiquitous computing & wireless connectivity (e.g. Bluetooth); e.g. eyeglasses & watches. • Important components include a custom CPU, sensors, imaging optics (display), microphone, network device, and integrated camera. • Emerging mobile health (mHealth) era presents new opportunities & challenges in Pathology. Cell Phone Attachments Changing cell phones into small, high-resolution, handheld microscopes Mobile Cell Phone Telemicroscopy Bellina & Missoni. Diag Pathol. 2009; 4:19

Image captured with m-phone directly from microscope's ocular. Plasmodium falciparum rings in erythrocytes (May-Grünwald-Giemsa stain0 M-phone and MMS penetration in Africa. M-phone subscriptions with declared availability of Multimedia Messaging Service (MMS, per 100 inhabitants. Map elaborated by the authors based on last available information (2007–2009)

Gosky $20 Carson $60 Orion $50 SVBONY $10

Solomark $11 MeoPix $70 Skylight $75 LabCam $250 Smartphone Adapter Images Roy et al. J Pathol Inform 2014, 5:24

iPhone & adapter (a+b = Magnifi) & (c+d = Skylight) UPMC Mobile Device Teleconsultation Hartman DJ et al. J Pathol Inform. 2014; 5:10

• Our app (Pocket Pathologist) launched in Aug 2013 • App augments functionality of our web- based portal (https://pathconsult.upmc.com) • Users can quickly upload digital images from their phone's gallery for teleconsultation Smartphone-Microscope Camera Adapter: An option for cytology consultation in low-resource environments Miller et al. JASC 2012; 1 (supplement):S124-5

• SkyLight smartphone adapter • 20 cases (10 Paps & 10 FNAs) • 1 static image/case reviewed • 64% diagnostic accuracy Live Streaming via FaceTime FaceTime validation study: Low-cost streaming video for cytology adequacy assessment Agarwal et al. Cancer Cytopathol 2016; 124:213-20

Setting: - iPhone/iPad + microscope adapter + FaceTime Method: - FNAs on-site reviewed with video streaming Results: - Retrospective review 22/25 (88%) agreement - Prospective review 14/14 (100%) agreement - Problems: software, camera alignment & audio Conclusion: - FaceTime can be used for ROSE Wearable Technology Battery Speakers (phone) CPU GL Λ SS TM TM Microphone Weighs42 only grams Digital Camera http://www.google.com/glass Prism

GLASS for Pathology?

• Aim at UPMC to explore the utility in Anatomic Pathology • ISD approval to obtain secure network connectivity • Use extended with 9000mAh USB external battery pack • Google+ hangout account created for multiple users & devices Instant messaging Users can perform a & video chat group video chat platform with multiple users at the same time Autopsy

• Pathologist in the morgue initiated video calls via Google hangout • Remote pathologists viewed live feeds from autopsy suite • Streaming video & audio quality to participants in other hospitals was clear • Hands-free capturing of static images was easy • Image quality of organs being examined was acceptable at 2-3 foot distance from the specimen Google Glass use during autopsy Autopsy Findings Syed et al. J Pathol Inform 2014, 5:S43-S44 • Finger-swipe navigations were difficult because of bloody gloves • Distracting background noise in the autopsy suite from ventilation, echoing & multiple • Voice navigation failed when the wearer used a surgical mask or shield • Problems can be alleviated by: – Adding headset microphone – Installing clear eye lenses provided with the glasses instead of a protective face shield Gross Pathology

• Glass during grossing allowed live video & audio feeds to be transmitted to users via our Google+ hangout

• Difficulties experienced: – Optimal lighting – Image resolution – Taking snapshots (during video calls) – Hearing voice clearly (noisy gross room) Training Videos

Acknowledgement Brad Clifford UPMC Glass-LIS Interface Taylor et al. J Pathol Inform 2015, 6:S2-S3 UPMC Glass Modality Benefits

• Useful in situations where the user cannot use their hands (e.g. during grossing, autopsy, FS scenarios). • These custom interfaces allowed: – Abstract query of patient & metadata – Storage of GLASS captured images – Integration with any UPMC information systems (LIS, PACS) Levels of Reality . Reality . The real world that the user is surrounded by . Augmented reality (AR) . Real-time integration of digital data with user’s reality . Digital overlay upon real surroundings . Virtual reality (VR) . Total artificial digital environment . Natural surroundings not visible Virtual Reality (VR)

. Replaces users entire (audio)visual sense with a complete simulated environment . Stereoscopic view for each eye to view a virtual screen that typically encompasses the users entire visual field . All viewable data is digital . Interactions have to be made through additional hardware . Increased likelihood of simulator sickness Augmented Reality (AR)

. Overlays digital data atop the user’s visual field . Natural surroundings are preserved . Visual field includes native world and digital data . Interactions can be made in the visual field What is Augmented Reality? Mixed Reality

. Subtype of AR . Key distinction: . Seamless integration of virtual data with user’s perception of the real world . i.e. walking around a “moving” virtual dog placed on the floor Oculus Rift from Facebook

Virtual reality head-mounted display VR for Pathology?

• WSI are typically viewed in 2D on a monitor. • Is a virtual reality (VR) environment better? • Aim: Explore OR for examining digital slides.

Oculus Rift Development Kit 2

UPMC Study: Methods

• Oculus Development Kit 2 (DK2) connected to a 64-bit computer • HP Z440 Workstation • Intel Xeon E5-1650v3 • 32GB DDR4-2133 • NVIDIA GeForce GTX Titan X GPU • 512GB PCIe SSD • 1TB 7200RPM HDD • Running Windows 7 & Virtual Desktop • Glass slides from 10 lymph nodes (5 benign, 5 malignant) were scanned • 3 pathologists reviewed WSI on: • 24-inch LCD monitor (HP ZR24w) • With OR (Figure) • Diagnoses & time to read slides recorded • Rated image quality (1-10), ease of navigation (1-10) & diagnostic confidence (1-10) for both modalities UPMC Study: Results

• WSI viewable in VR environment using the OR • 100% diagnostic concordance with both modalities • Time to read WSI: – LCD monitor: Average 80 seconds (range 45 sec – 3½ min) – Oculus Rift : Average 74 seconds (range 20 sec – 2min 40 sec) • Pathologists similarly rated image quality, ease of navigation, and diagnostic confidence for both sessions UPMC Study: Conclusion Farahani et al. J Pathol Inform 2016, 7:22 • Using Oculus Rift to view and navigate pathology WSI in a virtual environment is feasible • However, image quality is limited by the device’s resolution: – DK2 is just 1920x1080 (960x1080 per eye) • Although the VR atmosphere provided pathologists with a wide field of view that simulates microscopy, reading slides with this novel display did not enhance their navigation of digital slides • Viewing WSI in 2D on a computer monitor is equivalent to a VR environment Farahani et al. J Pathol Inform 2016, 7:S28-S29 HoloLens from Microsoft

A headset that projects holographic images into real space Using the HoloLens at UPMC Hanna et al. Modern Pathology 2017; 30 (suppl):396A-397A Use cases: . Autopsy . Telepathology . Frozen section . Gross examination . Virtual gross workstation . 3D holographic specimens . Whole slide image viewing . Radiology-Pathology correlation Autopsy

. Transportable hand’s free bi-directional audio & visual communication. . for HoloLens Beta . Microsoft Surface Pro . Successful bi-directional supervision & consultation. . Bi-directional virtual annotations. . Different user colors: blue & green . PPE modifications (e.g. face shield). . Cannot cover sensors

Telepathology

. Remote gross examination . Access to ancillary resources . Remote supervision . Remote intraoperative consultation . Accurate selection of areas from specimen for frozen section Gross Examination Virtual Gross Workstation

. Transportable, hands-free, wearable “computer”. . Voice recognition (non speech-to-text commands) . Speech-to-text dictation not yet supported . Access UPMC secure network applications: . LIS (CoPath, Cerner) . EMR (Cerner & Epic) . Whole slide imaging . Multiple dynamic windows (in/out user’s field of view) Virtual Gross Workstation 3D Holographic Specimens

. 3D scanning (gross specimens & autopsy brain) . Imported FBX file to HoloLens . Microsoft 3D Viewer application . Manipulate 3D images via hand gestures

Whole Slide Image Viewing

. Multiple WSI viewers tested – ImageScope, DigitalScope, etc

. Remote desktop application – Remote Desktop Preview App Radiology-Pathology Correlation

. To demonstrate time efficiency of identifying metallic breast biopsy clips using routine gross examination versus with the HoloLens. . PAs were requested to use the HoloLens to superimpose specimen radiographs atop gross specimens: . N=24 specimens . All cases had biopsy clip placement . End point: Time to clip identification Radiology-Pathology Correlation Path-Rad Correlation Outcome

. Identification of metallic biopsy clip (average): . Using conventional workflow: 10.6 minutes . Using the Microsoft HoloLens: 1.5 minutes . 100% of users rated Microsoft HoloLens ≥4/5 . Useful and usable technology . For radiograph-specimen co-registration HoloLens Demo: Pathology Use Cases at UPMC Take Home Message

•• TheThe HoloLens HoloLens is is better better than than Glass Glass & & OculusOculus Rift Rift for for digital digital pathology. pathology. •• DoesDoes wearable wearable technology technology have have real real potentialpotential for for pathology pathology practice practice or or is is thisthis just just another another gimmick? gimmick? •• SoSo far, far, wearable wearable technology technology with with VR/ARVR/AR feasible feasible niche niche applications applications include:include: – Education & simulation – Education & simulation – Telepathology – Telepathology – Novel 3D imaging – Novel 3D imaging – Path-Rad correlation – Path-Rad correlation Q&A

If you want to contact us: [email protected] [email protected]