Phasestain: the Digital Staining of Label-Free Quantitative Phase
Total Page:16
File Type:pdf, Size:1020Kb
Rivenson et al. Light: Science & Applications (2019) 8:23 Official journal of the CIOMP 2047-7538 https://doi.org/10.1038/s41377-019-0129-y www.nature.com/lsa ARTICLE Open Access PhaseStain: the digital staining of label-free quantitative phase microscopy images using deep learning Yair Rivenson1,2,3,TairanLiu1,2,3, Zhensong Wei1,2,3, Yibo Zhang 1,2,3, Kevin de Haan1,2,3 and Aydogan Ozcan 1,2,3,4 Abstract Using a deep neural network, we demonstrate a digital staining technique, which we term PhaseStain, to transform the quantitative phase images (QPI) of label-free tissue sections into images that are equivalent to the brightfield microscopy images of the same samples that are histologically stained. Through pairs of image data (QPI and the corresponding brightfield images, acquired after staining), we train a generative adversarial network and demonstrate the effectiveness of this virtual-staining approach using sections of human skin, kidney, and liver tissue, matching the brightfield microscopy images of the same samples stained with Hematoxylin and Eosin, Jones’ stain, and Masson’s trichrome stain, respectively. This digital-staining framework may further strengthen various uses of label-free QPI techniques in pathology applications and biomedical research in general, by eliminating the need for histological staining, reducing sample preparation related costs and saving time. Our results provide a powerful example of some of the unique opportunities created by data-driven image transformations enabled by deep learning. 1234567890():,; 1234567890():,; 1234567890():,; 1234567890():,; Introduction sections2,8, which can be considered a weakly scattering Quantitative phase imaging (QPI) is a rapidly emerging phase object, having limited amplitude contrast modula- field, with a history of several decades in development1,2. tion under brightfield illumination. QPI is a label-free imaging technique, which generates a Although QPI techniques result in quantitative contrast quantitative image of the optical-path-delay through the maps of label-free objects, the current clinical and specimen. Other than being label-free, QPI utilizes low- research gold standard is still mostly based on the intensity illumination, while still allowing for a rapid brightfield imaging of histologically labeled samples. The imaging time, which reduces phototoxicity in comparison staining process dyes the specimen with colorimetric to, e.g., commonly used fluorescence imaging modalities. markers, revealing the cellular and subcellular morpho- QPI can be performed on multiple platforms and devi- logical information of the sample under brightfield – ces3 7, from ultra-portable instruments all the way to microscopy. As an alternative, QPI has been demon- custom-engineered systems integrated with standard strated for the inference of local scattering coefficients of microscopes, with different methods of extracting the tissue samples8,9; for this information to be adopted as a quantitative phase information. QPI has also been diagnostic tool, some of the obstacles include the recently used for the investigation of label-free thin tissue requirement of retraining experts and competing with a growing number of machine learning-based image ana- lysis software10,11, which utilizes vast amounts of stained Correspondence: Aydogan Ozcan ([email protected]) 1Electrical and Computer Engineering Department, University of California, Los tissue images to perform, e.g., automated diagnosis, image Angeles, CA 90095, USA segmentation, or classification, among other tasks. One 2 Bioengineering Department, University of California, Los Angeles, CA 90095, possible way to bridge the gap between QPI and standard USA Full list of author information is available at the end of the article. image-based diagnostic modalities is to perform digital These authors contributed equally: Yair Rivenson, Tairan Liu, Zhensong Wei © The Author(s) 2019 Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a linktotheCreativeCommons license, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this license, visit http://creativecommons.org/licenses/by/4.0/. Rivenson et al. Light: Science & Applications (2019) 8:23 Page 2 of 11 23 Standard histological workflow Histological Brightfield Stained Slide staining microscope brightfield Preparation N O HO OH N N HO OH N OH Methenamine Periodic acid 5500 μm PhaseStain Unstained Slide Obtain quantitative Deep neural Network phase at 550 nm network output … 5050 μm Convolutional 5050 μm neural network Fig. 1 PhaseStain workflow. A quantitative phase image of a label-free specimen is virtually stained by a deep neural network, bypassing the standard histological staining procedure that is used as part of clinical pathology (i.e., virtual) staining of the phase images of label-free it transforms the phase image of a weakly scattering samples to match the images of histologically stained object (e.g., a label-free thin tissue section, which exhi- samples. One previously used method for the digital bits low amplitude modulation under visible light) into staining of tissue sections involves the acquisition of amplitude object information, presenting the same color multimodal, nonlinear microscopy images of the samples, features that are observed under a brightfield micro- while applying staining reagents as part of the sample scope, after the histological staining process. preparation, followed by a linear approximation of the We experimentally demonstrated the success of our absorption process to produce a pseudo-Hematoxylin and PhaseStain approach using label-free sections of human Eosin (H&E) image of the tissue section under investi- skin, kidney, and liver tissue that were imaged by a – gation12 14. holographic microscope, matching the brightfield micro- As an alternative to model-based approximations, scopy images of the same tissue sections stained with deep learning has recently been successful in various H&E, Jones’ stain, and Masson’s trichrome stain, computational tasks based on a data-driven approach, respectively. solving inverse problems in optics, such as super- The deep learning-based virtual-staining of label-free – resolution15 17, holographic image reconstruction and tissue samples using quantitative phase images provide – phase recovery18 21, tomography22, Fourier ptycho- another important example of the unique opportunities – graphic microscopy23, localization microscopy24 26,and enabled by data-driven image transformations. We believe ultrashort pulse reconstruction27. Recently, the appli- that the PhaseStain framework will be instrumental for cation of deep learning for the virtual staining of auto- the QPI community to further strengthen various uses of – fluorescence images of nonstained tissue samples has label-free QPI techniques30 34 for clinical applications also been demonstrated28. Following the success of and biomedical research, helping to eliminate the need for these previous results, here, we demonstrate that deep histological staining, and reduce sample preparation learning can be used for the digital staining of label-free associated time, labor, and costs. thin tissue sections using their quantitative phase ima- ges. For this image transformation between the phase Results image of a label-free sample and its stained brightfield We trained three deep neural network models, which image, which we term PhaseStain, we used a deep neural correspond to the three different combinations of tissue network trained using the generative adversarial net- and stain types, i.e., H&E for skin tissue, Jones’ stain for work (GAN) framework29. Conceptually, PhaseStain kidney tissue, and Masson’s trichrome for liver tissue. (see Fig. 1) provides an image that is the digital Following the training phase, these three trained deep equivalent of a brightfield image of the same sample networks were blindly tested on holographically recon- after the histological staining process; stated differently, structed quantitative phase images (see the Methods Rivenson et al. Light: Science & Applications (2019) 8:23 Page 3 of 11 23 QPI of label-free skin tissue section Network output — digital H&E staining 4/5 Trained network 100 μm 0 110000 μm Brightfield image of the QPI of label-free skin Network output — H&E histologically stained tissue section (zoom in) digital H&E staining skin tissue (20×/0.75NA) 4/5 μ 50 m 5050 μm 5050 μm 0 Fig. 2 Virtual H&E staining of label-free skin tissue using the PhaseStain framework. Top: QPI of a label-free skin tissue section and the resulting network output. Bottom: zoom-in image of a region of interest and its comparison to the histochemically stained gold standard brightfield image Brightfield image of the QPI of label-free Network output — histologically stained tissue section digital staining tissue (20×/0.75 NA) 4/5 Kidney Jones’ stain μ μ 5500 m 0 5500 m 4/5 Liver Masson’s trichrome μ μ 5050 m 0 5050