sign in sign up
<<
Home , Microscopy, Molecular vibration

All-digital histopathology by -optical hybrid

Martin Schnella, Shachi Mittala,b, Kianoush Falahkheirkhaha,c, Anirudh Mittala,b, Kevin Yeha,b, Seth Kenkela,d, Andre Kajdacsy-Ballae, P. Scott Carneyf, and Rohit Bhargavaa,b,c,d,g,h,i,1

aBeckman Institute for Advanced Science and Technology, University of Illinois at Urbana–Champaign, Urbana, IL 61801; bDepartment of Bioengineering, University of Illinois at Urbana–Champaign, Urbana, IL 61801; cDepartment of Chemical and Biomolecular Engineering, University of Illinois at Urbana– Champaign, Urbana, IL 61801; dDepartment of Mechanical Science and Engineering, University of Illinois at Urbana–Champaign, Urbana, IL 61801; eDepartment of , University of Illinois at Chicago, Chicago, IL 60612; fThe Institute of Optics, University of Rochester, Rochester, NY 14620; gCancer Center at Illinois, University of Illinois at Urbana–Champaign, Urbana, IL 61801; hDepartment of Electrical and Computer Engineering, University of Illinois at Urbana–Champaign, Urbana, IL 61801; and iDepartment of Chemistry, University of Illinois at Urbana–Champaign, Urbana, IL 61801

Edited by Christian Huck, University of Innsbruck, Innsbruck, Austria, and accepted by Editorial Board Member John A. Rogers December 20, 2019 (received for review July 19, 2019) Optical microscopy for biomedical samples requires expertise in fidelity imaging in both point-scanning (15) and wide-field (16–19) to visualize structure and composition. Midinfrared (mid-IR) modalities, IR pixel sizes are ∼100-fold larger than those easily spectroscopic imaging offers label-free molecular recording and achieved in visible microscopy, and data acquisition speed is still virtual staining by probing fundamental vibrational modes of three to four orders of magnitude slower than in visible imaging molecular components. This quantitative signal can be combined (20). The abundance of applications, ease of use, and ubiquity of with machine learning to enable microscopy in diverse fields visible microscopy points to the significant potential of an inte- from cancer diagnoses to forensics. However, absorption of IR grated approach with IR . by common optical imaging components makes mid-IR light Several transformative applications of a hybrid of visible mi- incompatible with modern optical microscopy and almost all croscopy and IR spectroscopy are apparent. For example, the biomedical research and clinical workflows. Here we conceptualize microscopic examination of stained tissue using visible micros- an IR-optical hybrid (IR-OH) approach that sensitively measures copy has been the standard method for detecting and grading molecular composition based on an optical with wide- most forms of human cancer in research and clinical care (21, 22). CHEMISTRY field interferometric detection of absorption-induced sample expan- Combinations of staining and microscopy with advanced artificial sion. We demonstrate that IR-OH exceeds state-of-the-art IR micros- copy in coverage (10-fold), spatial resolution (fourfold), and spectral intelligence (23, 24) now provide new capabilities, but this avenue consistency (by mitigating the effects of ). The combined is ultimately limited by available exogenous labels. Complemen- impact of these advances allows full slide infrared absorption im- tarily, IR imaging (25) offers detailed molecular contrast without ages of unstained breast tissue sections on a visible microscope the need to stain tissue, and its combination with machine learning – platform. We further show that automated histopathologic segmen- is being used to augment and automate histopathology (26 28), tation and generation of computationally stained (stainless) images discover cell physiology (29), inform therapeutic decisions (30), and MEDICAL SCIENCES is possible, resolving morphological features in both color and spa- aid discovery of new biomarkers (31). Making visible microscopy tial detail comparable to current pathology protocols but without stains or human interpretation. IR-OH is compatible with clinical and Significance research pathology practice and could make for a cost-effective alternative to conventional stain-based protocols for stainless, This study reports the ability to provide label-free molecular all-. information from infrared (IR) spectroscopy via the ubiquitous . Modeling the thermal-mechanical coupling | imaging | quantum cascade | pathology | of samples, we design, build, and validate an IR-optical hybrid (IR-OH) microscope that uses optical interferometry to measure the dimensional change in materials arising from spectral ab- ptical microscopy is ubiquitous in biomedical research for sorption. We show that the seamless compatibility of IR-OH with Othe examination of microscopic structure of tissues and forms routine optical microscopy and with emerging computational a cornerstone of all development and disease studies and much ubiquity enables all-digital pathology with applications across medical decision-making (1). Visualizing structure and chemical the spectrum of biomedical science. IR-OH microscopy provides a composition in biomedical samples, however, requires the use of means to retain the ease of use and universal availability of stains or labels. Label-free optical techniques have been proposed optical microscopy, add a wide palette of IR molecular contrast, to visualize molecular content without or labels to monitor and utilize emerging computational capabilities to change how processes unperturbed, to observe composition that is not easily we routinely handle, image, and understand microscopic tissue amenable to staining (2). Molecular spectroscopy through more structure. than 125 y of progress (3) enables the absorption of midinfrared (mid-IR) light to be used as an identifying molecular signature (4). Author contributions: M.S., P.S.C., and R.B. designed research; M.S., S.M., K.F., A.M., K.Y., S.K., A.K.-B., and R.B. performed research; M.S., S.M., K.F., K.Y., and A.K.-B. contributed Extant applications of mid-IR microscopy span biomedical tissue new reagents/analytic tools; M.S., S.M., K.F., A.M., S.K., P.S.C., and R.B. analyzed data; and diagnostics (5) and cell (6) to polymeric (7), (8), and M.S., S.M., P.S.C., and R.B. wrote the paper. forensic samples (9) and interstellar analyses (10) over nearly 70 y The authors declare no competing interest. (11). IR imaging, however, is incompatible with visible microscopy. This article is a PNAS Direct Submission. C.H. is a guest editor invited by the Detectors used in visible and near-IR imaging are nonresponsive, Editorial Board. and most are highly absorbing over the IR spectral band Published under the PNAS license. (2 to 12 μm), precluding the use of commonly available imaging 1To whom correspondence may be addressed. Email: [email protected]. components that have advanced other spectroscopic approaches This article contains supporting information online at https://www.pnas.org/lookup/suppl/ (12). Despite exciting recent advances in quantum cascade doi:10.1073/pnas.1912400117/-/DCSupplemental. (QCL) (13, 14) providing a powerful impetus for rapid, high-

www.pnas.org/cgi/doi/10.1073/pnas.1912400117 PNAS Latest Articles | 1of9 Downloaded by guest on September 25, 2021 and IR imaging compatible could thus yield tremendous benefits. IR-OH microscope as shown schematically in Fig. 1C.Detailedin Using visible imaging components for recording IR absorption has Materials and Methods, our instrument consists of a pulsed mid-IR recently been the focus of several approaches using up-conversion QCL that is used to illuminate a large area on the sample at the of light or secondary sample effects. Strong absorption causes repetition rate, Ω. We probe the induced sample deformations secondaryeffectssuchasrefractiveindexchangesforultra- by illuminating with visible light from a narrowband LED sensitive measurements (32) or local photothermal expansion ðλ0 = 660 nmÞ and collecting backscattered light from the sample for photoacoustic imaging (33), which provide a means to over- with a Mirau interference objective. While the sample is vertically come the extant limitations of IR imaging. The concept of dynamic (z) translated, we capture interferograms, IxyðzÞ,atahighcamera photothermal changes in morphology (34, 35), force (36, 37), or frame rate of F = 500 Hz (Fig. 1 D and E). Demodulation at near-field coupling (38–40), using an atomic force microscope frequency Ω − S of the interferograms, Ixy, yields a signal pro- cantilever as local probe, has been reported for point-by-point IR portional to the first-order harmonic of the surface deformation, measurements. Noncontact optical photothermal microscopy is eðΩÞ,whereS is the fringe frequency as determined by the rate at more recent (41–50) and typically utilizes a local IR illumination which interference fringes pass by as a function of the vertical coincident with a highly focused visible probe beam to measure sample position z (61). Subsequent normalization of the demodu- local change by beam scattered out of the angular lated signal to the incident IR intensity, IIR, constructs the IR acceptance of the objective . This method has been applied to sðΩÞ of tissue and live cells (44, 46, 47), (48), photothermal image, p . High SNR is provided by a high-intensity and pharmaceutical tablets (49). These point-scanning approaches, QCL and a million-electron, full-well camera that offers signifi- just as for point scanning FT-IR spectroscopy, involve long scan cantly reduced shot noise over conventional camera technology. A times (51). The need for sample scanning renders them imprac- The 1D model from Fig. 1 connects photothermal expansion, sðΩÞ Q tical for the rapid acquisition of spectrally resolved photothermal p , to IR absorption through the locally deposited heat ,surface datasets in cases of large samples such as tissue sections for pa- reflectivity RS, and thermomechanical properties M of the sample: thology. Recent studies have demonstrated wide-field measures π pffiffiffiffiffiffi QðωÞ of photothermal absorption and scattering (52, 53). Despite sðΩÞðωÞ ∝ 4 A2 R · · MðΩÞ [1] p 0 S , providing optical compatibility, however, these methods pro- λ0 IIRðωÞ vide fields of view and pixel counts that are much smaller than direct absorption IR microscopy (54). More importantly, data which correlates well with conventional IR spectroscopy (Fig. 2). exhibit relatively poor sensitivity compared to direct absorp- Intriguingly, experimental surface vibration amplitudes obtained tion IR microscopy as the beam deflection is a measure of the photothermal change in the refractive index. Data quality is a primary driver of accuracy in obtaining biomedical information A B (55), thus limiting the utility of these methods and precluding any reports of sensitive histopathologic imaging. High-quality, high-speed, and high-resolution imaging of absorption of large areas of tissue remains elusive, but is needed for histopathology and the effective application of artificial intelligence. Here we report a solution to these challenges by combining wide-field visible microscopy with IR sample illumination from a QCL, a modality we term IR-optical hybrid (IR-OH) imaging. We demonstrate that infrared absorption in thin films and tissue sections can be measured by detecting the IR absorption-induced sample expansion in a wide-field interferometric arrangement. Particularly, our design derives a synergy from both visible and IR microscopy, rather than merely combining them in a single unit, to CD provide an architecture for high-speed, superresolution IR imag- ing. One of the central tasks in all of biomedical sciences is the recognition of tissue structure as a measure of function, disease, or development associated changes. We acquire and segment multi- hundred megapixel IR datasets on a breast tissue microarray and a breast biopsy at high accuracy, which provides experi- mental evidence of the potential of our method to enable all- E digital pathology based on visible microscopy and at a level of detail near current pathology practice. In IR-OH, we directly measure sample expansion in an inter- ferometric arrangement. We first developed a simple model to predict photothermal expansion in thin film samples to mimic those typical in histopathology (see SI Appendix,Note1, for de- tails). Our simulation involves the analysis of a uniform film placed Fig. 1. Concept of infrared-optical hybrid (IR-OH) microscopy. (A)Tempera- on a semiinfinite to understand the coupled thermal- ture modulation profile, juT ðz, ΩÞj, obtained with a 1D model of an absorbing mechanical response of this assembly. We solved heat conduc- thin film placed on a semiinfinite substrate. (B) Predicted and experimentally eðΩÞ tion to obtain the temperature modulation profile, juT ðz, ΩÞj,in measured surface deformation amplitude, , as a function of the laser pulse response to pulsed heating from the mid-IR laser at pulse fre- rate, Ω. The horizontal dashed line marks the noise floor of our instrument. (C) Ω μ IR-OH microscope setup. QCL, single-frequency, midinfrared QCL tunable to quency . For a 5- m thin SU-8 polymer film and laser pulse − Ω = A k = 900...1,900cm 1; MO, Mirau interference objective; PM, parabolic mirror; frequency 600 Hz (Fig. 1 ), the model predicts a temperature CM, cylindrical mirror; CMOS, camera; CHOP, optical chopper. The QCL beam modulation profile within the film on the order of 1 K and a for IR sample illumination is shown in yellow. The optical probe beam is shown resulting expansion of the film between 0.1 and 1 nm, which is in red. The sample is mounted on a piezo stage (not shown) for z translation. – readily measurable with wide-field interferometry (56 60). Based (D) Interferogram Ixy at a single pixel on SU-8 and (E) its Fourier transform, on this prediction, we conceptualized, built, and tested a wide-field showing fringe frequency peak S and surface deformation signals Ω ± S.

2of9 | www.pnas.org/cgi/doi/10.1073/pnas.1912400117 Schnell et al. Downloaded by guest on September 25, 2021 with a thin SU-8 film for a variety of pulse frequencies, Ω, show We validated IR-OH against the established standards of both good agreement with the model without requiring any fitting Fourier transform infrared (FT-IR) spectroscopic imaging and (Fig. 1B), which provides further confidence in our design. In optical microscopy using a United States Air Force (USAF) 1951 addition to providing IR absorption, demodulation at frequency standard pattern synthesized for IR microscopy. S constructs an optical image of the sample akin to en face scat- contrast (at 1,502 cm−1Þ reveals that IR-OH improves spatial tering maps of full-field optical tomography (59, 60). resolution by a factor of ∼4 (380 line pairs per mm vs. 100 line Perfectly coregistered to the IR absorption image, this provides pairs per mm) compared to high-definition FT-IR imaging (Fig. opportunity for IR-optical synergistic approaches as described 2 A and B), as quantified by contrast analysis (Fig. 2D). To later. Our IR-OH design leverages technological advances in vis- validate IR spectral fidelity, we performed spectroscopic imaging ible imaging technology (refractive optics and modern sensors), in the range of 950 and 1,700 cm−1 and observed spectral peak while preserving compatibility with the existing knowledgebase of frequencies and line shapes consistent with FT-IR (Fig. 2C) IR imaging of tissues (62). IR-OH allows us to access both the spectra. Spectral noise of the IR-OH data are analyzed in SI spatial content and the spectral information needed for advanced Appendix, Fig. S1. Observed differences between spectra from algorithms to enable all-digital pathology. IR-OH and FT-IR microscopy may be attributed to differ- ences in optical configurations that cause well-known changes in recorded FT-IR spectra (63, 64) and the possibility of in- homogeneous heating in a thin film absorber rather than a A uniform film on which our model is based. More accurate re- construction of IR absorption might be achieved with more so- 6

p phisticated thermal models, just as for scattering models in FT- s IR imaging (65, 66). Importantly, IR-OH imaging with an optical camera delivered ∼2 megapixels (MP) of measurements at a

IR-OH time (Fig. 2A), while typical mid-IR uncooled (0.3-MP) or 0 cooled (16-kP) cameras do not offer this large multichannel advantage nor the possibility of rapidly increasing sensor sizes or quality that visible cameras have experienced over the past de- cades. Together with the high sensitivity afforded by interfero- B metric detection, our approach allows for 10-fold larger area coverage than state-of-the-art wide-field FT-IR imaging (460 × CHEMISTRY 0.75 460 μm2 vs. 140 × 140 μm2) at a smaller pixel size (0.32 × 0.32 μm2 vs. ∼1.1 × 1.1 μm2), equating to an ∼125-fold increase in coverage. Finally, by directly measuring absorption, we mitigate scattering and its effects on the recorded data (67). As illustrated FT-IR Abs. FT-IR 0 by a spectral line scan taken across an edge in the SU-8 film, IR- OH data yielded consistent spectra on the SU-8 film (spectra 1 to 4inFig.2E) and no signal on the substrate (spectra 5 to 8). The MEDICAL SCIENCES 100 m 10 m approach improves considerably over wide-field FT-IR imaging, C D whose large baseline fluctuations and peak distortion (Fig. 2F) 1.0 present a challenge to accurate analyses, prompting the quest for 6 IR-OH FT-IR .8 many approaches to corrections, slowing data processing and trig-

[a.u.] 4 gering considerable debate over the interpretation of data (68–70). p 0.5 s .4 The suppression of the effects of scattering in the data in IR-OH 2 Rel. Mod. presents a new opportunity for improved quantitative analyses, FT-IR Abs. FT-IR

IR-OH 0 0 0 obviating spectral corrections such as baselining or Mie corrections 1600 1400 1200 1000 0 200 400 600 800 -1 and simplifying data processing for histopathology. While progress Wavenumber [cm ] Frequency (cycles/mm) in classical IR microscopy is hindered by fundamental physical E 1 F limitations (e.g., limit), our results reveal the potential of 9 2 0.6

p 1 3 IR-OH for superresolution IR imaging, largely improved coverage, s 6 8 4 — and suppression of IR scattering that can be employed for un- 5 0.3 foreseen, transformative applications in histopathologic imaging. 3 6 IR-OH

7 Abs. FT-IR Further, the interferometric detection in IR-OH provides a sensi- 8 0 0 tivity advantage over previous photothermal techniques that relied 1600 1400 1200 1000 1600 1400 1200 1000 on beam deflection from IR absorption-induced refractive index Wavenumber [cm-1] Wavenumber [cm-1] change (41, 42, 44, 45, 48, 53). While our method measures the Fig. 2. Validation of IR-OH against FT-IR. (A and B) Spatial comparison. (A) physical response of expansion directly, deflection methods measure Single field of view of IR-OH imaging, showing absorption at 1,502 cm−1 (Left) the refractive index change that occurs upon deflection and need and digital zoom on group 8 (Right; indicated by yellow arrow in Left)ofan higher probe intensity than we have employed. The direct expansion SU-8 test target wherein all elements from groups 5 to 8 were clearly resolved. measurement is challenging but rewarding. It is notable that our (B) Composite high-definition FT-IR image of the same area, obtained by method provides both phase and amplitude measurements that mosaicking due to the small field of view (tiles are indicated by white lines), allow us to conduct further analyses. For example, we identified that did not resolve any element of group 8. (C) Spectral validation: comparison of the contribution of surface expansion to the optical signal is one IR-OH spectrum, taken from the square of group 6, against FT-IR spectrum of a magnitude larger than that contributed by a refractive index change, bulk SU-8 film. (D) The improved spatial detail in IR-OH is quantitatively as measured by sample reflectivity (SI Appendix,Fig.S2). High- compared to FT-IR imaging via the contrast as a function of spatial frequency derived from the USAF standard. (E and F) IR-OH and FT-IR spectral profiles sensitivity, wide-field detection with large area of view and pixel recorded along a line across an SU-8 film edge (elements 1 to 4 and 5 to 8 are count enables spectrally imaging of large samples, as we will dem- on the SU-8 and substrate, respectively) showing greater consistency in spectra onstrate in the following sections. due to reduced scattering contributions in IR-OH compared to the measured We evaluated the accuracy of IR-OH for pathology by imaging beam attenuation in FT-IR imaging. breast tissue as a test case. For over 125 y, stains have been used

Schnell et al. PNAS Latest Articles | 3of9 Downloaded by guest on September 25, 2021 for histopathologic recognition in clinical and research activities, applied random forest (RF) classification on a subset of only allowing differentiation of cellular and subcellular components seven frequency bands (Materials and Methods). Fig. 3 B–D show − in tissue. For example, the hematoxylin and eosin (H&E) stain IR absorption (1,550 cm 1) and classification maps of selected allows a trained observer to contrast epithelial cells from the tissue types, in accordance to the gold standard (H&E images, surrounding stroma (Fig. 3A). Morphological features of epi- Fig. 3A). Classifier performance was assessed using receiver thelial cells are the basis for cancer diagnoses and studying operating characteristic (ROC) curves (Fig. 3E), demonstrating progression. In contrast, vibrational imaging of tissue can pro- an overall area under the curve (AUC) of 0.93, which is com- vide label-free diagnoses by machine learning techniques applied parable to state-of-the-art IR imaging (71). Classification of the to chemical constitution of tissue without stains. We demonstrate epithelium into disease (malignant and noncancerous subtypes) this method enabled by IR-OH imaging of an unstained breast showed an AUC of ∼0.90. Representative IR-OH spectra (Fig. tissue microarray at 22 discrete IR frequencies and subsequently 3G) revealed differences that allow for classification and are

ABC DE

F

G

H

Fig. 3. IR-OH imaging of breast tissue and spectral cell type recognition. (A) H&E image of a breast tissue microarray section. (Scale bar: 500 μm.) (B) IR-OH − absorption of an adjacent, unstained tissue section (at 1,550 cm 1). (C) A four-class model (blood, epithelium, stroma, and other) allows rapid tissue com- ponent visualization based on five IR bands. (D) Epithelial classification (five-class model) permits both histologic cellular identification and recognition of cancer based on seven IR bands. (E) ROC curves quantify the accuracy of recognition in D by IR-OH. (F) AUC of the ROC curve increases with number of IR frequencies used for classification, showing a small number provides good recognition (G and H) Representative class spectra obtained with IR-OH and FT-IR imaging. Full classification maps are shown in SI Appendix, Figs. S3 and S4. Class mean spectra and variance are shown in SI Appendix, Figs. S5 and S6,and further AUC and ROC curves are shown in SI Appendix, Fig. S7.

4of9 | www.pnas.org/cgi/doi/10.1073/pnas.1912400117 Schnell et al. Downloaded by guest on September 25, 2021 consistent with FT-IR spectra (Fig. 3H). We conducted feature QCL were shown to enable rapid spectral IR imaging based on selection on the original 22 frequency point dataset (Fig. 3F and discrete-frequency approaches, reducing full slide spectral im- Materials and Methods). We observed saturation of classifier aging to less than a day. However, wide-field QCL microscopy is performance for the first seven most relevant frequency bands still fundamentally limited by diffraction; further, both image and (Fig. 3D, five-class model) with only little improvement provided spectral distortions (see SI Appendix,Note5, for discussion) mit- by using all 22 bands for classification. Interestingly, the first five igate the widefield detection advantage and present challenges for most relevant bands already provided tissue type identification high-quality imaging needed in pathology. We note that the use of with >0.90 AUC as shown in Fig. 3C (four-class model), which QCL sources requires preselection of the IR frequencies prior to could make IR-OH a rapid screening tool for guiding robotic imaging, which might depend on the specific tissue type to be surgery where localization and identification of anatomical segmented. This could be done on the same instrument by spectral structures is needed. Importantly, mitigation of scattering arti- imaging at many data points or by FT-IR imaging of a represen- facts in IR-OH does not require the acquisition of additional IR tative sample and by performing subsequent feature selection. We frequencies for baselining spectral data. We only required ratios further note that with IR-OH, tighter integration between machine of the signals at various frequencies for our IR-OH protocol, learning and optical components will extend the abilities of this which provides a speed advantage compared to QCL-based far- hybrid approach further (72). field IR imaging. With the subset of seven bands, our first- IR-OH enables a different dimension in histopathology by generation IR-OH instrument already provides full slide classi- combining visible microscopic morphology with chemical com- fication after ∼2.5 d of imaging time, and there is certainly room position. The data can be used to produce computational H&E for further improvement by improving the sensitivity of the in- contrast, providing a stain-free means of tissue imaging that terferometer. In comparison, FT-IR microscopy would require maps to extant clinical practice (Fig. 4). While absorbance im- about 26,000 individual frames with a 128 × 128 array detector to ages from FT-IR measurements (Fig. 4A) can now be obtained cover an entire TMA slide owing to the small format of the in higher detail by IR-OH imaging (Fig. 4B), the visible mi- detector, as well as a large number of coadditions owing to the croscopy channel (Fig. 4C) provides a further and complemen- weak irradiance of a thermal source. Assuming a typical value of tary contrast derived from a combination of factors, including 32 coadditions, full slide imaging time is estimated to be on the density of the tissue, scattering, path length, and visible absorp- order of 40 d for FT-IR microscopy. Recently, high-brightness tion that exceed simple brightfield images (16, 18). We combine CHEMISTRY

ABCDFTIR Abs. IR-OH snp IR-OH Optical Comput. H&E EH&E MEDICAL SCIENCES

00.506E-30max

Fig. 4. Morphology-based inspection of tissue with IR-OH to generate stainless staining images that mimic current clinical practice. (A) Uncorrected ab- − − sorbance from FT-IR imaging (at 1,550 cm 1), showing limited contrast due to limited optical resolving power and scattering. (B) IR-OH absorption (1,550 cm 1). (C) Low-coherence interferometry images from the optical channel that are perfectly coregistered to those in B.(D) Computed H&E image using the stainless staining approach and (E) H&E image of adjacent tissue section. Full core images are shown in SI Appendix, Fig. S8. (Scale bar: 100 μm.)

Schnell et al. PNAS Latest Articles | 5of9 Downloaded by guest on September 25, 2021 absorption and morphologic data using machine learning to classification accuracy was limited by 1) inhomogeneity and fluc- generate stainless H&E images at the resolution of optical micros- tuations in the IR illumination of the sample, 2) low illumination copy (Fig. 4D). Sample morphology, e.g., the individual stromal fi- power at the crossover points of the QCL preventing reliable mea- − bers and structure of the breast acini, were reproduced well in both surements in some of the relevant spectral regions (e.g., 1,200 cm 1 − color and spatial detail compared to H&E images (Fig. 4E). Thus, and 950 cm 1 regions), and 3) the limited vertical sensitivity of our our approach provides stain-free pathology images that rival current interference microscope (SI Appendix,Note4). Optimized inter- stain-based practice. IR-OH is thus the enabling technology needed ferometric have been shown to reach picometer vertical for truly all-digital pathology without stains or human interpretation. sensitivity, promising more accurate classification and faster imaging In the short term it opens the door to research in stainless pathology speeds (77). by spectroscopists, microscopists, pathologists, and computer scien- In terms of technological progress, IR-OH merges wide-field tists for improved human interpretation and the translation of ad- optical microscopy with vibrational spectroscopy into a hybrid vanced machine learning algorithms developed on archival stained microscopy platform that overcomes the diffraction limit of IR images (73). Extraction of texture features from the visible micros- microscopy, and provides for a class of practical and cost-effective copy channel is a possibility (74, 75), such as, for example, based on instrumentation for IR imaging. It enables a contrast mechanism Gabor filter banks, and could be combined with chemical in- for visible microscopy, providing molecular information without formation from the IR channel for novel synergistic tissue seg- specialized stains or dyes. We thus access more than a century mentation approaches. The required, perfect coregistration between of knowledge and progress in quantitative molecular analyses visible and IR data is intrinsically provided by IR-OH. While this through vibrational spectroscopy for visible microscopy. IR-OH initial implementation can be greatly improved in terms of data also considerably enhances IR spectroscopic imaging technology quality by optimizing the geometry further, the combination of by doing away with the need for IR-specialized imaging compo- widefield microscopy in IR-OH is also compatible with other means nents and by leveraging recent developments in optical technology of providing the sensitive visible microscopy measurements, such as such as the megapixel visible camera employed here for increased methods based on quantitative phase imaging in transmission, which coverage. High-frame rate, multimegapixel visible camera tech- will allow greater morphologic detail for thin samples and un- nology and QCL technology with consistent output power across ambiguously detect subcellular morphologies more precisely. the spectrum could lead to high-throughput chemical imaging at Our IR-OH system can be applied to large pathology samples mm2 field of view, submicron pixel size, and subsecond integration and is useful for whole-slide imaging. Full-slide absorbance im- time. Building a unit of multiplexed, fixed-frequency QCL chips ages (400 mm2) were computationally segmented (Fig. 5A) using could further replace tunable QCL at the fraction of the cost. IR- the machine-learned model previously developed in Fig. 3 and OH could thus provide a new gamut of cost-effective IR micros- computationally stained (Fig. 5B). The image quality agreed well copy that proves practical for routine biomedical imaging appli- with a reference H&E image (cf. Fig. 5 B and C), providing cations. We anticipate many variations on the basic IR-OH conventional and additional disease state information for tumor configuration presented here with innovations in QCL illumina- detection and determination of tumor margins (further examples tion, measurement optics of the sample expansion, and phase and in SI Appendix, Fig. S9). Minimal as provided amplitude contrast in optical detection. This activity will also spur by entirely backscattering geometry and rejection of IR scattering for IR-OH directions that are of contemporary interest in quan- makes this technique appealing not only for minimally preparative titative optical microscopy, including computational methods for histopathology (76) but also for analysis in dynamic polymer improved image acquisition, emerging methods that make great systems or for forensics. In the current implementation of IR-OH, use of morphology such as deep learning, extraction of enhanced

A 5 mm BC5 mm 5 mm

500 m 500 m 500 m

Non-canc. Epithelium Malignant Epi. Blood Other Stroma

Fig. 5. All-digital histopathology of an unstained breast surgical resection, combining automated recognition and traditional pathology. (A) Classification of surgical resection from IR-OH data and (B) its derived computational H&E. (C) H&E image of adjacent section. (Top) Whole-slide images. (Bottom) Digital zooms into region marked by arrow in A.

6of9 | www.pnas.org/cgi/doi/10.1073/pnas.1912400117 Schnell et al. Downloaded by guest on September 25, 2021 detail from the data, and reconstruction of the modified optical The IR beam is intensity-modulated with an optical chopper at frequency signals. The spectrally integrative approach of IR-OH is especially Ω = 600 Hz (MC2000B; Thorlabs) and directed toward the sample for illumi- amenable to advanced machine learning algorithms that make use nation from the side using a combination of a concave and parabolic mirror of chemical composition; morphology data; and, in future imple- to shape the IR beam. To probe the sample deformation induced by IR ab- mentations, tomographic information afforded by the underlying sorption, we apply stroboscopic wide-field interference microscopy. In detail, the sample is illuminated with light from a narrow-band LED emitting at a low coherence interferometry principle. nominal λ0 = 660 nm and 20 nm spectral bandwidth (M660L4; Thorlabs). The In terms of use, the synergy of IR absorption contrast with LED is strobed in synchronization with the beginning of each camera frame optical microscopy in a backscattering geometry clearly holds for a time of ∼500 μs. The reflected light ES from the sample is collected with further promise of IR-OH for minimally preparative imaging a Mirau interference objective (50× CF IC Epi Plan DI; Nikon) where ES is across the spectrum of applications where either visible micro- interfered with a reference field ER for phase detection. To scan the delay τ copy or IR spectroscopy are useful. IR-OH especially brings an between sample and reference field, the sample can be translated vertically important functionality to optical microscopy and removes a (z) by a linear piezo stage (P-611.3; Physik Instrumente). The range of vertical major barrier of using IR absorption as a contrast for biomedical translation was chosen to be 4.5 and 7 μm for the USAF test target (Fig. 1) – analyses. This compatibility with optical microscopy can obviate and the tissue samples (Figs. 2 4), respectively. The slight increase in travel the need for stains or specialized knowledge for routine chemical with tissue was needed to keep the uneven surface of the tissue sections in focus and within the coherence length of the LED across the entire sample. analysis in biomedical research, making optical microscopy While the sample is translated at constant velocity zðtÞ = vt, a sequence of cheaper, more informative, and easier to apply. For example, the images is registered with a monochromatic camera at high frame rate B results of Fig. 5 demonstrate a potential to eliminate staining. F = 500 Hz, yielding interferograms Ixy ðzÞ at each pixel (x, y). We optimize This could cut down on precious research time and need for signal-to-noise ratio (SNR) by employing a million-level electron full well safely maintaining reagents and supplies, as well as human labor complementary metal-oxide semiconductor (CMOS) camera for image reg- for a routine task. Since the tissue is not changed in any manner istration (Q-2A750; Adimec), a recently developed camera technology that by IR-OH microscopy, it is still available for conventional or ad- significantly reduces shot-noise contributions in bright imaging conditions. vanced analyses. With modern machine learning undergoing a The image sequence is continuously streamed to computer memory at 2 similar democratization in availability and use, the combined ap- GByte/s. To reduce data volume and extract the IR absorption signal, signal demodulation is implemented in C++ for real-time processing. Demodula- plication of vibrational molecular contrast and machine learning Ω − can greatly expand the analytical tools accessible to biomedical tion is performed at frequencies S and S by 1) applying a Gaussian window to each interferogram and 2) subsequent discrete Fourier trans- scientists. Together, the compatibility of IR-OH microscopy with form. The resulting datasets are saved to disk. To enable extraction of 3D optical microscopy and its synergy with emerging computational information based on interferogram analysis, a decimated version of the full CHEMISTRY ubiquity to handle the wide palette of IR molecular contrast can data was saved to disk as well, keeping every 20th frame. Note that since in change how we routinely handle, image, and understand micro- our case, Ω > F, the vibration signal is detected at the low-frequency alias scopic tissue structure, enabling all-digital pathology with appli- Ω′ = 100 Hz of the pulse rate Ω (remainder of division Ω=F). For clarity we do cations across the spectrum of biomedical science. not distinguish between Ω and Ω′ in the main text. The IR absorption image is obtained by dividing Ω − S by S datasets, as described by SI Appendix,Eq. Materials and Methods S19. The optical image is obtained by demodulation at frequency S. Sample Preparation. The use of tissue for this study was approved by the University of Illinois Institutional Review Board via project 06684. IR-OH Imaging. IR-OH imaging of the USAF test sample in Fig. 2 was per- A paraffin-embedded serial breast tissue microarray (BR1003a; US Biomax formed by acquiring a single field of view of groups 6 and 7 and part of MEDICAL SCIENCES Inc.) consisting of a total of 101 cores of nominal 1 mm diameter from 47 cases group 5. A set of frames was acquired at a total of 396 frequency points and −1 −1 was obtained. One section was stained with H&E and imaged with a light 2cm spacing between the spectral range of 910 to 1,700 cm . To remove microscope. A 5-μm-thick adjacent unstained section of the TMA was placed contributions of lines in our unoptimized system, a 2-pt Gauss filter was applied in the spectral domain. IR-OH imaging of the BR1003a breast on a BaF2 salt plate. We note that we chose IR transparent BaF2 salt plates as substrate to image the same sample with both IR-OH and transmission FT-IR tissue microarray (TMA) was performed by acquiring 2,917 tiles measuring 2 microscopy. As IR-OH can be based entirely on a backscattering geometry, ∼460 × 460 μm each and with 12% overlap. At each sample location, 22 IR imaging of samples placed on IR opaque substrates like glass or in aqueous frequencies were consecutively acquired before moving to the next sample media is in principle possible, which would constitute a major advantage for location. The 22 IR frequency points were selected according to a previously pathology proposes as it would remove the need for special IR transparent developed FT-IR classification model for breast tissue, where IR frequencies substrates and could allow live cell imaging and imaging of nonfixated tis- were manually determined (78, 79). The chosen IR frequencies were 1,038, sue. The unstained section was deparaffinized using a 16-h hexane bath. 1,086, 1,142, 1,170, 1,214, 1,254, 1,282, 1,302, 1,338, 1,362, 1,386, 1,406, −1 Next, a surgical biopsy was obtained on a BaF2 salt plate with a corre- 1,426, 1,450, 1,486, 1,534, 1,550, 1,590, 1,630, 1,662, 1,690, and 1,718 cm . sponding H&E-stained adjacent section on a glass slide. These are large tis- Inhomogeneous sample illumination of the QCL beam was corrected in sue sections of about 20 × 20 mm2 obtained from a single patient to obtain a postprocess by 1) calculating an average image over all tiles, 2) applying a 2D comprehensive diagnostic profile. These are commonly used in clinical de- polynomial fit, and 3) multiplying the inverse of the result to each tile. cision making, and hence, the models developed on the TMA were extended Stitching with in-house developed software yielded a set of 22 IR absorption as stainless staining on the biopsy data. images covering an area of 19 × 22 mm2, which covered the entire TMA. For A USAF 1951 optical resolution test target was fabricated in house. The classification purposes, data were binned by 4 × 4 pixels to improve SNR and target was patterned with SU-8 2005 polymer (MicroChem) on a polished to reduce the dataset to a more manageable size, yielding 295 Megapixel × μ 1-in.-diameter barium fluoride (BaF2) substrate (Spectral Systems). The sub- 22 frequencies with a pixel size of 1.2 m. Data were then normalized to the strate was first cleaned with acetone and isopropyl alcohol (IPA) and then Amide II band. No further spectral postprocessing was applied, e.g., base- rinsed with distilled water. A 5-μm-thick layer of SU-8 polymer was spun coat lining or noise reduction was not applied. Similarly, the breast biopsy was

on BaF2 at 500 rpm for 5 s then ramped to 3,000 rpm for 60 s at a ramp rate imaged by acquiring 2,808 tiles and at 25 IR frequencies, yielding 285 of 1,500 rpm/s. The sample was then soft baked at 65 °C for 1 min and then Megapixel image × 25 frequencies of an area of ∼20 × 20 mm2. In both cases, heated to 95 °C at a ramp rate of 300 °C/min and held for 2 min. A lithog- the integration time was 8 s per tile and frequency, and the total image time raphy mask was contact aligned, and SU-8 was exposed to i-line UV radiation was 10 s including overhead for data processing and storage. Image time of (365 nm) at 9 mW/cm2 for 5 min with an optical density filter (PN NE206B; the entire sample was ∼7 d for all 22 frequencies (equivalent to 1.7 h per Thorlabs) to filter out deep UV. The exposed sample was heated with the core). Potential for significant speed-up exists by improving the vertical same parameters as the soft bake and then developed in SU-8 developer for sensitivity of the interferometer (SI Appendix, Note 4). For spectral com- 4 min and rinsed with IPA. parison with FT-IR, we acquired a representative selection of 17 tiles at 221 − − IR frequencies with IR-OH covering the range from 910 cm 1 to 1,790 cm 1 in − IR-OH Setup. The setup of our IR-OH microscope is shown in Fig. 1C. The steps of 4 cm 1. To remove contributions of water lines and to improve SNR, sample is heated with a monochromatic IR laser beam (MIRcat; Daylight a 1-pt Gauss filter was applied in the spectral domain. Class spectra were Solutions) that can be tuned to molecular vibration modes in the sample. averaged from regions labeled in classification.

Schnell et al. PNAS Latest Articles | 7of9 Downloaded by guest on September 25, 2021 FT-IR Imaging. HD FT-IR imaging was performed on a 680-IR classification and thus speed up future IR-OH imaging and 2) learn how well coupled to a 620-IR imaging microscope (Agilent Technologies) with a 0.62 IR-OH data classify with only a few IR frequencies. Briefly, we developed an NA objective and a liquid nitrogen-cooled MCT 128 × 128 focal plane array algorithm that tested classifier performance (AUC) for all possible band −1 − detector. Data were acquired over the 900 to 3,800 cm spectral range and combinations, starting with n = 2 frequency points and using 1,550 cm 1 as averaged over 16 scans per pixel. Data were subsequently corrected against the single reference band to which all other bands were normalized. The a background acquired in an empty space of the BaF2 slide with 128 scans four best frequency combinations where kept, using a cutoff of the 80 −1 and Fourier transformed. The spectral resolution was 4 cm with a pixel size combinations with highest AUC scores, and the search algorithm was re- μ ∼ × of 1.1 m. Each core was imaged by acquiring several tiles measuring 140 peated for n + 1 bands. This search algorithm worked fully automatically. μ 140 m each and subsequent stitching using in-house software. Data were The found features were (most relevant first) 1,550 (reference band for further processed using minimum noise fraction for noise reduction in a normalization), 1,302, 1,630, 1,086, 1,338, 1,662, 1,718, 1,690, 1,038, 1,406, commercial software package, Environment for Visualizing Images (ENVI, 1,486, 1,450, 1,534, 1,590, 1,282, 1,386, 1,254, 1,362, 1,214, 1,142, 1,426, and Harris Geospatial Solutions, Inc., Broomfield, CO). − 1,170 cm 1. Classifier accuracy (AUC) was evaluated using the top n features in Fig. 3F with the breast TMA. Prior to segmentation of the biopsy, we Supervised Classification. The IR absorption images obtained with IR-OH were repeated the feature selection by training and validation on the TMA manually labeled using correlation with the consecutive marked H&E-stained dataset shown in Fig. 3 and testing classifier performance with a second glass slide images under the supervision of a pathologist as ground truth for dataset of the same TMA to provide feedback (SI Appendix, Fig. S10). This our analysis. The following four histological classes were used: epithelium, was done to assure robustness in performance and improvements in the stroma, red blood cells, and other, where “other” summarized the remaining developing instrument, particularly because the biopsy and TMA datasets types (necrosis, mucin, and secretions). Epithelium was further discriminated into noncancerous and malignant subtypes. As a first step for supervised were collected at different time points during instrument development. classification, a tissue mask based on the IR-OH signal of the Amide I band was Classifier performance was found to saturate for the top nine features: 1,550 applied to remove empty spaces, debris, and pixels exhibiting low IR-OH signal (manually selected reference band), 1,630, 1,302, 1,338, 1,662, 1,486, 1,038, −1 from further analysis. For each class, 10,000 training and 10,000 validation 1,690, and 1,406 cm . These features are contained in the top 11 features of pixels were randomly selected from the marked regions, and subsequently, the first feature search, which confirms feature consistency. Note that we IR-OH absorption spectra were extracted and normalized to the Amide II band. manually substituted feature 1086 by feature 1038, which captures similar These spectral signatures were then used to build a random forest classifier. tissue chemistry, because the former yielded poor segmentation of the bi- Notably, metrics were not defined—as typically done in the classification of opsy. We attribute this to the low reproducibility when imaging at the 1086 FT-IR data—but rather the normalized spectral data were used as direct input laser line for reasons yet to be determined but which could involve power for the classifier. A total of five predictors were used (1,086, 1,302, 1,338, 1,550 and pointing instabilities of the laser. − [reference band for normalization], and 1,630 cm 1) to build a 60-tree en- semble for the four-class tissue segmentation. A total of seven predictors were Computational H&E Images. We first assigned a specific color to each cell type used (1,086, 1,302, 1,338, 1,550 [reference band for normalization], 1,630, that represented the corresponding color obtained from the H&E stained −1 1,662, and 1,718 cm ) to build a 60-tree ensemble for five-class tissue seg- images and subsequently generated a color image from the four-class map. mentations. These frequency bands were selected in a feature selection pro- To add morphology information, we combined it pixel by pixel with the cedure (see below). The four-class classifier was based on the main cell types optical image using a pan-sharpening approach. Briefly, we transformed the comprising epithelium, stroma, red blood cells, and other. A five-class classifier color image to a single intensity band that was matched to the histogram of further discriminated between noncancerous and malignant epithelial sub- the optical image. Then, we applied a linear combination of the color image types. To assess the performance of the classifiers, we generated binary ROC with the optical image and so obtained the computational H&E image. curves and calculated the AUC metric for each class. AUC values were estimated Details about this pan-sharpening approach will be published elsewhere. All for each class for three separate classification runs. In each of these runs a data and software developed can be requested from the corresponding random subset of the pixels were sampled for training. This was done to en- author by email. sure that the all subsets of the sampled pixels are representative of the data population. Mean AUCs are reported for each class. The breast tissue micro- ACKNOWLEDGMENTS. The authors thank Ilia Rasskazov for discussions. M.S. array shown in Figs. 3 and 4 was classified at a pixel size of 1.2 μm(4× 4pixel acknowledges support by the European Union’s Horizon 2020 research and binning). The breast biopsy shown in Fig. 5 was classified at a pixel size of innovation programme under the Marie Sklodowska-Curie grant agreement 2.4 μm(8× 8 pixel binning) to further improve SNR and to obtain a more 655888. R.B. acknowledges support from the National Institutes of Health accurate classification. via award R01EB009745 and from the Agilent Thought Leader Award. S.K. was supported by an NIH T32 fellowship through the tissue microenviron- Feature Selection. We applied feature selection based on an iterative search ment training program (T32EB019944). S.M. was supported by a Beckman algorithm to 1) reduce the number of IR frequencies needed for accurate Institute graduate fellowship.

1. V. Kumar, A. Abbas, N. Fausto, J. Aster, Robbins and Cotran Pathologic Basis of Disease, 13. J. Faist et al., Quantum cascade laser. Science 264, 553–556 (1994). Professional Edition (Elsevier Health Sciences, 2014). 14. M. R. Kole, R. K. Reddy, M. V. Schulmerich, M. K. Gelber, R. Bhargava, Discrete fre- 2. J.-X. Cheng, X. S. Xie, Vibrational spectroscopic imaging of living systems: An quency infrared microspectroscopy and imaging with a tunable quantum cascade emerging platform for biology and . Science 350, aaa8870 (2015). laser. Anal. Chem. 84, 10366–10372 (2012). 3. E. F. Nichols, A study of the transmission spectra of certain substances in the infra-red. 15. S. Mittal et al., Simultaneous cancer and tumor microenvironment subtyping using Phys. Rev. Ser. I 1,1–18 (1893). confocal infrared microscopy for all-digital molecular histopathology. Proc. Natl. 4. M. Diem, Modern Vibrational Spectroscopy and Micro-Spectroscopy: Theory, Instrumentation Acad. Sci. U.S.A. 115, E5651– E5660 (2018). and Biomedical Applications (John Wiley & Sons, 2015). 16. K. Yeh, S. Kenkel, J.-N. Liu, R. Bhargava, Fast infrared chemical imaging with a 5. M. Pilling, P. Gardner, Fundamental developments in infrared spectroscopic imaging quantum cascade laser. Anal. Chem. 87, 485–493 (2015). for biomedical applications. Chem. Soc. Rev. 45, 1935–1957 (2016). 17. S. Tiwari et al., Towards translation of discrete frequency infrared spectroscopic imaging 6. H.-Y. N. Holman, M. C. Martin, E. A. Blakely, K. Bjornstad, W. R. McKinney, IR spec- for digital histopathology of clinical biopsy samples. Anal. Chem. 88, 10183–10190 (2016). troscopic characteristics of cell cycle and cell death probed by synchrotron radiation 18. M. J. Pilling, A. Henderson, P. Gardner, Quantum cascade laser spectral histopathol- based Fourier transform IR spectromicroscopy. Biopolymers 57, 329–335 (2000). ogy: Breast cancer diagnostics using high throughput chemical imaging. Anal. Chem. 7. R. Bhargava, S.-Q. Wang, J. L. Koenig, “FTIR microspectroscopy of polymeric systems” 89, 7348–7355 (2017). in Liquid /FTIR Microspectroscopy/Microwave Assisted Synthesis 19. C. Kuepper et al., Quantum cascade laser-based infrared microscopy for label-free (Springer, Berlin, 2003), pp. 137–191. and automated cancer classification in tissue sections. Sci. Rep. 8, 7717 (2018). 8. S. Türker-Kaya, C. W. Huck, A review of mid-infrared and near-infrared imaging: 20. F. Ghaznavi, A. Evans, A. Madabhushi, M. Feldman, in pathology: Principles, concepts and applications in plant tissue analysis. 22, 168 (2017). Whole-slide imaging and beyond. Annu. Rev. Pathol. 8, 331–359 (2013). 9. C. Ricci, K. L. A. Chan, S. G. Kazarian, Combining the tape-lift method and Fourier 21. P. L. Fitzgibbons et al., Prognostic factors in breast cancer. College of American Pa- transform infrared spectroscopic imaging for forensic applications. Appl. Spectrosc. thologists Consensus Statement 1999. Arch. Pathol. Lab. Med. 124, 966–978 (2000). 60, 1013–1021 (2006). 22. V. Kumar, A. K. Abbas, J. C. Aster, Robbins Basic Pathology (Elsevier Health Sciences, 10. P. Colarusso et al., Infrared spectroscopic imaging: From planetary to cellular systems. 2017). Appl. Spectrosc. 52, 106A–120A (1998). 23. A. H. Beck et al., Systematic analysis of breast cancer morphology uncovers stromal 11. E. R. Blout, R. C. Mellors, Infrared spectra of tissues. Science 110, 137–138 (1949). features associated with survival. Sci. Transl. Med. 3, 108ra113 (2011). 12. C. W. Freudiger et al., Label-free biomedical imaging with high sensitivity by stimu- 24. A. Esteva et al., Dermatologist-level classification of skin cancer with deep neural lated microscopy. Science 322, 1857–1861 (2008). networks. Nature 542, 115–118 (2017).

8of9 | www.pnas.org/cgi/doi/10.1073/pnas.1912400117 Schnell et al. Downloaded by guest on September 25, 2021 25. J. D. Pallua et al., Clinical infrared microscopic imaging: An overview. Pathol. Res. 52. R. M. Sullenberger, S. M. Redmond, D. Crompton, A. M. Stolyarov, W. D. Herzog, Pract. 214, 1532–1538 (2018). Spatially-resolved individual particle spectroscopy using photothermal modulation of 26. D. C. Fernandez, R. Bhargava, S. M. Hewitt, I. W. Levin, Infrared spectroscopic imaging Mie scattering. Opt. Lett. 42, 203–206 (2017). for histopathologic recognition. Nat. Biotechnol. 23, 469–474 (2005). 53. Y. Bai et al., Ultrafast chemical imaging by widefield photothermal sensing of in- 27. B. Bird et al., Infrared spectral histopathology (SHP): A novel diagnostic tool for the frared absorption. Sci. Adv. 5, eaav7127 (2019). accurate classification of lung cancer. Lab. Invest. 92, 1358–1373 (2012). 54. K. Yeh, D. Lee, R. Bhargava, Multicolor discrete frequency infrared spectroscopic 28. D. Mayerich et al., Stain-less staining for computed histopathology. Technology imaging. Anal. Chem. 91, 2177–2185 (2019). (Singap. World Sci.) 3,27–31 (2015). 55. R. Bhargava, Towards a practical Fourier transform infrared chemical imaging pro- 29. M. J. Walsh et al., Fourier transform infrared microspectroscopy identifies symmetric tocol for cancer histopathology. Anal. Bioanal. Chem. 389, 1155–1169 (2007). PO(2)(-) modifications as a marker of the putative stem cell region of human in- 56. A. F. Fercher, W. Drexler, C. K. Hitzenberger, T. Lasser, Optical coherence tomography— testinal crypts. Stem Cells 26, 108–118 (2008). Principles and applications. Rep. Prog. Phys. 66, 239–303 (2003). 30. J. T. Kwak et al., Improving prediction of prostate cancer recurrence using chemical 57. A. Bosseboeuf, S. Petitgrand, Characterization of the static and dynamic behaviour of imaging. Sci. Rep. 5, 8758 (2015). M(O)EMS by optical techniques: Status and trends. J. Micromech. Microeng. 13, S23 31. F. Großerueschkamp et al., Spatial and molecular resolution of diffuse malignant (2003). mesothelioma heterogeneity by integrating label-free FTIR imaging, laser capture 58. M. R. Hart, R. A. Conant, K. Y. Lau, R. S. Muller, Stroboscopic interferometer system – microdissection and proteomics. Sci. Rep. 7, 44829 (2017). for dynamic MEMS characterization. J. Microelectromech. Syst. 9, 409 418 (2000). 32. D. Boyer, P. Tamarat, A. Maali, B. Lounis, M. Orrit, Photothermal imaging of 59. A. Dubois, L. Vabre, A.-C. Boccara, E. Beaurepaire, High-resolution full-field optical – nanometer-sized metal particles among scatterers. Science 297, 1160–1163 (2002). coherence tomography with a Linnik microscope. Appl. Opt. 41, 805 812 (2002). 33. L. V. Wang, S. Hu, Photoacoustic tomography: In vivo imaging from to 60. O. Assayag et al., Large field, high resolution full-field optical coherence tomography: organs. Science 335, 1458–1462 (2012). A pre-clinical study of human breast tissue and cancer assessment. Technol. Cancer – 34. A. Dazzi, C. B. Prater, AFM-IR: Technology and applications in nanoscale infrared Res. Treat. 13, 455 468 (2014). “ ” spectroscopy and chemical imaging. Chem. Rev. 117, 5146–5173 (2017). 61. P. d. Groot, Coherence scanning interferometry in Optical Measurement of Surface 35. A. Dazzi et al., AFM-IR: Combining and infrared spectroscopy Topography, R. Leach, Ed. (Springer, Berlin, 2011), chap. 9. for nanoscale chemical characterization. Appl. Spectrosc. 66, 1365–1384 (2012). 62. M. J. Baker et al., Using Fourier transform IR spectroscopy to analyze biological ma- – 36. F. Lu, M. Jin, M. A. Belkin, Tip-enhanced infrared nanospectroscopy via molecular terials. Nat. Protoc. 9, 1771 1791 (2014). expansion force detection. Nat. Photonics 8, 307–312 (2014). 63. B. J. Davis, P. S. Carney, R. Bhargava, Theory of midinfrared absorption micro- – 37. D. Nowak et al., Nanoscale chemical imaging by photoinduced force microscopy. Sci. spectroscopy: I. Homogeneous samples. Anal. Chem. 82, 3474 3486 (2010). 64. D. Mayerich et al., On the importance of image formation optics in the design of Adv. 2, e1501571 (2016). infrared spectroscopic imaging systems. Analyst (Lond.) 139, 4031–4036 (2014). 38. M. Schnell et al., Nanofocusing of mid-infrared energy with tapered transmission 65. T. van Dijk, D. Mayerich, P. S. Carney, R. Bhargava, Recovery of absorption spectra lines. Nat. Photonics 5, 283–287 (2011). from Fourier transform infrared (FT-IR) microspectroscopic measurements of intact 39. I. Amenabar et al., Structural analysis and mapping of individual complexes spheres. Appl. Spectrosc. 67, 546–552 (2013). by infrared nanospectroscopy. Nat. Commun. 4, 2890 (2013). 66. A. K. Kodali et al., Narrowband midinfrared reflectance filters using guided mode 40. H. A. Bechtel, E. A. Muller, R. L. Olmon, M. C. Martin, M. B. Raschke, Ultrabroadband resonance. Anal. Chem. 82, 5697–5706 (2010). infrared nanospectroscopic imaging. Proc. Natl. Acad. Sci. U.S.A. 111, 7191–7196 67. B. Mohlenhoff, M. Romeo, M. Diem, B. R. Wood, Mie-type scattering and non-Beer- (2014). Lambert absorption behavior of human cells in infrared microspectroscopy. Biophys. CHEMISTRY 41. R. Furstenberg, C. A. Kendziora, M. R. Papantonakis, V. Nguyen, R. A. McGill, J. 88, 3635–3640 (2005). “Chemical imaging using infrared photothermal microspectroscopy“ in Next-Gener- 68. P. Bassan et al., FTIR microscopy of biological cells and tissue: Data analysis using ation Spectroscopic Technologies V, M. A. Druy, R. A. Crocombe, Eds. (SPIE, Baltimore, resonant Mie scattering (RMieS) EMSC algorithm. Analyst (Lond.) 137, 1370–1377 MD, 2012), Vol. 8374, p. 837411. (2012). 42. A. Mërtiri et al., Mid-infrared photothermal heterodyne spectroscopy in a liquid 69. A. Kohler et al., Estimating and correcting mie scattering in synchrotron-based mi- crystal using a quantum cascade laser. Appl. Phys. Lett. 101, 44101 (2012). croscopic Fourier transform infrared spectra by extended multiplicative signal cor- 43. M. Pfeifer, A. Ruf, P. Fischer, Indirect using quantum cascade rection. Appl. Spectrosc. 62, 259–266 (2008). lasers: Mid-infrared refractometry and photothermal spectroscopy. Opt. Express 21, 70. P. Lasch, Spectral pre-processing for biomedical vibrational spectroscopy and micro- – 25643 25654 (2013). spectroscopic imaging. Chemom. Intell. Lab. Syst. 117, 100–114 (2012). 44. D. Zhang et al., Depth-resolved mid-infrared photothermal imaging of living cells and 71. M. J. Nasse et al., High-resolution Fourier-transform infrared chemical imaging with MEDICAL SCIENCES organisms with submicrometer spatial resolution. Sci. Adv. 2, e1600521 (2016). multiple synchrotron beams. Nat. Methods 8, 413–416 (2011). 45. A. Totachawattana et al., Vibrational mid-infrared photothermal spectroscopy using 72. X. Lin et al., All-optical machine learning using diffractive deep neural networks. – a fiber laser probe: Asymptotic limit in signal-to-baseline contrast. Opt. Lett. 41, 179 Science 361, 1004–1008 (2018). 182 (2016). 73. B. Ehteshami Bejnordi et al.; The CAMELYON16 Consortium, Diagnostic assessment of 46. Y. Bai, D. Zhang, C. Li, C. Liu, J.-X. Cheng, Bond-selective imaging of cells by mid- deep learning algorithms for detection of lymph node metastases in women with infrared photothermal microscopy in high wavenumber region. J. Phys. Chem. B 121, breast cancer. JAMA 318, 2199–2210 (2017). 10249–10255 (2017). 74. M. N. Gurcan et al., Histopathological image analysis: A review. IEEE Rev. Biomed. 47. A. Totachawattana, M. S. Regan, N. Y. R. Agar, S. Erramilli, M. Y. Sander, Label-free Eng. 2, 147–171 (2009). Mid-Infrared Photothermal Spectroscopy and Imaging of Neurological Tissue in 75. R. Bhargava, A. Madabhushi, Emerging themes in image informatics and molecular Conference on Lasers and Electro-Optics, (OSA, 2017), p. ATu4A.3. analysis for digital pathology. Annu. Rev. Biomed. Eng. 18, 387–412 (2016). 48. Z. Li, K. Aleshire, M. Kuno, G. V. Hartland, Super-resolution far-field infrared imaging 76. F. Fereidouni et al., Microscopy with ultraviolet surface excitation for rapid slide-free by photothermal heterodyne imaging. J. Phys. Chem. B 121, 8838–8846 (2017). . Nat. Biomed. Eng. 1, 957–966 (2017). 49. C. Li, D. Zhang, M. N. Slipchenko, J.-X. Cheng, Mid-infrared photothermal imaging of 77. E. Leirset, H. E. Engan, A. Aksnes, Heterodyne interferometer for absolute amplitude active pharmaceutical ingredients at submicrometer spatial resolution. Anal. Chem. vibration measurements with femtometer sensitivity. Opt. Express 21, 19900–19921 89, 4863–4867 (2017). (2013). 50. R. Chatterjee, I. M. Pavlovetc, K. Aleshire, G. V. Hartland, M. Kuno, Subdiffraction 78. B. Bird et al., Infrared micro-spectral imaging: Distinction of tissue types in axillary infrared imaging of mixed cation perovskites: Probing local cation heterogeneities. lymph node histology. BMC Clin. Pathol. 8, 8 (2008). ACS Energy Lett. 3, 469–475 (2018). 79. F. N. Pounder, R. K. Reddy, R. Bhargava, Development of a practical spatial-spectral 51. B. J. Davis, P. S. Carney, R. Bhargava, Theory of mid-infrared absorption micro- analysis protocol for breast histopathology using Fourier transform infrared spec- spectroscopy: II. Heterogeneous samples. Anal. Chem. 82, 3487– 3499 (2010). troscopic imaging. Faraday Discuss. 187,43–68 (2016).

Schnell et al. PNAS Latest Articles | 9of9 Downloaded by guest on September 25, 2021