X-Ray Ptychography

WIR SCHAFFEN WISSEN – HEUTE FÜR MORGEN

Andreas Menzel :: Coherent X-Ray Scattering Group :: Paul Scherrer Institut X-Ray Ptychography

“Phaseless Imaging in Theory and Practice” IMA, University of Minnesota, Aug 14–18, 2017 Ana Andreas Esther Johannes Klaus Manuel Mariana Michal Viviane Xavier Diaz Menzel Tsai Ihli Wakonig Guizar-Sicairos Verezhak Odstrčil Lütz-Bueno Donath

Andreas Menzel :: Coherent X-Ray Scattering Group :: Paul Scherrer Institut and Special Thanks to: • Mirko Holler, Jörg Raabe, … (OMNY) • Christian David and his X-Ray Optics Group • Bernd Schmitt and his Detector Group • Derek Feichtinger and his Scientific Computing Group • J.C. da Silva (now ESRF) • Alessandro Sepe (Adolphe Merkle Institute, Fribourg) • Sarah Shahmoradian (BIO @ PSI) • and other users of cSAXS [email protected] “Phaseless Imaging in Theory and Practice” IMA, University of Minnesota, Aug 14–18, 2017 X-Ray Microscopy

[email protected] “Phaseless Imaging in Theory and Practice” IMA, University of Minnesota, Aug 14–18, 2017 Hierarchical Structures Organized Over Many Orders of Magnitude

image credit: H.D. Barth et al., Bone 46 (2010) 1475

Spongy Bone

Compact Bone

Amino Acids Tropocollagen Fibrils Arrays Fiber Patterns Osteons and Bone Tissue ~1nm ~300nm ~10µm ~50µm Haversian Canals ~50cm ~100µm Mineralized Collagen Fibrils ~1µm

[email protected] “Phaseless Imaging in Theory and Practice” IMA, University of Minnesota, Aug 14–18, 2017 Contrast

• X-rays can be element specific. • X-rays can be chemically specific. • X-rays can be magnetically specific.

further advantages: • X-rays can probe buried structures. • Photon-in and photon-out (i.e., electric and/or magnetic fields don’t disturb the measurement.) • X-rays are weakly interacting, i.e., nano-magnets, for instance, are not switched. E || antiferromagnetic axis E ⊥ antiferromagnetic axis

[email protected] “Phaseless Imaging in Theory and Practice” IMA, University of Minnesota, Aug 14–18, 2017 Arguments for Low Energies High Contrast, e.g., the “Water Window”

image credit: Kirz et al., Q. Rev. Biophys. 28 (1995) 33 [email protected] “Phaseless Imaging in Theory and Practice” IMA, University of Minnesota, Aug 14–18, 2017 Arguments for High Energies Penetration Depth, Depth of Focus, …

A rather general limitation of the depth of ﬁeld:

r2 T<↵ 100

T sample thickness 80 for λ = 2Å δr (lateral) resolution λ sample thickness 60 α factor in the range 2–5.2 40 resolution (nm) Assuming a cylindrical object, 20 of which you’d like to resolve 3 T ≈10 ⋅δr, requires λ≲δr/200. 0 0 50 100 150 200 sample thickness (µm)

[email protected] “Phaseless Imaging in Theory and Practice” IMA, University of Minnesota, Aug 14–18, 2017 X-Ray Microscopy An Equivalent to Optical Microscopes

Resolution limited by • the size and quality of the lens • mirrors oﬀer high eﬃciency but introduce aberrations that are hard to correct for

image credit: U. Neuhäusler et al. J Phys D Appl Phys 36(10A) (2003) 79 [email protected] “Phaseless Imaging in Theory and Practice” IMA, University of Minnesota, Aug 14–18, 2017 X-Ray Microscopy Projection Based

Resolution limited by Depending on geometric details one distinguishes • extent of the point source • contact regime, • pixel size of the detector • Fresnel regime, and • holographic regime

image credit: T. Gorniak et al. Optics Express 19(12) (2011) 11059 [email protected] “Phaseless Imaging in Theory and Practice” IMA, University of Minnesota, Aug 14–18, 2017 X-Ray Microscopy Scanning Probe

Resolution limited by Frequently used with only a point detector • spot size on the sample Alternative contrast mechanisms easily available • positioning accuracy in parallel

image credit: W. De Nolf (Univ. Antwerpen) [email protected] “Phaseless Imaging in Theory and Practice” IMA, University of Minnesota, Aug 14–18, 2017 X-Ray Microscopy A Brief Summary

TransmissionTransmission X-Ray X-ray Microscope Microscope Projection-Based

Scanning Probe Oftentimes, the limiting factor are X-ray optical elements that are simultaneously • highly resolving, • eﬃcient, and • aberration free.

[email protected] “Phaseless Imaging in Theory and Practice” IMA, University of Minnesota, Aug 14–18, 2017 Coherent Diffractive Imaging

xtrial B Fourier constraint image is consistent with measured intensity xsolution

support constraint xsolution ∈ A ∩ B the object is inside a ﬁnite support

[email protected] “Phaseless Imaging in Theory and Practice” IMA, University of Minnesota, Aug 14–18, 2017 Coherent Diffractive Imaging

xtrial B Fourier constraint image is consistent with measured intensity xsolution

atomicity constraint xsolution ∈ A ∩ B density is consistent with predetermined number of atoms

[email protected] “Phaseless Imaging in Theory and Practice” IMA, University of Minnesota, Aug 14–18, 2017 Coherent Diffractive Imaging

xtrial B Fourier constraint image is consistent with measured intensity xsolution

positivity constraint xsolution ∈ A ∩ B all pixels/voxels are non-negative

[email protected] “Phaseless Imaging in Theory and Practice” IMA, University of Minnesota, Aug 14–18, 2017 Coherent Diffractive Imaging

xtrial B Fourier constraint image is consistent with measured intensity xsolution

some form of sparsity xsolution ∈ A ∩ B

[email protected] “Phaseless Imaging in Theory and Practice” IMA, University of Minnesota, Aug 14–18, 2017 Coherent Diffractive Imaging

This is an ill-posed problem. xtrial B I.e., at least one of the following conditions is not met: existence a solution exists xsolution uniqueness the solution is unique stability the solution’s behavior changes continuously with the initial conditions

xsolution ∈ A ∩ B

[email protected] “Phaseless Imaging in Theory and Practice” IMA, University of Minnesota, Aug 14–18, 2017 Coherent Diffractive Imaging

• “Lensless” microscopy • detection requires high dynamic numerical aperture is limited only range by the detectable scattered signal • specimens need to be isolated, i.e., • Quantitative Information far removed from any other • Numerical propagation scattering objects • refocusing (3D information) • slow convergence rates • aberration correction in particular, low-frequency • Increase in space-bandwidth product information is hard to recover

[email protected] “Phaseless Imaging in Theory and Practice” IMA, University of Minnesota, Aug 14–18, 152017 4. Missing center tolerance Because far ﬁeld diffraction patterns tend to have their signal decrease as the fourth power of spatial frequency, the center pixels can be saturated on many present-day detectors. While one can use a variety of beamstop positions and assemble multiple images to minimize this limitation, it is often the case that one works with coherent diffraction intensities with unreliable or unknown values at some number of pixels near the center. For this reason, we carried out a series of reconstruction simulations with various numbers of missing central pixels in the diffraction intensities, ranging from 2x2 pixels up to 15x15. The 2D oversampling ratio σ, i.e. the entire image array pixel number divided by reconstructed object pixel number is about 4.6 in these simulations, so the corresponding missing speckle number can be calculated by dividing the missing pixel number with σ. To test the oversampling ratio dependency of reconstructions, we generatedCoherent a second simulated Diffractive cell with the same features Imaging and an oversampling ratio of 7.5 by increasing the ice cube size to 512 512 512 pixels and keeping the cell size the same. The missingSusceptibility center sizes for this cellto were Systematic× tested× up to 19 19Errors pixels. × HIO DM RAAR ER 5x5 pixels 5 speckles

7x7 pixels 10 speckles

8x8 pixels 13 speckles

9x9 pixels 17 speckles

Fig. 5. Magnitudes of reconstructed images of the σ = 4.6 simulated cell from different algorithms with varying sizes of missing centers. Only the central part of diffraction patterns and reconstructed images are displayed. The green line indicates where artifacts becomes noticeable in reconstructed images.

Some reconstructed images of the σ = 4.6 simulated cell are shown in Fig. 5. HIO has the highest tolerance for missingX. Huang center et al size. Opt. with Express no signiﬁcant, 18 (2010) artifact 26441 showing up to 13 missing [email protected] speckles. DM works well up to 10 missing speckles.“Phaseless RAAR Imaging and in Theory ER do and not Practice” reconstruct IMA, University images of Minnesota, Aug 14–18, 2017 well with missing data. Artifacts presented in reconstructed images using diffraction pattern with missing center can be modeled as Fourier transform pairs which satisfy both support and Fourier modulus constraints so that they are allowed by algorithms using those constraints. We used the approach proposed in [19] to approximate these missing modes using harmonic oscillator modes as an Coherent Diffractive Imaging Susceptibility to Systematic Errors

Correct Bump-out Bite-in Loose Tight

HIO

RAAR

Fig. 3. Reconstructed image magnitudes with HIO, DM, RAAR and ER algorithms using correct, “bump-out”, “bite-in”, “loose” and “tight” supports. Only the central part of images are displayed.

X. Huang et al. Opt. Express, 18 (2010) 26441 [email protected] “Phaseless Imaging in Theory and Practice” IMA, University of Minnesota, Aug 14–18, 2017 Coherent Diffractive Imaging in 3 Dimensions – via Tomography

H. Jiang et al., P. Natl. Acc. Sci. USA, 107 (2010) 11234 [email protected] “Phaseless Imaging in Theory and Practice” IMA, University of Minnesota, Aug 14–18, 2017 Coherent Diffractive Imaging in 3 Dimensions – Ab Initio Reconstructions in 3D

Approach: • “Oversampling” the intensity distribution in three dimensions. Chapman et al. (2006) J. Opt. Soc. Am. A 23 1179.

(easier said than done in forward scattering, more convenient around Bragg peaks)

Pfeifer et al. (2006) Nature 442 63.

[email protected] “Phaseless Imaging in Theory and Practice” IMA, University of Minnesota, Aug 14–18, 2017 Coherent Diffractive Imaging

This is an ill-posed problem. B I.e., at least one of the following conditions is not met: existence a solution exists xsolution uniqueness the solution is unique stability theunique solution’sstability behaviorthe changessolution’s continuously behavior withchanges the initial continuously conditions with the initial conditions

How to design data collection and how many measurement are required to turn this problem into a better-posed problem? xsolution ∈ A ∩ B

[email protected] “Phaseless Imaging in Theory and Practice” IMA, University of Minnesota, Aug 14–18, 2017 Ptychography

[email protected] “Phaseless Imaging in Theory and Practice” IMA, University of Minnesota, Aug 14–18, 2017 Pty⋅chog⋅ra⋅phy noun from Greek: πτυξ = fold (crease) and: γραφή = writing, drawing

Hoppe, Acta Cryst A 25 (1969) 508; Hegerl and Hoppe, Ber Physik Chemie 74 (1970) 1148 [email protected] “Phaseless Imaging in Theory and Practice” IMA, University of Minnesota, Aug 14–18, 2017 c c

R R R R q r Ptychographic Sampling Ratio f c R

Q q q q q a b c f f

Q q QQ Q r Q QQ Q Q R r Q rr r r rr r r

1 I(q; R)= (q; R) 2 = dr O(r) P (r R)exp( iq r) | | · Z1 r P ⇤ (r) ⌘ R,q | {z } 1Q = dr O(r)PR⇤ ,q(r) The measurable quantity I(q;R) is a spectrogram (a). It represents the energy Z1 density of O(r) in the phase space neighborhood of the position (q;R). The analysis task includes the deconvolution into object (b) and illumination (c).

J.C. da Silva, A. Menzel, Opt Express 23(26) (2015) 33812 [email protected] “Phaseless Imaging in Theory and Practice” IMA, University of Minnesota, Aug 14–18, 2017 Ptychographic Sampling Ratio

The measurable quantity I(q;R) is a spectrogram (a). It represents the energy density of O(r) in the phase space neighborhood of the position (q;R). The analysis task includes the deconvolution into object (b) and illumination (c).

J.C. da Silva, A. Menzel, Opt Express 23(26) (2015) 33812 [email protected] “Phaseless Imaging in Theory and Practice” IMA, University of Minnesota, Aug 14–18, 2017 Ptychography Wigner Distribution Deconvolution

Hoppe, Acta Cryst. A 25 (1969) 508; Hegerl and Hoppe, Ber. Physik. Nellist, McCallum, Rodenburg, Chapman, Ultramicroscop 66 Chemie 74 (1970) 1148 Nature 374 (1995) 630 (1996) 153

Theory: J.M. Rodenburg,, R.H.T. Bates, Philos T Roy Soc A 339 (1992) 630

[email protected] “Phaseless Imaging in Theory and Practice” IMA, University of Minnesota, Aug 14–18, 2017 Ptychographic Sampling Ratio

2⇡ =1 “critical sampling” qR poor localization properties of the window functions. Numerical stability will be poor, and the reconstruction may be reduced to phase retrieval on individual, decoupled “views.”

2⇡ in real space: “overlap” > 1 qR in reciprocal space: “oversampling”

2⇡ the window function may not completely ﬁll < 1 qR 4D phase space

T.B. Edo et al., Phys Rev A 87(5) (2013) 053850 J.C. da Silva, A. Menzel, Opt Express 23(26) (2015) 33812 [email protected] “Phaseless Imaging in Theory and Practice” IMA, University of Minnesota, Aug 14–18, 2017 REPORTS

Fig. 2. Preliminary analysis specimen. The amplitude component carries the of a measured data set of absorption information, and the phase expresses 201-by-201 diffraction pat- the phase shift experienced by a wave when terns. This test specimen is a traversing the specimen. In the present case, the Fresnel zone plate of 30-mm phase difference caused by the 1.0-mm-thick gold diameter and with 70-nm structure in the zones is measured to be about outer zone width. (A) SEM 0.67 p,inverygoodagreementwiththeexpected image. The zone plate struc- value of 0.65 p. We note that a complex recon- ture is completely buried under struction carries the necessary information to alayerofgold.Afullrecon- reproduce any traditional microscopy arrange- struction of the region of in- ARTICLEment. IN For instance, PRESS a differential phase contrast terest (solid line) is shown in image, similar to Fig. 2, C and D, is simply Fig. 3. The reconstructed im- given by the gradient of Fig. 3B. An example 230 age of the dashed line region J.M. Rodenburg et al. / Ultramicroscopy 107 (2007) 227–231 is shown in fig S1. (B)Trans- of a simulated phase contrast image is shown in mission of the specimen, ob- fig. S1C. tained from the total transmitted The wavefield of the probe, which is retrieved intensity at each probe posi- at the same time as the object, is shown in Fig. wave (Fig. 2b). Thetion. background (C)Phasegradientofthe phase ramp arises from4A. The knowledgeThis of both is the a phase crucial and the advantage of the technique: it is not transmitted wave in the hori- amplitude of the wave gives the most complete zontal direction. (D)Phasegra- information for subsequent analysis. Computed the cover slip not beingdient in the vertical parallel direction. with the mounting slide.propagationdependent of this wavefield shows that on the wavelength-scale stability or coherence: a 2500 (B) to (D) involve no phase re- specimen was placed 1.2 mm behind the focal Note that conventionaltrieval and can beZernike produced as phase imaging (or con-plane andparticularly was thus illuminated with a slightly severe limitation in electron imaging. Fig. 2c data are acquired. The pixel size is equal to the scanning grid spacing (100 nm), but the resolution is curved and diverging wavefront. Ptychographylimited by the size of the probe (about 300 nm). Unlike the few other known methods (11, 28), ventional bright-field transmission electron imaging), losethe probeshows reconstruction does the not require image any we obtain by scanning our object across on July 29, 2008 Fig. 3. SXDM reconstruction knowledge of the pupil of the Fresnel zone plate. low frequenciesA Brief image History…of the complex components, optical trans- and so would beIn fact, it doesthe not require illumination a pupil plane. Probes function and simply detecting the transmission function of the zone formed with refractive (29)orreflective(30)op- plate specimen. (A)Amplitude tics can be equally well reconstructed, as well as unable to detect suchand (B large)phaseoftherecon- scale phase variation. unfocused probesmitted formed with aintensity. mask. The door For comparison, Fig. 2d shows the struction of a selected region to a general wavefront characterization procedure We thus reconstructof 61-by-61 the diffraction total pat- exit wavefield, similar toappears toconventional be even more widely open when one bright-field lens image of the same object 2000 terns, from the same data set realizes the symmetry in the multiplicative re- that was used for Fig. 2. The lation between P and O in Eq. 1. Transferring the www.sciencemag.org that encoded in a hologram,magnified region shows im-but obtained without the useoversamplingobtained requirement to the object on function a 100 objective optical microscope. perfections in the nanofab- placed downstream—for instance, by using a Â of a reference wave,ricated lens zones. Other or defects, interferometer of any kind.simple pinhole—Theallows the characterization resolution of of our reconstruction is determined by corroborated by a SEM image, arbitrary large and extended regions of an are shown in fig. S1. The out- incoming wavefield.the angular In this wavefront-sensing size of the diffraction pattern we process, which ermost rings of the zone plate configuration, what was the “probe” becomes the (70 nm wide) are very well resolved. The pixel size is 18 nm. The total acquisition time for this image is main object of interest: It can be the propagated 186 s, with a total dose of 2 × 109 photons.

wave perturbationis limited from any specimen geometrically or optical Downloaded from by the size of our CCD camera and device. This feature will be useful for the Fig. 4. Reconstructed characterizationthe of focusing camera devices to be length. used In the image presented in Figs. 2a and b, 1500 probe. (A)Colorrendition at future coherent x-ray light sources. of the complex-valued probe We have demonstrated an imaging technique at the specimen plane. (B) that is fullywe compatible process with STXM and only has the a semi-angle of 37 mrad, giving an intrinsic The computed wavefield high-resolution potential of CDI approaches. The propagated back to the reconstructionresolution method removes the main of diffi- 17 mm. The reconstruction has been per- focal plane, situated 1.2 mm culty of ptychography by retrieving the complex- upstream. (C)Asectionof valued image of both the specimen and the probe. the probe wavefield along formed with an image pixel size of 13 mm. The important The method is noninvasive,(a) and radiation dam- (b) (c) the propagation direction, age can be reduced by a combination of fast as calculated from the data large-area scans with longer high-resolution im- in (A). The dashed line, in- difference to note is between Fig. 2c and Figs. 2a and 2b: aging of regions of interest. The technique can be dicating the plane of (B), is at the waist of the wave used with hardthe and soft effective x-rays, can be combined gain in resolution over the intrinsic resolution 1000 distribution. (D)Thesquared with other scanning methods such as fluores- amplitude of the Fourier cence imaging, and can be extended to nano- transform of the probe. (E) diffraction mappingof our or nanospectroscopy. only Higher imaging component—a simple aperture—is a Measured diffraction pat- coherent flux and improved focusing optics (31) tern of the probe without should soonfactor provide conditions of for 47. better-than- In the case of scanning transmission electron aspecimen.Thisdata,which 10-nm resolutions, making possible the imaging was not used in the re- of the finest structures(d) in state-of-the-art elec- (e) (f) construction, is in good agreement with the retrieved probe. tronics devicesmicroscopy or the macromolecular assemblies (STEM), it is routinely possible to obtain a 0.4 1 Rodenburg, Hurst, Cullis, Thibault et al., Science 321 Guizar-Sicairos,PIE Fienup, 0.2 nm probe (illumination)0.8 function. A similar gain in www.sciencemag.org SCIENCE VOL 321 18 JULY 20080.3 NL optimization381 Ultramicroscopy 107 (2007) 227 (2008) 379 Opt Express 16 (2008) 72640.6 E resolution would∆r then achieve ultimate wavelength-limited 500 0.4 0.1 1 imaging (about 1000.2 of an atomic diameter): only the zero- 0 0 0 point100 200 energy300 400 of500 the0 atomic100 200 vibrations300 400 500 would deﬁne useable Iteration Iteration resolution.(g) Even in conventional(h) light optical imaging, no Fig. 3. Reconstruction results. Amplitude and phase after 500 iterations are shown in (a) andlonger (b) for the PIE and requiring (d) and (e) for the nonlinear a lens optimization that approach. is (c) mounted and (f) show close to the sample the amplitude of the initial estimate of p(x,y) and the result after nonlinear optimization respectively.would (g) Normalized have invariant important error metric, E, vs. iteration beneﬁts. number. (h) Shift error, For example, remote ∆r vs. iteration number for the nonlinear optimization algorithm. Phase is shown from 0.4 − 0 to 0.4 radians in (b) and (e). A 172 210 portion of the 256 256 array is shown in (a)-(f). 1969 1971 1973 1975 1977 1979 1981 1983 1985 1987 1989 1991 1993 1995 1997 1999imaging2001 2003 2005 will2007× in principle2009 2011× 2013 allow2015 uncompromised2017 resolution

Duringto the reconstruction be obtained the aperture was assumed for to objects have a radius of 52 held pixels (instead within pressurised or Number of citations of all preceding publications markedof 50), as shown by in Fig. the 3(c). Noticekeywords that an error of this nature would occur not only from an inaccurateenvironmental measurement of the aperture, but also enclosures. from an inaccurate knowledge Furthermore, of the distance since we have “ptychography,” “inhomogeneous primary wave,” or “Wignerfrom the distribution sample to the detector, sincedeconvolution.” the sampling in object domain scales proportionally to the propagationrecovered distance. Additionally, the zero-mean entire Gaussian-distributed wavefunction, noise was added all to the depth information is Fig. 2. Images of the object function; illumination conditions are as in Fig.actual translations of the object, thus yielding a shift error of ∆r = 0.8 pixel. The amplitude and Data according to Thomson Reuters, Web of Knowledge,phase of accessed thealso reconstruction encoded with Jun 500 iterations 13, in our of2017 the PIE data, algorithm. is leading shown in Figs. 3(a) to and the possibility of 3D 1. (A) Modulus of the reconstructed image of the object after 1503(b), respectively.imaging Notice that without the quality of the lenses.image has degraded significantly. [email protected] iterations: dark features correspond to absorption.“Phaseless (B) Phase Imaging of in Theory the4. Nonlinear and Practice” optimization IMA, approach University of Minnesota, Aug 14–18, 2017 In Figs. 3a and b, we present the amplitude and phase of reconstructed object, corresponding to its optical thickness. Sharp changesFor phase retrieval techniques that use diversity, where the problem is robustly constrained, nonlinear optimization algorithms have been shown to be a valuable tool for reconstruction. in intensity correspond to phase wraps of 2p in thick areas of the object.The principala reconstruction benefit of this approach is that we using can straightforwardly no lenses include any at form all. of non- In order to make our The background phase ramp is due to the cover slip not being perfectly simple detector possess a large angle effective numerical #94330 - $15.00 USD Received 27 Mar 2008; accepted 24 Apr 2008; published 5 May 2008 parallel to the mounting slide, indicating that our method reconstructs(C) 2008 OSAaperture, we12 have May 2008 / Vol. moved 16, No. 10 / OPTICS it toEXPRESS a 7273 position just 15 mm large scale phase changes which are normally invisible in conventional downstream of the object. The aperture we employ in this Zernike phase plate imaging. (C) Image obtained by scanning the aperture across the object to the same positions used in the reconstruction: this experiment is 150 mm in diameter. The nominal resolution represents the intrinsic, unprocessed resolution of the optical configuration. of the reconstruction (part of the ant shown in Fig. 2) is (D) Conventional bright-field image obtained on a 100 microscope. 3 mm: this increase is due to the larger scattering angle we Â

50 50

100 100

150 150

200 200

(A) 50 100 150 200 (B) 50 100 150 200

Fig. 3. Modulus (A), and phase (B), obtained with the detector close (15 mm) to the object. The aperture used was 150 mm in diameter and the nominal resolution of this image is 3 mm. However, because the detector is no longer formally in the Fraunhofer diffraction plane (by several factors), errors occur. The reconstruction is nevertheless reasonable. Ptychography A Brief History…

Dierolf et al., Esmaeili et al., Nature 467 (2010) 436. Macromolecules 46 (2013) 434.

Maiden et al., Schropp et al., Vine et al., Rev. Sci. Instrum. Nature Commun 4 (2013) 1669. Sci. Rep. 3 (2013) 1633. 83 (2012) 033703.

[email protected] “Phaseless Imaging in Theory and Practice” IMA, University of Minnesota, Aug 14–18, 2017 Ptychography Recent Developments

3D reconstruction using a multiple modes, diﬀerent geometries multi-slice approach state mixtures and variations, e.g., near-ﬁeld ptychography, instabilities Fourier ptychography… Maiden et al., J Opt Soc Am A 29 (2012) 1606. Thibault, Menzel, Stockmar et al., Sci. Rep. Suzuki et al., Phys Rev Lett 112 Nature 494 (2013) 68 3 (2013) 1927. (2014) 053903. Odstrčil et al., Opt. Express 24 Zhen et al., Nat. Photonics Shimomura et al., Phys Rev B 91 (2016) 8360. 7 (2013) 739. (2015) 214114.

[email protected] “Phaseless Imaging in Theory and Practice” IMA, University of Minnesota, Aug 14–18, 2017 A Rather General Mission Statement (from the applications’ point of view)

DAQ speed, throughput

field of view processing speed

resolution

image analysis

specificity, sensitivity dose efficiency 2.5ms acquisition [email protected] “Phaseless Imaging in Theory and Practice” IMA, University of Minnesota, Aug 14–18, 2017 Some Sort of Roadmap Imaging of Integrated Circuits

achieved* achieved† demanded# 3D “3D” 2D (tomography) (via laminography)

ﬁeld of view 500 × 288 µm2 ⌀ ~ 10µm 10 × 10 mm2

resolution ~ 40 nm ~ 15 nm 10 nm

measurement ~ 70 min ~ 1 day 25 days time ~ 4 × 103 s ~ 9 × 104 s ~ 2 × 106 s eﬀective ~ 40 µs ~ 170 µs ~ 0.1 µs dwell time

*Guizar-Sicairos et al., Opt. Express 22 (2014) 14859 †M. Holler et al., Nature 543(7645) (2017) 402 #http://www.iarpa.gov/index.php/research-programs/raven [email protected] “Phaseless Imaging in Theory and Practice” IMA, University of Minnesota, Aug 14–18, 2017 OMNY (tomography nano cryo endstation)

[email protected] “Phaseless Imaging in Theory and Practice” IMA, University of Minnesota, Aug 14–18, 2017 OMNY (tomography nano cryo endstation) …inside the chamber…

Microscope Sample Changer for sample/optics alignment

Tracking Interferometer Sample Stage Optics Mount (FZP, central stop)

Sample Parking

[email protected] “Phaseless Imaging in Theory and Practice” IMA, University of Minnesota, Aug 14–18, 2017 Ptychography Brain Tissue

unstained cryo-preserved mouse brain

S. Shahmoradian et al., Sci. Rep. 7 (2017) 6291

myelinated axons nuclei lysosomal lipofuscin?

[email protected] “Phaseless Imaging in Theory and Practice” IMA, University of Minnesota, Aug 14–18, 2017 flOMNI (flexible tomography nano imaging)

Interferometer

X-Ray Fluorescence Microscope

Sample Changer

X-Ray Optics

Sample Tray

Sample Stages (i.e., scanner and rotation stage)

M. Holler et al., Rev Sci Instrum 83 (2012) 073703 M. Holler et al., Sci Rep 4 (2014) 3857 [email protected] “Phaseless Imaging in Theory and Practice” IMA, University of Minnesota, Aug 14–18, 342017 Applications Characterization of Organic Solar Cells

• 3D resolution: 20 nm • Defects in the structure causing failure could be identified by ptychographic tomography

E.B.L. Pedersen et al., Nanoscale 7 (2015) 13765 [email protected] “Phaseless Imaging in Theory and Practice” IMA, University of Minnesota, Aug 14–18, 2017 Applications FCC (Fluid Catalytic Cracking) Particles

• 3D resolution: 40 nm • Scale Bars: 5 µm • Resonant X-ray ptychographic tomography on and off the Fe edge

J. Ihli, …, J. van Bokhoven; Nature communications, in print [email protected] “Phaseless Imaging in Theory and Practice” IMA, University of Minnesota, Aug 14–18, 2017 Imaging of Integrated Circuits Intelprocesser, 22nm Technology

M. Holler et al., Nature 543(7645) (2017) 402 [email protected] “Phaseless Imaging in Theory and Practice” IMA, University of Minnesota, Aug 14–18, 2017 M. Yabashi and H. Tanaka, Nat Photonics 11 (2017) 12 [email protected] “Phaseless Imaging in Theory and Practice” IMA, University of Minnesota, Aug 14–18, 2017 [email protected] “Phaseless Imaging in Theory and Practice” IMA, University of Minnesota, Aug 14–18, 2017 Comprehensive Approach

empty excluding blue 2D red 3D overhead Detector Technology 100000

kHz data acquisition 10000

1000

Eiger “selﬁe” Experimental Design 100

optimizing illumination and sample positioning Resolution Element / Second 10 2007 2008 2010 2011 2012 2013 2015 2016 2017

1000 OMNY 100 Analysis Development

“multi mode” “on the ﬂy” 10 A B

1 2007 2008 2010 2011 2012 2013 2015 2016 2017 C 38.9% 27.6% 16.7% Resolution Estimate [nm]

7.8% 2.7% Acquisition rate could be doubled yearly, , 0 2.5ms acquisition resolution approaching 10 nm (in 3D) and below (in 2D) [email protected] “Round Beams” Workshop, SOLEIL, June 14–16, 2017 Comprehensive Approach

All parts of data acquisition and analysis will have to be matched to the drastically increased ﬂux that will soon be available.

This includes sample/beam positioning hardware, detector technology, and the need to make analysis more eﬃcient and performant.

It also suggests (re-)visiting sampling strategies. How to condition the data set for fast and reliable analysis? (In particular for “advanced” analysis schemes including multi-slice, ab initio 3D, multi-mode, etc.)

[email protected] “Phaseless Imaging in Theory and Practice” IMA, University of Minnesota, Aug 14–18, 2017 Illumination Design

pinhole

lens

lens + pinhole

Structured illuminations were found beneficial to reconstruction stability, image resolution, and dose efficiency and allow for “super-resolution” even in case of formally “under-sampled” data.

A. Maiden et al., JOSAA 28 (2011) 604. Guizar-Sicairos et al., Phys Rev B 86 (2012) 100103. [email protected] “Phaseless Imaging in Theory and Practice” IMA, University of Minnesota, Aug 14–18, 2017 Challenges

[email protected] “Phaseless Imaging in Theory and Practice” IMA, University of Minnesota, Aug 14–18, 2017 3D Ptychography

or ?

We’re likely to impose too much radiation onto 3D samples by requiring 2D projections to reconstruct reliably. How to avoid the “detour” via 2D projections? [email protected] “Phaseless Imaging in Theory and Practice” IMA, University of Minnesota, Aug 14–18, 2017 Good Practice — Figure of Merit

F. Hue et al. Ultramicroscopy, 111 (2011) 1117

The community needs to establish criteria to asses the reliability of ptychographic measurements. Beyond the image, what meta-data needs to be communicated? Resolution is rather crude a measure for image quality, as is self consistency. What would be suitable ﬁgures of merit for both measurement and reconstruction? [email protected] “Phaseless Imaging in Theory and Practice” IMA, University of Minnesota, Aug 14–18, 2017 Information Theory

exploiting concise notions, such as • sparsity, • constraints, • degrees of freedom, • parameterization, • articulations, …

information level << sparsity level << pixels*grey levels Claude Shannon (1916–2001)

Despite the measurement success of ptychography, the theoretical foundation is still insuﬃcient to address fundamental issues of experimental design, data acquisition strategy, and reconstruction modalities.

[email protected] “Phaseless Imaging in Theory and Practice” IMA, University of Minnesota, Aug 14–18, 2017 Questions? Comments? Debate…

• image credit: Nick Veasey [email protected] “Phaseless Imaging in Theory and Practice” IMA, University of Minnesota, Aug 14–18, 472017