Modelling Point Spread Function in Fluorescence Microscopy

Modelling point spread function in ﬂuorescence microscopy with a sparse Gaussian mixture: trade-off between accuracy and efﬁciency Denis K. Samuylov, Prateek Purwar, Gábor Székely, and Grégory Paul

Abstract—Deblurring is a fundamental inverse problem in space, digital or analog, the task is qualitatively different. In bioimaging. It requires modelling the point spread function the digital case, the number and position of the light sources (PSF), which captures the optical distortions entailed by the in physical space are lost: only the integrated intensity at each image formation process. The PSF limits the spatial resolution attainable for a given microscope. However, recent applications pixel position is reconstructed. On the contrary, an analog require a higher resolution, and have prompted the development reconstruction aims at resolving the number, the position, and of super-resolution techniques to achieve sub-pixel accuracy. This the intensity of the emitting light sources in physical space. requirement restricts the class of suitable PSF models to analog The PSF and the inherent noise in the acquisition process make ones. In addition, deblurring is computationally intensive, hence this inverse problem ill-posed and ill-conditioned. Therefore, further requiring computationally efficient models. A custom candidate fitting both requirements is the Gaussian model. “good” models about the forward problem and the imaged However, this model cannot capture the rich tail structures objects are necessary for solving this problem. In this work, found in both theoretical and empirical PSFs. In this paper, we focus on modelling the first step of the image formation we aim at improving the reconstruction accuracy beyond the process by finding a PSF representation, suitable for solving Gaussian model, while preserving its computational efficiency. We imaging inverse problems in an analog reconstruction space. introduce a new class of analog PSF models based on Gaussian mixtures. The number of Gaussian kernels controls both the The quality of the PSF model determines the accuracy of the modelling accuracy and the computational efficiency of the model: imaged object reconstruction, thus its knowledge is fundamen- the lower the number of kernels, the lower accuracy and the tal for solving bio-imaging inverse problems [4], [5], [6]. One higher efficiency. To explore the accuracy–efficiency trade-off, we approach is to assume that the PSF is unknown and to include propose a variational formulation of the PSF calibration problem, its estimation in the inverse problem (blind deconvolution, where a convex sparsity-inducing penalty on the number of Gaussian kernels allows trading accuracy for efficiency. We see e.g. [7], [8]). However, it makes the problem more ill- derive an efficient algorithm based on a fully-split formulation of conditioned. Therefore, we focus on estimating the PSF as alternating split Bregman. We assess our framework on synthetic a separate task. We also restrict ourselves to shift-invariant and real data and demonstrate a better reconstruction accuracy imaging systems, excluding space-varying PSF models. in both geometry and photometry in point source localisation—a PSF models can be classified according to different criteria: fundamental inverse problem in fluorescence microscopy. model-based vs. phenomenological, digital vs. analog, and Index Terms—Quantitative fluorescence microscopy, model- calibrated vs. uncalibrated. The PSF model can be derived based image processing, Bayesian modelling, alternating split either from modelling the imaging systems or from imaging Bregman, point spread function, parametric dictionary, virtual microscope framework point-like objects. A digital PSF is well-suited for a digital reconstruction space, but not adapted to analog reconstructions; whereas an analog PSF can be used both in analog and digital I.INTRODUCTION reconstructions (after appropriate integration and sampling). NCOHERENT imaging systems are linear in power (i.e. Finally, if the PSF model is calibrated experimentally on a arXiv:1809.01579v2 [eess.IV] 28 Feb 2019 I intensity) and are well described by linear system the- specific imaging system, we call this PSF calibrated. These ory: the mean intensity image results from the superposition criteria define a range of possible models. At the two extremes integral of the true object intensity weighted by the point stand the theoretical (derived from optics) and the empirical spread function (PSF). The PSF encodes the propagation of (punctual source measurements) PSF models. The former the incoherent light emitted by a unit-intensity point source is analog and can potentially achieve sub-pixel resolution, object through the optics up to the array of photo-detectors. In but at the expense of costly convolution algorithms. On the turn, the mean intensity parameterises the statistical model that contrary, the latter is digital and therefore amenable to efficient accounts for the intrinsic stochastic nature of light emission convolution algorithms (based on the fast Fourier transform, (see e.g. [1], [2], [3]). Together with other sources of distortion, FFT), but at the expense of a loss in accuracy. this defines the image formation process, i.e. the forward Theoretical PSF models are derived from modelling the problem. microscope optics. Their differences lie in their level of simpli- Solving the inverse problem for an incoherent imaging sys- fication. The simplest models use the paraxial approximation, tem amounts to reconstructing the number (amount), the posi- which is valid for objectives with a small numerical aperture tion (geometry), and the intensity (photometry) of the emitting (NA), thin imaged specimens, and ideal acquisition conditions light sources. Depending on the nature of the reconstruction [9]. More advanced models account for the spherical aberra- 2 tions due to sample thickness and refractive indexes mismatch number and placement of the basis functions. Nonetheless, [10]. Recent models use the vectorial theory of diffraction to to the best of our knowledge, this trade-off has not yet been extend PSF models to objectives with a high NA [11], [12], systematically explored in the literature. [13], [14]. In practice, this class of PSF models is difficult In this paper, we aim at improving the PSF reconstruction to use because hard or impossible to calibrate (e.g. unknown accuracy beyond single Gaussian models that misapproximates refractive index of the immersion medium [6]). In addition, the PSF tails [28]. Therefore, we propose a new class of a specific imaging system can deviate from these idealised phenomenological PSF models with an adjustable level of models in at least two ways (e.g. [10], [15]): the experimental complexity that can be tuned to achieve a desired accuracy– setup does not fit the model assumptions or variations in the efficiency trade-off. We model the PSF as a sparse mixture of optics (e.g. lens imperfections, optics misalignment) create multivariate Gaussian distributions. The motivation behind this distortions not accounted for by the models. choice is a fast algorithm to compute non-gridded convolutions Empirical PSF models are derived from calibration data, with Gaussian kernels: the improved fast Gaussian transform namely an image stack of point-like objects, such as quantum (IFGT, [33]). dots, fluorescent beads (e.g. [16], [17], [18], [19], [20]) or small structures within the sample [21]. Calibration data II.FORWARD PROBLEM provide an accurate representation of the PSF only for specific To ease the reading of the presentation of the forward imaging conditions [16]. Any deviation from the expected PSF problem we provide in Table V a summary of the notations image can be used to identify problems in the experimental used in the forthcoming sections in the appendix. setup (e.g. oil mismatch, vibrations from the heating system [15], [20]). However, it also imposes a restriction: calibration A. Object model data must be acquired under the same conditions as the speci- We assume that the PSF measurements result from imaging men, i.e. from point-like objects embedded within the sample a fluorescent bead with a size below the resolution limit [5], [15]. This is rarely feasible in practice and may require of the microscope objective, thus approximating an idealised advanced measurement techniques [22]. Moreover, empirical point source (PS) object [15]. We further assume that dur- PSF models require correcting the corruptions entailed by the ing acquisition, the fluorescent bead is immobile and emits image formation process, e.g. by removing the background photons at a constant rate (i.e. we neglect photo-bleaching). signal, denoising, or accounting for the extended geometry of The total photon flux decomposes as the contribution of the point-like objects (e.g. [23], [4], [15], [24]). photons emitted by the PS located at xps ∈ 3 at rate The two ends of the spectrum of possible PSF models R ϕps ∈ and the photons emitted by the background (due illustrate a trade-off between accuracy and efficiency. On R+ to autofluorescence) at a constant rate ϕbg ∈ . We use one end, theoretical PSF models can be used to resolve the R+ the measure-theoretical framework of [34] to model the total localisation of individual point sources in physical space, at photon flux as a spatio-temporal object measure: the expense of robustness (difficulty to fit a specific setup) obj ps bg and computational efficiency (analog PSF are typically more φ (dy × dt) = ϕ δxps (dy) + ϕ dy dt . (1) expensive to evaluate than digital ones). On the other end, digital PSF models are computationally efficient to use (e.g. B. Image formation process thanks to fast algorithms such as FFT), but their resolution is The image formation in fluorescence microscopy can be limited by the pixel grid. Between these two extremes, one modelled as a two-stage process [34], [1]. The object-to-pixel can trade accuracy for efficiency or vice versa. For example, mapping models the optical distortions due to the random theoretical PSF models can be adjusted to a specific setup by nature of light emission and propagation: it integrates the using PSF measurements (e.g. [6], [25], [24]). This improves expected photon flux emitted by a set of fluorescent objects robustness, but does not solve the computational efficiency located in physical space over the pixel surface during the problem. For this purpose, a well-spread solution is to use a exposure time and models the number of photons hitting Gaussian PSF model approximating the real PSF of a specific each photo-detector. The pixel-to-image mapping models the imaging system (e.g. [26]). Such models are well suited for conversion of photon counts collected at each pixel to image modelling the in-focus section of the PSF [27], but results in a grey values and accounts for the measurement noise, e.g. due poor approximation of the PSF tails [28] and are not suitable to the photoelectric effect, signal amplification, noise in the for modelling the 3D PSF of a widefield microscope [26]. In electronic circuitry and quantisation [35], [36]. order to explore the trade-off between accuracy and efficiency, 1) Object-to-pixel mapping: The PSF tails are in the low- a solution is to use analog phenomenological models. These intensity region where the fluctuations inherent to light emis- models are flexible enough to accommodate real PSF shapes, sion are measurable and photon shot-noise is significant. and still computationally efficient. Moreover, their complexity Therefore, we use a Poissonian model for the photon counting can be adjusted according to the desired trade-off between statistics. We denote the imaging volume as Ω ⊂ R3 and accuracy and efficiency. For example, Zernike moments can model the photon count at pixel Pj ⊂ Ω during the exposure encode reflection-symmetric PSFs [29], [30] with a complexity photon time t as a random variable N following a Poisson controlled by the number of moments used in the model. e j distribution with mean intensity µ : More recently, polynomial B-splines were proposed [31], [32]. j photon In that case, the trade-off can be explored by varying the Nj ∼ Poisson(µj) . (2) 3

40 45 50 55 60 5 10 15 20 25 10 30 50 70 90

(a) Synthetic Born and Wolf (b) Wideﬁed ﬂuorescence microscopy (c) Laser scanning confocal microscopy

Figure 1. Fluorescent bead image data for different imaging modalities. (a) Synthetic image stack of an idealised PS object using the forward model in Section II and the BW PSF model. (b)-(c) Real image data of nanoscale beads acquired using a widefied fluorescence microscopy (b) and a laser scanning confocal microscopy (c). Orthogonal z- and x- mean projections (first row) and sample focal planes (second row) shown in log-colour scale.

In incoherent imaging, the mean intensity at a given location 2) Pixel-to-image mapping: Modelling the conversion of in the image plane results from the superposition of the photons hitting the detector surface into grey values depends independent contributions of the light sources distributed in on the type of camera [35], [36]. We model the conversion by physical space. For each individual light source, the photon a deterministic affine mapping, denoted ν: flux observed under a specific imaging system is entirely char- N grey = q M f −1N photon + b =: ν(N photon) , (6) acterised by the PSF. After a proper normalisation, the PSF j λ j j can be understood microscopically as the transition probability where qλ is the quantum efficiency at emission wavelength kernel, denoted κ(dx | y), translating individual photons in λ, M is the multiplication gain, f is the analog-to-digital the image plane around dx, conditionally on the position y of proportionality factor, and b is the camera bias. the emitting light source [37], [1], [38]. For a general object modelled as a spatio-temporal measure, C. PSF models φ j denoted , the expected photon count at -th pixel is described We introduce three PSF models spanning the complexity as a superposition integral: range. The classical Born and Wolf (BW) model is used as Z Z a reference idealised model, assuming that the microscope is µj(φ) = κ(dx | y) φ(dy × dt) . (3) operated under design conditions. The single Gaussian (SG) Pj ×[0,te] Ω model is a widespread PSF approximation because of its We assume that the imaging setup is well described by a shift- computational advantages. The Gaussian mixture (GM) model, invariant PSF. In this case, the superposition integral becomes which we propose in this work, represents a whole class of a convolution integral: models that will allow us exploring the accuracy–efficiency trade-off. Each model is parameterised by a set of parameters, Z generically denoted θ, belonging to a set of admissible values, κ(dx | y) φ(dy × dt) = (κ ∗ φ)(dx × dt) . (4) denoted Θ. When ambiguous, we will add a superscript to Ω θ and Θ with the PSF model name. We denote by x := (x, y, z) ∈ 3 an arbitrary point in physical space. For a set of PSs with position and intensity {(xs, ϕs)}s, R equation (4) transforms in a linear combination of shifted PSF: 1) Born and Wolf (BW) model: The model is parameterised by three parameters, i.e. the emission wavelength, the nu- ! ! X X merical aperture, and the refractive index of the immersion κ ∗ ϕsδx (dx × dt) = ϕsκ(x − xs) dx dt . bw bw 3 s medium, denoted θ := {λ, NA, ni} ∈ Θ := R+. The s s BW PSF kernel writes as: Z 1 2 We specialise the above equation to the object model (1) and bw : bw −i Φ(ρ,z) κθ (x) = Cθ J0(k0 r NA ρ) e ρ dρ , use a mid-point quadrature to approximate the photon flux (3): 0 where Cbw is the normalisation constant, J denotes the µ (φ) ≈ µ (xps, ϕps) := c ϕpsκ(x − xps) + ϕbg , (5) θ 0 j j j Bessel function of the first kind of order zero, Φ(ρ, z) = 2 2 k0zNA ρ 2π 2n is the phase term, k0 = λn is the wavenumber where xj ∈ Ω is the centre of the j-th pixel, c := |P|te i i p 2 2 the spatio-temporal integration volume, and |P| the pixel in vacuum, and r = x + y . area. For clarity we have omitted the dependence on ϕbg, 2) Single Gaussian (SG) model: The SG model is simply that is estimated independently in practice. From now on, a Gaussian density, with a particular covariance matrix: this we rescale the photon emission rates by c to obtain the model is radially symmetric around the z-axis. Therefore, it integrated intensities, that we denote for simplicity with the is parameterised by the lateral and axial standard deviations, sg : sg 2 same symbols: ϕps := c ϕps and ϕbg := c ϕbg. We assume denoted θ = (σxy, σz) ∈ Θ ⊂ R+. The SG PSF sg sg 1 T −1 − x Σθ x that the pixels are back-projected onto the object space, i.e. kernel is defined as κθ (x) = Cθ e 2 , where Σθ := 2 2 2 3×3 the lateral pixel coordinates are divided by the magnification diag σxy, σxy, σz ∈ R is the covariance matrix. The sg 3 4 2−0.5 factor and the axial coordinate is aligned with the object [27]. normalisation constant is given by Cθ = 8π σxyσz . 4

n ps ps ps ps o 3) Gaussian mixture (GM) model: (x + xk1, vk1 ϕ ),..., (x + x X , v X ϕ ) that are knk knk a) Definition: To explore the accuracy–efficiency trade- sg convolved independently with κk . The resulting mean images off, we introduce a Gaussian mixture model, namely a convex are summed to produce the mean image of the original imaging combination of Gaussian kernels shifted to different positions. system. The general GM model is described by a parametric dictio- c) Computational efficiency: The GM model can be nary, denoted D, where each atom is a shifted SG kernel: efficiently evaluated by the FFT or the IFGT (see [39]). The S sg D := D κ (· − x ). However, in this paper we (θa,xa)∈θ θa a FFT is suitable for the digital scenario when the object model structure θD by a set of nC covariance matrices: for each and the PSF model are sampled on the same grid. The IFGT sg C θk with K := {1, . . . , n } we associate a set of positions, allows fast evaluation of the forward model based on the S X defined as Xk := xkm with Mk := {1, . . . , nk }: virtual source formulation discussed above. In both scenarios, m∈Mk the computational efficiency of the evaluation of the Gaussian θD := S {θsg} × X =: S θD. The dictionary k∈K k k k∈K k mixture PSF model depends on the size of its support. size corresponds to the number of Gaussian kernels in the P X mixture, defined as |D| = k∈K nk . To ensure the proper normalization of the PSF, a GM model is defined as a convex III.INVERSEPROBLEM combination of the Gaussian kernels in the dictionary D. The We aim to estimate the parameters of the PSF models mixture weight vector is denoted v and belongs to the |D|- from calibration data. In scanning fluorescence microscopy, simplex, denoted ∆|D|, i.e. v must have positive components the image stack is acquired by moving the focal plane over summing to one. The parameters are the dictionary parameters the imaging volume [15]. The camera detector of size nh ×nw gm D and the mixture weights vector, i.e. θ := θ ∪ {v}. The pixels measures the incident light intensity during the exposure GM model then writes: s time te and encodes it into grey values at n positions of the gm X X sg focal plane. The pixel size and the displacement between two κθ (x) = vkm κθ (x − xkm) . (7) k consecutive focal planes are denoted ∆ and ∆ , respectively. k∈K m∈Mk xy z D As a result, the image stack is an array of grey values, denoted To capture the structure of the dictionary parameter set θ , s× h× w Z ∈ n n n , holding measurements from np := nsnhnw we regroup the kernels in D and the mixture weights into nX - Z+ k pixels. We make the standard assumption that the images in a dimensional vectors, denoted κ (x) and v respectively. Their k k stack are acquired simultaneously in time. m-th elements are the translated kernel [κ (x)] := κsg (x − k m θk xkm) and the mixture weight [vk]m := vkm respectively. Then, the GM model (7) writes: A. Estimation problem: maximum a posteriori formulation

gm X T 1) Problem formulation: Given a PSF model κθ, we solve κθ (x) = κk(x) vk . (8) k∈K a maximum a posteriori (MAP) optimisation problem to estimate θ. In general, the position xps and intensity ϕps of We define supp(v) := {(k, m) ∈ KM | [v] > 0}, the km the PS object are also unknown. Therefore, the general inverse support of the mixture weights, and KM := S {k} × M . k∈K k problem aims at jointly estimating the parameters of the PS For a fixed dictionary size, the support of v sets the effective and the PSF models: computational effort to evaluate the GM model by reducing the number of SG kernel evaluations (i.e. the sum in equation (7) arg min nll(xps, ϕps, θ | Z) + η R(xps, ϕps, θ) ps ps runs only over supp(v)). The size of the support defines our x , ϕ , θ (9) ps ps measure of efficiency for the MG model. If the set of kernel s.t. x ∈ Ω, ϕ ∈ R+, θ ∈ Θ positions X := ∪k∈KXk is aligned with the pixel centres, we where nll is called the data-fitting term, R the regularisation call the dictionary digital, otherwise we call it analog. term encoding our prior knowledge about the model parame- b) Virtual source interpretation: We can interpret the ters, and η the regularisation parameter controlling the trade- GM model within the virtual microscope (VM) framework off between them. We call the problem (9) the parametric [34]. The MG model (7) can be rewritten as a superposition blind PS deconvolution. It specialises to two problems: C of n convolutions: a) PS localisation: The PSF is assumed known, and only ! gm X sg X the position and intensity of the PS are estimated. In this case, κ (x) dx dt = κ ∗ v δx (dx × dt) . ps ps θ θk km km the data-fitting term reduces to nll(x , ϕ | Z, θ), and the k∈K m∈Mk PSF model parameters are fixed. X Using the terminology of [34], the nk light sources modelling b) PSF calibration: The PS is assumed known, and only the k-th object measure (located at xkm and of intensity vkm) the PSF model parameters are estimated. The data-fitting term are virtual, in the sense that they do not represent physical light reduces to nll(θ | Z, xps, ϕps), and the PS intensity and sources, but they approximate a photon flux. This observation position are fixed. leads to a representation of the convolution of the GM model 2) Data-fitting functional: In a Bayesian setting, the data- with an arbitrary object that is well-suited for the VM frame- fitting term corresponds to the negative log-likelihood (nll) work [34]: the GM model amounts to approximating the PSF derived from the stochastic image formation model described C sg by n optical systems modelled using κk . For each system, by (2). We convert the grey values into what we call the ps ps X each point source (x , ϕ ) is replaced by nk point sources: raw photon counts using (6) and vectorise it into n := 5

−1 np vec ν (Z) ∈ R+ . Specialising the object-to-pixel map- 4) Convex relaxation: Problem (9) has multiple sources ping defined by (5) for the PSF model parameterised by of non-convexity for the GM model: the joint estimation of θ, we compute the expected photon count at each pixel PS position and intensity, the PSF parameters estimation, and j ∈ {1, . . . , np} =: J and stack the results into a vector, the cardinality regularisation functional for the GM model. ps ps np denoted µ(x , ϕ , θ) ∈ R+ . To compare different models, Nevertheless, it has also convex components: the constraint we use the normalised version of the nll, also called deviance sets are convex, and nll is convex in the mean photon count or Bregman divergence [3], which is zero when over-fitting vector. If the PS position xps and the dictionary parameters (i.e. the model prediction at each pixel is its raw photon count): θD are fixed, the remaining non-convexity is Rgm. To handle more tractable problems, we use a convex sparsity-inducing ps ps : ps ps gm nll(x , ϕ , θ | Z) = nll(x , ϕ , θ | n) regulariser, denoted Rco , based on the popular `1 norm [40]: D n ps ps E = 1 p , n log + µ(x , ϕ , θ) − n . X ηk n µ(xps, ϕps, θ) Rgm(ϕnet) := np kϕnetk . (11) co η k 1 k∈K a) Vector of expected photon counts: The vector of 5) Problem formulation for BW and SG PSF models: For expected photon counts is computed using (5). For the BW BW and SG, problem (9) amounts to solving the maximum and SG models, the formulation is straightforward. However, likelihood problem: the GM model requires more attention in its formulation. Substituting (8) into (5), we evaluate the expected photon (xps, ϕps, θb) := arg min nll(xps, ϕps, θ | Z) b b ps ps count at each pixel j ∈ J and stack the results into a vector: x , ϕ , θ (12) ps ps s.t. x ∈ Ω, ϕ ∈ R+, θ ∈ Θ . ps ps gm ps X ps bg µ(x , ϕ , θ ) = ϕ Kk(x ) vk + ϕ 1np , (10) 6) Problem formulation for GM PSF models: We propose k∈K solving problem (9) in three steps. ps a) PS position estimation, xb : We solve the parametric where Kk is the blurring matrix defined for the k-th kernel as: blind PS deconvolution problem for the SG model given by (12). ps T ps ps Kk(x ) := [κk(x1 − x ),..., κk(xnp − x )] . b) Mixture weight support estimation: We fix the PS position to its estimated value and solve the GM PSF calibration b) Identifiability of Gaussian mixture weights: From (10) problem using the regulariser (11): it is apparent that only the product between the PS intensity X net : ps D and the GM weights is identifiable. We define the net mixture ϕbη = arg min nllp(ϕ | Z, xb , θ ) + ηkkϕkk1 , (13) X |D| net ps nk ϕ∈R+ k∈K intensity as ϕk := ϕ vk ∈ R+ : the PS intensity is spread p among the kernels in the dictionary as virtual intensities. where n nllp := nll. We estimate the support by thresholding We re-parameterise the vector of expected photon count as the weights above ϕmin: µ(xps, ϕnet, θD) = µ(xps, ϕps, θgm), where ϕnet ∈ |D| is R+ suppϕnet := (k, m) ∈ KM | [ϕnet ] > ϕ . the stacked vector of net mixture intensities. [ bη bη,k m min 3) Regularisation functional: In general, the regularisation The effective dictionary is defined by keeping the kernels in term R in (9) accounts for the prior knowledge about the the estimated support: PS position and intensity, and the PSF model parameters. thr sg net Db := κ (· − xm) ∈ D | (k, m) ∈ supp ϕ . (14) However, in this paper we aim at showing how to explore η θk [ bη the accuracy–efficiency trade-off based on the GM model. c) Debiasing: The convex problem (13) brings a known Therefore, we focus on regularising only the PSF model problem due to using the `1 norm: a bias (see e.g. [41]) in the i.e. Rgm(θ) net parameters, . intensity estimates ϕbη . We use a refitting strategy based on One way to trade accuracy for efficiency is to penalise the maximum likelihood to debias the mixture weights: effective number of Gaussian kernels, by penalising the size deb ps thr ϕη := arg min nll ϕ | Z, x , Dbη . (15) of the support of the net mixture intensity vector: b thr b |Dbη | ϕ∈R+ gm net p X ηk net R (ϕ ) := n supp ϕ , ps : η k We compute the estimates of the PS intensity as ϕb = k∈K deb ps −1 deb kϕbη k1 and mixture weights as vbk := (ϕb ) ϕbη . where we scale η by the number of pixels to have a comparable regularisation for different image data. Solving problem (9) B. Algorithms with this regularisation functional aims precisely at finding The main difference in solving the parametric blind PS the optimal trade-off between accuracy (minimising the nll) deconvolution problem for different PSF models is the di- and efficiency (minimising the number of non-zero mixture mensionality of the solution space. For the SG and BW PSF weights). The regularisation parameters are stacked into a models, problem (12) requires estimating 6 and 7 parameters, nC vector, denoted η := (η1, . . . , ηnC ) ∈ R+ , and allow explor- respectively. Both are low-dimensional smooth non-convex ing this trade-off: small values of |η| favour accuracy over optimisation problems that can be solved by a general-purpose efficiency, and large values vice versa. black-box optimiser. In contrast, the GM PSF calibration 6

Algorithm 1: Fully-split ASB for solving problem (13) the notations, we implicitly assume the conditional dependence of nll on the image data Z, the estimated point source bg 0 0 p Input : Z, ϕ , η, γ, {Kk, b , w } ps D k k k∈K position xb , and the dictionary parameters θ . net ∞ Output: ϕbη := w3 b) Fully-split Bregman: The next step is to fully split 0 0 0 0 w from ϕnet by applying split Bregman to the equivalent ∀k ∈ K : w = n, w = w = nk, b = 0 p X 1k 2k 3k k n +2nk problem arg min h0|D|, ϕi + E(w), writing at iteration i + 1: while NOTCONVERGED do ϕ,w for k = 1 to nC do i+1 i+1 i LS sub-problem: (ϕ , w ) = arg min h0|D|, ϕi + Ψ (ϕ, w) ϕ,w ϕi+1 = arg min kbi + O ϕ − wi k2 (16) k k k k k 2 bi+1 = bi + Oϕi+1 − wi+1 , ϕk nll sub-problem: i −1 i 2 where Ψ (ϕ, w) := E(w) + (2γ) kb + Oϕ − wk2, i+1 i ps i+1 C p w = Prox b + K (x )ϕ (17) n n +2|D| 1k γnllp 1k k k b ∈ R is the vector of the Bregman dual variables 2 stacked similarly to w, and γ is the dual-ascent step-size. `1–`2 sub-problem: c) Gauss-Seidel-like alternation: We exploit the second wi+1 = Prox (bi + ϕi+1) (18) 2k γηkk·k1 2k k level of additivity by splitting over the kernels the optimisation Positivity constraint sub-problem: problem in the Bregman iteration. The algorithm at iteration i for kernel k reads: i+1 i i+1 w = Proj nX (b + ϕ ) (19) 3k k 3k k R + i+1 1 i i 2 ϕk = arg min h0|D|, ϕki + kbk + Okϕk − wkk2 Bregman update (dual gradient ascent): ϕk 2γ (21) bi+1 = bi + O ϕi+1 − wi+1 (20) k k k k k 1 wi+1 = arg min Ei (w ) + kbi + O ϕi+1 − w k2 end k k k k k k k 2 wk 2γ end i i+1 =: Prox i b + O ϕ (22) γEk k k k i+1 i i+1 i+1 bk = bk + Okϕk − wk . (23) problem (13) is a large-scale non-differentiable convex optimi- i sation problem. We derive an efﬁcient and modular algorithm where the energy Ek(wk) introduces the coupling be- to solve (13) based on the alternating split Bregman strategy, tween kernels and relies on the set of k − 1 weights i and discuss the algorithms for the debiasing step. that are already updated at the i-th iteration: Ek(wk) := i+1 i 1) Estimating the support of MG PSF model: We deﬁne nllp(..., w1k−1, w1k, w1k+1, ...) + ηkkw2kk1 + ιS (ϕ3k). the energy functional in problem (13) as: d) Sub-problems solution: The fully-split ASB strategy X amounts to solving three standard problems: a least-squares E(ϕ) := nll (ϕ | Z, xps, θD) + η kϕ k + ι (ϕ) , p b k k 1 S (LS) problem (21), a proximal map evaluation (22), and a k∈K linear update (23). Further, it leads to a decomposition of the |D| where ιS (ϕ) with S := R+ is the indicator functional that proximal map (22) in three independent proximal maps (17), assumes 0 if ϕ is componentwise non-negative and +∞ (18), and (19), that decouple across pixels. The algorithm is otherwise. This problem has an additive structure at two levels. shown in Algorithm 1, where nk denotes the image data at First, the functional is the sum of three terms: the nll, the `1 the virtual source positions. regulariser, and the indicator function ιS (ϕ). The second level The solution to the LS sub-problem (21) is straightforward: of additivity comes from the sum over kernels in the mean −1 vector (10). We exploit this additivity structure and derive the wi+1 = OT O OT wi − bi , (24) algorithm in three steps: k k k k k k

a) Operator splitting: We exploit the first level of addi- T T T i i with O Ok = 2InX + K Kk and O w − b = tivity by introducing dummy variables to split the different k k k k k k T i i i i i i T p X K (w − b ) + w − b + w − b , where Kk := terms: w = wT wT wT := O ϕnet ∈ n +2nk k 1k k 2k 2k 3k 3k k 1k 2k 3k k k R K (xps). The inverse operator can be pre-computed outside T ps T (np+2nX )×nX k with O := K (x ) I X I X ∈ k k , k k b nk nk R the main iteration. Moreover, digital dictionaries can be effi- nX ×nX where I X ∈ k k is the identity matrix. We define nk R ciently evaluated using a spectral solver based on the FFT. C p the stacked vector w ∈ Rn n +2|D| and the block-diagonal The proximal maps (18) and (19) are also known: (18) is a (nC np+2|D|)×|D| i i+1 matrix O := diag(O1,..., OnC ) ∈ R , such componentwise soft-thresholding of b2k +ϕk with threshold net i i+1 that w = Oϕ . In addition, we denote the subset of dummy γηk, and (19) is a componentwise projection of b3k + ϕk variables for the operator o ∈ {1, 2, 3} stacked into a vector on R+. The solution to the proximal (17) is derived in [42], as wo: := vec {wok}k∈K . We can rewrite energy E(ϕ) to [3]. However, due to the additive structure of the mean vector highlight its additive structure: E(w) := E(w1:, w2:, w3:) := and the Gauss-Seidel-like alternation, we provide more details P nllp(w1:)+ k∈K ηkkw2kk1+ιS (w3k), where for simplifying for this proximal. 7

At iteration i, when the alternation reaches kernel k, the 3) GM model with a one-kernel dictionary: We use a digital 1d mean vector (10) can be rewritten in terms of the dummy dictionary, denoted D1 , made of Gaussian kernels of standard D1d variables: deviation 0.1 µm, placed at every pixel centres, i.e. θ 1 =

{0.1} × {xj}j∈J . We estimate the net mixture intensities by k−1 nC i+1 X i+1 X i bg i+1 applying Algorithm 1 for different values of the regularisation µ := w +w + w 0 +ϕ 1 p =: w +r . k 1k0 1k 1k n 1k k parameter and build the GM model by using a threshold of k0=1 k0=k+1 ϕmin = 0.1 in (14). The results are shown on Fig. 2b. For smaller regularisation, the estimated support of the GM model Instead of solving (17) in terms of w1k, we solve the problem i+1 covers the support of the true signal. For higher regularisation, in terms of µk and update w1k by subtracting the residuals i+1 the estimated support overlaps only with the regions of high rk : signal: the central mode and side lobes for η1 = 0.04, only the central mode for η = 0.07. However, we observe that the µi+1 = Prox bi + K ϕi+1 + ri+1 (25) 1 k γnllp 1k k k k estimated intensity is biased, as expected (see Section III-A6c). wi+1 = µi+1 − ri+1 . 1k k k (26) We solve problem (15) using CMA-ES and two gradient- based algorithms that allow enforcing positivity constraints: Evaluating (25) amounts to choosing the positive solution of Levenberg-Marquardt (LM), and the limited-memory Broy- the following quadratic equation in [µk]j deﬁned for the j-th den–Fletcher–Goldfarb–Shanno algorithm with box con- pixel: straints (L-BFGS-B). The three algorithms result in similar re- 2 γ T i γ constructions, but L-BFGS-B has the lowest run-time (Fig. 2). [µk] + − ej ck − [nk] = 0 , j np np j We observe that the debiasing step induces even more sparsity. The estimated GM model improves compared to SG model with ci := bi + K ϕi+1 + ri+1. k k k k k in two ways: the width and intensity of the central lobe are 2) Debiasing mixture weights of MG PSF model: The accurately reconstructed, even for a high regularisation, i.e. gradient of nll with respect to the mixture weights can be when the PSF is modelled with only a few kernels; for smaller derived analytically: regularisation, the GM model also captures the side lobes.

p 4) GM model with a two-kernel dictionary: We add to n [n ] ! X k j T the previous dictionary a second kernel of standard deviation ∂ϕ nll(ϕ) = 1 − K e . k [µ(xps, ϕ)] k j 1d j 0.2 µm such that the new dictionary, denoted D2 , has param- j=1 1d 1d D2 D1 eters θ = θ ∪ {0.2} × {xj}j∈J . Similarly to the one- To solve (15), we use gradient-based algorithms allowing kernel dictionary, we observe that the amount of regularisation positivity constrains. controls the number of each kernel in the effective dictionary (Fig. 3a). If the regularisation is equally increased for both kernels (plots on the diagonal), the support size decreases. IV. EXPERIMENTS For a higher regularisation (η1 = η2 = 0.07), only the smaller kernel contributes to modelling the PSF. If the regularisation A. Illustrative example: synthetic 1-D parameter of one kernel changes, the cardinality of the support of the other kernel remains the same. At the debiasing step, 1) Imaging settings: We place an idealised PS object at if the regularisation parameter for both kernels are equal, the xps = (2.5 µm, 0 µm, 1.5 µm) in the 3D imaging volume. preference is given to the smaller kernel as it allows capturing We model the optical distortions using the BW PSF. Using finer details of the PSF (Fig. 3b, plots on the diagonal). When the forward problem in Section II, we model the signal we regularise more the smaller kernel, the bigger kernel is used acquired by a one-dimensional camera detector along the x- to approximate the wider lobes, whereas the smaller kernel axis in the range x ∈ [0 µm , 5 µm], at y = z = 0 µm (the is used to approximate the narrow central mode (Fig. 3b, imaging parameters are summarised in Table I). This one- η = 0.04, η = 0.01). dimensional synthetic signal captures realistic features found 1 2 in experimental PSF image data: a prominent central mode with side lobes that are characterised by a rich structure and an B. GM PSF model: exploring accuracy–efficiency trade-off amplitude significantly above the background signal. In what follows, we assume the background intensity known. The one-dimensional synthetic example illustrates how the 2) SG model: We solve the parametric blind PS deconvo- GM model improves the model accuracy compared to SG, lution problem using the covariance matrix adaptation evo- while controlling the model complexity by designing the lutionary strategy (CMA-ES) algorithm [43]. It is a popular dictionary structure (i.e. one/two kernels) and the amount and well-tested derivative-free algorithm designed to solve of regularisation. We now apply our framework to three- low-to-moderate dimensional non-convex optimisation prob- dimensional image data: a synthetic image stack generated by lems (Fig. 2a). We observe that the estimated PS position a BW model, and real fluorescent bead measurements acquired (xˆ = 2.539 µm) is close to the true value. However, the by a widefield and a laser scanning confocal microscope. approximation accuracy of the central mode is poor: its width 1) Imaging settings: The parameter of the image formation is overestimated resulting in an underestimated intensity. process are summarised in Table I. 8

Table I PARAMETERSOFTHEIMAGEFORMATIONPROCESS

Dataset Object-to-pixel Sampling Pixel-to-image Object h w s λ [nm] NA ni n × n × n ∆xy [nm] ∆z [nm] te [ms] qλ M f b diameter [nm] Synthetic 1-D 474 1.45 1.518 1 × 101 × 1 50 − 1 1 1 1 0 δ SBW 474 1.45 1.518 81 × 81 × 101 65 200 21 0.81 1 2.0 100 δ WFFM 620 1.45 n.a. 81 × 81 × 31 65 200 21 0.81 1 2.14 98.24 50 LSCM 520 1.45 n.a. 61 × 61 × 101 133 50 12 0.70 200 6.44 398.06 100

80 80 400 102 Expected signal PS position (true) Expected signal η = 0. 01 η = 0. 01 1 101 1 60 SG PSF PS position (fit) 60 Measurements η = 0. 04 300 η = 0. 04 1 100 1 Measurements η = 0. 07 200 η = 0. 07 40 40 1 10-1 1

100 run time [s] 10-2 Photon counts 20 Photon counts 20 -3

0 Net mixture intensity 10 0 1 2 3 4 5 0 1 2 3 4 5 CMA-ES LM L-BFGS-B Position [um] Position [um]

(a) SG PSF. (b) GM PSF before debiasing. (c) Run-time of the debiasing step. 80 4000 80 4000 80 4000 Expected signal η1 = 0. 01 Expected signal η1 = 0. 01 Expected signal η1 = 0. 01 3000 3000 3000 60 Measurements η1 = 0. 04 60 Measurements η1 = 0. 04 60 Measurements η1 = 0. 04 η = 0. 07 2000 η = 0. 07 2000 η = 0. 07 2000 40 1 40 1 40 1 1000 1000 1000

Photon counts 20 Photon counts 20 Photon counts 20

0 Net mixture intensity 0 Net mixture intensity 0 Net mixture intensity 0 1 2 3 4 5 0 1 2 3 4 5 0 1 2 3 4 5 Position [um] Position [um] Position [um]

(d) GM PSF debiased using CMA-ES. (e) GM PSF debiased using LM. (f) GM PSF debiased using L-BFGS-B.

Figure 2. SG PSF model and GM PSF model with a one-kernel dictionary for a one-dimensional synthetic signal. (a) Parametric blind PS deconvolution for the SG model. (b) GM model (continuous line) and mixture weights support (weighted Dirac comb shown as vertical bars) estimated using Algorithm 1 for different amount of regularisation. (c) Run-time of debiasing using the covariance matrix evolution evolution strategy (CMA-ES), the Levenberg-Marquardt (LM) and the limited-memory Broyden–Fletcher–Goldfarb–Shanno algorithm with box constraints (L-BFGS-B). (d) - (f) GM model after debiasing.

η2 = 0.01 η2 = 0.04 η2 = 0.07 80 400 80 400 80 400 Expected signal D1 Expected signal D1 Expected signal D1 300 300 300 60 Measurements D2 60 Measurements D2 60 Measurements D2 01

. D ∪ D 200 D ∪ D 200 D ∪ D 200 40 1 2 40 1 2 40 1 2 100 100 100 = 0 Photon counts 20 Photon counts 20 Photon counts 20 1