Interpretable Functional Principal Component Analysis

Biometrics 72, 846–854 DOI: 10.1111/biom.12457 September 2016

Zhenhua Lin,1,* Liangliang Wang,2,** and Jiguo Cao2,*** 1Department of Statistical Sciences, University of Toronto, Toronto, Ontario, M5S 3G3, Canada 2Department of Statistics and Actuarial Science, Simon Fraser University, Burnaby, British Columbia, V5A 1S6, Canada ∗email: [email protected] ∗∗email: [email protected] ∗∗∗email: [email protected]

Summary. Functional principal component analysis (FPCA) is a popular approach to explore major sources of variation in a sample of random curves. These major sources of variation are represented by functional principal components (FPCs). The intervals where the values of FPCs are significant are interpreted as where sample curves have major variations. However, these intervals are often hard for na¨ıve users to identify, because of the vague definition of “significant values”. In this article, we develop a novel penalty-based method to derive FPCs that are only nonzero precisely in the intervals where the values of FPCs are significant, whence the derived FPCs possess better interpretability than the FPCs derived from existing methods. To compute the proposed FPCs, we devise an efficient algorithm based on projection deflation techniques. We show that the proposed interpretable FPCs are strongly consistent and asymptotically normal under mild conditions. Simulation studies confirm that with a competitive performance in explaining variations of sample curves, the proposed FPCs are more interpretable than the traditional counterparts. This advantage is demonstrated by analyzing two real datasets, namely, electroencephalography data and Canadian weather data. Key words: EEG; Functional data analysis; Null region; Penalized B-spline; Projection deflation; Regularization; Sparse PCA.

1. Introduction An introductory exposition of FPCA can be found in Chap- Functional principal component analysis (FPCA) is a com- ters 8 and 9 of Ramsay and Silverman (2005). Although the mon tool for dimensionality reduction in functional data anal- literature on functional data analysis is abundant, little has ysis. It aims to discover the major source of variability in been written on interpretability, besides FLiRTI (functional observed curves. Since being introduced by Rao (1958) for linear regression that is interpretable) proposed by James comparing growth curves, it has attracted considerable atten- et al. (2009) and the interpretable dimension reduction pro- tion. For example, Castro et al. (1986) related FPCA to the posed by Tian and James (2013). Inspired by their work, we Karhunen–Loève theorem and the best m-dimensional func- consider the interpretability of FPCs in this article. tional linear model. Dauxois et al. (1982) studied the asymp- If an FPC is nonzero in some intervals while strictly zero in totic properties of empirical eigenfunctions and eigenvalues other intervals, it is easier to interpret the FPC. For instance, when sample curves are fully observable. Zhang and Chen such an FPC can be interpreted as suggesting that the major (2007) and Benko et al. (2009) extended this work to a more variation of sample curves exists only in the nonzero inter- practical setting where sample curves are observed at finitely vals. However, FPCs produced by existing FPCA methods many design points. Hall and Hosseini-Nasab (2006, 2009) such as Silverman (1996) are nonzero almost everywhere. It studied the estimation errors of empirical eigenfunctions and is non-trivial to propose an estimate of FPC that is more in- eigenvalues. To overcome excessive variability of empirical terpretable, and at the same time enjoys desired properties, eigenfunctions, Rice and Silverman (1991) proposed smooth- such as consistency and accounting for a major portion of ing estimators of eigenfunctions via a roughness penalty. unexplained variance. Consistency of these estimators was established by Pezzulli In this article, we tackle the problem by proposing a and Silverman (1993). Subsequently, Silverman (1996) pro- novel penalty-based method to derive smooth FPCs that are posed an alternative way to obtain smoothing estimators of nonzero in the intervals where curves have major variations eigenfunctions through modifying the norm structure, and while are strictly zero in others, whence the proposed FPCs established the consistency of his estimators. Qi and Zhao are more interpretable than traditional FPCs. In light of this (2011) established the asymptotic normality of the estimators advantage, we call our method the interpretable functional of Silverman (1996). A kernel-based method for smoothing principal component analysis, or in short, iFPCA and an eigenfunctions was proposed by Boente and Fraiman (2000). estimated functional principal component is called an iFPC. The extension of FPCA to sparse data such as longitudinal The main contribution of this article is the efficient esti- data was studied by James et al. (2000) and Yao et al. (2005). mation of iFPCs and the demonstration that the estimated

846 © 2015, The International Biometric Society Interpretable Functional Principal Component Analysis 847 iFPCs are asymptotically normal and strongly consistent incorporates a roughness penalty on eigenfunctions through under some regularity conditions. modifying the default norm ·. Below we briefly describe The basic idea of iFPCA is to penalize the support of this methodology in which the roughness penalty is placed on smooth FPCs. When the iFPCs are estimated via our non- the second-order derivative. 2 2 def={ parametric approach, the penalty on the support can be ap- Let W2 be the Sobolev space W2 f : f, f are absolutely 2 proximated by an L0 penalty on the coefficients of carefully continuous in I and f ∈ L (I )}. Define the linear operator D2 2 D2 = D2 chosen basis functions. This feature distinguishes our method in W2 by f f , i.e. is the second-order derivative 2 ≥ from various sparse PCA methods in multivariate analysis operator in W2 . For a given scalar parameter γ 0, define a = + D2 D2 proposed in Jolliffe et al. (2003), Zou et al. (2006), and Witten new inner product f, g γ f, g γ f, g and its corre- et al. (2009). These sparse PCA methods utilize an L penalty, = 2 + D2 2 2 1 sponding norm f γ f γ f in W2 . Then, the which was originally proposed by Tibshirani (1996), to pe- roughness penalty is enforced by using the new norm ·γ nalize the loadings of principal components. In contrast, our rather than the default one. That is, the kth smooth FPC ξk is method penalizes the support of FPCs, using an L0 penalty found to maximize ξ, Cξ subject to ξγ = 1 and ξ, ξjγ = 0 on coefficients to serve as a surrogate for a penalty on the for jstochastic process in a com- ξ2 + γD2ξ2 pact interval I . That is to say X ∈ L2(I ) almost surely, subject to ξ = 1 and ξ,ξ = 0 for j 0 is a prespecified parameter. The first term in (1) is the tions in I equipped with the usual inner product f, g = k √ roughness penalty formulated in Silverman (1996) to guar- f (t)g(t)dt and the corresponding norm f = f, f for I antee smoothness of iFPC, while the second term is a new f, g ∈ L2(I ). Since our focus is on estimating eigenfunctions penalty term that yields the potential null subregions of iFPC. which are invariant to the overall mean function of X, without Note that S(ξ) is always nonnegative. It equals the length of loss of generality, we assume EX(t) = 0 for all t ∈ I . Then the domain if ξ is nonzero in the entire domain, and is zero the covariance function C of X is C(s, t) = E{X(s)X(t)} for if ξ = 0 identically. This observation indicates that for a fixed s, t ∈ I . Its corresponding covariance operator C is defined by ρ , those ξ’s with a larger support get penalized more heav- the mapping (Cf )(s) = C(s, t)f (t)dt. Assume X and C sat- k I ily. In addition, the penalty applies to functions which are isfy the conditions of the Karhunen-Loève theorem so that X = ∞ nonzero over more than one subinterval. In this case, S(ξ) admits a Karhunen-Loève expansion X(t) = Zkξk(t) and ∞ k 1 represents the sum of the lengths of all nonzero subintervals C admits a decomposition C(s, t) = λ ξ (s)ξ (t), where k=1 k k k of ξ. Therefore, all nonzero subintervals of ξ get penalized Z ,Z ,... are uncorrelated random variables, the functions 1 2 simultaneously. The tuning parameter ρ controls the trade- {ξ : k = 1, 2,...} form an orthonormal basis of L2(I ), and k k off between fidelity and interpretability. For example, when each ξ (t) is an eigenfunction of C corresponding to the eigen- k ρ = 0, the iFPC ξ (t) is identical to the smooth FPC pro- value λ . k k k posed by Silverman (1996), which is usually nonzero over the The task of functional principal component analysis is to estimate the first m eigenfunctions of the unknown operator C entire domain. In contrast, a large ρk prefers an iFPC ξ(t) which is zero in a large region. In particular, as ρk approaches based on a sample of n observed curves X1(t),X2(t),...,Xn(t). infinity, ξ (t) tends to be zero everywhere. Ordinary FPCA is formulated to find the first m FPCs ξ k k (k = 1, 2,...,m) such that ξk maximizes ξ, Cξ subject to 3.2. Method = ⊥ ξ 1, and ξ Hk−1, where Hk−1 denotes the space spanned We now develop a practical method to estimate the iFPCs C by ξ1, ξ2,...,ξk−1, and denotes the empirical covariance defined by (1) via a basis approach. First, we choose a set operator corresponding to the empirical covariance function of basis functions {φj(t):1≤ j ≤ p}. We assume that all basis = 1 n C(s, t) n i=1 Xi(s)Xi(t). functions are nonzero only in short subintervals with the same However, eigenfunctions that are estimated based on C(s, t) length except for a few basis functions on the boundaries. This generally exhibit excessive variability. Therefore, smoothing is assumption is satisfied by a B-spline basis, which will be used often desirable. Regularization through a roughness penalty is in simulation studies and applications. A B-spline basis is de- a popular approach to smooth estimators of functions. For in- fined by a sequence of knots that divide the compact interval stance, the estimation method proposed by Silverman (1996) I into subintervals. Over each subinterval, a B-spline basis 848 Biometrics, September 2016 function is a polynomial of specified order M. An order M kth iFPC is found to maximize B-spline basis function is nonzero over no more than M con- secutive subintervals. This property is called the local support ξ, Ckξ−ρkS(ξ) (4) property, which is important for efficient computation. In usual practice, each curve X (t), i = 1,...,n, is observed i = at some discrete design points and contaminated by random subject to ξ γ 1. Analogous to the projection deflation of a errors. If this is the case, as a pre-process step, we first esti- matrix described in Mackey (2009), we call our deflation “projection deflation” where C is the deflated covariance function. mate Xi(t) by the smoothing spline method (Ramsay and Sil- k verman, 2005) for dense design points, or the PACE method We now show how to compute iFPCs using projection de- (Yao et al., 2005) for sparse design points. However, it should flation based on the chosen B-spline basis system that we be noted that if the design points are too sparse, then infor- previously introduced. First of all, each residual can be writ- k = p (k) mation on X(t) in some subregions might be lost, and our ten as ri j=1 sij φj. Let Sk be the matrix with elements method might not be able to identify the sparsity pattern in = (k) Sk(i, j) sij . An efficient method to compute Sk is provided those subregions. = in Section S.2.3 in the supplementary file. Now define Qk Now let φ denote the column vector (φ ,φ ,...,φ )T , and −1 T 1 2 p n WSk SkW. Also, as aforementioned, it is assumed that T = × T T write (X1, X2,...,Xn) Sφ, where S is an n p matrix ξ = a φ. Then ξ, Ckξ=a Q a. Using a0 to serve as surro- = k with elements S(i, j) sij. Suppose each function ξ(t)in(1) gate for S(ξ), the optimization problem (4) is transformed into p has an expansion ξ(t) = ajφj(t). Then the number of j=1 finding the coefficient vector ak of the kth iFPC ξk to maximize nonzero coefficients aj can serve as a surrogate for S(ξ)because of the local support property of the basis functions φj(t). T − T T a Qka ρk a 0 (5) Let a denote the vector (a1,a2,...,ap) , and write ξ = a φ. Then the empirical covariance function is written as C(s, t) = T 1 T T subject to a Ga = 1. Or alternatively we find the vector ak to n φ (s)S Sφ(t) and the variance explained by ξ is estimated 1 T T by ξ, Cξ= a WS SWa, where W(i, j) =φi,φj. Also, n T ξ2 =aT φ, aT φ=aT Wa. Similarly, if R denotes a matrix maximize a Qka =D2 D2 D2 2 = T such that R(i, j) φi, φj , then ξ a Ra. Define subject to aT Ga = 1 and a ≤ τ , (6) = + = 1 T 0 k G W γR and Q n WS SW. Then the first term of (1) becomes where the constant τk is a prespecified positive integer. By using formulation (6), we can relax the orthogonality of esti- ξ, Cξ 1 aT WST SWa aT Qa = n = . (2) mated FPCs in exchange for interpretability. Also, the opti- ξ2 + γD2ξ2 aT (W + γR)a aT Ga mization problem (6) is equivalent to the optimization problem (5) in the sense that if ak is the optimal solution of (5), = Let a0 denote the number of nonzero loadings of a. Using then ak is also the optimal solution of (6) with τk ak 0 a0 as an approximate surrogate for S(ξ), the optimization (d’Aspremont et al., 2008). The advantage of (6) is that it is problem (1) is approximated by finding the vector a that easier to choose a value of τ than ρ ,asτ is directly related k k k k maximizes to the penalty on the support of ξk. Unfortunately, the optimization problem (6) is NP-hard (Wang and Wu, 2012), which aT Qa means that no efficient algorithm is known to solve it. In the − ρ a (3) aT Ga k 0 next subsection, we develop a greedy backward elimination method to approximately solve the optimization problem (6). T = T = subject to a Ga 1 and a Gaj 0 for j

Backward elimination to find such an α removes a nonzero plementary file, we show that λ1(Qα, Gα) can be computed loading of a iteratively until there are only τ nonzero loadings in O(p2 log p) operations. Thus, in total it takes O(p2 log p) left. In each iteration, the loading with the minimum impact operations for the AMVL approach to solve the problem (7) on explaining the variance will be chosen in a greedy fashion. approximately. More precisely, if αj denotes the value of α when the j-th The greedy backward elimination algorithm is outlined in 0 ={ } −1 iteration is completed, and by convention, α 1, 2,...,p , Algorithm 1. Note that in Algorithm 1, the updating of Gα −1 2 then the ijth loading of a such that and Gα Qα in each iteration can be done in O(p ) operations (see Section S.2.2 in the supplementary file). Therefore, 3 = { − − − } ij arg min λ1(Qαj−1 , Gαj 1 ) λ1(Qαj−1−{i}, Gαj 1−{i}) (7) the computational complexity of Algorithm 1 is O(p log p) i∈αj−1 by noting that τ ≤ p. The iFPCA algorithm is outlined in Algorithm 2, which uses the projection deflation procedure j = j−1 −{ } will be nullified in the jth iteration, and α α ij . Note described in Section S.2.3 of the supplementary file to iter- that in (7), λ (Q j−1 , G j−1 ) − λ (Q j−1−{ }, G j−1−{ })isthe 1 α α 1 α i α i atively compute ak by applying Algorithm 1 on matrices Qk loss of explained variance due to the elimination of the ith and G. In total, the iFPCA algorithm takes O(nmp2 + m2p2 + loading. mp 3 log p) operations to complete. When n = O(p) and m is 3 It requires O(p ) floating point operations to solve the op- fixed, the computational complexity of the iFPCA algorithm 2 timization (7), because it takes at least O(p ) operations to grows on the order of O(p3 log p). This complexity is almost compute the leading eigenvalue of a generalized eigenvalue the same as the regularized FPCA approach described in problem with two p × p matrices. When p is large, this is Ramsay and Silverman (2005), which requires at least O(p3) still quite computationally inefficient. Instead of computing operations. − − − the exact value of λ1(Qαj−1 , Gαj 1 ) λ1(Qαj−1−{i}, Gαj 1−{i}), we borrow the idea of the approximate minimum variance Algorithm 1 The Greedy Backward Elimination Algorithm loss (AMVL) criterion by Wang and Wu (2012), originally (1) Initialize β =∅, α ={1, 2,...,p} and s = p; Compute proposed in the setting of ordinary principal component anal- − − G 1 and G 1Q ; ysis: choose i with the smallest upper bound on the loss of α α α j (2) Repeat the following steps until s ≤ τ:a)for variance due to the elimination of the i th loading, provided − j all i ∈ α, compute h = [v y 1vT g {λ(Q , G ) − c}+ that the upper bound is easy to compute. We now derive an i i α i α α v vT p ]/(1 − v2) as in inequality (8); b) set i = min h , upper bound on the loss of variance due to the elimination of i α 2 i s i i update β := β ∪{is}, α := α −{is} and set s := s − 1; c) the ith loading. The inequality below generalizes the one in − − update G 1 and G 1Q according to Section S.2.2 in Wang and Wu (2012). The proof is relegated to the supple- α α α the supplementary file; mentary file. (3) Denote by a the vector such that a(β) = 0 and a(α) = v. Normalize a so that aT Ga = 1. Finally output Proposition 1. Suppose A and B are p × p symmetric λ (Q , G ) and the vector a. The corresponding es- matrices, and B is invertible. Denote by λ (A, B) the largest 1 α α 1 timated iFPC is ξ(t) = aT φ(t). eigenvalue of the generalized eigenvalue problem Az = λBz, and by v the corresponding eigenvector. Let vi be the i-th ele- ={ }−{} ment of the vector v, α 1, 2,...,p i , and 3.4. Tuning Parameter Selection First, we assume basis functions have been chosen in advance. Ff P1 p B−1 = i , B−1A = 2 . As we pointed out in Section 3.2, the basis functions must T T fi y p3 c have a local support property, which means that each basis function is nonzero only in a small interval of the domain. Then In addition, the length of the nonzero interval for each basis function must be the same except for a few basis functions. − ≤ −1 T { − } λ1(A, B) λ1(Aα, Bα) [viy fi vα λ1(A, B) c We recommend the B-spline basis since it satisfies these re- quirements. + T − 2 vivα p2]/(1 vi ). (8) Algorithm 2 The iFPCA Algorithm

This proposition enables us to approximate λ1(A, B) − (1) Compute matrices S, W, R based on the chosen basis λ1(Aα, Bα) in an efficient way. More specifically, we approxi- functions, and Set G = W + γR; T mate λ1(A, B) − λ1(Aα, Bα) by the RHS of (8), because the (2) For k = 1, 2,...,m, let Q = WS SW, compute ak and RHS of (8) is much easier to compute than the LHS. For λk using Algorithm 1, and update S according to Sec- −1 −1 example, provided λ1(A, B), B , and B A have been com- tion S.2.3 in the supplementary file; puted, the upper bound in (8) can be computed in O(p) extra (3) Output (a1, λ1), (a2, λ2),...,(am, λm). Note that the = operations. It is easy to see that the matrix Gα is positive- total variance of data is estimated by λtol definite, symmetric and hence invertible. Then, Proposition 1 n T n i=1 S(i, :)WS(i, :) , and the estimated first m iF- T T T 1 applies to the matrix pair (Qα, Gα). Based on inequality = = = PCs: ξ1(t) a1 φ(t), ξ2(t) a2 φ(t), ..., ξm(t) amφ(t). (8), by using the AMVL criterion, the optimization (7) can Also, the percentages of variance accounted by each be approximately solved in O(p2) operations provided that iFPC are λ1/λtol, λ2/λtol,...,λm/λtol. λ1(Qα, Gα) has been computed. In Section S.2.1 in the sup- 850 Biometrics, September 2016

Once a basis is chosen, the number of basis functions p can this section, we will show that our iFPCA estimators also be set accordingly. Since the smoothness of iFPCs is regular- enjoy these nice properties under more general conditions on ized by the smoothing parameter γ, the choice of p has little the ρk’s. We present the main results in this section while impact on the smoothness of iFPCs. However, p should be set providing their proofs in the supplementary file. large enough to make a0 serve as a good surrogate for S(ξ) In our setup, the sample functions, X1(t),X2(t),...,Xn(t), of ξ(t). In practice, we recommend setting p ≥ 50. Note that are assumed to be independent and identically distributed re- p should not be set too large either, because the computa- alizations of the stochastic process X(t) that is introduced in tional burden increases significantly as p increases. Since the Section 2. We further assume the following technical condi- proposed iFPCA method depends only on the local support tions, which are identical to those in Silverman (1996): property of the basis, other parameters specific to the basis can be set to values that are recommended in the literature. Assumption 1. The covariance function C(s, t) is strictly For example, if a B-spline basis is used, then the order M is positive-definite and the trace I C(t, t)dt is finite. This as- recommended to be set equal to 4, which means that a cubic sumption ensures that the covariance operator C is a Hilbert- B-spline basis is used. Schmidt operator. Thus, the operator C has a complete se- After the number of basis functions is determined, we can quence of eigenfunctions ξ1(t),ξ2(t),...,and eigenvalues λ1 ≥ 4 choose the parameter γ that controls roughness of iFPCs, and λ2 ≥···> 0. Furthermore, it implies E(X ) < ∞. τk’s that control the sparsity of iFPCs in the optimization criterion (6). For example, a two-dimensional cross-validation Assumption 2. Each eigenfunction ξk(t) falls in the space procedure can be employed to search for good values of γ 2 W2 . When a roughness penalty is applied on derivatives of and τk simultaneously. Alternatively, we propose a two-step 2 other orders, the space W2 is replaced by the corresponding procedure, in which γ is chosen in the first step, and the τk’s space of “smooth” functions in I . are determined in the second step. Our computational experi- ments show that the two-step procedure produces reasonably Assumption 3. The first m eigenvalues have multiplicity good results and is computationally efficient. Below we give 1 and hence λ1 >λ2 > ···>λm >λm+1 ≥···> 0. This condi- the details of this two-step procedure. tion is assumed for simplicity. The result can be extended to In the first step, γ can be chosen based on prior knowl- the general case. edge or cross-validation (CV) as described in Ramsay and Silverman (2005). This γ remains fixed for different FPCs, as Note that even when an eigenvalue λ has multiplicity 1, it represents the choice of the norm defined in the space of k both ξ (t) and −ξ (t) are its corresponding eigenfunctions, eigenfunctions, and this norm must be the same for all eigen- k k although they have opposite directions. In the sequel, when functions. Once γ is chosen, τ can be chosen by CV in the k we mention eigenfunctions, their directions are assumed to be second step. Details are provided in Section S.2.4 of the sup- given. Also, we assume that the directions of iFPCs ξ (t) and plementary file. In principle, the tuning parameter τ is chosen k k ≥ by minimizing the CV score or its variants such as the well Silverman’s FPCs ξk(t) are chosen so that ξk, ξk γ 0 and ≥ = known “+1 SE” rule (Breiman et al., 1984). Alternatively, if ξk,ξk 0 for all k 1, 2,...,m. we want to control the loss of explained variance due to the pursuit of interpretability, then we can choose the smallest Theorem 1(Asymptotic Normality). Let ξk(t) be the τk such that the variance explained by iFPCA is at least δ% estimated iFPC that maximizes (1), and λk =ξk, Cξk. Sup- of the variance explained by the regularized FPCA method pose Assumptions 1–3 hold. As n →∞, (Silverman, 1996) where δ is a prespecified number between 0 and 100. In applications, we also often need to select the √ √ { − − number of principal components. This can be done by using (i) the joint distribution of n(λ1 λ1), n(λ2 √ the information criterion proposed in Li et al. (2013). λ2),..., n(λm − λm)} converges to a Gaussian distribution with mean zero, if γ = o(n−1/2) and −1/2 ρk = o(n ) for all k = 1, 2,...,m; 4. Asymptotic Properties √ √ √ (ii) the vector { n(ξ1 − ξ1), n(ξ2 − ξ2),..., n(ξm − ξm)} Although our method can be applied to functional data with converges to a Gaussian random element with mean sparse design points by utilizing the PACE method proposed −1 −1 zero, if γ = o(n ) and ρk = o(n ) for all k = by Yao et al. (2005), the following theoretical analysis only 1, 2,...,m. applies to dense functional data. This is because, according to Li and Hsing (2010), for eigenvalues and eigenfunctions to achieve a root-n convergence rate, the smoothing errors are Theorem 1 implies that our iFPCA estimators are consis- ignorable only if the design points are dense enough. tent. We will show that they are strongly consistent in Theo- In the special case that ρ = 0 for all k = 1, 2,...,m rem 2. k in (1), our iFPCA estimators, λ1, λ2,...λm, and ξ1(t), ξ2(t),...,ξm(t), reduce to the regularized FPCA Theorem 2(Strong Consistency). Let ξk(t) be the es- estimators, λ1, λ2,...,λm, and ξ1(t), ξ2(t),...,ξm(t), proposed timated iFPC that maximizes (1), and λk =ξk, Cξk. Under by Silverman (1996). Strong consistency and asymptotic Assumptions 1–3, if γ → 0 and ρ → 0 for all k = 1, 2,...,m, k normality of these estimators have been established by then λk → λk and ξk → ξk almost surely as n →∞ for all Silverman (1996) and Qi and Zhao (2011), respectively. In k = 1, 2,...,m. Interpretable Functional Principal Component Analysis 851

FPC #1 FPC #2 FPC #3 8 8 8 True FPC True FPC True FPC 6 iFPCA 6 iFPCA 6 iFPCA FPCA FPCA FPCA 4 4 4 (t) (t) (t) ξ ξ ξ 2 2 2

0 0 0

0 0.5 1 0 0.5 1 0 0.5 1 t t t FPC #1 FPC #2 FPC #3 0.8 0.3 iFPCA 0.3 iFPCA iFPCA FPCA FPCA FPCA 0.6 0.2 0.2 0.4

MSE 0.1 MSE 0.1 MSE 0.2

0 0 0 0 0.5 1 0 0.5 1 0 0.5 1 t t t

Figure 1. The top three panels display the estimated FPCs using our iFPCA method and the traditional FPCA method in one random simulation replicate in Simulation 1. The bottom three panels show the pointwise mean squared errors (MSE) of the estimated FPCs using the two methods. This ﬁgure appears in color in the electronic version of this article.

2 = 2 = 2 ∈ In Section 3, we use the projection deflation technique to 3 , λ2 2.5 and λ3 2 , t [0, 1]. The three true FPCs are relax the orthogonality constraint. In this case, our iFPCA es- = 53 simulated as ξk(t) =1 ak φ (t), where each φ (t) is one of timators are still strongly consistent, as shown in the following the 53 basis functions of the cubic B-spline basis system with theorem. 51 knots equally spaced in [0,1]. For each FPC ξk (k = 1, 2, 3), all of its basis coefficients ak except 2k + 2 of them, are zero. Theorem 3(Strong Consistency using Deflation Figure 1 displays the true FPCs in the top panels. The sim- Technique). Let ξk(t) be the estimated iFPC that maximizes ulated data are generated as pairs (tj,yij ) for i = 1, 2,...,500 = (4), and λk =ξk, Ckξk. Under Assumptions 1–3, if γ → 0 and j 1, 2,...,51, where tj is the jth design point equally- = + and ρk → 0 for all k = 1, 2,...,m, then λk → λk and ξk → ξk spaced in [0,1], and yij Xi(tj) ij is the value of Xi at tj →∞ = i.i.d almost surely as n for all k 1, 2,...,m. with a random measurement error ij ∼ N(0, 1). Both methods first estimate Xi(t), i = 1,...,500, by the 5. Simulation Studies penalized spline smoothing method (Ramsay and Silverman, We evaluate the finite sample performance of our iF- 2005) and using a cubic B-spline basis system with 51 equally- PCA method and compare it with the traditional FPCA spaced knots in [0,1]. Both methods express an estimated FPC method (Silverman, 1996) in two simulation studies. In the in terms of the same basis system. The tuning parameter γ first simulation study, the true FPCs are nonzero in some is set to be γ = 10−7. This number is selected by the cross- short subintervals and strictly zero elsewhere. This sce- validation method proposed in Ramsay and Silverman (2005) nario favors our iFPCA method. We will show that our iF- in a randomly chosen simulation replicate. We fix γ for all PCA method produces more accurate FPC estimators, and simulation replicates, as it is not sensitive across different sim- the loss of variance explained by the estimated iFPCs is neg- ulation replicates. We choose τk, k = 1, 2, 3, by 10-fold gener- ligible. In the second simulation study, the true FPCs are alized cross-validation in the iFPCA method. The simulation nonzero over almost the entire domain. This scenario favors is implemented with 100 simulation replicates. the traditional FPCA method, but we will show that our iF- The top three panels in Figure 1 display the estimated PCA method is still competitive. Below we present the first FPCs of the two methods from a single simulation. They show simulation study, while relegate the second one to the supple- that the FPCs produced by the traditional FPCA method are mentary file. nonzero in almost the entire interval [0, 1], and all FPCs have In the simulation study, 500 curves are simulated by three modes instead of the true one mode. In contrast, each Xi(t) = bi1ξ1(t) + bi2ξ2(t) + bi3ξ3(t) where bik ∼ N(0,λk), λ1 = iFPC has only one mode, and is nonzero only in almost the 852 Biometrics, September 2016

4.5 100 4

3.5 80

3 60 2.5

2 40 1.5

1 20 iFPCA

Diff. of cumulative variance (%) 0.5 FPCA 0 123 Cumulative variance percentage (%) 1 2 3 Number of FPCs Number of principal components

Figure 2. The left panel displays the boxplot of the loss of cumulative percentages of variance explained by the estimated FPCs using our iFPCA method compared with the traditional FPCA method in 100 simulation replicates in Simulation 1. The right panel shows the average cumulative percentages of variance explained by the estimated FPCs using our iFPCA method in comparison with the traditional FPCA method in 100 simulation replicates. This ﬁgure appears in color in the electronic version of this article.

same interval as the corresponding true FPC. Similar results The EEG data is based on a case study on genetic are seen in most of the other 99 simulation replicates. In fact, predisposition to alcoholism (Zhang et al., 1995). This data the iFPCs have the same nonzero intervals as the correspond- is available from the UCI machine learning repository ing true FPCs in 97, 98, and 100% of the simulations for the (https://archive.ics.uci.edu/ml/datasets/EEG+Database). first, second, and third FPCs, respectively. The bottom panels In the study, 122 subjects were separated into two groups, in Figure 1 show the pointwise mean squared errors (MSE) of namely, alcoholic and control. Each subject was exposed the estimated FPCs using the two FPCA methods. All three to either a single stimulus or two stimuli in the form of iFPCs have smaller pointwise MSEs than traditional FPCs pictures. If two pictures were presented, the pictures might over nearly the entire interval. The iFPCA method also seems be identical (matching condition) or different (non-matching to give more robust estimates of FPCs than the traditional condition). At the same time, 64 electrodes were placed on FPCA method. the subjects’ scalps to record brain activities. Each electrode In comparison with the traditional FPCA method, the was sampled at 256 Hz for 1 second. iFPCA method has the additional constraint that FPCs have We focus our study on the alcoholic subjects and channel to be zero in most intervals to increase their interpretability. CP3. The standard sites of electrodes can be found in Guide- Therefore, the iFPCs may explain less variance in the lines for Standard Electrode Position Nomenclature, Ameri- sample curves. Figure 2 summarizes the average cumulative can Electroencephalographic Association 1990. The electrode percentages of variance explained by both methods. It CP3 is located over the motor cortex which is related to body shows that the iFPCs explain almost the same cumulative movements. Previous studies have shown that alcohol has a percentages of variance of the sample curves. The differences negative impact on the motor cortex (e.g., see Ziemann et al. in cumulative percentage of variance explained are less than 1995). Therefore, it is of interest to understand fluctuation 3.2% on average. Figure 2 also displays a boxplot of the patterns of CP3 among the 77 alcoholic subjects. More specif- differences of cumulative percentages of explained variance ically, we examine one epoch of CP3 when the subjects are by the two methods in 100 simulations. The boxplot clearly exposed to non-matching stimuli where we attempt to iden- shows that the differences are all less than 4.5% in 100 simu- tify the time intervals of major fluctuations. Results for other lations. These differences are partly due to the fact that the epochs are similar to the one presented below. traditional FPCA method undesirably catches more random The data consists of n = 77 curves measured at 256 time measurement errors, as it is observed that FPCs estimated by points in a selected epoch, which are displayed in Figure this method are nonzero in almost the entire domain and each 3. The underlying continuous curves are first recovered of them has three modes, rather than just one true mode. from the discrete measurements by the smoothing spline method (Ramsay and Silverman, 2005). Then the functional 6. Application principal components are estimated using the proposed We apply our method to two real datasets, namely, electroen- method and the traditional FPCA method. Cubic B-spline cephalography (EEG) data and Canadian weather data. Be- basis functions are used in both methods with one knot low we present the study on the EEG data. The study on the placed in each design point, and the tuning parameter γ Canadian weather data is relegated to the supplementary file. is chosen by cross-validation. The iFPCA method chooses Interpretable Functional Principal Component Analysis 853

40 mean there is no brain activity in that time interval, since the overall mean curve is nonzero in [1, 121].

20 7. Discussion Functional principal component analysis (FPCA) has been 0 widely used to identify the major sources of variation of random curves. The variation is represented by functional principal components (FPCs). The intervals where FPCs −20 are signiﬁcantly far away from zero are interpreted as the regions where the sample curves have major variations. Our

Sensor Reading interpretable FPCA (iFPCA) method enables us to locate −40 precisely these intervals. The approach developed and our theoretical analysis assume the data is densely sampled. In practice, when data −60 0 50 100 150 200 250 curves are sparsely observed, curve recovery procedures such Time as PACE (Yao et al., 2005) can be applied to recover the data curves before applying our iFPCA method. In this case, the Figure 3. The readings of the CP3 channel over one epoch quality of the iFPCs depends on the performance of the curve of non-matching stimuli from all alcoholic subjects in an EEG recovery procedure and the sparseness of the data. As data case study on genetic predisposition to alcoholism. This figure get sparser, information contained in the data for recovering appears in color in the electronic version of this article. curves diminishes, and the performance of the iFPCA method is expected to deteriorate. For example, in the extreme case that no data is observed in the interval where an FPC van- ishes, it is impossible to identify the null sub-domain of the the other tuning parameters τk,k = 1, 2, 3, which control the sparseness of iFPCs, by 10-fold generalized cross-validation. FPC. Figure 4 shows the first three iFPCs and the traditional In some applications, curves may be densely sampled but FPCs for the EEG data. Both iFPCs and traditional FPCs corrupted by noise. In this case, curves can be smoothed be- have similar trends so that their interpretations are consis- fore applying the iFPCA method. Various curve smoothing tent. However, we see that all traditional FPCs are nonzero methods can be found in Ramsay and Silverman (2005). As in- almost everywhere while the three iFPCs are strictly zero in vestigated by Kneip (1994), the performance of the approach certain subintervals. For each component, the null subregion that first smooths the data and then extracts principal com- suggests that brain activities have no significant variation in ponents depends on the sample size, the sampling rate, and the corresponding time interval among subjects, whereas the the noise level. When the noise level is not too high relative to nonzero intervals indicate where the EEG curves fluctuate the signal strength, we expect that the performance of the iFPCA most. For example, the first iFPC that captures more than method is robust to the noise. This has been demonstrated in 65% of the total variation is zero in the subinterval [1, 121]. our simulation studies where measurement errors have been On one hand, this indicates that the brain activities of all taken into consideration. 77 subjects are similar over the time interval [1, 121] in the response to the stimuli, whereas there is significant variation 8. Supplementary Material over the time interval (121, 256]. This insight into the data is Figures, proofs, computational details of the algorithms, an not provided by the traditional FPCA method. On the other additional simulation study and an additional data applica- hand, the fact that the first iFPC is zero in [1, 121] does not tion referenced in Sections 4, 5, and 6 are available with this

FPC #1 FPC #2 FPC #3 0.15 0.2 0.2

0.1 0.1 0.1 0.05 0 0 0 −0.1 −0.1 Sensor Reading −0.05 Sensor Reading Sensor Reading iFPCA (65.4%) iFPCA (18.5%) −0.2 iFPCA (4.6%) FPCA (79.7%) −0.2 FPCA (7.9%) FPCA (6.2%) −0.1 0 100 200 0 100 200 0 100 200 Time Time Time

Figure 4. The estimated three FPCs using our iFPCA method (solid lines) and the traditional FPCA method (dashed lines) for the EEG data. Numbers in the parentheses are percentages of variance explained by the FPCs. This figure appears in color in the electronic version of this article. 854 Biometrics, September 2016 article at the Biometrics website in the Wiley Online Library. Li, Y., Wang, N., and Carroll, R. J. (2013). Selecting the number Matlab code for applying our method is also available at the of principal components in functional data. Journal of the Wiley website. American Statistical Association 108, 1284–1294. Mackey, L. (2009). Deflation methods for sparse PCA. Advances in Acknowledgements Neural Information Processing Systems 21, 1017–1024. Pezzulli, S. and Silverman, B. W. (1993). Some properties of The authors are grateful for the invaluable comments and sug- smoothed principal components analysis for functional data. gestions from the editor, Dr Jeremy M. G. Taylor, an asso- Computational Statistics 8,1–16. Qi, X. and Zhao, H. (2011). Some theoretical properties of Sil- ciate editor, and a reviewer. We thank Professor Tim Swartz verman’s method for smoothed functional principal com- and Professor Richard Lockhart for their constructive component analysis. Journal of Multivariate Analysis 102, ments on this article. This research was supported by Discov- 741–767. ery grants (Wang and Cao) from the Natural Sciences and Ramsay, J. O. and Silverman, B. W. (2005). Functional Data Anal- Engineering Research Council of Canada (NSERC). ysis. Springer Series in Statistics. New York: Springer, 2nd edition. Rao, R. C. (1958). Some statistical methods for comparison of References growth curves. Biometrics 14,1–17. Rice, J. A. and Silverman, B. W. (1991). Estimating the mean Benko, M., Härdle, W., and Kneip, A. (2009). Common functional and covariance structure nonparametrically when the data principal components. The Annals of Statistics 37,1–34. are curves. Journal of the Royal Statistical Society, Series B Boente, G. and Fraiman, R. (2000). Kernel-based functional princi- 53, 233–243. pal components. Statistics and Probability Letters 48, 335– Silverman, B. W. (1996). Smoothed functional principal compo- 345. nents analysis by choice of norm. The Annals of Statistics Breiman, L., Friedman, J., Olshen, R., and Stone, C. J. (1984). 24,1–24. Classification and Regression Trees. Wadsworth statis- Tian, T. S. and James, G. M. (2013). Interpretable dimensionality tics/probability series. Wadsworth International Group. reduction for classification with functional data. Computa- Castro, P. E., Lawton, W. H., and Sylvestre, E. A. (1986). Princi- tional Statistics and Data Analysis 57, 282–296. pal modes of variation for processes with continuous sample Tibshirani, R. (1996). Regression shrinkage and selecting via the curves. Technometrics 28, 329–337. lasso. Journal of the Royal Statistical Society, Series B d’Aspremont, A., Bach, F. R., and Ghaoui, L. E. (2008). Optimal (Methodological) 58, 267–288. solutions for sparse principal component analysis. Journal of Wang, Y. and Wu, Q. (2012). Sparse PCA by iterative elimina- Machine Learning Research 9, 1269–1294. tion algorithm. Advances in Computational Mathematics 36, Dauxois, J., Pousse, A., and Romain, Y. (1982). Asymptotic the- 137–151. ory for the principal component analysis of a vector random White, P. A. (1958). The computation of eigenvalues and eigenvec- function: Some applications to statistical inference. Journal tors of a matrix. Journal of the Society for Industrial and of Multivariate Analysis 12, 136–154. Applied Mathematics 6, 393–437. Hall, P. and Hosseini-Nasab, M. (2006). On properties of functional Witten, D. M., Tibshirani, R., and Hastie, T. (2009). A penalized principal components analysis. Journal of the Royal Statis- matrix decomposition, with applications to sparse principal tical Society, Series B 68, 109–126. components and canonical correlation analysis. Biostatistics Hall, P. and Hosseini-Nasab, M. (2009). Theory for high-order 10, 515–534. bounds in functional principal components analysis. Mathe- Yao, F., Muller, H.-G., and Wang, J.-L. (2005). Functional data matical Proceedings of the Cambridge Philosophical Society analysis for sparse longitudinal data. Journal of the Ameri- 146, 225–256. can Statistical Association 100, 577–590. James, G. M., Hastie, T. J., and Sugar, C. A. (2000). Principal Zhang, J.-T. and Chen, J. (2007). Statistical inferences for func- component models for sparse functional data. Biometrika tional data. The Annals of Statistics 35, 1052–1079. 87, 587–602. Zhang, X., Begleiter, H., Porjesz, B., Wang, W., and Litke, A. James, G. M., Wang, J., and Zhu, J. (2009). Functional linear re- (1995). Event related potentials during object recognition gression that’s interpretable. The Annals of Statistics 37, tasks. Brain Research Bulletin 38, 531–538. 2083–2108. Ziemann, U., Lönnecker, S., and Paulus, W. (1995). Inhibition of Jolliffe, I. T., Trendafilov, N. T., and Uddin, M. (2003). A modified human motor cortex by ethanol. a transcranial magnetic principal component technique based on the lasso. Journal stimulation study. Brain 118, 1437–1446. of Computational and Graphical Statistics 12, 531–547. Zou, H., Hastie, T., and Tibshirani, R. (2006). Sparse principal Kneip, A. (1994). Nonparametric estimation of common regressors component analysis. Journal of Computational and Graphi- for similar curve data. The Annals of Statistics 22, 1386– cal Statistics 15, 265–286. 1427. Li, Y. and Hsing, T. (2010). Uniform convergence rates for nonparametric regression and principal component analysis in functional/longitudinal data. The Annals of Statistics 38, Received December 2014. Revised October 2015. 3321–3351. Accepted October 2015.