arXiv:0807.1659v1 [math.FA] 10 Jul 2008 n loihs e 2,1,4.I hsfaeok h basic the framework, functions this In of 4]. space 12, [25] [20, book see algorithms, the ing see reference standard esta are a been RKHS For has the [1]. RKHS of scalar paper elements for seminal the theory setting mathematical exampl usual The for the tions. see In algorithms, [9]. learning book the designing (RKH for spaces tool Hilbert tant kernel reproducing theory, learning In Introduction 1 ic 2 -03 iao tl.emi:veronica.umanita M e-mail: di Italy. Politecnico Milano, I-20133 Brioschi”, 32, “F. Vinci Matematica e di Italy. Dipartimento Genova, 16146 33, Dodecaneso Via Genova, di Sezione 1 33, Dodecaneso Via Genova, di [email protected] Sezione I.N.F.N., and Italy, [email protected] e-mail: Italy. Genova, 16146 33, caneso rpryta,frany for that, property ∗ § ‡ † nmcielann hr sa nraigitrs o vect for interest increasing an is there learning machine In .Uai`,DS,Uiestad eoa i oeaeo35 Dodecaneso Via Genova, Universit`a di DISI, Umanit`a, V. 16 35, Dodecaneso Via Universit`a Genova, DISI, di Toigo, A. Agostin S. Stradone Genova, Universit`a di DSA., Vito, De E. Sezi I.N.F.N., and Genova, Universit`a di DIFI, Carmeli, C. e trvle erdcn enlHilbert kernel reproducing valued Vector ftasainivratkreso bla rusadw re we and groups problem. abelian universality on the kernels invariant t characterize value translation we vector particular of In aspects: two kernels. universal on and focus maps We spaces. Hilbert kernel .Carmeli C. hsppri eoe otesuyo etrvle reproduci valued vector of study the to devoted is paper This f pcsaduniversality and spaces rmaset a from .D Vito De E. , , ∗ x ∈ X uy1,2008 10, July , k f X Abstract ( x noanre etrspace vector normed a into ) ≤ k 1 .Toigo A. , † C x k f k t @polimi.it o oiieconstant positive a for .Umanit`a V. , ‡ 16Gnv,Iay e-mail: Italy. Genova, 6146 ln,Paz enroda Leonardo Piazza ilano, mi:[email protected] -mail: n iGnv,VaDode- Via Genova, di one 4 eoa n I.N.F.N., and Genova, 146 64 eoa and Genova, 16146 , 7 62 Genova, 16123 37, o beti Hilbert a is object estructure he 8 9 1 and 31] 29, [8, e )aea impor- an are S) rvle learn- valued or . feature d aei to it late lse nthe in blished clrfunc- scalar Y , § ihthe with ng C x independent of f. The theory of vector valued RKHS has been completely worked out in the seminal paper [27], devoted to the characterization of the Hilbert spaces that are continuously embedded into a locally convex topological , see [23]. In the case is itself a , the theory can be simplified as Y shown in [21, 6, 5]. As in the scalar case, a RKHS is completely characterize by a map K from X X into the space of bounded operators on such that × Y N K(x , x )y ,y 0 h i j j ii ≥ i,j=1 X for any x ,...,x in X and y ,...,y in . Such a map is called a - 1 N 1 N Y Y reproducing kernel and the corresponding RKHS is denoted by K. This paper focuses on three aspects of particular interest in vectorH valued learning problems:

vector valued feature maps; • universal reproducing kernels; • translation invariant reproducing kernels. • The feature map approach is the standard way in which scalar RKHS are pre- sented in learning theory, see for example [26]. A feature map is a mapping the input space X into an arbitrary Hilbert space in such a way H that can be identified with a unique RKHS. Conversely, any RKHS can be realizedH as a closed subspace of a concrete Hilbert space, called feature space, by means of a suitable feature map – typical examples of feature spaces are ℓ2 and L2(X,µ) for some µ. The concept of feature map is extended to the vector valued setting in [6, 5], where a feature map is defined as a function from X into the space of bounded operators between and the feature space . In the first part of ourY paper, Section 3 showsH that sum, product and compo- sition with maps of RKHS can be easily described by suitable feature maps. In particular we give an elementary proof of Schur lemma about the product of a scalar kernel with a vector valued kernel. Moreover, we present several examples of vector valued RKHS, most of them considered in [22, 5]. For each one of them we exhibit a nice feature space. This allows to describe the impact of these examples on some learning algorithms, like the regularized least-squares [13]. In the second part of the paper, Section 4 discusses the problem of char- acterizing universal kernels. We say that a -reproducing kernel is universal Y 2 2 if the corresponding RKHS K is dense in L (X,µ; ) for any probability measure µ on the input spaceHX. This definition is motivatedY observing that in learning theory the goal is to approximate a target function f ∗ by means of a prediction function fn K, depending on the data, in such a way ∗ ∈ H the distance between f and fn goes to zero when the number of data n goes to infinity. In learning theory the “right” distance is given by the in L2(X,µ; ), where µ is the (unknown) probability distribution modeling Y the sample of the input data, see [8]. The possibility of learning any target ∗ function f by means of functions in K is precisely the density of K in L2(X,µ; ). Since the probability measureH µ is unknown, we requireH that Y the above property holds for any choice of µ – compare with the definition of universal consistency for a learning algorithm [18]. Under the condition that the elements of are continuous functions vanishing at infinity, we prove HK that universality of K is equivalent to require that K is dense in 0(X; ), the ofH continuous functions vanishingH at infinity withC the uni-Y form norm. If X is compact and = C, the density of K in 0(X; ) is precisely the definition of universalityH given in [30, 32]. ForH arbitraryC XYand , another definition of universality is suggested in [5] under the assumption Y that the elements of K are continuous functions. We show that this last H 2 notion is equivalent to require that K is dense in L (X,µ; ) for any prob- ability measure µ with compact support,H or that is denseY in (X; ), HK C Y the space of continuous functions with the compact-open topology. If X is not compact, the two definitions of universality are not equivalent, as we show in two examples. To avoid confusion we refer to the second notion as compact-universality. We characterize both universality and compact-universality in terms of the injectivity of the integral operator on L2(X,µ; ) whose kernel is the repro- ducing kernel K. For compact-universal kernels,Y this result is presented in a slightly different form in [5] – compare Theorem 2 below with Theorem 11 of [5]. However, our statement of the theorem does not require a direct use of vector valued measures, our proof is simpler and it is based on the fact that any bounded linear functional T on (X; ) is of the form C0 Y T (f)= f(x), h(x) dµ(x), h i ZX where µ is a probability measure and h is a bounded measurable function from X to – see Appendix A. Notice that, though in learning theory Y 2 the main issue is the density of the RKHS K in L (X,µ; ), however, our results hold if, in the definition of universalH kernels, we replaceY L2(X,µ; ) p Y with L (X,µ; ) for any 1 p < . In particular, we show that K is dense in (XY; ) if and only≤ if there∞ exists 1 p < such that H is C0 Y ≤ ∞ HK 3 p dense in L (X,µ; ) for any probability measure µ. In that case, K is dense in Lq(X,µ; Y) for any 1 q < . H Y ≤ ∞ In the third part of the paper, under the assumption that X is a group, Section 5 studies translation invariant reproducing kernels, that is, the kernels such that K(x, t) = K (t−1x) for some operator valued function K : X e e → ( ) of completely positive type. In particular, we show that any translation LinvariantY kernel is of the form

∗ K(x, t)= Aπx−1tA for some unitary representation π of X acting on a Hilbert space , and a H A : . If X is an abelian group, SNAG theorem [16] provides a more explicitH→Y description of the reproducing kernel K, namely

K(x, t)= χ(t x)dQ(χ), ˆ − ZX where Xˆ is the dual group and Q is a positive operator valued measure on Xˆ. The above equation is precisely the content of Bochner theorem for operator valued functions of positive type [2, 15]. In particular, we show that the corresponding RKHS can be always realized as a closed subspace of HK L2(X,ˆ ν,ˆ ) whereν ˆ is a suitable positive measure on Xˆ. In this setting, we give aY sufficient condition ensuring that a translation invariant kernel is universal. This condition is also necessary if X is compact or = C. For Y scalar kernels and compact-universality this result is given in [22]. We end the paper by discussing in Section 6 the universality of some of the examples introduced in Section 3.

2 Background

In this section we set the main notations and we recall some basic facts about vector valued reproducing kernels.

2.1 Notations and assumptions In the following we fix a locally compact second countable topological space X and a complex separable Hilbert space , whose norm and scalar product are denoted by and , respectively. LocalY compactness of X is needed in order to prove Theoremk·k h· 7·i in the appendix, which is at the root of Theorem 1. The separability of X and will avoid some problems in measure theory. All these assumptions are alwaysY satisfied in learning theory.

4 We denote by (X; ) the vector space of functions f : X , by (X; ) the subspaceF of continuousY functions, and by (X; ) the→ subspace Y C Y C0 Y of continuous functions vanishing at infinity. If = C, we set (X) = Y C (X; C) and 0(X)= 0(X, C). If X is compact, 0(X; )= (X; ). CWe regard (CX; ) asC a locally convex topologicalC vectorY spaceC by endowingY C Y 1 it with the compact-open topology and 0(X; ) as a Banach space with respect to the uniform norm f = maxC fY(x) . k k∞ x∈X k k Let (X) be the Borel σ-algebra of X. Bya measure on X we mean a σ-additiveB map µ : (X) [0, + ] which is finite on compact sets2. We say that µ is a probabilityB −→ measure∞ if µ(X) = 1. For 1 p < , ≤ ∞ Lp(X,µ; ) denotes the Banach space of (equivalence classes of) measur- able3 functionsY f : X such that f p is µ-integrable, with norm → Y k k f = f(x) p dµ(x) 1/p. If p = 2 we denote the scalar product in k kp X k k L2(X,µ, ) by , . For p = , L∞(X,µ; ) is the Banach space of µ- RY h· ·i2  ∞ Y essentially bounded measurable functions f : X with norm f µ,∞ = µ ess sup f(x) . → Y k k − x∈X k k If µ is a probability measure, clearly

(X; ) Lp(X,µ; ) Lq(X,µ; ) C0 Y ⊂ Y ⊂ Y for all 1 q

5 called a -reproducing kernel if Y N K(x , x )y ,y 0 h i j j ii ≥ i,j=1 X for any x1,...,xN in X, y1,...,yN in and N 1. Given x X, Kx : (X; ) denotes the linear operator whoseY action≥ on a vector∈ y isY the → FfunctionY K y (X; ) defined by ∈ Y x ∈F Y (K y)(t)= K(t, x)y t X. (1) x ∈

Given a -reproducing kernel K, there is a unique Hilbert space K (X; ) satisfyingY H ⊂F Y

K ( , ) x X (2) x ∈L Y HK ∈ f(x)= K∗f x X, f , (3) x ∈ ∈ HK ∗ where Kx : K is the adjoint of Kx, see Proposition 2.1 of [6]. The space isH called→ the Y reproducing kernel Hilbert space associated with K, HK the corresponding scalar product and norm are denoted by , K and K, respectively. As a consequence of (3), we have that h· ·i k·k

K(x, t) = K∗K x, t X x t ∈ = span K y x X,y . HK { x | ∈ ∈ Y}

As discussed in the introduction, the space K can be realized as a closed subspace of some arbitrary Hilbert space by meansH of a suitable feature map, as shown by the next result, see Proposition 2.4 of [6]. Proposition 1. Let be a Hilbert space and γ : X ( ; ). Then the operator W : H(X; ) defined by −→ B Y H H −→F Y (Wu)(x)= γ∗u, u , x X, (4) x ∈ H ∈ is a partial isometry from onto the reproducing kernel Hilbert space K with reproducing kernel H H

K(x, t)= γ∗γ , x, t X. (5) x t ∈ Moreover, W ∗W is the orthogonal projection onto

ker W ⊥ = span γ y x X, y , { x | ∈ ∈ Y} and f = inf u u , Wu = f . k kK {k kH | ∈ H }

6 The map γ is usually called the feature map, W the feature operator and the feature space. Since W is an isometry from ker W ⊥ onto , the map H HK W allows us to identify with the closed subspace ker W ⊥ of . With a HK H mild abuse of notation, we say that K is embedded into by means of the feature operator W . H H Comparing (4) with (3), we notice that any RKHS admits a trivial feature HK map, namely γx = Kx. In this case the feature operator is the identity. Conversely, if is a Hilbert space of functions from X to such that f H Y k k ≤ Cx f H for some positive constant Cx, then there exists a bounded operator γ k: k such that f(x) = γ∗f. Hence, the above proposition implies x Y → H x that is a RKHS with kernel given by (5) and that the feature operator is the identity.H

2.3 Mercer and -kernels C0 In this paper, we mainly focus on reproducing kernel Hilbert spaces, whose elements are continuous functions. In particular we study the following two classes of reproducing kernels.

Definition 1. A reproducing kernel K : X X ( ) is called × →L Y (i) Mercer provided that is a subspace of (X; ); HK C Y (ii) provided that is a subspace of (X; ). C0 HK C0 Y

The choice of (X; ) and 0(X; ) is motivated in Section 4 where we discuss the universalityC Y problem.C Y The following proposition directly characterizes Mercer and 0-kernels in terms of properties of the kernels. C

Proposition 2. Let K be a reproducing kernel.

(i) The kernel K is Mercer iff the function x K(x, x) is locally bounded and K y (X; ) for all x X and7−→y k . k x ∈ C Y ∈ ∈Y

(ii) The kernel K is 0 iff the function x K(x, x) is bounded and K y (X; ) forC all x X and y 7−→. k k x ∈ C0 Y ∈ ∈Y If K is a Mercer kernel, the inclusion ֒ (X; ) is continuous. If K is HK → C Y a 0-kernel the inclusion K ֒ 0(X; ) is continuous. In both cases, the spaceC is separable. H → C Y HK

7 Proof. We prove only (ii), since the other proof is similar – see Proposition 5.1 of [6]. If K 0(X; ), it is clear that Kxy is an element of 0(X; ). H ⊂ C ∗ Y C Y Moreover, since Kxf = f(x) f ∞ f K, by the principle of k k k k≤k k ∀ ∈ H ∗ uniform boundedness there exists M < such that Kx M for all x. Therefore, K(x, x) = K∗ 2 M 2 for∞ all x. k k ≤ k k k xk ≤ Conversely, assume that the function x K(x, x) is bounded and Kxy (X; ). Given f , we have 7−→ k k ∈ C0 Y ∈ HK f(x) f K(x, x) 1/2 M f . k k≤k kK k k ≤ k kK In particular, convergence in implies uniform convergence, so that the HK closure (in ) of the linear span of K y x X,y is contained in HK { x | ∈ ∈ Y} 0(X; ), i.e. K 0(X; ). CThe continuityY H of⊆ the C inclusionY of in (X; ) follows from f HK C0 Y k k∞ ≤ M f . Finally K is separable by Corollary 5.2 of [6]. k kHK H If K is defined by means of a feature map γ, the above characterization can beH expressed in terms of γ, as shown by the following result.

Corollary 1. With the notations of Proposition 1 the following conditions are equivalent.

(a) The kernel K is Mercer [resp. ]. C0 (b) There is a total set in such that W ( ) (X; ) [resp. W ( ) (X; )] and the functionS Hx γ is locallyS ⊂ C boundedY [resp. boundedS ⊂]. C0 Y 7−→ k xk Proof. We give the proof only in the case of a -kernel, the other case being C0 simpler. Suppose hence (a) holds true, i.e. K 0, then W ( ) ran W = (X; ) for all subset of . Moreover,H ⊂ Cγ 2 = K(Sx, x⊂) M by HK ⊂ C0 Y S H k xk k k ≤ item (ii) of Proposition 2. Conversely, if condition (b) holds, we have that for all x X and u ∈ ∈ H ∗ ∗ 1 1 (Wu)(x) = K (Wu) K W u K(x, x) 2 u M 2 u , k k k x k≤k xkk kk kH ≤k k k kH ≤ k kH where W 1 being W a partial isometry. Then W maps into the space k k ≤ H of bounded functions and W is continuous from onto endowed with H HK the uniform norm. Since W ( ) 0(X; ) and 0(X; ) is complete, then (X; ). S ⊂ C Y C Y HK ⊂ C0 Y

8 2.4 Mercer theorem For a Mercer kernel K, there is a canonical feature map, based on Mercer theorem, which relates the spectral properties of the integral operator with kernel K, and the structure of the corresponding reproducing kernel Hilbert space. This result will be also used in the examples. To state this result for vector valued reproducing kernels, we need some preliminary facts. First of all, if K is a Mercer kernel and µ is a probability 2 measure on X, the space K is a subspace of L (X,µ; ), provided that K(x, x) is bounded on theH support of µ. This last conditionY is always k k satisfied if K is a 0-kernel or if µ has compact support. If K is a subspace of L2(X,µ; ), weC denote the canonical inclusion by H Y .( ;i : ֒ L2(X,µ µ HK → Y

Next lemma states some properties of iµ and its proof is a consequence of Propositions 3.3, 4.4 and 4.8 of [6].

Proposition 3. Let K be a Mercer kernel and µ a probability measure such that K is bounded on the support of µ. The inclusion iµ is a bounded operator, its adjoint i∗ : L2(X,µ; ) is given by µ Y −→ HK

∗ (iµf)(x)= K(x, t)f(t)dµ(t), ZX ∗ where the integral converges in norm, and the composition iµiµ = Lµ is the integral operator on L2(X,µ; ) with kernel K Y

(Lµf)(x)= K(x, t)f(t)dµ(t). ZX In particular, if K(x, x) is a for all x X, then L is a ∈ K compact operator.

The fact that LK is a compact operator implies that there is a family (fi)i∈I of eigenvectors in (X; ) and a family (σi)i∈I of eigenvalues in ]0, [ C Y ⊥ ∞ such that (fi)i∈I is an orthonormal basis of ker Lµ = ran Lµ and

Lµfi = σifi. (6)

With this notation we are ready to state Mercer Theorem for vector valued kernels. Its proof is consequence of Proposition 6.1 and Theorem 6.3 of [6].

9 Proposition 4. Let µ be a probability measure with supp µ = X. Suppose K is a Mercer kernel such that sup K(x, x) < , and K(x, x) is a x∈X k k ∞ compact operator x X. With the notation of (6), we have that ∀ ∈ f, f 2 = f (X; ) ker L ⊥ |h ii2 | < (7) HK { ∈ C Y ∩ µ | σ ∞} i∈I i X f, fi 2 fi,g 2 f,g K = h i h i (8) h i σi Xi∈I K(x, t)= σ f (x) f (t) (9) i i ⊗ i Xi∈I where the last converges in the strong operator topology of ( ). L Y

Equations (7) and (8) imply that (√σifi)i∈I is an orthonormal basis in K . In particular the vectors √σifi are ℓ2-linearly independent in (X; ), H F Y namely, if (c ) is a family such that c 2 < and c √σ f (x)= i i∈I i∈I | i| ∞ i∈I i i i 0 for all x X, then ci = 0 for all i I. As said∈ at the beginning of Section∈P 2.4, PropositionP 4 gives a feature operator, which is often used in learning theory.

Example 1. With the assumptions and notations of Proposition 4, the re- ⊥ producing kernel Hilbert space K is unitarily equivalent to ker Lµ = ran Lµ by means of the feature operatorH

1 2 2 (W f)(x)= √σ f (x) f, f =(Lµ f)(x) , f L (X,µ; ) . (10) i i h ii2 ∈ Y Xi∈I Proof. Given x X, define ∈ γ : L2(X,µ; ) γ y = √σ y, f (x) f , x Y → Y x i h i i i Xi∈I which is well defined since (fi)i∈I is orthonormal family of continuous func- 2 tions and (9) ensures that i∈I σi y, fi(x) < . Using (9) again, one ∗ |h i| ∞ checks that γ γt = K(x, t). The fact that feature operator is given by (10) x P is clear by definition of γx. Since ker W = ker Lµ, W is a from ker L ⊥ onto . µ HK 2.5 Trivial examples We give two examples of trivial vector valued kernels.

10 Example 2. Let B ( ) be a positive operator and define K(x, t)= B for all x, t X, then K∈Lis a Y -reproducing kernel, is unitarily equivalent to ∈ Y HK ker B⊥ = ran B by means of the feature operator

1 ⊥ (Wy)(x)= B 2 y x X, y ker B . ∈ ∈

The kernel K is of Mercer type and it is a 0-kernel if and only if X is compact. C

⊥ 1 Proof. Apply Proposition 1 with = ker B and γ = B 2 . Since B is H x injective on , then W is unitary. The claims about the continuity are clear. H

Example 3. Let f : X , f = 0. Define K(x, t) = f(x) f(t), then → Y 6 ⊗ K is a reproducing kernel, K is unitarily equivalent to C by means of the feature operator H

(Wc)(x)= cf(x) x X, c C. ∈ ∈ In particular K is Mercer [resp. ] if and only if f (X; ) [resp. f C0 ∈ C Y ∈ (X; )]. C0 Y Proof. Apply Proposition 1 with = C and γ y = y, f(x) . Since f = 0, H x h i 6 W is injective. The characterization about Mercer and is trivial. C0 3 Operations with kernels

In this section we characterize reproducing kernel Hilbert spaces whose kernel is defined by algebraic operations, like sum, product and composition. Most of the results are well known for scalar kernels, whereas for vector valued ker- nels they are consequences of the theory developed in [27] in a more general context. We provide a direct and simple proof of these results, based on the use of suitable feature maps. In some cases, our approach can be of interest also in the scalar case, like, for example, in proving Schur lemma about the product of kernels. As an application, we present a large supply of examples of vector valued re- producing kernels and, for most of them, we realize the corresponding RKHS by elegant and simple structures. This characterization will be used to an- alyze some learning algorithm, like regularized least-squares, in the vector valued setting.

11 3.1 Sum of kernels The following result extends to vector valued kernels the relation between sum of kernels and sum of the corresponding reproducing kernel Hilbert spaces.

i Proposition 5. Denote by I a countable set and let (K )i∈I be a family of -reproducing kernels such that Y Ki(x, x)y,y < y and x X. ∞ ∀ ∈Y ∀ ∈ i∈I X Given x, t X the series Ki(x, t) converges to a bounded operator ∈ i∈I K(x, t) in the strong operator topology, and the map K : X X ( ) defined by P × → L Y K(x, t)y = Ki(x, t)y i∈I X is a -reproducing kernel. The corresponding space is embedded in Y HK i by means of the feature operator i∈I HK L W (f)(x)= f (x) where f = f i ⊕i∈I i i∈I X where the sum converges in norm. Moreover, if each Ki is a Mercer kernel [resp. -kernel] and x Ki(x, x) C0 7→ i∈I k k is locally bounded [resp. bounded], then K is Mercer [resp. 0]. C P Proof. We apply Proposition 1. Letting = i , we regard each i H i∈I HK HK as a closed subspace of so that any two of them are orthogonal. Given x X, we define the boundedH operator γ : L by γ = Ki , where ∈ x Y → H x i∈I x the series converges in the strong operator topology since, given y , P ∈Y 2 Ki y = Ki(x, x)y,y < x Ki ∞ i∈I i∈I X X by assumption, see [7]. Given i I and f i , then ∈ i ∈ HK γ∗f ,y = f ,Ki y = f (x),y h x i i i x Ki h i i ∗ ∗ by reproducing property (3), so that γxfi = fi(x). Since γx is continuous, for any f = f , ⊕i∈I i ∗ ∗ (W f)(x)= γxf = γxfi = fi(x) Xi∈I Xi∈I 12 where the series converges in norm. ∗ i Finally, K(x, t)y = γxγty = i∈I (γty)i(x)= i∈I K (x, t)y, that is K(x, t)= Ki(x, t) in the strong operator topology. i∈I P P The second part is a consequence of Corollary 1 with = i . P S i∈I HK As an application, we have the following example. S

Example 4. Let (fi)∈I a countable family of functions fi : X such that 2 →Y i∈I fi(x),y is finite for all x X and y Y . Define K : X X ( ) |has i| ∈ ∈ × → LPY K(x, t)= f (x) f (t) . i ⊗ i Xi∈I Then, the sum converges in the strong operator topology, K is a reproducing kernel and

= f (X; ) f(x)= c f (x), c 2 < . (11) HK { ∈F Y | i i | i| ∞} Xi∈I Xi∈I In particular (f ) is a normalized tight frame in . It is an orthonormal i i∈I HK basis if and only if (f ) is ℓ -linearly independent in (X; ). i i∈I 2 F Y Proof. Apply Proposition 5, with Ki(x, t) = f (x) f (t), observing that i ⊗ i i C K = by Example 3, so that i∈I Ki ℓ2. The feature operator is Hexplicitly given by ⊕ H ≃

W (c)(x)= c f (x) where c =(c ) , c 2 < , i i i i∈I | i| ∞ Xi∈I Xi∈I so that (11) is clear. If (ei)i∈I is the canonical orthonormal basis of ℓ2, then W e = f and, for any f , i i ∈ HK 2 ∗ 2 ∗ 2 2 f = W f = W f, ei = f, fi , k kK k kℓ2 |h iℓ2 | |h iK | i i X X i.e. (fi)i∈I is a normalized tight frame in K. Clearly, it is an orthonormal basis if and only if W is unitary, i.e. WH is injective. This is precisely the condition that (f ) is ℓ -linearly independent in (X; ). i i∈I 2 F Y Proposition 4 shows that any RKHS with a bounded compact Mercer kernel can be realized as in the above example, where the functions fi are the eigenfunctions (with f 2 = σ ) of the integral operator L with eigenvalues k ik2 i µ σi > 0, and µ is any probability measure with supp µ = X, see (6).

13 3.2 Composition with maps We now describe the reproducing kernel Hilbert spaces whose kernel is defined in terms of a mother kernel and suitable maps acting either on the input space X or on the output space . The following result characterizes the action of Y a bounded operator on . Y Proposition 6. Let K be a -reproducing kernel. Let ′ be another Hilbert space and w : ′ be a boundedY operator. Define Y Y→Y K : X X ( ′) K (x, t)= wK(x, t)w∗, w × →L Y w then K is a ′ reproducing kernel and is embedded in by means of w Y HKw HK W : , (W f)(x)= wf(x) x X. HK −→ HKw ∈

If w is injective, Kw is unitarily equivalent to K. Moreover, if K is Mercer [resp. ], then KH is Mercer [resp. ]. H C0 w C0 Proof. Let γ : ′ , γ = K w∗ and apply Proposition 1 with = . x Y → HK x x H HK The feature operator from K onto Kw is explicitly given by (W f)(x) = ∗ H H γxf = wf(x). If w is injective, then W is unitary. The second claim is evident. We now study the action of an arbitrary map on X. Proposition 7. Let K be a -reproducing kernel on X. Let T be another locally compact second countableY topological space, and Ψ : T X. Define → K : T T ( ) K (t , t )= K(Ψ(t ), Ψ(t )) t , t T. Ψ × →L Y Ψ 1 2 1 2 1 2 ∈

Then KΨ is a -reproducing kernel on T , the space KΨ is unitarily equiv- alent to Y H

span K y x ran Ψ = f f(x)=0 x ran Ψ ⊥ { x | ∈ } { ∈ HK | ∀ ∈ } by means of the feature operator

W : W (f)(t)= f(Ψ(t)) f , t T. HK −→ HKΨ ∈ HK ∈

If K is a Mercer kernel and Ψ is continuous, then KΨ is Mercer. If K is a -kernel and Ψ is continuous and proper, then K is . C0 Ψ C0 Proof. Apply Proposition 1 with = and, for any t T , γ = K , H HK ∈ t Ψ(t) observing that ker W = f K f(x)=0 x ran Ψ . The claims about Mercer{ and∈ H -kernels| are∀ clear.∈ } C0 14 In the above proposition observe that ker W ⊥ can be identified with the quotient space / ker W , so that one has also the natural identification HK f f (12) HKΨ ≃{ |ran Ψ | ∈ HK} where, the r.h.s. is endowed with the norm f = inf g g , g = f |ran Ψ {k kK | ∈ HK |ran Ψ |ran Ψ}

As a consequence, we describe the relation between a kernel and its re- striction to a subset.

Corollary 2. Let X0 be a subset of X. Let KX0 be the restriction of K to X X , then 0 × 0 K = f| : f K . H X0 { X0 ∈ H }

If K is Mercer and X0 is locally closed, then KX0 is Mercer. If K is 0 and X is closed, then K is . C 0 X0 C0 Proof. Apply Proposition 7 and identification (12), with Ψ the canonical inclusion of X0 in X. We end this part by describing the reproducing kernel Hilbert space as- sociated with the kernel proposed in [5]. Proposition 8. Let κ be a scalar reproducing kernel on X. Let T be an- other locally compact second countable topological space. Let Ψ1,..., Ψm be functions from T to X and define K(t , t ) as the m m-matrix 1 2 × K(t , t ) = κ(Ψ (t ), Ψ (t )) i, j =1,...,m, t , t T. 1 2 ij i 1 j 2 1 2 ∈ Then K is a Cm-reproducing kernel on T , the space is embedded in HK Hκ by means of the feature operator W : (W (ϕ)(t)) = ϕ(Ψ (t)) ϕ , t T. Hκ −→ HK i i ∈ Hκ ∈

If one of Ψ1, . . ., Ψm is surjective, then W is unitary. Proof. Apply Proposition 1 with = and γ : Cm , H Hκ t → Hκ m γ (y ,...,y )= y κ , t 1 m i Ψi(t) i=1 X ∗ so that γt (ϕ)i = ϕ(Ψi(t)). If Ψi is surjective for some index i = 1,...,m, the condition ϕ(Ψi(t)) = 0 for all t T implies that ϕ(x)=0 for all x X, that is, ϕ = 0. Hence W is ∈ ∈ injective and, hence, unitary.

15 3.3 Product of kernels The following proposition extends Schur lemma about products of reproduc- ing kernels to the vector valued case.

Proposition 9. Let K be a -kernel and κ a scalar kernel. Define Y (κK)(x, t)= κ(x, t)K(x, t) x, t X, ∈ then κK is a -reproducing kernel and is embedded into by Y HκK Hκ ⊗ HK means of the feature operator

W (ϕ f)(x)= ϕ(x)f(x) ϕ , f . ⊗ ∈ Hκ ∈ HK If both κ and K are Mercer kernels, so is κK, whereas if

κ (X) and K v (X; ) x ∈ C0 x ∈ C Y sup κ(x, x), K(x, x) < and or (13)  x∈X{ k k} ∞ κ (X) and K v (X; )  x ∈ C x ∈ C0 Y then K is a kernel.  C0

Proof. Let = κ K . Since κ is a scalar kernel, κx κ. Define H H ⊗ H ∗ ∈ H γx : by means of γxy = κx Kxy, then γx(ϕ f)= ϕ(x)f(x). First claimY is → a H consequence of Proposition⊗ 1. ⊗ If both κ and K are Mercer kernels, clearly κK is Mercer. To prove that if (13) hold then K is 0, we apply Corollary 1 with = ϕ f ϕ , f , and observeC that S { ⊗ | ∈ Hκ ∈ HK} γ κ K C, k xk≤k xkκ k xk ≤ by assumption. Based on the above results, we characterize the RKHS whose kernel is given in [5].

Example 5. Let κ be a scalar reproducing kernel and B a positive bounded operator on . Define K : X X ( ) as Y × →L Y K(x, t)= κ(x, t)B x, t X ∈ (i) The map K is a -reproducing kernel and is unitarily equivalent Y HK to ker B⊥ by means of the unitary operator Hκ ⊗ 1 W (ϕ y)(x)= ϕ(x)B 2 y. ⊗

16 (ii) If κ is Mercer [resp. ], then K is Mercer [resp. ], too. C0 C0 ⊥ (iii) If there is an orthonormal basis (yi)i∈I of ker B such that Byi = σiyi (so that σi > 0 for all i I), then K is unitarily equivalent to i∈I κ by means of the unitary∈ operator H ⊕ H

W ( ϕ )(x)= √σ ϕ (x)y , (14) ⊕i∈I i i i i Xi∈I f where the series converges in norm.

Proof. First two items are a consequence of Proposition 9 and Example 2. We prove item (iii) in two steps. Apply first Proposition 6 with w : ℓ2, (wy) = y,y , so that is embedded in , by means of theY feature → i h ii HKw HK operator Ww(f) = w f for all f K . The corresponding ℓ2-kernel is ◦∗ ∈ H Kw(x, t)= κ(x, t)wBw . By definition of w, the kernel Kw is diagonal with respect to (ei)i∈I , the canonical basis of ℓ2, namely

K (x, t)= σ κ(x, t)e e =: Ki(x, t), w i i ⊗ i Xi∈I Xi∈I where the series converges in the strong operator topology. Now observe that, for each i I, ker(σ e e )⊥ = Ce , so that for item (i) ∈ i i ⊗ i i of this example, the space Ki is unitarily equivalent to κ C ei κ, through the feature operatorH H ⊗ ≃ H

i i W : i , W (ϕ)(x)= ϕ(x)√σ e Hκ → HK i i i Applying Proposition 5 to the family (K )i∈I , we obtain a unitary operator

W : , W ( ϕ )(x)= ϕ (x)√σ e , Hκ −→ HKw ⊕i i i i i i∈I i M X

(the operator W is unitary since σi > 0 for all i I, so that W is injective). ∈∗ Equation (14) is finally obtained letting W = WwW . If in Example 5, is a RKHS of scalar functions over some set X′, then Y f there is a particular choice for the operator B, suggested by Example 1.

Example 6. Let X and X′ be two locally compact second countable topo- logical spaces. Let κ : X X C and κ′ : X′ X′ C be two scalar × → × → reproducing kernels on X and X′, respectively.

17 ′ (i) If I denotes the identity operator on ′ , define Hκ K : X X ( ) K(x, t)= κ(x, t)I′, × →L Hκ then K is a ′ -reproducing kernel on X and the corresponding RKHS Hκ is unitarily equivalent to ′ by means of the feature operator HK Hκ ⊗Hκ W : ′ , W (ϕ ϕ )(x)= ϕ (x)ϕ . Hκ ⊗ Hκ −→ HK 1 ⊗ 2 1 2 (ii) Define κ κ′ :(X X′) (X X′) C as × × × × → (κ κ′)(x, x′; t, t′)= κ(x, t)κ′(x′, t′), × ′ ′ then κ κ is a scalar kernel on X X and ′ is unitarily equivalent × × Hκ×κ to by means of the feature operator HK ′ ′ ′ W (f)(x, x ) = [f(x)](x )= f(x), κ ′ ′ f . h x iκ ∈ HK ′ Proof. The firstf part follows from Example 5 with = κ′ and B = I , which is injective. The second part is a consequence ofY PropoHsition 1 applied to

′ ′ ′ γ : X X (C; ) , (x, x ) W (κ κ ′ ) , × −→ L HK ≃ HK 7−→ x ⊗ x taking into account the injectivity of W and the equalities

′ ′ W (ϕ ϕ ),γ ′ = ϕ ϕ , κ κ ′ = ϕ (x) ϕ , κ ′ ′ 1 ⊗ 2 (x,x ) K h 1 ⊗ 2 x ⊗ x i 1 h 2 x iκ ′ ′ = W (ϕ1 ϕ2)(x), κx′ ′ = W (W (ϕ1 ϕ2))(x, x ). h ⊗ iκ ⊗ f By using Proposition 4 on the space X′, the above example can be realized in an alternative way. Example 7. Let X and X′ be two locally compact second countable topo- logical spaces. Let κ : X X C and κ′ : X′ X′ C be two scalar × → ′ × ′ → 0-reproducing kernels on X and X , respectively. Let µ be a probability mea- C ′ ′ ′ 2 ′ ′ sure on X with supp µ = X and Lµ′ be the integral operator on L (X ,µ ) with kernel κ′. Define

2 ′ ′ K : X X (L (X ,µ )) K(x, t)= κ(x, t)L ′ , × →L µ 2 ′ ′ then the kernel K is a L (X ,µ )-reproducing kernel and the space b is b b HK unitarily equivalent to ′ by means of Hκ ⊗ Hκ b W (f g)(x)= f(x)i ′ (g) f , g ′ , ⊗ µ ∈ Hκ ∈ Hκ 2 ′ where i ′ is the inclusion of ′ in L (X ,µ). In particular, K is a -kernel. µ c Hκ C0 18 b Proof. Apply Proposition 6 with K = κI′, as in the previous example, and w = i ′ , which is injective. Clearly K = K, so that b is unitarily equiva- µ w HK lent to ′ . The thesis follows immediately from Example 6. HκI b The above example shows that and b are the same RKHS, where HK HK the elements of K are regarded as functions from X into κ′ , whereas the H 2 H′ ′ elements of b are regarded as functions from X into L (X ,µ ). HK 3.4 Application to learning theory We end this section considering an application of some of the above examples to vector valued regression problems. In learning theory, a popular algorithm is the minimization on a RKHS K of the empirical error with a penalty term proportional to the square of theH norm [13], namely

n ⋆ 1 ℓ ℓ 2 2 f = argmin y f(x ) Y + λ f K . (15) f∈H n − k k K ℓ=1 ! X

Here (x1,y1),..., (xn,yn) is the training set of n input-output pairs (xℓ,yℓ) X Y{ and λ> 0 is the regularization} parameter. If the reproducing kernel∈ × K is as in Example 5, then it can be checked that

⋆ ⋆ f (x)= ϕi (x)yi Xi∈I ⋆ where each ϕi is given by

n ⋆ 1 ℓ ℓ 2 λ 2 ϕi = argmin yi ϕ(x ) + ϕ κ , ϕ∈Hκ n | − | σi k k ! Xℓ=1 ℓ ℓ and yi = y ,yi . In many applications = Cm so that B is a m m positive semi-definite Y × matrix. The above observation reduces the problem of computing the mini- mizer of (15) to I scalar problems, where the cardinality I is the rank of the matrix B. | | | | With the choice of K as in Proposition 8, let f ⋆ be the minimizer given by (15), where the n-examples in the training set are the pairs (tℓ,yℓ) T Rm. By using the fact that W is a partial surjective isometry, one can∈ × check that ⋆ ⋆ ⋆ f (t)=(ϕ (Ψi(t)),...,ϕ (Ψm(t)),

19 where ϕ⋆ is given by

n m ⋆ 1 ℓ ℓ 2 2 ϕ = argmin yi ϕ(xi ) + λ ϕ κ , ϕ∈H n | − | k k κ i=1 ! Xℓ=1 X where yℓ R are the components of the output yℓ Rm and xℓ =Ψ (tℓ) X. i ∈ ∈ i i ∈ With this choice the problem (15) is reduced to a minimization problem on the scalar RKHS . Hκ 4 Universal kernels: main results

In this section we address the problem of defining and characterizing the universality of a kernel K. As pointed out in the introduction, in learning theory a necessary condition in order to have universally consistent algo- rithms is the assumption that the reproducing kernel Hilbert space K is dense in L2(X,µ; ) for any probability measure µ. From this point ofH view Y next definition is very natural.

Definition 2. Let K : X X ( ) be a reproducing kernel. × →L Y (i) A -kernel K is called universal if is dense in L2(X,µ; ) for C0 HK Y each probability measure µ.

(ii) A Mercer kernel K is called compact-universal if is dense in HK L2(X,µ; ) for each probability measure µ with compact support. Y We briefly comment on the above definitions. In item (i) the assumption that the kernel is ensures both that is a subspace of L2(X,µ; ) C0 HK Y and that universality is equivalent to the density of K is 0(X; ) (see Theorem 1). In item (ii), since µ has compact support,H itC is enoughY to 2 assume that K is a Mercer kernel in order to have K L (X,µ; ). This last property turns out to be equivalent to the definitionH ⊂ of universalityY given in [5]. Clearly a universal kernel is also compact-universal. Conversely, a 0-kernel can be compact-universal but not universal, as shown by Examples 8C and 11. Notice that in Definition 2 if we replace L2(X,µ; ) with Lp(X,µ; ) for an arbitrary 1 p < , we have in principle aY different notionY of universality. Nevertheless≤ Theorem∞ 1 clarifies that there is no difference. We state the results for p = 2, since it is the natural choice in learning theory. The following corollary shows that universality is preserved by restriction to a subset.

20 Corollary 3. Let X0 be a subset of X.

(i) If X0 is closed and K is universal, then KX0 is universal.

(ii) If X0 is locally closed and K is compact-universal, then KX0 is compact- universal.

Proof. We only prove (i). Since X0 is closed, Corollary 2 implies that KX0 is a 0-kernel, and a function f belongs to K if and only if there exists C H X0 g K such that f = g| . Given a probability measure µ on X0, let ν be ∈ H X0 the probability measure on X, ν(E)= µ(E X0) for any Borel subset E of X. By universality of K, is dense in L2∩(X,ν, ) L2(X , µ, ), where HK Y ≃ 0 Y the equivalence is given by the restriction from X to X0, so that K is H X0 dense in L2(X , µ, ). 0 Y The converse is clearly not true. Notice that the compact-universal ker- nels are precisely the Mercer kernels such that KX0 is universal for any com- pact subset X0 of X. In the next subsections we discuss separately the two notions of univer- sality and then we make a comparison between them.

4.1 Universality and -kernels C0 In this section we characterize the universal 0-kernels. First result shows 2 C that the density of K in L (X,µ; ) for any probability measure µ is equiv- alent to the densityH in (X; ) andY that one can replace L2(X,µ; ) with C0 Y Y Lp(X,µ; ), 1 p< . Y ≤ ∞ Theorem 1. Suppose K is a -kernel. The following facts are equivalent. C0 (a) The kernel K is universal.

(b) The space is dense in (X; ). HK C0 Y p (c) There is 1 p < such that K is dense in L (X,µ; ) for all probability measures≤ ∞µ on X. H Y Proof. Clearly (a) implies (c). Since X is locally compact and second count- able, (X; ) is dense in L2(X,µ; ) where the inclusion is continuous, so C0 Y Y that (b) implies (a). We show that (c) implies (b). Suppose hence that K is not dense in (X; ). Then, there exists T (X; )∗, T = 0 suchH that T (f)=0 C0 Y ∈ C0 Y 6 for all f K . By Theorem 7, there is a probability measure µ on X and a function∈h H L∞(X,µ; ) such that T (f) = f(x), h(x) dµ(x). Since ∈ Y X h i R 21 T = 0, then h = 0. Since6 µ is a probability6 measure, h is a non-null element in Lp/(p−1)(X,µ; )= Y Lp(X,µ; )∗ (where we set 1/0= ) such that Y ∞ f(x), h(x) dµ(x)=0 f . h i ∀ ∈ HK ZX It follows that is not dense in Lp(X,µ; ). HK Y As a consequence of the previous theorem, we have the following nice corollary.

Corollary 4. Let K be a 0- kernel. Given 1 p q < , the space K is dense in Lp(X,µ; ) for allC probability measures≤ µ≤if and∞ only if it isH dense Y in Lq(X,µ; ) for all probability measures µ. Y The previous result is not trivial. Clearly, if q p, the space Lq(X,µ; ) ≥ Y is always a dense subspace of Lp(X,µ; ) and the inclusion is continuous. q Y Hence, if a RKHS K is dense in L (X,µ; ), then K is always dense in Lp(X,µ; ). However,H in general Lp(X,µ; )Y is not containedH in Lq(X,µ; ), Y Y Y so that, if is dense Lp(X,µ; ), the density of in Lq(X,µ; ) has to HK Y HK Y be proved. Corollary 4 shows this result under the assumption that K is 0. Now, we give a characterisation of universality of K in terms of the injec-C tivity property of the integral operators Lµ, for µ varying over the probability measures on X.

Theorem 2. Suppose K is a 0-kernel. Then the following facts are equiv- alent. C

(a) The kernel K is universal.

∗ 2 (b) The operator iµ : L (X,µ; ) K is an injective operator for all probability measures µ on XY. → H

2 2 (c) The integral operator Lµ : L (X,µ; ) L (X,µ; ) is injective for all probability measures µ on X. Y → Y

The proof is an immediate consequence of Theorem 1 and the next propo- sition.

Proposition 10. Let K be a Mercer kernel and µ a fixed probability measure on X such that K is bounded on the support of µ. The following facts are equivalent.

(a) The space is dense in L2(X,µ; ). HK Y 22 ∗ (b) The operator iµ is injective.

(c) The integral operator Lµ is injective. Proof. The space is dense in L2(X,µ; ) if and only if the range of i is HK Y µ dense in L2(X,µ; ). This last condition is equivalent to the injectivity of ∗ Y ∗ ∗ iµ, that is, (a) is equivalent to (b). Since Lµ = iµiµ and ker Lµ = ker iµ, then (b) and (c) are equivalent.

4.2 Compact-universality In this section, we characterize compact-universality of Mercer kernels and we show that compact-universality is precisely what is called universality in [5]. Next theorem characterizes compact-universality. Theorem 3. Suppose K is a Mercer kernel. The following facts are equiva- lent. (a) The kernel K is compact-universal.

(b) The space is dense in (X; ) endowed with compact-open topology. HK C Y p (c) There is 1 p < such that K is dense in L (X,µ; ) for all compactly supported≤ ∞ probability measures.H Y Proof. Clearly (a) implies (c). We prove that (b) implies (a). Indeed, fixed a probability measure µ with compact support Z, the fact that K is dense in (X; ) implies that := f f is dense in H(Z; ), but C Y HK |Z { |Z | ∈ HK } C Y (Z; ) is clearly dense in L2(Z,µ; ) L2(X,µ; ) with continuous injec- C Y 2 Y ≃ Y tion. Hence K is dense in L (X,µ; ). It only remains to prove that (c) implies (b).H For this, it is enough toY prove that is dense in (Z; ) HK |Z C Y with the uniform norm, for all compact subset Z of X. But this is a simple consequence of Theorem 1 since is clearly dense in Lp(Z,µ; ) for all HK |Z Y probability measure µ on Z, and (Z; )= (Z; ). C Y C0 Y The analog of theorem 2 also holds. Theorem 4. Suppose K is a Mercer kernel. Then the following facts are equivalent. (a) The kernel K is compact-universal.

∗ 2 (b) The operator iµ : K L (X,µ; ) is an injective operator for all compactly supportedH probability→ measuresY µ on X.

23 2 2 (c) The integral operator Lµ : L (X,µ; ) L (X,µ; ) is injective for all probability measures µ on X withY compact→ support.Y The proof is a simple consequence of Proposition 10. Clearly universality of a 0-kernel K implies compact-universality. The converse is not true as shownC by the following example, see also Example 11. The reason of this phenomenon is the fact that 0(X; ) endowed with the compact-open topology is not continuously embeddedC inY Lp(X,µ; ). Y 2 Example 8. Let X = Z+, and let ℓ be the Hilbert space of square summable . Then, ℓ2 is a RKHS of scalar functions on X with reproducing kernel K(i, j) = δi,j, where δi,j is the Kronecker delta. We fix the following 2 f Z in ℓ { k}k∈ + fk(j)= δj,k + eδj,k+1, and we let 2 ˜ = ℓ cl span f k Z (16) HK − { k | ∈ +} 2 2 (ℓ cl denotes the closure in ℓ ). K˜ is also a RKHS of scalar functions − H 2 on X, whose reproducing kernel we denote by K˜ . Since ℓ c0 ( = the sequences going to 0 at infinity), K˜ is a -reproducing kernel.⊂ C0 For all n Z+, let Zn = 1, 2 ...n . Zn is compact in X, and every compact set Z∈ X is contained{ in some}Z . Clearly, ⊂ n (Z ) = span (f ) k n , C n k |Zn | ≤ hence ˜ is dense in (X) with the topology of uniform convergence on HK C compact subsets. Let µ be the probability measure on X such that µ( j )=(e 1)e−j. 2 { } 2 − We claim that K˜ is not dense in L (X,µ). In fact, let f L (X,µ) be the H j ∈ function f(j)=( 1) . We have f , f 2 = 0 for all k. By (16) and − h k iL (X,µ) continuity of the inclusion ℓ2 ֒ L2(X,µ), we see that f is in the orthogonal 2 → complement of ˜ in L (X,µ). The claim then follows. HK A universal kernel is strictly positive definite, but the converse in general fails, as shown by the following corollary and example. Corollary 5. Suppose K is a compact-universal kernel. Then K is strictly positive definite, i.e. for all finite subsets x1, x2 ...xN of X such that xi = x if i = j, the condition { } 6 j 6 N K(x , x )y ,y =0 (y , i =1 ...N) h i j j ii i ∈Y i,j=1 X implies yi =0 for all i =1,...,N.

24 Proof. Assume N K(x , x )y ,y = 0 for some finite subset x , x ...x i,j=1 h i j j ii { 1 2 N } ∈ X, xi = xj if i = j, and y1,y2 ...xN in . Taking 6 6 P { } Y 1 N N µ = δ and ϕ = y δ , N xi i xi i=1 i=1 X X we obtain a probability measure µ on X with compact support and a function ϕ L2(X,µ; ) such that ∈ Y N 0= K(x , x )y ,y = N 2 K(x, y)ϕ(y),ϕ(x) dµ(y) dµ(x) h i j j ii h i i,j=1 X×X X Z = N 2 (L ϕ)(x),ϕ(x) dµ(x)= N 2 L ϕ, ϕ . h µ i h µ i2 ZX

Since Lµ is positive and injective by Theorem 4, we have ϕ(xi) = 0 for all i =1,...,N. Since x = x if i = j, then y = 0 for all i =1,...,N. i 6 j 6 i The converse of the above corollary fails to be true, as shown by the following example.

Example 9. Let K : R R C be the kernel × → 1 sin 2π(x t) K (x, t)= e2πi(x−t)pdp = − . π(x t) Z−1 −

The map K is a scalar 0-kernel, which is strictly positive definite, but not universal. C

Proof. We show that it is strictly positive definite. Indeed, let x ,...x X 1 N ∈ such that x = x if i = j, c ,...,c C and suppose i 6 j 6 1 N ∈ N 1 N 0= c c K (x , x )= c e2πixip 2dp i j i j | i | i,j=1 −1 i=1 X Z X

2πixip 2 2πixip 2 Since p i cie is continuous, it follows that i cie = 0 7→ | | | 2πixj t | for all p [ 1, 1]. Observing that the functions fj(t) = e are linearly independent∈ −P on [ 1, 1] since x = x , it follows that c = 0P for all j. Clearly − i 6 j j K is a -kernel, but it is not universal (see Example 11). C0 In the next remark we show that compact-universality is exactly what is called universality in [5].

25 Remark 1. In [5], a Mercer kernel K is said to be universal if, for each compact set Z X ⊆ (Z; )= cl span K ( , x) v x Z, v , (17) C Y k·kZ − · |Z | ∈ ∈Y where cl denotes the closure in (Z; ) with the uniform norm topol- k·kZ − C Y ogy. This is equivalent to require that K is dense (X; ) with the compact- open topology, that is, by Theorem 1H that K is compact-universal.C Y Indeed, by definition of the compact-open topology, is dense in (X; ) if and HK C Y only if (Z; )= cl (18) C Y k·kZ − HK |Z for all compact Z X. Clearly (17) implies⊆ (18). Suppose on the other hand that (18) holds true. Denote with K the restriction of K to Z Z. Since convergence in e × HK implies uniform convergence we have e cl span K( , x)v x Z, v e k·kZ − · |Z | ∈ ∈Y ⊇ HK  On the other hand, Ke = K Z as a linear space of functions (see Corol- lary 2). Hence (18)H impliesH (17).|

5 Translation invariant kernels and univer- sality

In this section we assume that X is a locally compact second countable topological group with identity e and we study the reproducing kernels that are translation invariant, namely

K(zx, zt)= K(x, t) forall x, t, z X. (19) ∈ In particular we characterize all the translation invariant kernels in terms of a unitary representation of X acting on an arbitrary Hilbert space and an operator A : . If X is an abelian group, we give a moreH explicit characterizationH→Y in Theorem 5 and Theorem 13 provides a sufficient condition ensuring that the corresponding reproducing kernel Hilbert space is universal. This condition is also necessary if X is compact or = C. For Y scalar kernels on Rd our result has been already proved in [22]. For a representation π of X on a vector space V we mean a group ho- momorphism from X to the automorphisms of V . In particular, if V is a Hilbert space, π is unitary if it takes values in the group of unitary operators

26 on V . In this framewok the representation is called continuous if π is strongly continuous (see [16]). We denote by λ the left regular representation of X acting on (X; ), namely F Y (λ f)(t)= f(x−1t) t, x X, f (X; ). x ∈ ∈F Y We recall that a function Γ : X ( ) is of completely positive type if →L Y N Γ(x−1x )y ,y 0 (20) j i j i ≥ i,j=1 X for all finite sequences xi i=1...N in X and yi i=1...N in . The following facts{ are} easy to prove. { } Y Proposition 11. Let K : X X ( ) be a reproducing kernel. The × → L Y following conditions are equivalent. (a) K is a translation invariant reproducing kernel.

(b) There is a function Ke : X ( ) of completely positive type such −1 → L Y that K(x, t)= Ke(t x). If one the above conditions is satisfied, then the representation λ leaves in- variant , its action on is unitary and HK HK ∗ K(x, t)= K λ −1 K x, t X (21) e x t e ∈ K(x, x) = K (e) x X (22) k k k e k ∈

The notation Ke for the function of completely positive type associated with the reproducing kernel K is consistent with the definition given by (1) since (K y)(x)= K (x)y y , x X. e e ∈Y ∈ Proof of Proposition 11. Assume (a). Given x, t X, (1) and (19) give ∈ −1 −1 Ke(t x)= K(t x, e)= K(x, t).

Since K is a reproducing kernel, Ke is of completely positive type, so that (b) holds true. Assume (b). Clearly K is a translation invariant reproducing kernel, so that (a) holds true. Suppose now that K is a translation invariant reproducing kernel. Ob- serve that, given t X and y , ∈ ∈Y (λ K y)(z)=(K y)(x−1z)= K(x−1z, t)y = K(z, xt)y =(K y)(z) x, z X, x t t xt ∈ 27 that is, λxKt = Kxt. Moreover λ K y ,λ K y = K y ,K y = K(xt , xt )y ,y h x t1 1 x t2 2iK h xt1 1 xt2 2iK h 2 1 1 2i = K(t , t )y ,y = K y ,K y . h 2 1 1 2i h t1 1 t2 2iK This means that λ leaves the set K y x X,y invariant and its ac- { x | ∈ ∈ Y} tion is unitary. First two claims now follow recalling that Kxy x X,y is total in . To prove (21) observe that { | ∈ ∈ Y} HK ∗ ∗ ∗ ∗ K(x, t)= KxKt = Ke λxλtKe = Ke λx−1tKe for all x, t X. ∈ Notice that, if K is a translation invariant kernel, (22) implies that the elements of K are bounded functions. The following lemma characterizes the translationH invariant kernels that are Mercer or . C0 Lemma 1. Let Ke : X be a function of completely positive type and let K be the corresponding→ Y translation invariant reproducing kernel. The following conditions are equivalent. (a) The map K is a Mercer kernel. (b) For all y , K ( )y (X; ). ∈Y e · ∈ C Y (c) The representation λ is continuous on . HK Moreover, the map K is a -kernel if and only if K ( )y (X; ) for all C0 e · ∈ C0 Y y . ∈Y Proof. The equivalence between (a) and (b) as well as the statement about −1 0-kernel is a consequence of Proposition 2, observing that (Kxy)(t)= Ke(x t)y Cand (22) holds. Assume that K is a Mercer kernel. Since λ is a unitary representation and the set K y t X,y is total in , it is enough to check that for { t | ∈ ∈ Y} HK any t X and y the function x λxKty is continuous at the identity. Indeed,∈ observe that∈Y 7→ λ K y K y 2 = K y K y 2 k x t − t kK k xt − t kK = (K(xt, xt) K(t, xt) K(xt, t)+ K(t, t)) y,y , h − − i = 2K (e) K (t−1x−1t) K (t−1xt) y,y e − e − e which is continuous at the identity by assumption on Ke. Conversely, if λ is continuous, (21) gives that ∗ Ke(x)y = K(x, e)y = Ke λx−1 Key, so that K ( )y is continuous. e · 28 The following theorem characterizes the translation invariant reproducing kernels.

Proposition 12. Let π be a unitary representation of X acting on a separable Hilbert space and A : a bounded operator. Define H H→Y

W : (X; ) , (W v)(x)= Aπ −1 v . (23) H→F Y x W is a unitary map from ker W ⊥ onto the reproducing kernel Hilbert space with translation invariant kernel HK ∗ K(x, t)= Aπ −1 A x, t X. (24) x t ∈ Moreover W intertwines the representations π and λ. Finally W is unitary if and only if the only π-invariant closed subspace of ker A is the null space.

∗ ∗ Proof. Define γ : as γ = π A , so that (W v)(x) = γ v = Aπ −1 v. x Y → H x x x x The claim is now consequence of Proposition 1, up the last statement. The fact that W intertwines π with λ is trivial. Finally, by Proposition 1, W is unitary if and only if is injective. By definition

ker W = v π v ker A x X . { ∈H| x ∈ ∀ ∈ } Hence ker W is a closed subspace of ker A invariant with respect to π. Con- versely any π-invariant closed subspace of ker A is contained in ker W . Proposition 11 and 12 show that any translation invariant kernel is of ∗ the form K(x, t) = Aπx−1tA for some unitary representation π acting on a Hilbert space and a bounded operator A : . In particular, if π is a continuous representation,H then K is a MercerH→Y kernel and for any Mercer kernel π can be assumed to be continuous and separable. Moreover, the H reproducing kernel Hilbert space K is embedded in by the feature oper- ator W defined by (23). ObserveH that if the representationH π is irreducible or if A is injective, then W is unitary. C If = , the operator A is of the form Av = v,w H for some w , so thatY (W v)(x)= v, π w . This operator is wellh knowi in harmonic∈ analysis H h x iH as wavelet operator [17].

Remark 2. Notice that any translation invariant kernel K is the sum of translation invariant kernels associated with cyclic representations. Indeed, let π be a unitary representation defining K by means of (24). Since any unitary representation is the direct sum of a family of cyclic representations, then = where each is a closed π-invariant subspace and the H ⊕i∈I Hi Hi

29 action of π on i is cyclic. Denote by Pi the orthogonal projection on i, then H H ∗ i K(x, t)= APiπx−1tPiA = K (x, t), i∈I i∈I X X where the series converges in the strong operator topology and the reproduc- i i i ∗ i ing kernels K are K (x, t)= Aiπx−1tAi where π and Ai are the restrictions of π and A to i, respectively. Proposition 5 implies that K = i∈I Ki . For scalar kernels,H we can always assume that π is cyclic itself.H Indeed,H the P wavelet operator is (W v)(x) = v, πxw H for some w , so that the as- sociated kernel K is determinedh only byi the cyclic subrepresentation∈ H of π containing w.

5.1 Abelian groups In this section, we specialize the previous discussion to the case in which X is an abelian group. With this assumption, we can give a more explicit construction of translation invariant Mercer kernels, which is related to a generalization of Bochner theorem for scalar functions of positive type, [2, 15]. We denote the product in X additively and the identity by 0, since the main example is Rd. We let Xˆ be the dual group of X and we denote by dx the Haar measure on X. Now, we briefly recall the definition of Fourier transform, see for example [16]. If φ L1(X, dx; ), its Fourier transform (φ) : Xˆ is given by ∈ Y F →Y (φ)(χ)= χ(x) φ(x)dx. F ZX We denote by dχ the Haar measure on Xˆ normalized so that extends to a unitary operator from L2(X, dx; ) onto L2(X,ˆ dχ; ). If µFis a positive measure on X and ϕ L1(X,µ; ),Y let (ϕµ) : Xˆ Y be given by ∈ Y F →Y (ϕµ)(χ)= χ(x)ϕ(x) dµ(x). F ZX If µ is a complex measure4 on X, we denote (µ)= (h µ ) where µ is the of µ and h L1(X, µ ) is theF densityF of µ|with| respect| | to µ . By general properties of∈ Fourier| transform,| (φ) and (µ) are bounded| | F F continuous functions on Xˆ (actually, (φ) 0(X; )). Moreover, (φ)=0 [respectively, (µ) = 0] if and only ifFφ = 0∈ in C L1(X,Y dx; ) [resp., Fµ = 0]. F Y 4That is, a σ-additive map µ : (X) C. B →

30 We recall that a positive operator valued measure (POVM) on Xˆ with values in is a map Q : (Xˆ) ( ) such that Q(Zˆ) 0 for all Y B −→ L Y ≥ Zˆ (Xˆ), and ∈ B Q(Zˆ )= Q( Zˆ ), i ∪i i i X for every denumerable sequence of disjoint Borel sets Zˆi i where the sum converges in the weak operator topology. A positive operato{ }r valued measure Q is a projection valued measure if Q(Zˆ)2 = 1 for all Zˆ (Xˆ). If fˆ : Xˆ C ˆ ∈ B → is a bounded measurable function, Xˆ f(χ)dQ(χ) is the unique bounded operator fˆ(Q) defined by R

ˆ ′ ˆ ′ f(Q)y,y = f(χ)dQy,y′ (χ) y,y , Xˆ ∈Y D E Z ′ where Qy,y′ is the on Xˆ given by Qy,y′ (Zˆ) = Q(Zˆ)y,y for all Borel subsets Zˆ. D E Next theorem shows that there is a one to one correspondence between translation invariant Mercer kernels on X and positive operator valued mea- sures on Xˆ. For scalar kernels this result is Bochner theorem [2]. For vector valued kernels, it is proved in [14, 15] under the weaker assumption that K0 is a function of positive type, namely that

N c c K (x x )y,y 0 (25) i j h 0 i − j i ≥ i,j=1 X for all finite sequences xi i=1...N in X, ci i=1...N in C and y . The fact that conditions (20){ and} (25) are equivalent{ } for abelian groups∈ Y is a consequence of [10, Lemma 3.1]. In the following, assuming (20), we give a proof simpler than the one provided in [14, 15]. Theorem 5. If Q : (Xˆ) ( ) is a positive operator valued measure, then B −→ L Y K(x, t)= χ(t x)dQ(χ) (26) ˆ − ZX is a translation invariant -Mercer kernel on X. Conversely, if K is a trans- lation invariant -MercerY kernel on X, then there exists a unique positive Y operator valued measure Q such that (26) holds. We say that Q in (26) is the positive operator valued measure associated to the translation invariant Mercer kernel K.

31 Proof of Theorem 5. If Q : (Xˆ) ( ) is a positive operator valued measure, by Neumark dilationB theorem−→ L [24]Y there exist a separable Hilbert space , a projection valued measure P : (Xˆ) ( ) and a bounded operatorH A : such that B −→ L H H −→Y Q(Zˆ)= AP (Zˆ)A∗ Zˆ (Xˆ). (27) ∀ ∈ B Let π be the continuous unitary representation of X acting on given by H π(x)= χ(x)dP (χ), (28) ˆ ZX ∗ see [16]. Eq. (26) then becomes K(x, t)= Aπt−xA , so that K is a translation invariant Mercer kernel by Proposition 12 and Lemma 1. Conversely, by Proposition 12 and Lemma 1, every translation invariant Mer- ∗ cer kernel is of the form K(x, t) = Aπt−xA for some continuous unitary representation π of X in a separable Hilbert space and some bounded operator A : . By SNAG theorem [16], thereH is then a projection H −→ Y valued measure P : (Xˆ) ( ) such that (28) holds and (26) follows defining the POVM QB as in−→ (27). L H Finally, uniqueness of Q follows from

′ K0(x)y,y = χ(x)dQy,y′ (χ)= (Qy,y′ )(x) h i ˆ F ZX by injectivity of Fourier transform of measures on Xˆ. The next proposition is a useful tool to construct translation invariant Mercer kernels. Theorem 6. Let νˆ be a measure on Xˆ and A : L2(X,ˆ νˆ; ) be a bounded operator. For all y,y′ let Y →Y ∈Y K(x, t)y,y′ = χ(t x) (A∗y)(χ), (A∗y′)(χ) dˆν(χ). (29) h i ˆ − h i ZX Then K is a translation invariant Mercer kernel and the corresponding re- producing kernel Hilbert space is embedded in L2(X,ˆ νˆ; ) by means of the Y feature operator W : L2(X,ˆ νˆ; ) Y → HK (W fˆ)(x)= Afˆx where fˆx(χ)= χ(x)fˆ(χ) (30)

(W fˆ)(x),y = χ(x) fˆ(χ), (A∗y)(χ) dˆν(χ). Xˆ D E Z D E Conversely, any translation invariant Mercer kernel is of the above form for some positive measure νˆ and bounded operator A : L2(X,ˆ νˆ; ) . Y →Y 32 Proof. Ifν ˆ is a measure on Xˆ and A : L2(X,ˆ νˆ; ) is a bounded operator, then Y →Y

Q(Zˆ)y,y′ = (A∗y)(χ), (A∗y′)(χ) dˆν(χ) Zˆ (Xˆ), y,y′ Zˆ h i ∀ ∈ B ∈Y D E Z defines a positive operator valued measure Q : (Xˆ) ( ), since Q(Zˆ)= AP (Zˆ)A∗ where P (Zˆ) is the multiplication byB the characteristic−→ L Y function of Zˆ. The kernel K given in (29) is then the translation invariant Mercer kernel associated to Q by (26). To prove (30), set γ : L2(X,ˆ νˆ; ) (γ y)(χ)= χ(x)(A∗y)(χ), x Y −→ Y x ∗ so that K(x, t)= γxγt and

∗ ˆ ˆ ˆ ∗ γxf,y = f,γxy = f(χ), χ(x)(A y)(χ) dˆν(χ) 2 Xˆ D E D E Z D E = χ(x) fˆ(χ), (A∗y)(χ) dˆν(χ)= Afˆx,y Xˆ Z D E D E for all fˆ L2(X,ˆ νˆ; ). ∈ Y Conversely, assume that K is a translation invariant Mercer kernel. We first consider the case that is infinite-dimensional. Propositions 11 and 12 show Y ∗ that K is of the form K(x, t)= Aπt−xA for some unitary continuous repre- sentati¡on π acting on a separable Hilbert space and a bounded operator A : . H H→Y A basic result of commutative harmonic analysis (see [16]) ensures that, for each n N := N , there exist a complex separable Hilbert space ∈ ∗ ∪ {∞} Yn of dimension n, and a measurable subset Xˆn of Xˆ endowed with a positive measureν ˆn such that the Xˆn are disjoint and cover Xˆ. Without loss of gen- erality, we can assume thatν ˆ (Xˆ ) 2−n andν ˆ (Xˆ ) 1. Moreover there n n ≤ ∞ ∞ ≤ exists a unitary operator U : L2(Xˆ , νˆ , ) such that H→ n n n Yn (Uπ U ∗fˆ )(χ)= χ(x)fˆL(χ) fˆ L2(Xˆ , νˆ , ) . x n n n ∈ n n Yn For each n N∗, let Jn : n be a fixed isometry, which always exists since is infinite∈ dimensional,Y → and Y consider the Hilbert space L2(X,ˆ νˆ; ), Y Y whereν ˆ = n νˆn, which is a bounded measure by assumption onν ˆn. Define the isometry V : L2(X,ˆ νˆ; ) as P H→ Y (Vu)(χ)= J (Uv)(χ) χ Xˆ . n ∈ n A simple calculation shows that

∗ πx = V λˆxV

33 2 where (λˆxfˆ)(χ) = χ(x)fˆ(χ) is the diagonal representation on L (X,ˆ νˆ; ). Now Y ∗ ∗ˆ ∗ K(x, t)= Aπt−xA = AV λt−xV A . ∗ Redefining A = AV , (29) is a consequence of the explicit form of λˆx. If is finite dimensional, let (ˆν, B) be the pair associated to K as in Y Proposition 13 below. Eq. (29) follows defining A : L2(X,ˆ ν,ˆ ) Y →Y

1 Af,yˆ = B(χ) 2 fˆ(χ),y dˆν(χ). Xˆ D E Z D E

If = Cm, K(x, t) can be regarded as a m m-matrix and A is uniquely definedY by a family of functions fˆ ,..., fˆ L×2(X,ˆ νˆ; ) through A∗e = fˆ. 1 m ∈ Y i i Hence, (29) becomes

K(t x)ij = χ(t x) fj(χ), fi(χ) dˆν(χ) i, j =1,...,m. (31) − ˆ − h i ZX As an application, we give the following example that generalizes the one given in [5]. Example 10. Let X = Rd, regarded as vector abelian group, and = Cm. d i2πx·p Y The dual group is isomorphic to R by means of χp(x)= e . Let νˆ = dp be the on Rd and

2 1 −σ2 |p| fˆ(p)= e i 2 v v , σ > 0, i (2π)d/4 i i ∈Y i then the translation invariant Mercer kernel given by (31) is

i2π(t−x)·p K(t x)ij = e fj(p), fi(p) dp − Rd h i Z 2 −2π2 |x−t| 1 σ2 σ2 i + j = 2 2 d/2 e vj, vi . (σi + σj ) h i

The example in [5] corresponds to the choice vi = vj and σi = σj for any i, j =1,...,m. Theorems 5 and 6 give two different characterizations of a translation invariant kernel K, but the POVM Q defining K through (26) is always unique, whereas there are many pairs (ˆν, A) defining the same K by (29). These two descriptions are related observing that, given a pair (ˆν, A), the ∗ ∗ ′ scalar bounded measure Q ′ has density (A y)(χ), (A y )(χ) with respect y,y h i 34 ′ toν ˆ for any y,y . On the other hand, given the POVM Q, letν ˆQ be the bounded positive∈Y measure defined by

νˆ (Zˆ)= 2−n y −2n Q(Zˆ)y ,y Zˆ (Xˆ) (32) Q k nk n n ∀ ∈ B n X D E where y N is a dense sequence in . Clearly, given Zˆ (Xˆ),ν ˆ (Zˆ)=0 { n}n∈ Y ∈ B Q if and only if Q(Zˆ)=0,andν ˆQ is uniquely defined by Q up to an equivalence. Moreover, by Neumark dilation theorem, see (27), there exists an operator A : L2(X,ˆ νˆ ; ) such that the pair (ˆν , A ) gives the kernel K Q Q Y → Y Q Q associated with Q. We notice that in general it is not true that the POVM Q has an operator valued density. We recall that Q has operator density if there exists a map B : Xˆ ( ) and a positive measureν ˆ such that B( )y,y′ L1(X,ˆ νˆ) for all y,y−→′ L Y and h · i ∈ ∈Y ′ B(χ)y,y dˆν(χ)= Qy,y′ (Zˆ) Zˆ (Xˆ). (33) ˆ h i ∀ ∈ B ZZ The following proposition will characterize the kernels having a POVM with an operator density. To prove the result, we need the following technical lemma. Lemma 2. Let νˆ be a positive measure on Xˆ and B : Xˆ ( ) such that −→ L Y B( )y,y′ L1(X,ˆ νˆ) for all y,y′ . Then, the sesquilinear form h · i ∈ ∈Y L1(X,ˆ νˆ), (y,y′) B( )y,y′ (34) Y×Y→ 7→ h · i is continuous. Proof. For fixed y [resp. y′ ] the map y′ B( )y,y′ [resp. y B( )y,y′ ] is continuous∈ Y from into∈ Y L1(X,ˆ νˆ) by the7→ closed h · graphi theorem,7→ h · i Y i.e. the application defined in (34) is separately continuous in y and y′. So, the closed graph theorem again assures the joint continuity. Proposition 13. Let νˆ be a positive measure on Xˆ and B : Xˆ ( ) such that B( )y,y′ L1(X,ˆ νˆ) for all y,y′ and B(χ) 0 for−→νˆ-almost L Y h · i ∈ ∈Y ≥ all χ. Then K(x, t)= χ(t x)B(χ) dˆν(χ), (35) ˆ − ZX is a translation invariant Mercer kernel, and the space K is embedded in L2(X,ˆ νˆ; ) by means of the feature operator H Y 1 (W fˆ)(x)= χ(x)B(χ) 2 fˆ(χ)dˆν(χ), (36) ˆ ZX 35 where both the above integrals converge in the weak sense. If is finite dimensional or X is compact, any translation invariant kernel Y is of the above form for some pair (ˆν, B). If = C, one can always assume that B = 1 and νˆ is a bounded positive measure.Y Proof. Letν ˆ and B as in the assumptions. Given a Borel subset Zˆ of Xˆ define Q(Zˆ) as the unique bounded operator satisfying

Q(Zˆ)y,y′ = B(χ)y,y′ dˆν(χ). Zˆ h i D E Z The fact that Q(Zˆ) is a bounded operator follows from Lemma 2 and from the continuity of the map L1(X,ˆ νˆ) φ φ(χ)dˆν(χ) C. Clearly, ∋ 7→ Zˆ ∈ Q(Zˆ) is a positive operator and monotone convergence theorem implies that R Zˆ Q(Zˆ) is a POVM on Xˆ. By construction K(x, t)= χ(t x)dQ(χ), 7→ Xˆ − so K is a translation invariant Mercer kernel by Theorem 5. Setting R γ : L2(X,ˆ νˆ; ) (γ y)(χ)= χ(x)B(χ)1/2y, x Y −→ Y x ∗ we see that K(x, t)= γxγt and

∗ ˆ ˆ ˆ 1/2 γxf,y = f,γxy = f(χ), χ(x)B(χ) y dˆν(χ) 2 Xˆ D E D E Z D E = χ(x) B(χ)1/2fˆ(χ),y dˆν(χ) Xˆ Z D E for all fˆ L2(X,ˆ νˆ; ), from which (36) follows. Assume now∈ that Yis finite dimensional or X is compact and K is a trans- Y lation invariant Mercer kernel. Theorem 5 ensures that there exists a POVM Q on Xˆ taking value in such that K(x, t)= χ(t x)dQ(χ). If X is Y Xˆ − compact, Xˆ is discrete. Letν ˆ be the counting measure and B(χ)= Q( χ ) R { } for all χ Xˆ, then (ˆν, B) satisfies the required properties. ∈ If is finite dimensional, chooseν ˆQ as in (32). It follows that for any Y′ 1 y,y , the complex measure Q ′ has density b ′ L (X,ˆ νˆ ) with ∈ Y y,y y,y ∈ Q respect toν ˆ . In particular, b (χ) 0 forν ˆ -almost all χ Xˆ. Let Q y,y ≥ Q ∈ y ,...,y be a basis of and by linearity extend b L1(X,ˆ νˆ ) to a map 1 N Y yi,yj ∈ Q B : Xˆ ( ), which clearly satisfies the required properties. →L Y If = C, the claim is clear. Y If = C, Proposition 13 is already given in [22]. Y We end by showing a sufficient condition ensuring that a translation in- variant Mercer kernel is of the form given in Proposition 13.

36 Proposition 14. Let K be a translation invariant Mercer kernel. Suppose that K ( )y,y′ L1(X, dx) for all y,y′ . Let h 0 · i ∈ ∈Y B(χ)y,y′ := χ(x) K (x)y,y′ dx y,y′ . (37) h i h 0 i ∀ ∈Y ZX Then

(i) B(χ) is a bounded nonnegative operator for all χ Xˆ; ∈ (ii) B( )y,y′ L1(X, dx) for all y,y′ ; h · i ∈ ∈Y (iii) for all x, t X, ∈ K(x, t)= χ(t x)B(χ)dχ, (38) ˆ − ZX where the integral converges in the weak sense.

Proof. The operator B(χ) defined in (37) is bounded as a consequence of Lemma 2 (applied to K ) and of the continuity of the map L1(X, dx) φ 0 ∋ 7→ (φ)(χ) C. F ∈ Since K0( )y,y is a function of positive type, by Fourier inversion theorem B( )y,yh · L1(iX,ˆ dχ), and h · i ∈

K0(x)y,y = χ(x) B(χ)y,y dχ, h i ˆ h i ZX which is (38).

5.2 Universality In this section we study the universality problem for translation invariant ker- nels on an abelian group in terms of the characterization given by Theorem 5 and Proposition 13. The assumptions and notations are as in Section 5.1. To state the following result, we recall that the support of a POVM Q is the complement of the largest open subset U such that Q(U)=0.

Proposition 15. Let K be a translation invariant Mercer kernel, and Q its associated positive operator valued measure. If the RKHS is dense in HK L2(X,µ; ) for any probability measure µ, then supp(Q)= Xˆ. Y Proof. Suppose there is an open set U Xˆ such that Q(U) = 0. Let χ U, so that χ U −1 is a neighborhood⊂ of the identity element of Xˆ. Let 0 ∈ 0

37 5 −1 µ be a probability measure on X such that supp (µ) χ0U and set ϕ(x)= χ (x)y with y 0 . Then (26) gives F ⊂ 0 ∈Y\{ }

Lµϕ,ϕ = χ(t x)χ0(x)χ0(t)dQy,y(χ)dµ(x)dµ(t) h i ˆ − ZX ZX ZX −1 2 = (µ)(χ0χ ) dQy,y(χ)=0. ˆ F ZX

This shows that Lµ is not injective, i.e. K is not universal. We now characterize the universality of the kernels defined in terms of the pair (ˆν, B) by means of (35).

Proposition 16. Given a positive measure νˆ on Xˆ and B : Xˆ ( ) −→ L Y such that B( )y,y′ L1(X,ˆ νˆ) for all y,y′ and B(χ) 0 for νˆ-almost all χ, let Kh be· the translationi ∈ invariant Mercer∈Y kernel given≥ by (35).

2 (i) If K is dense in L (X,µ; ) for any probability measure µ, then both suppH ν ˆ = Xˆ and supp B =YXˆ .

(ii) If suppν ˆ = Xˆ and B(χ) is injective for νˆ-almost all χ Xˆ, then ∈ HK is dense in L2(X,µ; ) for any probability measure µ. In the case X is compactY also the converse holds true.

(iii) If = C and B = 1, is dense in L2(X,µ; ) for any probability Y HK Y measure µ if and only if suppν ˆ = Xˆ.

Proof. Item (i) follows from Proposition 15 and (33). Let now µ be a probability measure on X. Using (35), we have

L ϕ,ϕ = χ(t x) B(χ)ϕ(t),ϕ(x) dˆν(χ)dµ(x)dµ(t) h µ i − h i ZZZ = B(χ) (ϕµ)(χ−1), (ϕµ)(χ−1) dˆν(χ). (39) ˆ F F ZX

(ii) If B(χ) is injective for almost all χ Xˆ and suppν ˆ = Xˆ, then, by the above equation, positivity of B(χ∈) and the injectivity of Fourier 2 transform, Lµϕ =0 if ϕ = 0 in L (X,µ; ). Therefore, K is dense in L2(X,µ; ) for any6 probability6 measure Yµ. H Suppose YX is compact, so that Xˆ is discrete. If is dense in HK 5For example, if V is a compact symmetric neighborood of the identity of Xˆ such 2 −1 that V χ0U , let h = 1V 1V , so that (up to a constant) the measure dµ(x) = 2 −1 ⊂ −1 ∗ (h)(x)dx = (1V )(x) dx has the required property. F F

38 L2(X,µ; ) for any probability measure µ, suppν ˆ = Xˆ by item (i). If χ XˆY and y ker B(χ ), choose dµ(x) = dx and ϕ(x) = χ (x)y, 0 ∈ ∈ 0 0 so that (ϕµ)(χ)= δχ,χ−1 y. We thus have F 0 L ϕ, ϕ = B(χ )y,y νˆ(χ )=0. h µ i h 0 i 0 Since Lµ is injective, this implies ϕ = 0, i.e. y = 0. (iii) Since B = 1, the ‘if’ part is clear from item (ii). The converse follows by item (i).

By inspecting the proofs of Propositions 15 and 16, one can easily replace L2(X,µ; ) with any Lp(X,µ; ), 1 p< , in the statements. The same Y Y ≤ ∞ holds for Corollary 6 below.

Remark 3. If the translation invariant kernel K is 0, then Propositions 15 and 16 characterize universality of K. C Remark 4. If X = Rd, = C, and suppν ˆ is a subset of Xˆ = Rd such that every entire functionY on Cd vanishing on it is identically zero, then K is κ-universal (see [22, Proposition 14]). This follows by (39), taking into account that, for compactly supported µ, the Fourier transform of ϕµ can be extended to an entire function defined on Cd. In particular, if d = 1 a sufficient condition for κ-universality is that suppν ˆ has an accumulation point. Based on the above remark, we give another example of compact-universal kernel, which is not universal, see also Example 8. Example 11. Let K : R R C be the -kernel × → C0 1 sin 2π(t x) K (x, t)= e2πi(t−x)pdp = − , π(t x) Z−1 − with νˆ the restriction of the Lebesgue measure to [ 1, 1]. Since the support of νˆ admits an accumulation point, K is compact-universal− by the last remark. On the other hand since suppν ˆ is not the whole R, K is not universal by Proposition 16. We now exhibit a particular case in which Proposition 16 applies. Corollary 6. Let K be a translation invariant Mercer kernel such that ′ 1 ′ K0( )y,y L (X, dx) for all y,y . Let B : Xˆ ( ) be as in h(37).· If Bi(χ ∈) is injective for dχ-almost∈ Y all χ, then the−→ reproducing L Y kernel Hilbert space is dense in L2(X,µ; ) for any probability measure µ. HK Y Proof. Since the support of the Haar measure dχ is Xˆ, the claim is then a consequence of Proposition 16.

39 6 Examples of universal kernels

In this section we present various examples of universal kernels, some of them has been already introduced in Section 3. We start with the gaussian kernel, which is a well known example of uni- versal kernel. The first proof about universality is given [30] with a different technique and in [22] by means of the Fourier transform. In both paper only compact-universality is taken into account.

Example 12. Let X be a closed subset of Rd, = C and Y 2 − kx−tk κ(x, t)= e 2σ2 x, t X, ∈ where σ> 0. Then K is a -universal kernel. C0 Proof. Assume first that X = Rd, regarded as abelian group, then κ is trans- lation invariant kernel with κ in (Rd) L1(Rd, dx). According to (37) 0 C0 ∩ 2 2 2 B(p)= (2πσ2)d e−2π σ kpk

p d i2πp·x where the dual group is identified with R by means of χp(x)= e . Since B(p) > 0 for all p Rd, universality is a consequence of Corollary 6. If X is an arbitrary∈ closed subset of Rd it is enough to apply Corollary 3. Next example is well known in (see, for example, [3]).

Example 13. Let X = R, = C and let Y κ(x, t)= e−π|x−t| .

1 Then the kernel κ is a 0-universal kernel and κ = W (R), the of measurable complexC functions f on R withH finite norm

2 2 ′ 2 f W 1 = f(x) + f (x) dx, k k X | | | | Z h i where f ′ is the weak derivative.

2 Proof. The same reasoning as above, observing that B(p) = π+4πp2 > 0 for all p R. ∈ Next example characterizes universal kernels of the form K = κB – see Example 5.

40 Example 14. Let κ be a 0-scalar reproducing kernel and B a positive op- erator. The kernel K = κBC is universal if and only if κ is universal and B is injective.

Proof. We have to show that, given a probability measure µ, κB is dense 2 H ⊥ in L (X,µ; ). The space κB is unitarily equivalent to κ ker B by Y H 1 H ⊗ means of W (ϕ y)(x) = ϕ(x)B 2 y, see Example 5. Hence, it is enough to ⊗ 1 2 prove that κ B 2 is dense in L (X,µ) . This is the case if and only H ⊗ 2 Y 1 ⊗Y if κ is dense in L (X,µ) and B 2 has dense range, and this last condition is equivalentH to the fact that B is injective since B is a positive operator.

The same result holds replacing 0-kernel with Mercer kernel and univer- sality with compact-universality. C

Example 15. Let κ : X X C and κ′ : X′ X′ C be two scalar × → × → C0 reproducing kernels on X and X′, respectively. Let I′ be the identity operator on ′ . Hκ ′ (i) The ′ -kernel K = κI if universal if and only if κ is universal. Hκ ′ ′ 2 ′ ′ (ii) Fixed a probability measure µ on X , the L (X ,µ )-kernel K = κLµ′ 2 ′ ′ is universal if and only if κ is universal and ′ is dense in L (X ,µ ). Hκ b (iii) The scalar kernel κ κ′ is universal if both κ and κ′ are universal. × Proof. Items (i) and (ii) follow immediately from Example 14 and Propo- sition 10. Item (iii) is a consequence of Proposition 9 and the density of (X) (X′) in (X X′). C0 ⊗ C0 C0 × The following class of examples is considered in [5].

Example 16. Let X be a locally compact second countable abelian group. i N i N Let B i=1 be a finite set of positive operators on and κ0 i=1 be a finite { } 1Y { } set of scalar functions of positive type in 0(X) L (X, dx). The translation invariant kernel K C ∩ N K(x, t)= κi (x t)Bi 0 − i=1 X is universal provided that kerBi = 0 and, for each i = 1,...N, there is ∩i { } an open dense subset Zˆi Xˆ such that (κi ) > 0 on Zˆi. ⊂ F 0

41 ′ 1 Proof. Clearly, K0( )y,y is in L (X, dx). Moreover, according to (37), for all y and χh Xˆ· i ∈Y ∈ N B(χ)y = (κi )(χ−1)Biy. F 0 i=1 X Each Zi is open and dense, hence Z = Zi is dense in Xˆ. Let χ Zˆ and y such that B(χ)y = 0; then Biy =∩ 0 for all i = 1,...,N, since∈ every ∈ Y Bi is ab positive operator and (κi ) b> 0 onbZˆi, so that by assumption y = 0. F 0 Therefore, K is universal by Corollary 6.

A Vector valued measures

In this appendix we describe the dual of 0(X; ). For = C, it is a well ∗ C Y Y known result that 0(X) can be identified with the Banach space of complex measures on X. ForC arbitrary , a similar result holds by considering the Y space of vector measures. If X is compact, this result is due to [28] and we slightly extend it to X being only locally compact. The proof we give is simpler than the original one also for X compact. Moreover, by using a version of Radon-Nikodym theorem for vector valued measures, it is possible to describe the dual of 0(X; ) in a simpler way. Indeed, the following result holds. C Y

∗ Theorem 7. Let T 0(X; ) . There exists a unique probability measure µ on X and a unique∈ function C Y h L∞(X,µ; ) such that ∈ Y T (f)= f(x), h(x) dµ(x) f (X; ) (40) h i ∈ C0 Y ZX with h(x) = T for µ-almost all x X. k k k k ∈ Proof. It follows combining Theorems 8 and 9 below. Observe that, given µ and h as in the statement of the theorem, if we define T by (40), then T 0(X; ). Hence (40) completely characterizes the dual of (X; ) in terms∈ C of pairsY (µ, h). C0 Y To prove the theorem, we recall some basic facts from the theory of vector valued measures (see [11, 19]). If A (X), we denote by Π(A) the family of partitions of A into finite or denumerable∈ B disjoint Borel subsets.

Definition 3. A vector measure on X with values in is a mapping M : (X) such that Y B −→ Y 42 (i) sup M(A ) < ; k i k ∞ {Ai}∈Π(X) i X (ii) for all A (X) and A Π(A) ∈ B { i} ∈

M(A)= M(Ai) i X where the sum converges absolutely by item (i). If M is a -valued vector measure on X, for all A (X) we define Y ∈ B M (A) = sup M(A ) . | | k i k {Ai}∈Π(A) i∈I X Then, M is a bounded positive measure on X, called the total variation of M. | | The integration of a function f L1(X, M ; ) with respect to M is ∈ | | Y n defined as it follows. Let St(X; ) be the space of functions f = 1 v , Y i=1 Ai i with Ai disjoint Borel sets and vi (1A is the characteristic function of the set A). For such f’s, define ∈ Y P n f(x), dM(x) := v , M(A ) . (41) h i h i i i X i=1 Z X Since n M M M vi, (Ai) vi (Ai) (Ai) vi = f 1 , h i ≤ k kk k ≤ | | k k k k i=1 i i X X X the integral (41) extends to a bounded functional on L1(X, M ; ), which is M | | Y denoted again by X f(x), d (x) . By Theorem 4.1 in [19], then there exists h L∞(X, M ; ) suchh that i ∈ | | YR f(x), dM(x) = f(x), h(x) d M (x) f L1(X, M ; ), h i h i | | ∀ ∈ | | Y Z Z and h(x) = 1for M -almost all x. These facts are collected in the following theorem.k k | | Theorem 8 (Radon-Nikodym). If M is a -valued vector measure on X, there exists a unique M -measurable functionYh : X such that h(x) = 1 for M -almost all x| and| −→ Y k k | | f(x), dM(x) = f(x), h(x) d M (x) f L1(X, M ; ). h i h i | | ∀ ∈ | | Y ZX ZX

43 The function h is called the density of M with respect to M . We denote by M(X; ) the space of -valued vector measures| | on X. The Y Y space M(X; ) is a Banach space with respect to the norm Y M = M (X) k k | | (see [11]). If = C, we let M(X)= M(X; C). The next duality theorem is shown in [28]Y for X compact – see also [11].

Theorem 9. If 0(X; ) is endowed with the Banach space topology induced C Y ∗ by the uniform norm, then (X; ) = M(X; ), the duality being given by C0 Y Y f, M = f(x), dM(x) f (X; ), M M(X; ). h i h i ∀ ∈ C0 Y ∈ Y ZX Proof. By Theorem 8, it is clear that, if M M(X; ), then ∈ Y

TM(f)= f(x), dM(x) = f(x), h(x) d M (x) h i h i | | ZX ZX defines a bounded functional TM on (X; ). C0 Y Clearly TM M . To show that TM = M , fix by Lusin theorem k k≤k k k k k k a function g 0(X; ) such that g(x) = h(x) for x X Z, Z being a M ∈ C Y M ∈ \ -measurable set with (Z) < ǫ, and g ∞ h |M|,∞ = 1. For ǫ small |enough,| we then have | | k k ≤k k

M (X) 2ǫ< M (X Z) M (Z) g(x), h(x) d M (x) M (X). | | − | | \ −| | ≤ h i | | ≤| | ZX

M This shows that TM = . k k k k ∗ Suppose now T 0(X; ) . For v , let iv : 0(X) 0(X; ) be the bounded operator∈ C givenY by ∈ Y C −→ C Y

[iv(ϕ)](x)= ϕ(x)v.

Since T i (X)∗, by Riesz theorem there exists a measure µ M(X) v ∈ C0 v ∈ such that

T i (ϕ)= ϕ(x)dµ (x) and T i = µ . v v k vk k vk ZX For all A (X), let M(A) be the vector in such that ∈ B Y v, M(A) = µ (A) h i v

44 (M(A) is well defined, since µv(A) µv = T iv T v ). We now show that, if A | (X)|≤k and Ak k Π(k≤kA), thenkk k ∈ B { i} ∈

M(Ai) T , i k k≤k k X so that item (i) of Definition 3 holds. It is enough to prove it for all finite partitions A . Let v = M(A )/ M(A ) (we set v = 0 whenever { i}i=1...n i i k i k i M(Ai) = 0). We have

M(Ai) = vi, M(Ai) = µv (Ai). i k k i h i i i X X X Set ν = i µvi , which is ν a bounded positive measure, and every µvi has | | (i) density with respect to ν. For all i =1 ...n, fix a sequence ϕj j∈N in c(X) P (i) { } C such that limj ϕj (x)=1Ai (x) for ν-almost all x. Define

n −1 n (k) (i) ψj(x)= 1 ϕj (x) ϕj (x)vi. " ∨ # k=1 i=1 X X Then, ψ (X; ), and ψ (x) 1 for all x. Moreover, j ∈ Cc Y k j k ≤

n −1 1 ϕ(k)(x) ϕ(i)(x) 1 x, i ∨ j j ≤ ∀ " k=1 # X

and n −1 (k) (i) lim 1 ϕj (x) ϕj (x)=1Ai (x) for ν-almost all x. j " ∨ # k=1 X Therefore

M(Ai) T ψj i k k − −1 X (k) (i) = µv (Ai) T iv 1 ϕ ϕ i  i − i  ∨ j j  " k # X  X   −1 

 (k) (i)  1A (x) 1 ϕ (x) ϕ (x) dµv (x) ≤ i  i − ∨ j j  i ZX " k # X  X  j→∞ 0 −→  

45 by dominated convergence theorem. On the other hand, T ψj T ψj ∞ T . It follows that n M(A ) T , as claimed. | |≤k kk k ≤ k k i=1 k i k≤k k We now show that P M(A)= M(Ai) i X (absolutely) for all A (X) and A Π(A). We have just proved that ∈ B { i} ∈ the right hand side is absolutely convergent, and the equality follows by

v, M(A ) = µ (A )= µ (A)= v, M(A) v . i v i v h i ∀ ∈Y * i + i X X Therefore, M is a -valued measure. It remains to show that T = TM. Let h and M be associatedY to M as in Radon-Nikodym theorem. Then, for any Borel| set| A X, we have µ (A) = v, h(x) d M (x), from which it ⊂ v A h i | | follows that µ has density v, h(x) with respect to M . For ϕ (X) and v R c v , we thus have h i | | ∈ C ∈Y

T (ϕv)= ϕ(x)dµ (x)= ϕ(x)v, h(x) d M (x)= TM(ϕv). v h i | | ZX ZX Then, T = TM by density of (X) in (X; ). Cc ⊗Y C0 Y Acknowledgment. This work has been partially supported by the FIRB project RBIN04PARL and by the the EU Integrated Project Health-e-Child IST-2004-027749.

References

[1] N. Aronszajn. Theory of reproducing kernels. Trans. Amer. Math. Soc., 68:337–404, 1950.

[2] S. Bochner. Lectures on Fourier integrals. With an author’s supple- ment on monotonic functions, Stieltjes integrals, and harmonic analysis. Translated by Morris Tenenbaum and Harry Pollard. Annals of Math- ematics Studies, No. 42. Princeton University Press, Princeton, N.J., 1959.

[3] H. Br´ezis. Analyse fonctionnelle : th´eorie et applications. Dunod, Paris, 1983.

[4] A. Caponnetto and E. De Vito. Optimal rates for the regularized least- squares algorithm. Found. Comput. Math., 7(3):331–368, 2007.

46 [5] A. Caponnetto, C. A. Micchelli, M. Pontil, and Y. Ying. Universal kernels for multi-task learning. J. Mach. Learn. Res., 2008 (accepted). Preprint available at http://eprints.pascal-network.org/archive/00003780/.

[6] C. Carmeli, E. De Vito, and A. Toigo. Vector valued reproducing kernel Hilbert spaces of integrable functions and Mercer theorem. Anal. Appl. (Singap.), 4(4):377–408, 2006.

[7] J. B. Conway. A course in functional analysis, volume 96 of Graduate Texts in Mathematics. Springer-Verlag, New York, second edition, 1990.

[8] F. Cucker and S. Smale. On the mathematical foundations of learning. Bull. Amer. Math. Soc. (N.S.), 39(1):1–49 (electronic), 2002.

[9] F. Cucker and D.-X. Zhou. Learning theory: an approximation the- ory viewpoint. Cambridge Monographs on Applied and Computational Mathematics. Cambridge University Press, Cambridge, 2007. With a foreword by Stephen Smale.

[10] E. B. Davies. Quantum theory of open systems. Academic Press [Har- court Brace Jovanovich Publishers], London, 1976.

[11] J. Diestel and J. J. Uhl, Jr. Vector measures. American Mathemati- cal Society, Providence, R.I., 1977. With a foreword by B. J. Pettis, Mathematical Surveys, No. 15.

[12] T. Evgeniou, C. A. Micchelli, and M. Pontil. Learning multiple tasks with kernel methods. J. Mach. Learn. Res., 6:615–637 (electronic), 2005.

[13] T. Evgeniou, M. Pontil, and T. Poggio. Regularization networks and support vector machines. Adv. Comput. Math., 13(1):1–50, 2000.

[14] P. L. Falb. On a theorem of Bochner. Inst. Hautes Etudes´ Sci. Publ. Math., 36:59–67, 1969.

[15] P. L. Falb and U. Haussmann. Bochner’s theorem in infinite dimensions. Pacific J. Math., 43:601–618, 1972.

[16] G. B. Folland. A course in abstract harmonic analysis. Studies in Ad- vanced Mathematics. CRC Press, Boca Raton, FL, 1995.

[17] H. F¨uhr. Abstract harmonic analysis of continuous wavelet transforms, volume 1863 of Lecture Notes in Mathematics. Springer-Verlag, Berlin, 2005.

47 [18] L. Gy¨orfi, M. Kohler, A. Krzy˙zak, and H. Walk. A distribution-free the- ory of nonparametric regression. Springer Series in Statistics. Springer- Verlag, New York, 2002.

[19] S. Lang. Real and functional analysis, volume 142 of Graduate Texts in Mathematics. Springer-Verlag, New York, third edition, 1993.

[20] Y. Lee, Y. Lin, and G. Wahba. Multicategory support vector machines. In Proceedings of the 33rd Symposium on the Interface, 2001.

[21] C. A. Micchelli and M. Pontil. On learning vector-valued functions. Neural Comput., 17(1):177–204, 2005.

[22] C. A. Micchelli, Y. Xu, and H. Zhang. Universal kernels. J. Mach. Learn. Res., 7:2651–2667, 2006.

[23] G. Pedrick. Theory of reproducing kernels for hilbert spaces of vector valued functions. Technical report, Kansas Univ Lawrence, 1957.

[24] F. Riesz and B. Sz.-Nagy. Functional analysis. Dover Books on Ad- vanced Mathematics. Dover Publications Inc., New York, 1990. Trans- lated from the second French edition by Leo F. Boron, Reprint of the 1955 original.

[25] S. Saitoh. Theory of reproducing kernels and its applications, volume 189 of Pitman Research Notes in Mathematics Series. Longman Scientific & Technical, Harlow, 1988.

[26] B. Schoelkopf and J. Smola. Learning with Kernels. The MIT Press, Cambridge, MA, 2002.

[27] L. Schwartz. Sous-espaces hilbertiens d’espaces vectoriels topologiques et noyaux associ´es (noyaux reproduisants). J. Analyse Math., 13:115– 256, 1964.

[28] I. Singer. Linear functionals on the space of continuous mappings of a compact Hausdorff space into a Banach space. Rev. Math. Pures Appl., 2:301–315, 1957.

[29] S. Smale and D.-X. Zhou. Shannon sampling. II. Connections to learning theory. Appl. Comput. Harmon. Anal., 19(3):285–302, 2005.

[30] I. Steinwart. On the influence of the kernel on the consistency of support vector machines. Journal of Machine Learning Research, 2:67–93, Nov. 2001.

48 [31] H.-W. Sun and D.-X. Zhou. Reproducing kernel Hilbert spaces associ- ated with analytic translation-invariant Mercer kernels. J. Fourier Anal. Appl., 14(1):89–101, 2008.

[32] D. X. Zhou. Density problem and approximation error in learning theory. Technical report, City University of Hong Kong, 2003.

49