Max-Planck-Institut fur¨ Mathematik in den Naturwissenschaften Leipzig

On the Representation of Symmetric and Antisymmetric Tensors

(revised version: April 2017) by

Wolfgang Hackbusch

Preprint no.: 72 2016

On the Representation of Symmetric and Antisymmetric Tensors

Wolfgang Hackbusch Max-Planck-Institut Mathematik in den Naturwissenschaften Inselstr. 22–26, D-04103 Leipzig, Germany [email protected]

Abstract Various tensor formats are used for the data-sparse representation of large-scale tensors. Here we investigate how symmetric or antiymmetric tensors can be represented. The analysis leads to several open questions. Mathematics Subject Classi…cation: 15A69, 65F99 Keywords: tensor representation, symmetric tensors, antisymmetric tensors, hierarchical tensor format

1 Introduction

We consider tensor spaces of huge dimension exceeding the capacity of computers. Therefore the numerical treatment of such tensors requires a special representation technique which characterises the tensor by data of moderate size. These representations (or formats) should also support operations with tensors. Examples of operations are the addition, the scalar product, the componentwise product (Hadamard product), and the matrix-vector multiplication. In the latter case, the ‘matrix’belongs to the tensor space of Kronecker matrices, while the ‘vector’is a usual tensor. In certain applications the subspaces of symmetric or antisymmetric tensors are of interest. For instance, fermionic states in quantum chemistry require antisymmetry, whereas bosonic systems are described by symmetric tensors. The appropriate representation of (anti)symmetric tensors is seldom discussed in the literature. Of course, all formats are able to represent these tensors since they are particular examples of general tensors. However, the special (anti)symmetric format should exclusively produce (anti)symmetric tensors. For instance, the truncation procedure must preserve the symmetry properties. The formats in use are the r-term format (also called the canonical format), the subspace (or Tucker) format, and the hierarchical representation including the TT format. In the general case, the last format has turned out to be very e¢ cient and ‡exible. We discuss all formats concerning application to (anti)symmetric tensors. The r-term format is seemingly the simplest one, but has several numerical disadvantages. In §2 we discuss two di¤erent approaches to representing (anti)symmetric tensors. However, they inherit the mentioned disadvantages. As explained in §3, the subspace (or Tucker) format is not helpful. The main part of the paper discusses the question how the TT format can be adapted to the symmetry requirements. The analysis leads to unexpected di¢ culties. In contrast to the general case, the subspaces Uj involved in the TT format (see §4.3) have to satisfy conditions which are not easy to check. We can distinguish the following two di¤erent situations. In the …rst case we want to construct the TT format with subspaces Uj not knowing the tensor v to be represented in advance. For instance we change Uj to obtain a variation of v; or the dimension of Uj is reduced to obtain a truncation. In these examples, v is obtained as a result on the chosen Uj: It turns out that the choice of Uj is delicate. If Uj is too small, no nontrivial tensors can be represented. On the other hand, Uj may contain a useless nontrivial part, i.e., Uj may be larger than necessary. The algebraic characterisation of the appropriate Uj is rather involved. In the second case, we start from v and know the minimal subspaces Umin(v) (cf. (1.9)). Then Umin(v) j j  Uj is a su¢ cient condition. However, as soon as we want to truncate the tensor v; its result v0 must be determined from modi…ed subspaces Uj0 so that we return to the di¢ culties of the …rst case.

1 In Section 8 we describe the combination of the TT format and the ANOVA technique for symmetric tensors. This leads to a favourable method as long as the ANOVA degree is moderate. Quite another approach for antisymmetric tensors is the so-called ‘second quantisation’(cf. Legeza et al. [12, §2.3]) which does not …t into the following schemes.

1.1 Tensor Notation 1.1.1 Tensors Spaces

In the general case, vector spaces Vj (1 j d) are given which determine the algebraic tensor space d   V := j=1 Vj: The common underlying …eld of the vector spaces Vj is either R or C: In the following we write K for either of the …elds. In the particular case of N

Vj = V for all 1 j d (1.1)   we write V := dV: Set D := 1; : : : ; d (1.2) f g and consider any nonempty subset D: We set  V := Vj : (1.3) j 2 O Note that V = VD is isomorphic to V VD : n We assume that V is a pre-Hilbert space with the scalar product ; : Then V and each V is de…ned as a pre-Hilbert space with the induced scalar product uniquely de…nedh

i by (cf. [9, Lemma 4.124])

v(j); w(j) = v(j); w(j) : (1.4) j j j 2 2 2 DO O E Y D E 1.1.2 Functionals

Let ' V 0 be a linear functional. The same symbol ' is used for the linear map ' : V VD de…ned by 2 ! n d (j) (j) (j) ' v = ' v v (1.5) j=1 j j D 2 2 n O  O  O (it is su¢ cient to de…ne a linear map by its action on elementary tensors, cf. [9, Remark 3.55]). In the (k) d d 1 case of (1.1) and ' V 0 we introduce the following notation. The linear mapping ' : V V is de…ned by 2 ! d '(k) v(j) = ' v(k) v(j): (1.6) j=1 j=k 6 O    O 1.1.3 Permutations and (Anti)symmetric Tensor Spaces

A permutation  Pd is a bijection of D onto itself. For ;  D, the permutation  is the transposition 2 2 d swapping the positions  and : If  = ;  is the identity id: Let V = V . Then the symbol of the permutation  is also used for the linear map  : V V de…ned by ! d d 1  v(j) = v( (j)): j=1 j=1 O  O

Each permutation  is a (possibly empty) product of transpositions:  = 11 22 ::: kk with    k i = i (1 i k). The number k determines the parity 1 of the permutation: sign() = ( 1) : 6    A tensor v dV is called symmetric if (v) = v for all permutations, and antisymmetric if (v) = sign()v:2This de…nes the (anti)symmetric tensor space:

d Vsym := v V : (v) = v ; (1.7a) 2 d Vanti := v V : (v) = sign()v : (1.7b)  2  2 (d) (d) If the parameter d should be emphasised, we also write Vsym and Vanti. Correspondingly, if U V is a d (d) (d)  (d) subspace, the (anti)symmetric tensors in U are denoted by Usym, resp. U : Another notation for V anti anti is d V using the exterior product . Besides the well-known applications^ in physics (cf. [2]), symmetric and antisymmetric tensors occur in di¤erentV mathematical …elds. The symmetric tensor space is related to multivariate polynomials which are homogenous of degree d, i.e., p(x) = dp(x): These polynomials are called quantics by Cayley [6]. If n = dim(V ); the symmetric (d) tensor space Vsym is isomorphic to the vector space of n-variate quantics of degree d (cf. [9, §3.5.2]). n The antisymmetric spaces are connected with the Cli¤ord algebra C`d of R , which is isomorphic to the d j n direct sum j=1 R (cf. Lounesto [13, Chap. 22]). L V 1.1.4 Properties

Since all permutations are products of transpositions i;i+1; the next remark follows. d Remark 1.1 A tensor v V is symmetric (resp. antisymmetric) if and only if (v) = i;i+1(v) (resp. 2 (v) = i;i+1(v)) holds for all transpositions with 1 i < d:  Let V := dV . The linear maps

1 1 = d :=  : V V; = d := sign() : V V (1.8) S S d! ! A A d! !  Pd  Pd X2 X2 are projections onto Vsym and Vanti; respectively (For a proof note that =  and = sign()  so that the application of 1 yields = and = ). and areS calledS the symmetrisationA A and d!  Pd alternation, respectively. 2 S SS A AA S A P 1 (D) Remark 1.2 Let 'D VD0 be a functional (no symmetry condition assumed). If v Vsym or (D) n 2 ( n) ( ) 2 v Vanti, then 'D (v) Vsym or 'D (v) Vanti; respectively. 2 n 2 n 2 The following expansion lemma will be used in the following.

k Lemma 1.3 Let u1; : : : ; ur be a basis of the subspace U V: Any tensor v U can be written in the f g  2 form r k 1 v = v u` with v U: [`] [`] 2 `=1 X Let '1;:::;'r U 0 be a dual basis of u1; : : : ; ur ; i.e., 'i(uj) = ij: Then the tensors v are de…ned by f g  f g [`] v[`] = '`(v): A consequence of the last equation and Remark 1.2 is the following.

k k 1 Remark 1.4 If v U is (anti-)symmetric, then so is v U. 2 [`] 2 1.2 Minimal Subspaces

Given a tensor v V = j D Vj and a subset D; the corresponding minimal subspace is de…ned by 2 2  min N U (v) := 'D v : 'D VD0 V (1.9) n n 2 n 2 min n o min (cf. (1.5); [9, §6]). U (v) is the subspace of smallest dimension with the property v U (v) VD . 2 n The dual space VD0 in (1.9) may be replaced by j D Vj0: n min 2 nmin For a subset V0 V we de…ne U (V0) := span U (v): v V0 :  N f 2 g min min min Remark 1.5 Let = $ D be nonempty subsets. Then U (v) = U (U (v)): ; 6  A conclusion from Remark 1.2 is the following statement.

min ( ) min ( ) Conclusion 1.6 If v Vsym [or Vanti], then U (v) Vsym [or U (v) V ]. 2   anti 1 Compare the de…nition (1.5) with interchanged subsets and D : n

3 2 r-Term Format for (Anti)symmetric Tensors

d Let r N0 := N 0 . A tensor v V = V can be represented in the r-term format (or canonical 2 [ f(jg) 2 format) if there are v V for 1 j d and 1  r such that 2     r d (j) v = v : j=1 =1 X O We recall that the smallest possible r in the above representation is called the rank of v and denoted by rank(v): The number r used above is called the representation rank. Since the determination of rank(v) is NP hard (cf. Håstad [10]) we cannot expect that r rank(v) holds with an equal sign.  Two approaches to representing (anti)symmetric tensors by the r-term format are described in §2.1 and §2.2.

2.1 Indirect Representation

A symmetric tensor v Vsym may be represented by a general tensor w V with the property (w) = v; where ( ) is the symmetrisation2 (alternation) de…ned in (1.8). The representation2 of w VS uses the S A r d (j) 2 r-term format: w = i=1 j=1 wi : This approach is proposed by Mohlenkamp, e.g., in [3]. For instance, v = a a b + a b a + b a a Vsym is represented by w = 3a a b: This example indicates that w may be of a much P simpler N form than2 the symmetric tensor v = (w ): However, the cost (storage size) of the representation is onlyS one aspect. Another question concerns the tensor operations. In the following we discuss the addition, the scalar product, and the matrix-vector multiplication. The addition is easy to perform. By linearity of ; the sum of v0 = (w0) and v00 = (w00) is represented S S S by w0 + w00: Similar in the antisymmetric case. The summation within the r-term format does not require computational work, but increases the repres- entation rank r: This leads to the question how to truncate w = w0 + w00 to a smaller rank. It is known that truncation within the r-term format is not an easy task. However, if one succeeds to split w into ^w + w; where ^w has smaller rank and w is small, this leads to a suitable truncation of v = ^v + v with ^v = ( ^w); v = (w); since v w with respect to the Euclidean norm. S S k k  k k The computation of the scalar product v0; v00 of v0; v00 in Vsym or Vanti is more involved. In the h i antisymmetric case, v0; v00 with v0 = (w0); v00 = (w00) and h i A A d (j) d (j) w0 = w0 ; w00 = w00 i0 i00 i0 j=1 i00 j=1 X O X O can be written as the sum v0; v00 = si i with the terms h i i0;i00 0 00

P d d (j) (j) si i := w0 ; w00 : 0 00 A 0 i0 1 A 0 i00 1 * j=1 j=1 + O O @ A @ A The latter product coincides with the determinant

() () 0 00 si0i00 = det wi ; wi h 0 00 i 1 ; d      (cf. Löwdin [14, (35)]). If the respective representation ranks of v0 and v00 are r0 and r00; the cost amounts 3 to (r0r00d ): O While in the antisymmetric case the determinant can be computed in polynomial time, this does not hold for the analogue in the symmetric case. Instead of the determinant one has to compute the permanent.2 As proved by Valiant [17], its computation is NP hard. Hence the computation of the scalar product is only feasible for small d or in special situations.

2 d d d The permanent of A R  is Perm(A) =  P i=1 ai;(i): 2 2 d P Q 4 d Next we consider the multiplication of a symmetric Kronecker matrix A Lsym L(V ) (cf. §9) by d d 2  d (j) a tensor v Vsym=anti V: A is represented by B L(V ) via A = (B) and B =  j=1 B ; 2  2 d (j) S while v = (w) or v = (w) is represented by w = w : The property v V implies the S A  j=1 2 sym=Panti N respective property A v Vsym=anti. Unfortunately, A v is not the (anti)symmetrisation of Bw. Instead one may use (cf. Lemma2 9.1) P N

A v = (A w) = (B v) or A v = (A w) = (B v); resp. S S A A However, this requires that either the symmetric tensor A or the (anti)symmetric tensor v must be con- structed explicitly, which contradicts the intention of the indirect representation. Similarly, the Hadamard product v0 v00 and the convolution v0 ? v00 are hard to perform within this format.

Conclusion 2.1 The indirect representation is suited to antisymmetric tensors if only the addition and the scalar product is required. In the case of symmetric tensors, the computation of the scalar product is restricted to small d:

Let v V be a tensor of rank rv: The indirect representation uses the rw-term representation of 2 sym=anti some w V: The gain is characterised by the ratio rv=rw where rw is the smallest possible rank. According 2 (dim(V ) 1)d+1 to Seigal [16], the generic reduction factor rv=rw is dim(V ) which approaches d for large dim(V ): The proof uses the results of Abo–Vannieuwenhoven [1]. The conclusion for symmetric tensors is negative: For larger d the computation of the permanent causes di¢ culties, while for smaller d the gain is only moderate.

2.2 Direct Symmetric r-Term Representation While the previous approach uses general (nonsymmetric) tensors, we now represent the symmetric tensors by an r-term representation involving only symmetric rank-1 tensors:

r d v = i vi for suitable r N0 and vi V; i K (2.1) 2 2 2 i=1 X 3 (cf. [9, p. 65]). The minimal r in this representation is called the symmetric rank of v Vsym and is 2 denoted by ranksym(v): Details about symmetric tensors and the symmetric tensor rank are described, e.g., by Comon–Golub–Lim–Mourrain [7]. Since the symmetric rank is at least as large as the standard tensor rank, the required r may be large. A di¢ culty of the r-term format is caused by the fact that, in general, the set v dV : rank(v) r is not 2 1 3  3 4 closed. The simplest counterexamples are symmetric tensors of the form lim" 0 (v + "w) v :  ! " Therefore also the subset of the symmetric tensors (2.1) is not closed.
3 Subspace (Tucker) Format

Let v V be the tensor to be represented. The subspace format (Tucker format) uses subspaces Uj Vj 2 d  with the property v Uj: In the (anti)symmetric case one can choose equal subspaces U V (set, 2 j=1  e.g., U = d U ): Let u ; : : : ; u be a basis of U: Then the explicit Tucker representation of v takes the j=1 j N 1 r form f g T r d

v = ci1;:::;id uij (3.1) i ;:::;i =1 j=1 1 Xd O d r with the so-called core tensor c j=1 K : Obviously, v is (anti)symmetric if and only if c is so. Therefore the di¢ culty is shifted into the treatment2 of the core tensor. The representation (3.1) itself does not help to represent (anti)symmetric tensors.N One may construct hybrid formats, using one of the other representations for c: 3 If K = C or if d is odd, the factor i can be avoided since its d-th root can be combined with vi: 4 The described limit x satis…es ranksym(x) rank(x) = d (cf. Buczy´nski–Landsberg [5]), although it is the limit of tensors with symmetric rank 2. 

5 4 Hierarchical Format

In the general case the hierarchical format is a very e¢ cient and ‡exible representation (cf. [9, §§11–12]). Here we brie‡y describe the general setting, the TT variant, and …rst consequences for its application to (anti)symmetric tensors.

4.1 General Case

The recursive partition of the set D = 1; : : : ; d is described by a binary partition tree TD. It is de…ned by f g the following properties: (a) D TD is the root; (b) the singletons 1 ;:::; d are the leaves; (c) if TD 2 f g f g 2 is not a leaf, the sons 0; 00 TD are disjoint sets with = 0 00: 2 d [ The hierarchical representation of a tensor v V = Vj is algebraically characterised by subspaces 2 j=1 U V (V de…ned in (1.3)) for all TD with  2 N v UD; (4.1a) 2 U U U ( 0; 00 sons of ); if is not a leaf. (4.1b)  0 00 Let the dimensions of U be r := dim(U ): Since UD = span(v) is su¢ cient, rD = 1 is the general value.

4.2 Implementation

( ) The subspaces U are described by bases bk : k = 1; : : : ; r . For leaves TD; the basis is stored explicitly. Otherwise, condition (4.1b) ensuresf that g 2 r r 0 00 ( ) ( ;`) ( 0) ( 00) b = c b b ( 0; 00 sons of ): (4.2) ` ij i j i=1 j=1 X X ( ;`) D Therefore it is su¢ cient to store the coe¢ cients matrices (cij )1 i r ;1 j r ; as well as the vector c   0   00 2 rD D (D) K for the …nal representation v = i ci bi (cf. (4.1a)). P 4.3 TT Variant

The TT format is introduced in Oseledets [15] (cf. [9, §12]). It is characterised by a linear tree TD. That means that the non-leaf vertices TD are of the form = 1; : : : ; j with the sons 0 = 1; : : : ; j 1 and 2 f g f g 00 = j : The embedding (4.1b) is U 1;:::;j+1 U 1;:::;j U j : f g f g f g f g d  Below we shall consider the case V = V; i.e., Vj = V is independent of j: Also their subspaces are

independent of j and denoted by U j = U: We abbreviate U 1;:::;j by Uj and denote its dimension by f g f g rj := dim(Uj); r := r1 = dim(U) (note that U = U1). Now the nested inclusion (4.1b) becomes

Uj+1 Uj U: (4.3)  min min Similarly, we rewrite U 1;:::;j (v) as Uj (v): f g 4.4 (Anti)symmetric Case min Conclusion 1.6 proves that (anti)symmetric tensors v lead to (anti)symmetric minimal subspaces: U (v) ( ) min ( )  Vsym or U (v) Vanti; respectively. 2 (D) The hierarchical representation (4.1a,b) of v Vsym should also use subspaces with the property ( ) 2 U Vsym (similar in the antisymmetric case).  ( ) ( ) The basic task is the determination of a basis b` U Vsym (1 ` r := dim(U )) by suitable ( 0) ( 00) 2  ( 0)  ( 0) ( 00) ( 00) linear combinations of the tensors bi bj . The assumptions bi Vsym and bj Vsym lead to ( ) ( ) 2 2 a partial symmetry, but, in general, (b ) = b is not satis…ed for  0 and  00. ` ` 2 2 Using (4.3) and symmetry, we conclude that j (j) (j) Uj U V = U : (4.4)  \ sym sym
6 Remark 4.1 (a) Because of (4.4) we can restrict the vector space V in (1.7a,b) to U: (b) If we want to represent the tensor v, the subspace Uj must satisfy

min U (v) Uj: j  min 4.5 Dimensions of Uj in the (Anti)symmetric Case The following statement shows that, in the case of antisymmetric tensors, the hierarchical approach becomes costly for high dimensions d: The simplest antisymmetric tensor is the antisymmetrisation of an elementary tensor: d a := u(j) : A j=1 O  To ensure a = 0; the vectors u(j) must be linearly independent. In that case the minimal subspace Umin(v) 6 k k (ij ) k is spanned by all tensors u with 1 i1 < i2 < : : : < ik d. There are tensors of this A j=1   d   min k d min d form. Since they are linearly independent,N dim Uk (a) = d follows. The sum k=1 dimU
k (a) is 2 1: Hence, this approach cannot be recommended for large d.
P d (j) The situation is di¤erent in the symmetric case, since the vectors in ( j=1 u ) need not be linearly independent. The next lemma uses the symmetric rank de…ned in §2.2. S N min Lemma 4.2 All symmetric tensors v satisfy dim U (v) ranksym(v): k  r d min Proof. Let v = i=1 i vi with r = ranksym(v): Then all minimal subspaces Uk (v) are contained in k the r-dimensional space span vi : 1 i r : P   The following symmetric tensor describes, e.g., the structure of the Laplace operator (a; b of the next example are the identity map and one-dimensional Laplacian, respectively).

d 1 Example 4.3 An important example is the symmetric tensor v := ( a b), where a; b V are linearly independent. In this case we have S 2

dim(Umin(v)) = 2 for 1 k < d: k  min k k 1 More precisely, U (v) is spanned by a and k b a : k S

5 TT Format for Symmetric Tensors

In the following we focus on the representation of symmetric tensors in the TT format (cf. §4.3). In principle, the same technique can be used for antisymmetric tensors (but compare §4.5).

In §5.4 we try to construct the space Uj+1 from Uj. This will lead to open questions in §5.4.6. If we min min start from v and the related minimal subspace Uj (v); then an appropriate choice is Uj = Uj (v) (see §5.5).

5.1 The Space (Uj U) (Uj U) and the Principal Idea \S We want to repeat the same construction of nested spaces as in (4.3). In contrast to the general case, we also have to ensure symmetry. By induction, we assume that Uj contains only symmetric tensors:

(j) Uj V : (5.1)  sym

On the one hand, the new space Uj+1 should satisfy Uj+1 Uj U; on the other hand, symmetry (j+1) (j+1) Uj+1 Vsym is required. Together, Uj+1 (Uj U) Vsym must be ensured.   \ (j+1) Remark 5.1 (Uj U) Vsym = (Uj U) j+1 (Uj U) holds with the symmetrisation j+1 in (1.8). \ \S S

7 (j+1) (j+1) Proof. Let v (Uj U) Vsym : Since v Vsym ; (v) = v holds. Since v Uj U; v = (v) 2 \ 2(j+1) S 2 S 2 (Uj U) follows. This proves (Uj U) Vsym (Uj U) (Uj U) : The reverse inclusion follows S (j+1) \  \S from (Uj U) Vsym : S  This leads us to the condition

Uj+1 U^ j+1 := (Uj U) (Uj U) (5.2)  \S

for the choice of the next subspace Uj+1:

It must be emphasised that, in general, (Uj U) is not a subspace of Uj U: Example 5.14 will show S nontrivial subspaces Uj;U that may even lead to U^ j+1 = 0 . f g (j) (j) To repeat the construction (4.2), we assume that there is a basis b1 ;:::; brj of Uj and the basis (j+1) (j+1) f g u1; : : : ; ur of U. Then the basis b1 ;:::; brj+1 of Uj+1 can be constructed by (4.2) which now takes fthe form g f g rj r (j+1) (k) (j) b = c b u (1 k rj+1): (5.3) k     =1 =1 X X In order to check linear independence and to construct orthonormal bases, we also have to require that we are (j) (j) able to determine scalar products. Assuming by induction that the scalar products b ; b and u ; u h 0 00 i h 0 00 i are known, the value of b(j+1); b(j+1) follows from (1.4). Therefore, we are able to form an orthonormal h k0 k00 i basis of Uj+1:

To avoid di¢ culties with a too small intersection U^ j+1; an alternative idea could be to choose the (j+1) subspace Uj+1 in (Uj U) and not necessarily in Uj U. Then, instead of (5.3), we have bk = (k) (j) S ; c (b u): This would be a very ‡exibleapproach, were it not for the fact that we need knowledge S (j) (j) (j) (j) (j+1) of the scalar products s ; s for s := (b u) (otherwise, an orthonormal basis b cannot P 00 S f k g be constructed). One …ndsD that E

(j) (j) 0 (j) (j) j (j) (j) s ; s  = b ; b + b ; b 0 0 j + 1 0 j + 1 ;[0] 0;[] D E D E D E where the expression b(j) is de…ned in Lemma 1.3. The scalar products b(j) ; b(j) can be derived from ;[0] ;[0] 0;[] (j) (j) (j 1) (j 1) s ; s : This expression, however, requires the knowledge of Db ; b E (concerning the ;[`] 00;[`0] ;[`;0] 0;[`0;] D E D (d) E(d 1) (d 1) subscript [`; 0] compare (5.7)). Finally, we need scalar products of the systems b ; b ; b;[] ; (d 2) (j) f g f g f g b ;:::; b with j = min j; d j ;::: . This leads to a data size increasing exponentially ;[`m] ;[`1;`2;:::;` ]  f g f j g f g in d:

min 5.2 The Spaces (Uj U) (Uj U) and U (v) \S j+1 d (d) Let v ( V ) Vsym be the symmetric tensor which we want to represent. We recall the minimal subspaces 2 \ min min j de…ned in §1.2. According to the notation of the TT format, Uj (v) is the space U 1;:::;j (v) V de…ned min f g  in (1.9). The minimality property of Uj (v) (cf. [9, §6]) implies that the subspaces U and Uj must satisfy

min min U U (v); Uj U (v); (5.4)  1  j otherwise v cannot be represented by the TT format.

The next theorem states that (5.4) guarantees that there is a suitable subspace Uj with U^ j Uj min   Uj (v); so that the requirement (5.4) is also valid for j + 1:

(d) min Theorem 5.2 Let (5.4) be valid for v Usym and let j < d: Then U^ j+1 in (5.2) satis…es U^ j+1 U (v): 2  j+1

8 min min Proof. We have Uj U U (v) U (v): A general property of the minimal subspace is  j 1 Umin(v) U min(v) Umin (v) j 1  j+1 min (cf. [9, Proposition 6.17]). Since Uj+1(v) is symmetric (cf. Conclusion 1.6), it follows that min min min min (Uj U) (U (v) U (v)) (U (v)) = U (v): S  S j 1  S j+1 j+1 min This inclusion together with the previous inclusion Uj U U (v) yields the statement.  j+1 min So far, we could ensure that there exists a suitable subspace Uj+1 Uj+1(v). Concerning the practical implementation, two questions remain: 

(a) How can we …nd the subspace U^ j+1 Uj U?  min (b) Given U^ j+1; how can we ensure Uj+1 U (v)?  j+1 The next subsection yields a partial answer to the …rst question.

5.3 Criterion for Symmetry

According to Lemma 1.3, any v Uj U is of the form 2 r

v = v u` (v Uj): (5.5) [`] [`] 2 `=1 X The mapping v V(j+1) v V(j) can be iterated: 2 7! [`] 2 (j) (j 1) v V (v ) = v = v V : [`] 2 7! [`] [m] [`][m] [`;m] 2 j 1 In the case of j = 1; the empty product V is de…ned as the …eld K, i.e., v[`;m] is a scalar.

(j+1) Lemma 5.3 A necessary and su¢ cient condition for v Vsym is 2 v V(j) and v = v for all 1 `; m r: (5.6) [`] 2 sym [`;m] [m;`]   Here, v refers to (5.5), and v is the expansion term of v jV: [`] [`;m] [`] 2 (j+1) (j) Proof. (a) Assume v Vsym : v[`] Vsym is stated in Remark 1.4. Applying the expansion to v[`] in (5.5), we obtain 2 2 r

v = v um u`: (5.7) [`;m] `;m=1 X Note that the tensors um u` : 1 `; m r are linearly independent. Therefore, transposition f   g um u` u` um and symmetry of v imply that v[`;m] = v[m;`]: 7! (j) (b) Assume (5.6). Because of v Vsym; v is invariant under all transpositions i;i+1 for 1 i < j: [`] 2  Condition v[`;m] = v[m;`] ensures that v is also invariant under the transposition j;j+1: This proves the symmetry of v (cf. Remark 1.1).

To apply this criterion to the construction of U^ j+1 := (Uj U) (Uj U) ; we search for a symmetric tensor (5.3) of the form \S rj r (j) b = cb u: (5.8) =1 =1 X X rj (j) (j) The tensor b corresponds to v := c`b in (5.5). v Vsym is satis…ed because of (5.1). The [`] =1 [`] 2 condition v[`;m] = v[m;`] in (5.6) becomes P rj rj (j) (j) c`b;[m] = cmb;[`]: (5.9) =1 =1 X X (j) (j) The tensors b and b belong to Uj 1: The new (nontrivial) algebraic task is to …nd the set of ;[m] ;[`] coe¢ cients c satisfying (5.9) for all 1 `; m r:  

9 Remark 5.4 The ansatz (5.8) has rrj 1 free parameters (one has to be subtracted because of the nor- r(r 1) r(r 1) malisation). Condition (5.9) describes 2 equations in the space Uj 1 equivalent to 2 rj 1 scalar equations.

5.4 TT Symmetrisation

In the following approach we obtain a symmetric tensor in (Uj U) ; but not necessarily in Uj U: S

5.4.1 Symmetrisation Operator j+1 S In the present approach we directly symmetrise the tensors. The general symmetrisation map consists S of d! terms. However, since Uj is already symmetric, tensors in Uj U can be symmetrised by only j + 1 transpositions  i;j+1.

(j) (j) Theorem 5.5 Let v Vsym and w V: Applying 2 2 j+1 1 ^j+1 := i;j+1 S j + 1 i=1 X to v(j) w V(j+1) yields a symmetric tensor: 2 (j) (j+1) s := ^j+1 v w V : S 2 sym   Proof. According to Remark 1.1, we have to show that k;k+1s = s for all 1 k j: First we consider the   case of k < j: If i = k; k + 1 ; we have k;k+1i;j+1 = i;j+1k;k+1: For i k; k + 1 we obtain 2 f g 2 f g

k;k+1k;j+1 = k+1;j+1k;k+1; k;k+1k+1;j+1 = k;j+1k;k+1:

This proves k;k+1 ^j+1 = ^j+1k;k+1 and S S (j) (j) k;k+1s = ^j+1k;k+1 v w = ^j+1 k;k+1v w : S S     (j) (j) (j) Symmetry of v implies k;k+1v = v so that k;k+1s = s is proved. (j) (j) The remaining case is k = j: For i < j; the identity j;j+1i;j+1 = i;j+1i;j together with i;jv = v (j) (j) implies j;j+1i;j+1(v w) = i;j+1(v w): For i j; j + 1 we obtain 2 f g

j;j+1j;j+1 = id = j+1;j+1; j;j+1j+1;j+1 = j;j+1 id = j;j+1;
i.e., j;j+1 (i;j+1 + j+1;j+1) = i;j+1 + j+1;j+1: Hence, also j;j+1s = s is proved.

Corollary 5.6 The corresponding antisymmetrisation is obtained by

d 1 d i ^d := ( 1) id: A d i=1 X

Although the symmetrisation ensures that s (Uj U) ; there is no guaranty that s Uj U: Hence, 2 S 2 whether s U^ j+1 holds or not is still open. 2 5.4.2 Expansion of s

(j) j+1 (j) j+1 Since v w U; the symmetrisation s = ^j+1(v w) also belongs to U: By Lemma 1.3 there 2 r S (j) is a representation s = s u` with s Usym: `=1 [`] [`] 2 P

10 r (j) (j) (j) Lemma 5.7 Let w = c`u` and v Usym: Then s := ^j+1(v w) satis…es `=1 2 S Pr j 1 (j) (j) s = s u` with s := c`v + i;j (v w) : (5.10) [`] [`] j + 1 [`] `=1 i=1 ! X X j (j) (j) The latter sum i;j(v w) can be written as j ^j(v w): i=1 [`] S [`] P Proof. Using j+1;j+1 = id; we obtain

j r j (j) (j) (j) (j) (j + 1) s = v w + i;j+1(v w) = c`v u` + i;j+1(v w):

i=1 `=1 i=1 X X X (j) r (j) (j) Since i;j+1 = i;jj;j+1i;j for i j and v = v u` Usym, we have  `=1 [`] 2

(j) (j)P (j) i;j+1(v w) = i;jj;j+1 (i;jv ) w = i;jj;j+1 v w

r  r  r  (j) (j) (j) = i;jj;j+1 v u` w = i;j v w u` = i;j(v w) u`: [`] [`] [`] `=1 `=1 `=1 X X X   r (j) r (j) Together we obtain (j + 1) s = c`v + i;j(v w) u`: `=1 `=1 [`] The last equation explicitlyP provides the expansionP  of s de…ned in Lemma 1.3.

5.4.3 Scalar Products

(j) The de…nition of s := ^j+1(v w) seems a bit abstract, since (5.10) contains the permuted tensor which not S (j) necessarily belongs to Uj U. Even in that case it is possible to determine the scalar products s; b u (j ) h i with the basis vectors b u of Uj U: The …rst term in (5.10) yields

(j) (j) (j) (j) v u`; b u = v ; b u`; u :   h i D E D E (j) (j) By induction, we assume that the scalar product of v Uj and b is known. Usually, the basis u` is 2 f g chosen orthonormal so that u`; u = `: The other terms yield the products h i (j) (j) (j) (j) i;j v w u` ; b u = i;j v w ; b u`; u : [`]  [`]  h i D   E D   E (j) (j) Using the selfadjointness of i;j and b Vsym; we obtain 2 (j) (j) (j) (j) (j) (j) i;j v w ; b = v w; i;jb = v w; b [`]  [`]  [`]  D   E D r E D r E (j) (j) (j) (j) = v w; b uk = v ; b w; uk : [`] ;[k] [`] ;[k] h i k=1 k=1  X  X D E

If u` is an orthogonal basis, w; uk = ck holds (cf. Lemma 5.7). f g h i (j) (j) Remark 5.8 Let the bases b : 1  rj and u` : 1 ` r be orthonormal. If s := ^j+1(v w) f   g f   g S 2 Uj U; the explicit representation is given by

rj r (j) s = cb u (5.11) =1 =1 X X (j) with coe¢ cients c = s; b u , which are computable as explained above. h i

11 Even if s = Uj U; the right-hand side in (5.11) is computable and describes the orthogonal projection 2 P Uj U s of s onto the space Uj U:

The check whether s belongs to Uj U is equivalent to the check whether P Uj U s is symmetric (cf. §5.3), as stated next. ^ (j+1) Criterion 5.9 s Uj U [and therefore also s Uj+1; cf. (5.2)] holds if and only if P Uj U s = s Vsym 2 ^ 2 2 (implying P Uj U s Uj+1 in the positive case). 2

Proof. (a) Abbreviate P Uj U by P . Let s Uj U: This implies P s = s: Since, by construction, s is (j+1) 2 symmetric, P s Vsym holds. 2 (j+1) (b) Assume P s Vsym : Because of s = P s + (I P )s; also s? := (I P )s (Uj U)? is symmetric. The properties of projections2 show 2

(j) s?; s? = (I P )s; (I P )s = s; (I P )s = j+1(v w); s? : h i h i h i hS i Since j+1 is selfadjoint and s? is symmetric, we have S (j) (j) s?; s? = v w; j+1s? = v w; s? = 0 h i h S i h i (j) because of v w Uj U and s? (Uj U)? : This proves s? = 0 and s = PUj U s; i.e., s Uj U: 2 2 2

5.4.4 Geometric Characterisation (j) (j) (j) Let b : 1  rj and u : 1  r be orthonormal bases of Uj Usym and U; respectively. b;[`] f   g (jf)  (j) g  are the expansion terms: b = b u`: They give rise to the scalar products ` ;[`] P (j) (j) B(;);( ; ) := b ; b (1 ; 0 rj; 1 ; 0 r): 0 0 ;[0] 0;[]     D E I I (j) Let B K  be the corresponding matrix, where I = 1; : : : ; rj 1; : : : ; r : The orthonormality of b 2 fH g  f g f g is equivalent to ` B(;`);(0;`) = ;0 : Note that B = B . rj r (j) 2 Consider theP tensor v = =1 =1 cb u: The normalisation v = 1 gives ; c = 1: The I k k j j entries c de…ne the vector c K : P2 P P Theorem 5.10 The spectrum of B is bounded by 1: The above de…ned tensor v is symmetric if and only if c is an eigenvector of B corresponding to the eigenvalue 1.

Proof. Let s = j+1v: The projection property of j+1 implies that v; s 1: Criterion 5.9 states that v is symmetric (i.e.,S v = s) if and only if v; s = 1: CalculatingS the scalarh producti  according to §5.4.3 yields h i (j + 1) v; s = 1 + j (Bc; c) ; where ( ; ) is the Euclidean product of KI : The inequality v; s 1 shows that h i

h i  all eigenvalues of B are bounded by 1. The equality v; s = 1 requires that (Bc; c) = 1 = max (Bc0; c0): h i f c0 = 1 ; i.e., c is the eigenvector with eigenvalue  = 1. k k g The questions from above take now the following form: (a) How can we ensure that 1 belongs to the spectrum of B; (b) what is the dimension of the corresponding eigenspace?

5.4.5 Examples The following examples use tensors of order d = 3. The case d = 2 is too easy since tensors of 2U correspond r r to matrices via v = ;=1 cu u C := (c);=1: Hence symmetric tensors v are characterised by symmetric matrices C: 7! P 2 In the following examples u1 = a; u2 = b V are orthonormal vectors. A possible choice is V = K : 2 Example 5.11 We want to represent the symmetric tensor s := a a a: We use U = span a; b and (2) (2) f g the symmetric subspace U2 := span b1 (U U) U U with b1 := a a: Symmetrisation of f g  S  1 U2 U = span a a a; a a b yields (U2 U) = span a a a; (a a b + a b a + b a a) : f g S f 3 g Obviously, (U2 U) is not a subspace of U2 U. S

12 The reason for (U2 U) U2 U in Example 5.11 may be seen in the choice of U = span a; b : This S 6 min f g space is larger than necessary: U = U (s) = span a is su¢ cient and this choice leads to (U2 U) = 1 f g S U2 U: min In the next example, U is chosen as U1 (s): Example 5.12 We want to represent the symmetric tensor s := a a a+b b b: We use U = span a; b (2) (2) (2) (2) f g and the symmetric subspace U2 := span b ; b (U U) U U with b := a a and b := b b: f 1 2 g  S  1 2 The tensor space U2 U is spanned by a a a; b b b; a a b; b b a: The …rst two tensors are already symmetric. The symmetrisation of a a b leads to a tensor which is not contained in U2 U:

The same holds for the last tensor. Hence, (U2 U) U2 U. S 6 (2) In Examples 5.11 and 5.12, we can omit the tensors b uj whose symmetrisation does not belong to i U2 U; and still obtain a subspace containing the tensor s to be represented. The latter statement is not true in the third example. Example 5.13 We want to represent the symmetric tensor s := 3 (a + b) + 3 (a b) : We use U = (2) (2) (2) 2 span a; b and the symmetric subspace U2 := span b1 ; b2 (U U) U U with b1 := (a + b) f (2) g 2 f g  S  (2) and b2 := (a b) : The tensor space U2 U is spanned by four tensors bi uj. For i = j = 1; we (2) have b a = (a + b) (a + b) a; whose symmetrisation does not belong to U2 U: The same holds for 1 the other three tensors. Hence, (U2 U) U2 U. S 6 Note that the setting of Example 5.13 coincides with Example 5.12 when we replace the orthonormal basis u1 = a; u2 = b with u1 = (a + b)=p2; u2 = (a b)=p2 : f g f g min The next example underlines the important role of condition Uj (v) Uj. (2) (2)  Example 5.14 Let U2 := span b1 with b1 := a b + b a: A general tensor in U2 U has the form (2) f g b1 ( a + b): There is no symmetric tensor of this form, except the zero tensor ( + = 0). This shows min that U2 is too small: there is no nontrivial symmetric tensor v with U (v) U2: 2 

5.4.6 Open Questions About (Uj U) (Uj U) S \ We repeat the de…nition U^ j+1 := (Uj U) (Uj U) : The main questions are: S \ ^ min What is the dimension of Uj+1; in particular, compared with dim(Uj (v)), if v is the tensor to be  represented?

Is there a constructive description of U^ j+1?  The minimal set, which is needed for the construction of the tensors in U^ j+1; is  min Uj := Uj (v): v U^ j+1 2 X By de…nition, U j Uj holds, but it is not obvious whether U j = U^ j: This yields the next question:  Does U j = Uj hold?  In the negative case, there is a direct sum Uj = U j Zj; where Zj = 0 contains symmetric tensors in (j)  (j+1) 6 f g Vsym which cannot be continued to symmetric tensors in Vsym : Using Uj instead of U j would be ine¢ cient.

5.4.7 Answers for d = 3 and r = 2 The questions from above can be answered for the simple case of d = 3 (transfer from d = 2 to d = 3) and r = 2: Hence we have (2) dim(U) = 2; U1 = U; U2 U  sym and have to investigate the space U^ := (U U) (U U) : We recall that U = Umin(w) U 3 3 2 2 2 w U^ 3 2 2 S \ 2  is the smallest subspace of U2 with the property 3(U 2 U) U 2 U = U^ 3: Hence, if dim(U2) > dim(U 2); S \ P U contains tensor which are useless for the construction of symmetric tensor in U^ : 2
3 (2) The symmetric tensors v U2 correspond to symmetric 2 2 matrices. Since dim(Usym) = 3, the following list of cases is complete.2 The general assumption of the following theorems is dim(U) = 2:

13 (2) Theorem 5.15 (Case dim(U2) = 1) Let dim(U2) = 1 and U2 = span b1 Usym. If rank(b1) = 1 then f g 

U 2 = U2; dim(U^ 3) = 1;

otherwise we have rank(b1) = 2 and

U 2 = 0 U2; U^ 3 = 0 : f g  f g

Proof. Note that rank(b1) dim(U) = 2. rank(b1) = 0 is excluded because of b1 = 0 and the assumption  that U2 = span b1 is one-dimensional. Hence, rank(b1) only takes the values 1 and 2. f g If rank(b1) = 1; b1 = a a0 follows. Symmetry shows that b1 = a a (possibly after changing the sign5 ). Then U^ 3 = span b1 a = span a a a : f g f g If rank(b1) = 2; the general form of w U2 U is w = b1 (a + b). Assume (; ) = 0: Then the 23 6 only symmetric tensor of this form is w = (a + b); i.e., b1 = (a + b) (a + b): The contradiction

follows from rank(b1) = 1: Hence  =  = 0 leads to the assertion.   min ^ The statements about U2 follow from the de…nition U2 = U2 (U3):

Theorem 5.16 (dim(U2) = 2) Let dim(U2) = 2: Then

U 2 = U2; dim(U^ 3) = 2:

(3) The precise characterisation of U^ 3 Usym is given in the proof. 

Proof. (i) There are two linearly independent and symmetric tensors b1, b2 with U2 = span b1; b2 : Fixing linearly independent vectors a; b U = span a; b ; the tensors have the form f g 2 f g

b1 = a a + b b + (a b + b a) ;

b2 = 0a a + 0b b + 0 (a b + b a)

In part (vi) we shall prove that dim(U^ 3) 2: The discussion of the cases 1–3 will show that dim(U^ 3) 2;   so that dim(U^ 3) = 2 follows. (ii) Case 1: = 0 = 0: One concludes that U2 = span a a; b b : Then the …rst case in Theorem 5.15 f g shows that a a a and b b b belong to U^ 3 so that dim(U^ 3) 2 and part (vi) prove 

U^ 3 = span a a a; b b b : (5.12) f g

(iii) Case 2: ( ; 0) = 0: W.l.o.g. assume = 0: We introduce the matrices 6 6

0 0 M := ;M := : 0 0    

Since = 0; both matrices have a rank 1: If rank(M ) = rank(M ) = 1; ( ; ; ) and ( 0; 0; 0) would be 6  linearly dependent in contradiction to the linear independence of b1; b2 : Hence, at least one matrix has f g rank 2 and is regular. W.l.o.g. we assume that rank(M ) = 2 (otherwise interchange the roles of a and b). (iv) For any (A; B) K2 the system 2  A M = M (5.13)  B     can be solved for (; ) K2: Then the tensor 2

w := (Ab1 + Bb2) a + (b1 + b2) b

5 If K = C; the representation b1 = a a holds in the strict sense, If K = R; either b1 = a a or b1 = a a can be obtained. Since the purpose of b1 is to span the subspace, we may w.l.o.g. replace b1 = a a by b1 = a a:

14 (3) is symmetric and belongs to U^ 3: For a proof apply Lemma 5.3: w Usym is equivalent to 'b (Ab1 + Bb2) = 2 'a (b1 + b2) ; where the functionals de…ned by 'a(a) = 'b(b) = 1;'a(b) = 'b(a) = 0 apply to the last argument. The latter equation is equivalent to (5.13).

Let (; ) be the solution of (5.13) for (A; B) = (1; 0) ; while (0; 0) is the solution for (A; B) = (0; 1) : Hence we have found a two-dimensional subspace

span b1 a + (b1 + b2) b; b2 a + (0b1 + 0b2) b U^ 3: (5.14) f g   min ^ (v) In both cases (5.12) and (5.14) the minimal subspace U2 = U2 (U3) coincides with U2: min (vi) For an indirect proof of dim(U^ 3) 2 assume dim(U^ 3) 3: Let 'a : U^ 3 U (U^ 3) = U2 be the   ! 2 mapping w=v1 a+v2 b v1: Since dim(U^ 3) > dim(U2); there is some w U^ 3; w = 0 with 'a(w) = 0: 7! 2 6 This implies w = v2 b and therefore, by symmetry, w = b b b up to a nonzero factor: Similarly, there are an analogously de…ned functional 'b and w U^ 3; w = 0 with 'b(w) = 0 proving a a a U^ 3: From 2 min6 2 a a a; b b b U^ 3 we conclude that U 2 := U (U^ 3) span a a; b b : Then U 2 U2 and 2 2  f g  dim(U2) = 2 prove dim(U^ 3) 2:  (2) Theorem 5.17 (dim(U2) = 3) If dim(U2) = 3; U2 coincides with space Usym of all symmetric tensors in (3) U U and generates all tensors in Usym:

 (2) ^ (3) ^ U2 = U2 = Usym; U3 = Usym with dim(U3) = 4:

(2) Proof. The statements follow from dim(Usym) = 3:

min 5.5 Direct Use of Uj (v) min Statement (5.4) emphasises the important role of the minimal subspace Uj (v):

min 5.5.1 Case of Known Uj (v)

min (d) If the minimal subspaces Uj (v) of a symmetric tensor v Usym are given, the above problems disappear. min 2 In this case we may de…ne Uj := Uj (v): This ensures that

min U j = Uj and U^ j+1 U (v)  j+1 (cf. Theorem 5.4). (d) If we want to be able to represent all tensors of a subspace V0 = span v1;:::; vk Usym; we may use f g  k min min Uj := Uj (V0) = Uj (v ): =1 X Lemma 6.1 shows that Uj satis…es (6.2). min Next we consider the case that Uj (v) is not given explicitly, but can be determined by symmetrisation.

5.5.2 Case of v = (w) S (d) As in §2.1 we assume that the symmetric tensor 0 = v Usym is the symmetrisation (w) of a known tensor w dV: Unlike in §2.1, we assume that w is given6 in2 the TT format with minimalS subspaces6 Umin(w): 2 j The obvious task is to transfer Umin(w) into Umin(v) = Umin( (w)): j j j S We solve this problem by induction on d = 1;:::: The proof also de…nes an recursive algorithm. min For d = 1 nothing is to be done since v = (w) = w: Formally, U0 (v) is the …eld K and U1 K U corresponds to (4.3). S  The essential part of the proof and of the algorithm is the step from d 1 to d: 6 In the case of hierarchical tensor representations it is easy to reduce the subspaces to the minimal ones by introducing the HOSVD bases (cf. Hackbusch [9, §11.3.3]).

15 d d Lemma 5.18 Let [j] : V V (1 j d) be the symmetrisation operator d 1 in (1.8) applied to S !   S the directions D j (cf. (1.2)). Using the transpositions 1j; j1; the explicit de…nition is nf g

[j] = 1j (id d 1) j1 : S S

Then the symmetrisation operator d is equal to S 1 d d = dj : S d S[j] j=1 X This lemma proves the next result.

d Conclusion 5.19 Let w U and ' U 0: Then 2 2 d (d) 1 (j) ' ( dw) = d 1 ' (w) S dS j=1 X holds with '(j) de…ned in (1.6).

d The 1; : : : ; d 1 -plex rank rd 1 of x U introduced by Hitchcock [11] is the smallest rd 1 with rdf 1 g d 1 2 x = =1 x y (x U; y U): For instance, this representation is the result of the HOSVD 2 2 min representation (cf. [9, §11.3.3]). The minimal subspace Ud 1(x) is the span of x : 1  rd 1 : P f   g r Alternatively, choose the standard basis u : 1  r of U and the representation x = z u f   g =1 together with the dual basis ' of u : Then the tensors z = ' (x) may be linearly dependent so that f g f g min P some of them may be omitted. Let x (1  rd 1) be the remaining ones: Ud 1(x) = span x : 1    f   rd 1 : g Remark 1.5 states that rd 1 min min min min Ud 2(x) = Ud 2(Ud 1(x)) = Ud 2(x ): (5.15) =1 X This allows us to determine the minimal subspaces recursively. Let the TT representation of w dU be given. The TT format is also called the matrix product representation since w dU can be2 written as 2 w = v(1) v(2) v(3) v(d) ; 1;k1 k1;k2 k2;k3 kd 1;1


k1;k2;:::;kd 1 X where the vectors v(j) U are data available from the TT representation. k varies in an index set I kj ;kj+1 j j min 2 (j) d 1 with #Ij = dim(U (w)). The tensor ' (w) U takes the matrix product form j 2 (j) (j) (1) (j 1) (j+1) (d) ' (w) = '(v ) v v v v : kj 1;kj 1;k1 kj 2;kj 1 kj ;kj+1 kd 1;1





kj 1;kj k1;:::;kj 2; X kj+1X;:::;kd 1

d (j) d 1 These tensors can be added within the TT format: w := ' (w) U: We conclude that =1 2 min min P Ud 1(v) = Ud 1( dw) = span d 1w : 1  rd 1 : S fS   g min min According to (5.15) the next minimal subspace Ud 2(v) can be written as  Ud 2( d 1w ) so that we can apply the inductive hypothesis. S P

16 6 Operations

The computation of scalar products is already discussed. Next we investigate the tensor addition.

Assume that v0 and v00 are two symmetric tensors represented by subspaces Uk0 and Uk00 and corresponding bases. For k = 1; we use the notation U 0 := U10 and U 00 := U100: By assumption, we have min U (v0) U0 k+1(U0 U 0) (U0 U 0) ; (6.1a) k+1  k+1  S k \ k min U (v00) U00 k+1(U00 U 00) (U00 U 00) : (6.1b) k+1  k+1  S k \ k

Lemma 6.1 The sum s := v0 + v00 can be represented by the subspaces

Uk := Uk0 + Uk00;U := U 0 + U 00: These spaces satisfy again the conditions

min U (s) Uk+1 k+1(Uk U) (Uk U) : (6.2) k+1   S \ Proof. The inclusions (6.1a,b) imply

min min U (v0) + U (v00) Uk+1 = U0 + U00 k+1 k+1  k+1 k+1 ( k+1(U0 U 0) (U0 U 0)) + ( k+1(U00 U 00) (U00 U 00)) :  S k \ k S k \ k min min min min Since U (s) U (v0) + U (v00); the …rst part of (6.2) follows: U (s) Uk+1. The inclusion k+1  k+1 k+1 k+1  U0 U 0 Uk U implies that k  k+1(U0 U 0) (U0 U 0) k+1(Uk U) (Uk U) : S k \ k  S \ The analogous statement for k+1(U00 U 00) (U00 U 00) yields S k \ k ( k+1(U0 U 0) (U0 U 0)) + ( k+1(U00 U 00) (U00 U 00)) S k \ k S k \ k k+1(Uk U) (Uk U) :  S \ Hence also the second part of (6.2) is proved.

The computation of the orthonormal basis of Uk is performed in the order k = 1; 2;::: . As soon as orthonormal bases of Uk and U are given, the orthonormal basis of Uk+1 can be determined.

Since the spaces Uk = Uk0 + Uk00 may be larger than necessary, a truncation is advisable.

7 Truncation

The standard truncation procedure uses the SVD. Formally, we have for any tensor u V and any k 1; : : : ; d 1 a singular value decomposition 2 2 f g ru u =  v w ; (7.1) =1 X k d k where v : 1  ru V and w : 1  ru V are orthonormal systems and f   g  f   g  1 2 ::: are the singular values. The usual approach is to choose some s < ru and to de…ne the ftensor  g s

^u =  v w

=1 X which can be represented with subspaces of lower dimension. (d) (k) (d k) In the case of symmetric tensors u Vsym, we have v Vsym and w Vsym : However, the standard 2 2 2 s truncation cannot be used since there is no guarantee that the truncated tensor ^u =  v w again =1 belongs to V(d) : sym P

17 7.1 Truncation for k = 1 and k = d 1 In the cases k = 1 and k = d 1; the truncation can be performed as follows. For k = 1; the standard truncation u ^u can be written as ^u = (P I) u; where P : V V is the orthogonal projection onto the subspace 7! ! U^ := span v : 1  s V; f   g  d 1 d 1 while I = I is the identity on V:

In the symmetric case, we need a symmetric mapping. If we delete the components v :  > s in the …rst direction, the same must be done in the other directions. The corresponding mappingf is the orthogonalg projection P := dP

(d) (d) onto the subspace U^ sym Vsym: Note that the error u Pu does not only consist of the omitted terms ru  s d 1  v w ; but also of  v I P w : =s+1 =1 In the case of k = d 1; w belongs to V and the analogous construction can be performed. If d = 3; P  P
the cases k = 1 and k =d 1 cover all possible ones. 7.2 Open Problem for 1 < k < d 1 If d > 3; there are integers k with 1 < k < d 1: Then both v and w in (7.1) are tensors of order 2: It is not obvious how the analogue of the previous mapping P could look like. The advantage of a  s symmetric projection P would be the existence of a tensor v0 = Pv Usym: De…ne w0 := =1  v w ru 2 and w00 :=  v w by SVD. Assume that Pw0 = 0 while Pw00 = 0: Then =s+1 6 P P Pv = P (w0 + w00) = (Pw0 + Pw00) = (Pw0) S S S min (cf. Lemma 9.1) does not vanish, i.e., v0 = 0 ensures the existence of a nontrivial subspaces Uj (v0) (1 j d): 6   (k) (k) Let Pk : Vsym Vsym be the orthogonal projection onto span v : 1  s ; so that Pk := d k ! s f   g Pk ( I) maps u to the SVD truncation ^u = =1  v w : The symmetrisation P = (Pk) de…nes (d) S u0 := Pu Usym: Since 2 P

u; u0 = u; Pu = u; (Pk)u = u; (Pku) = (u); Pku = u; Pku = ^u; ^u ; h i h i h S i h S i hS i h i h i min the tensor u0 does not vanish. However, it is not obvious that dim(Uk (u0)) s holds, as intended by the truncation. 

A remedy is a follows. Assume that the (not truncated) tensor uses the subspaces Uj satisfying (5.2). (k) Let the SVD for index k reduce Uk to U0 : Since U0 Uk; U0 still belongs to Usym: Moreover k k  k

U^ 0 := (U0 U) (U0 U) (Uk U) (Uk U) = U^ k+1 k+1 k \S k  \S guarantees the existence of a subspace U0 U^ 0 so that the construction can be continued. However, k+1  k+1 may it happen that U0 is too small and U^ 0 = 0 holds? k k+1 f g 8 Combination with ANOVA

As already mentioned in [9, §17.4.4] the ANOVA7 representation has favourable properties in connection with symmetric tensors. The ANOVA technique is brie‡ydescribed in §8.1 (see also Bohn–Griebel [4, §4.3]). The ANOVA approximation uses terms with low-dimensional minimal subspaces. Therefore the combination with the TT format is an e¢ cient approach.

7 ANOVA abbreviates ‘analysis of variance’.

18 8.1 ANOVA Technique Let 0 = e V be a special element. In the case of multivariate functions, e may be the constant function e(x) =6 1: In2 the case of mappings, e is the identity mapping. De…ne

E := span e ; V := E?: (8.1) f g

The choice of the orthogonal complement E? simpli…es later computations. Theoretically, it would be su¢ cient to choose V such that there is a direct sum

V = E V:  (d) The space V gives rise to the symmetric tensor space Vsym: (d) (k) We introduce the following notation of symmetric tensors in Vsym generated by tensors from Vsym:

d k (k) S(v; d) := d(v ( e)) for v V with 1 k d: (8.2) S 2 sym   (d) The tensors in (8.2) form the following subspaces of Vsym:

d (k) d k V0 := E; Vk := d(V ( E)) for 1 k d: S sym   (d) d  Lemma 8.1 If (8.1) holds, there is an orthogonal decomposition Vsym = j=0 Vj:

  L d (j) Proof. Let k > `; v Vk; w V`: Tensor v can be written as a sum of elementary tensors v = j=0 v 2 (j) 2 d (j) (j) containing k vectors v V. Correspondingly, w is a sum of w = w with d ` vectors w = e: 2 j=0 N (j)  (j) Because of ` < k; there must be some j with v V orthogonal to wN = e: Hence v ; w = 0 holds for all pairs implying v; w = 0: 2 h i h i (d) The ANOVA representation of a (symmetric) tensor v Vsym is the sum 2 L

v = vk with vk Vk for some 0 L d: (8.3) 2   k=0 X We call L the ANOVA degree.

Remark 8.2 (a) The motivation of ANOVA is to obtain an approximation (8.3) for a relative small degree L: d (b) Let vex = k=0 vk be the exact tensor. In order to approximate vex by v from (8.3) we have to assume that the terms vk are (rapidly) decreasing. P 8.2 Representation

(k) We assume that (8.3) holds with vk = S(xk; d). The tensors xk Vsym are given together with their min 2 minimal subspaces Uk (xk) as described in §5.5. We generalise the notation S( ; d) in (8.2) to subspaces:
S(X; d) = span S(x; d): x X for X V(k) : f 2 g  sym

If X = span x : 1  N ; we have S(X; d) = span S(x ; d) : 1  N : f   g f   g We remark that Umin(v) = span v for v dV: d f g 2

Lemma 8.3 Let vk Vk. Then 2 max k;d j f g min min Uj (S(vk; d)) = S(U (vk); j): (8.4) =min j;k j Xf g

19 min Proof. By de…nition of Uj ( ); functionals in d j directions are to be applied. Let  be associated with d k
vk and d j  with e: This leads to the inequalities 0  k and 0 d j  d k: Together they imply min j; k j  max k; d j :     f g   f g Consider the case of j = d 1: Applying ' with '(e) = 1 and '(v) = 0 (v V), we obtain (d) 2 ' (S(vk; d)) = S(vk; d 1); provided that k < d. On the other hand, ' with '(e) = 0 yields (d) ' (S(vk; d)) = S('(vk); d 1): This proves min min Ud 1(S(vk; d)) = S(vk; d 1) + S(Uk 1(vk); d 1): min Since span S(vk; d 1) = S(Uk (vk); d 1); this coincides with (8.4) for j = d 1: For the other j apply Remark 1.5f recursively. g min min min The ANOVA tensor is the sum v = k vk: As Uj (a + b) Uj (a) + Uj (b); we obtain from (8.4) that  min P min Uj S vk; d S(U (vk); j): k  k;  X  X min The dimension of the right-hand side may be larger than Uj (v); but here we want to separate the spaces E and U V: For instance, the tensor v = (a + e) (a + e) has the one-dimensional minimal subspace U min(v) = span a + e ; but here we use E + U with U = span a : 1 f g f g 8.3 Example

(2) Consider an expression of the form S(a0 ; d) + S(b; d); where a0 V and b Vsym: We assume that b i i i 2 2 can be approximated by (b0 c0 + c0 b0 ). Orthogonalisation of a0 ; b0 ; c0 with respect to some e yields k k kP k k i k k the vectors ai; bk; ck and the ANOVA form P

N1 N2 v = S(1; d) + S(xi; d) + S(ak bk + bk ak; d); (8.5)

i=1 k=1 X X d where xi represents ai and multiples of bk and ck. Note that S(1; d) = e: In the following we give the details for v = S(1; d) + S(x; d) + S(a b + b a; d):

The combined ANOVA-TT format uses the spaces

j = 1 : U1 = span e; x; a; b ; (8.6) f g j = 2 : U2 = span S(1; 2);S(x; 2);S(a; 2);S(b; 2);S(a b + b a; 2) ; f g . .

j < d : Uj = span S(1; j);S(x; j);S(a; j);S(b; j);S(a b + b a; j) ; f g j = d : Ud = span v : f g The essential recursive de…nition (5.3) of the basis reads as follows:

S(1; j) = S(1; j 1) e; S(x; j) = S(x; j 1) e + S(1; j 1) x; S(a; j);S(b; j): analogously, S(a b + b a; j) = S(a b + b a; j 1) e + S(a; j 1) b + S(b; j 1) a: The …nal step follows from

v = S(1; d 1) + S(x; d 1) e + S(1; d 1) x + S(a b + b a; d 1) e + S(a; d 1) b + S(b; d 1) a:

20 Remark 8.4 The terms N2 S(1; d) and S(ak bk + bk ak; d)

k=1 X in (8.5) lead to 3N2 + 1 basis vectors in Uj: Let N0 vectors xi be linearly independent of span ak; bk : 1 N1 f  k N2 : Then S(xi; d) requires N0 additional basis vectors in Uj:  g i=1 P 8.4 Operations

Concerning scalar product one can exploit Lemma 8.1: S(v; d);S(w; d) = 0 for v Vk; w V` with h i 2 2 k = `: If the basis of Uj should be orthonormalised, it is su¢ cient to orthonormalise only the contributions 6 S(v ; j) for all v Vk separately (cf. (8.6)). 2 One can possibly use that for tensors v ; w Vk; k j; the scalar product S(v; j);S(w; j) is a …xed multiple of v; w ; provided that e; e = 1: 2  h i h i h i j! S(v; j);S(w; j) = v; w : h i k! h i In principle, the operations within the TT format are as usual. However, one has to take care that the result is again of the ANOVA form. As an example we consider the Hadamard product (pointwise product) for multivariate functions. For the standard choice that e is the constant function with value 1, we have e e = e (and a e = e a = a

for any a). If v is of the form (8.3) with L = Lv; while w corresponds to L = Lw, the product z := v w satis…es (8.3) with degree Lz = min d; Lv + Lw : Enlarging L increases the storage cost and the arithmetic f g cost of operations involving z. A truncation Lz Lz0 < Lz could be helpful, provided that omitted terms are small. Here we need that z satis…es the assumption7! of Remark 8.2b. Let Z = L(V ) be the space of linear maps of V into V: Another example is the multiplication of an LA (d) Lv (d) operator (Kronecker matrix) A = k=0 Ak Zsym and a tensor v = k=0 vk Vsym: Let the identity be the special element I of Z (replacing e in the2 general description). This guarantees2 Ie = e: Again w := Av is P P min of the form (8.3) with Lw = min d; LA + Lv : Only under the assumption that all maps in U (A) possess f g 1 e as an eigenvector, we have Lw = Lv:

9 Operators, Kronecker Matrices

If V is a matrix space, the corresponding tensor space contains Kronecker matrices. More generally, linear operators on multivariate functions can be described by tensors. In this case, there is an operator A and a vector v as well as the product Av. For completeness we list the relations between (anti)symmetric operators and (anti)symmetric tensors v. The proofs are obvious and therefore omitted. The expression A used below means the application of the permutation  to the tensor A d L(V ) L(V); where L(V ) are the linear maps from V into itself. Concerning dL(V ) L(V) compare2 [9, Propos- ition 3.49]. 

Lemma 9.1 Let V = dV; A : V V a linear map and  a permutation. Then the action of A can be expressed by the action of A: !

1 (A) u =  A  u for all u V: 2

If A is symmetric then (Au) = A(u): If u Vsym

then ( (A))u = (Au): If A : V V is symmetric then A (u) = (Au): 2 S S ! S S

The last statement implies that if A : V V is symmetric, then A : Vsym Vsym and A : Vanti Vanti: ! ! ! 1 The adjoint of A is denoted by A; i.e., Au; v = u; Av : Any permutation satis…es  =  and h i h i (A) = A: In particular, permutations of selfadjoint operators are again selfadjoint.

21 References

[1] H. Abo, N. Vannieuwenhoven: Most secant varieties of tangential varieties to Veronese varieties are nondefective. Trans. Amer. Math. Soc. (to appear) [2] Bach, V., Delle, L. (eds.): Many-Electron Approaches in Physics, Chemistry and Mathematics. Springer, Cham (2014) [3] Beylkin, G., Mohlenkamp, M.J., Pérez, F.: Approximating a wavefunction as an unconstrained sum of Slater determinants. J. Math. Phys. 49, 032,107 (2008) [4] Bohn, B., Griebel, M.: An adaptive sparse grid approach for time series prediction. In: Garcke and Griebel [8], pp. 1–30.

[5] Buczy´nski, J., Landsberg, J.M.: On the third secant variety. J. Algebraic Combin. 40, 475–502 (2014) [6] Cayley, A.: An introductory memoir on quantics. Philos. Trans. R. Soc. Lond. 144 (1854) [7] Comon, P., Golub, G.H., Lim, L.H., Mourrain, B.: Symmetric tensors and symmetric tensor rank. SIAM J. Matrix Anal. Appl. 30, 1254–1279 (2008) [8] Garcke, J., Griebel, M. (eds.): Sparse grids and applications, Lect. Notes Comput. Sci. Eng., vol. 88. Springer, Berlin (2013). (Bonn, May 2011) [9] Hackbusch, W.: Tensor Spaces and Numerical Tensor Calculus, SSCM, vol. 42. Springer, Berlin (2012) [10] Håstad, J.: Tensor rank is NP-complete. J. Algorithms 11, 644–654 (1990) [11] Hitchcock, F.L.: The expression of a tensor or a polyadic as a sum of products. Journal of Mathematics and Physics 6, 164–189 (1927) [12] Legeza, O., Rohwedder, T., Schneider, R., Szalay, S.: Tensor product approximation (DMRG) and coupled cluster method in quantum chemistry. In: Bach and Delle [2], pp. 53–76 [13] Lounesto, P.: Cli¤ord Algebras and Spinors, 2nd edn. Cambridge University Press, Cambridge (2001) [14] Löwdin, P.O.: Quantum theory for many-particle systems. I. Physical interpretations by means of density matrices, natural spin-orbitals, and convergence problems in the method of con…g- urational interaction. Phys. Rev. 97(6), 1474–1489 (1955) [15] Oseledets, I.V.: Tensor-train decomposition. SIAM J. Sci. Comput. 33, 2295–2317 (2011) [16] Seigal, A.: private communication (2016) [17] Valiant, L.G.: The complexity of computing the permanent. Theoret. Comput. Sci. 8, 189–201 (1979)

22