Graphical Calculus for Products and Convolutions
Graphical Calculus for products and convolutions
Filippo M. Miatto Institut Polytechnique de Paris and T´el´ecom ParisTech, LTCI, 46 Rue Barrault, 75013 Paris, France
Graphical calculus is an intuitive visual notation for manipulating tensors and index contractions. Using graphical calculus leads to simple and memorable derivations, and with a bit of practice one can learn to prove complex identities even without the need for pen and paper. This manuscript is meant as a demonstration of the power and flexibility of graphical notation and we advocate exploring the use of graphical calculus in undergraduate courses. In the first part we define the following matrix products in graphical language: dot, tensor, Kronecker, Hadamard, Kathri-Rao and Tracy-Singh. We then use our definitions to prove several known identities in an entirely graphical way. Despite ordinary proofs consist in several lines of quite involved mathematical expressions, graphical calculus is so expressive that after writing an identity in graphical form we can realise by visual inspection that it is in fact true. As an example of the intuitiveness of graphical proofs, we derive two new identities. In the second part we develop a graphical description of convolutions, which is a central ingredient of convolutional neural networks and signal processing. Our single definition includes as special cases the circular discrete convolution and the cross-correlation. We illustrate how convolution can be seen as another type of product and we derive a generalised convolution theorem. We conclude with a quick guide on implementing tensor contractions in python.
I. INTRODUCTION convolution which includes the standard convolution and the cross-correlation as special cases. These operations When we manipulate mathematical expressions (either are extremely common in signal processing and in the in writing or mentally), the behaviour of the typographi- field of machine learning, in particular in convolutional cal symbols can acquire its own physicality. For example, neural networks [2] and signal processing. we might feel that as soon as we wrap a product of various Graphical calculus was introduced in the seventies by terms in a logarithm, the “force” that is binding them to- Penrose [3] in the context of general relativity, it was then gether comes loose, as log(xyz) = log(x)+log(y)+log(z). slowly picked up and/or rediscovered by various authors Even the simple operation of bringing the denominator including Lafont [4], Coecke and Abramsky [5, 6], Grif- of one side of an equation to the numerator of the op- fiths [7], Seilinger [8], Baez [9], Wood [10], Biamonte [11], x posite side ( 2 = y x = 2y) is a physical interpre- Jaffe [12] and others [13, 14]. Because of such diversity tation of the actual operation→ of multiplying both sides of scopes and the scarcity of mainstream adoption, the by the same quantity.Such physical familiarity with sym- notation is not yet standardised. Some authors proceed bolic manipulation comes with time and practice, and vertically, others horizontally. Some right to left, others arguably the more a notion is described by symbols that left to right. Obviously, there is no actual difference, but behave somewhat physically, the easier it is to have such different notations can make it easier or harder to tran- an experience. Graphical calculus is an extreme example sition between regular and graphical notations. In our of such notation: in graphical calculus all the operations case, we opted for a horizontal notation (as it is more consist in connecting “bendable” and “stretchable” wires, similar to the way in which we write) and right to left and one can perform complicated tensor manipulations (as that is how we compose successive matrix multiplica- entirely within this framework. tions). Graphical calculus offers a representation of concepts A few words about our conventions. When we write arXiv:1903.01366v1 [quant-ph] 4 Mar 2019 that complements the standard mathematical notation: tensors with indices (usually roman literals), we use the it is by providing alternative representations of the same convention that repeated indices are implicitly summed concept that we help the recognition networks in the over, to avoid an excessive use of the summation sym- brain of a learner, as recommended in the principles of bol. When we contract a pair of indices we assume that universal design for learning [1]. It is with this student- their dimensions match. In graphical notation, a tensor centered spirit that we present this paper. of rank R is drawn as a shape with R wires, where each In the first part of this work we define six types of ma- wire represents an index. When we connect two wires it trix product (dot, tensor, Hadamard, Kronecker, Kathri- means we are summing over those indices, like so: Aij Rao, Tracy-Singh) in graphical calculus notation. Our and Bjk both have two indices (i.e. we can think of A aim is to make them available in a form that is easy to and B as matrices) which means that in graphical nota- remember, easy to work with and easy to explain to oth- tion they both have two wires. If we join the two wires ers. To stress the power of graphical calculus, we show corresponding to the j indices (assuming they have the two new (to the best of the author’s knowledge) identi- same dimension), we obtain the tensor AijBjk which has ties. In the second part we describe a generalized discrete two leftover indices because we sum over the repeated 2
P j index: j AijBjk (see Fig. 8). Technically it doesn’t iAAAC6HicjVLLSgMxFD0dX7W+qi7dDBbBVZkRQZeiG5cVbSvUUmbStMbOi5mMIMVPEFxZ3PpX/kH9C29iCmrxkTCZk3PPuclN4ieByKTjvBasmdm5+YXiYmlpeWV1rby+0cjiPGW8zuIgTi99L+OBiHhdChnwyyTlXugHvOkPTlS8ecvTTMTRhbxLeDv0+pHoCeZJos5Fx+2UK07V0c2eBq4BFZhWi8tjXKGLGAw5QnBEkIQDeMiot+DCQUJcG0PiUkJCxznuUSJvTipOCo/YAY19mrUMG9Fc5cy0m9EqAX0pOW3skCcmXUpYrWbreK4zK/an3EOdU+3tjv6+yRUSK3FN7F++ifK/PlWLRA+HugZBNSWaUdUxkyXXp6J2bn+qSlKGhDiFuxRPCTPtnJyzrT2Zrl2drafjY61UrJozo83x9mt1PvVQ3Qk9Aff7hU+Dxl7VJXy2Xzk6No+hiC1sY5du/ABHOEUNdVqhjwc8YWTdWI/WyHr+kFoF49nEl2a9vAPZtpd1 1 matter how we orient the drawings of the tensors and
their wires, as long as we keep track of which index they iAAAC6HicjVLLSgMxFD0dX7W+qi7dDBbBVZkRQZeiG5cVbSvUUmbStMbOi5mMIMVPEFxZ3PpX/kH9C29iCmrxkTCZk3PPuclN4ieByKTjvBasmdm5+YXiYmlpeWV1rby+0cjiPGW8zuIgTi99L+OBiHhdChnwyyTlXugHvOkPTlS8ecvTTMTRhbxLeDv0+pHoCeZJos5Fx+2UK07V0c2eBq4BFZhWi8tjXKGLGAw5QnBEkIQDeMiot+DCQUJcG0PiUkJCxznuUSJvTipOCo/YAY19mrUMG9Fc5cy0m9EqAX0pOW3skCcmXUpYrWbreK4zK/an3EOdU+3tjv6+yRUSK3FN7F++ifK/PlWLRA+HugZBNSWaUdUxkyXXp6J2bn+qSlKGhDiFuxRPCTPtnJyzrT2Zrl2drafjY61UrJozo83x9mt1PvVQ3Qk9Aff7hU+Dxl7VJXy2Xzk6No+hiC1sY5du/ABHOEUNdVqhjwc8YWTdWI/WyHr+kFoF49nEl2a9vAPZtpd1 1 iAAAC6HicjVLLSgMxFD0dX7W+qi7dDBbBVZkpgi6LblxWtA+opcykaY2dFzMZoRQ/QXBlcetf+Qf1L7yJU1CLj4TJnJx7zk1uEjfyRCIt6zVnLCwuLa/kVwtr6xubW8XtnUYSpjHjdRZ6YdxynYR7IuB1KaTHW1HMHd/1eNMdnql4847HiQiDKzmKeMd3BoHoC+ZIoi5Ft9ItlqyypZs5D+wMlJC1Wlic4ho9hGBI4YMjgCTswUFCvQ0bFiLiOhgTFxMSOs5xjwJ5U1JxUjjEDmkc0KydsQHNVc5Euxmt4tEXk9PEAXlC0sWE1Wqmjqc6s2J/yj3WOdXeRvR3s1w+sRI3xP7lmyn/61O1SPRxomsQVFOkGVUdy7Kk+lTUzs1PVUnKEBGncI/iMWGmnbNzNrUn0bWrs3V0fKqVilVzlmlTvP1anUvdV3dCT8D+fuHzoFEp24QvjkrV0+wx5LGHfRzSjR+jinPUUKcVBnjAEybGrfFoTIznD6mRyzy7+NKMl3fcOJd2 2 correspond to. However, in order to help transition to regular notation, we orient row wires toward the left and FIG. 2. We can use multiple copies of the rank-1 Kronecker column wires toward the right. tensor to construct constant tensors whose entries have all value 1. In this figure we have a constant vector with all ones (on top) and a constant matrix with all ones (on the bottom). II. TWO MEDIATING TENSORS
mensions) Kronecker tensors yields other Kronecker ten- In order to compute the various products that we are sors, e.g. δ δ δ = δ . The Kronecker tensor going do describe, we will need the help of the Kronecker i1i2i3i4 i2i3 i4 i1 can mediate a dot product: A B is a tensor with four and the vectorization tensors, which are two “mediating” ij kl indices, but A B δ is equivalent to the dot product tensors that enable such products. In the following sub- ij kl jk of the two matrices A and B. We can also generalise it sections we will see them separately and get a feeling for to act on more than two matrices (or tensors), such as what they do, both at the level of indexed notation and A B C δ . The Kronecker tensor can extract the at the level of graphical notation. ij kl mn ikm diagonal of a matrix: diag(A)k = Aijδijk, create a diag- onal matrix from a vector: diag(a)jk = aiδijk or erase all A. The Kronecker tensor of the off-diagonal elements of a matrix: Aijδijkl. The Kronecker tensor can be used to compute the trace of a matrix: Tr(A) = A δ , or the partial trace of a higher The D-dimensional, rank-N Kronecker tensor is a gen- ij ij order tensor: Tr1(A)kl = Aijklδij, which is especially eralization of the familiar Kronecker delta δij. It is de- useful in quantum information. fined as: ( 1 if i1 = i2 = = iN diag(A) Tr(A)
AAAC93icjVLLSgMxFD0dX/U96tLNYBF0I1MRdFl147KCfYAWyUzTGpwXmUyhlP6J4Mri1s/xD+pfeBOnoBYfCZM5Ofecm9wkXhKIVLnua8GamZ2bXyguLi2vrK6t2xub9TTOpM9rfhzEsumxlAci4jUlVMCbieQs9ALe8O7PdbzR4zIVcXSl+glvhawbiY7wmSLq1rZvQqbuZDhoC9Yd7p3u39ol98A1zZkG5RyUkLdqbI9xgzZi+MgQgiOCIhyAIaV+jTJcJMS1MCBOEhImzjHEEnkzUnFSMGLvaezS7DpnI5rrnKlx+7RKQJ8kp4Nd8sSkk4T1ao6JZyazZn/KPTA59d769PfyXCGxCnfE/uWbKP/r07UodHBiahBUU2IYXZ2fZ8nMqeidO5+qUpQhIU7jNsUlYd84J+fsGE9qatdny0x8bJSa1XM/12Z4+7U6j3qo74SeQPn7hU+D+uFBmfDlUalylj+GIraxgz268WNUcIEqarRCDw94wsjqW4/WyHr+kFqF3LOFL816eQcz9p0z AAAC83icjVLLSgMxFD0dX7W+qi7dDBahbsqMCLqsunFZoS9oq8xM03bovMhkhFL6H4Iri1s/yD+of+FNnIJafCRM5uTcc25yk9iR58bCMF4z2tLyyupadj23sbm1vZPf3avHYcIdVnNCL+RN24qZ5wasJlzhsWbEmeXbHmvYwysZb9wzHrthUBWjiHV8qx+4PdexBFG3bd8SA+6Pq3xSvDi+yxeMkqGavgjMFBSQtkqYn6GNLkI4SOCDIYAg7MFCTL0FEwYi4joYE8cJuSrOMEGOvAmpGCksYoc09mnWStmA5jJnrNwOreLRx8mp44g8Iek4YbmaruKJyizZn3KPVU65txH97TSXT6zAgNi/fHPlf32yFoEezlUNLtUUKUZW56RZEnUqcuf6p6oEZYiIk7hLcU7YUc75OevKE6va5dlaKj5TSsnKuZNqE7z9Wp1N3Zd3Qk/A/H7hi6B+UjIJ35wWypfpY8jiAIco0o2foYxrVFCjFTge8ISplmiP2lR7/pBqmdSzjy9Ne3kHPhScHw==
AAAC5nicjVLLSgMxFD0dX7W+qi7dDBbBVZmKoMuqG5ct2FaoRWbStB06LzIZoRS/QHBlcetv+Qf1L7yJKajFR8JkTs495yY3iZcEfiod5zVnLSwuLa/kVwtr6xubW8XtnWYaZ4LxBouDWFx7bsoDP+IN6cuAXyeCu6EX8JY3vFDx1h0XqR9HV3KU8E7o9iO/5zNXElU/uy2WnLKjmz0PKgaUYFotLk5xgy5iMGQIwRFBEg7gIqXeRgUOEuI6GBMnCPk6znGPAnkzUnFSuMQOaezTrG3YiOYqZ6rdjFYJ6BPktHFAnph0grBazdbxTGdW7E+5xzqn2tuI/p7JFRIrMSD2L99M+V+fqkWih1Ndg081JZpR1TGTJdOnonZuf6pKUoaEOIW7FBeEmXbOztnWnlTXrs7W1fGpVipWzZnRZnj7tTqPeqjuhJ5A5fuFz4PmUblCuH5cqp6bx5DHHvZxSDd+giouUUNDr/6AJ0ysgfVoTaznD6mVM55dfGnWyzu/MZap δ = ··· (1) AAAAC5nicjVLLSgMxFD0dX7W+qi7dDBbBVZmKoMuqG5ct2FaoRWbStB06LzIZoRS/QHBlcetv+Qf1L7yJKajFR8JkTs495yY3iZcEfiod5zVnLSwuLa/kVwtr6xubW8XtnWYaZ4LxBouDWFx7bsoDP+IN6cuAXyeCu6EX8JY3vFDx1h0XqR9HV3KU8E7o9iO/5zNXElU/uy2WnLKjmz0PKgaUYFotLk5xgy5iMGQIwRFBEg7gIqXeRgUOEuI6GBMnCPk6znGPAnkzUnFSuMQOaezTrG3YiOYqZ6rdjFYJ6BPktHFAnph0grBazdbxTGdW7E+5xzqn2tuI/p7JFRIrMSD2L99M+V+fqkWih1Ndg081JZpR1TGTJdOnonZuf6pKUoaEOIW7FBeEmXbOztnWnlTXrs7W1fGpVipWzZnRZnj7tTqPeqjuhJ5A5fuFz4PmUblCuH5cqp6bx5DHHvZxSDd+giouUUNDr/6AJ0ysgfVoTaznD6mVM55dfGnWyzu/MZap A i1i2...iN 0 otherwise where each index has values in 0 ...D 1. As special diag(a) Tr (A)
AAADAHicjVLLSgMxFD0dX7W+qq7EzWARdCNTEXQpunFZwT6gSslM0zY4LzIZQUoR/BfBlcWtf+If1L/wJk5BLT4SZubcc+85mZvEjX2RKMd5zVlT0zOzc/n5wsLi0vJKcXWtlkSp9HjVi/xINlyWcF+EvKqE8nkjlpwFrs/r7vWpztdvuExEFF6o25hfBawbio7wmCKqVdy4DJjqyaDfFqw72DGR2+mzwW6rWHL2HDPsSVDOQAnZqETFES7RRgQPKQJwhFCEfTAkNJsow0FM3BX6xElCwuQ5BiiQNqUqThWM2Gt6dylqZmxIsfZMjNqjVXx6JCltbJMmojpJWK9mm3xqnDX7k3ffeOp/u6Wvm3kFxCr0iP1LN678r073otDBkelBUE+xYXR3XuaSml3Rf25/6kqRQ0ycxm3KS8KeUY732TaaxPSu95aZ/MhUalbHXlab4u3X7lyagT4TugLl7wc+CWr7e2XC5wel45PsMuSxiS3s0Ikf4hhnqKBKK9zhAU8YWvfWozW0nj9KrVymWceXYb28A5syoXM= AAAC93icjVLLSsNAFD2Nr1pfUZdugkWom5KIoMuqG5cV+oJaSpJOazAvJpNCKf0TwZXFrZ/jH9S/8M6Yglp8zJDJmXPPuTN3ZpzY9xJhmq85bWl5ZXUtv17Y2Nza3tF39xpJlHKX1d3Ij3jLsRPmeyGrC0/4rBVzZgeOz5rO/ZWMN4eMJ14U1sQoZp3AHoRe33NtQVRX128DW9zxYFzjk65Vujju6kWzbKpmLAIrA0VkrRrpM9yihwguUgRgCCEI+7CRUG/DgomYuA7GxHFCnoozTFAgb0oqRgqb2HsaBzRrZ2xIc5kzUW6XVvHp4+Q0cESeiHScsFzNUPFUZZbsT7nHKqfc24j+TpYrIFbgjti/fHPlf32yFoE+zlUNHtUUK0ZW52ZZUnUqcufGp6oEZYiJk7hHcU7YVc75ORvKk6ja5dnaKj5TSsnKuZtpU7z9Wp1DPZB3Qk/A+n7hi6BxUrYI35wWK5fZY8jjAIco0Y2foYJrVFGnFYZ4wBOm2kh71Kba84dUy2WefXxp2ss7lTec9A== AAAC73icjVLLSgMxFD0dX7W+qi7dDBbBVZkRQZdFNy4r2Ae2Iplp2g6dl5mMUEr/QnBlcesn+Qf1L7yJU1CLj4TJnJx7zk1uEif2vURa1mvOWFhcWl7JrxbW1jc2t4rbO/UkSoXLa27kR6LpsIT7Xshr0pM+b8aCs8DxecMZnKt4456LxIvCKzmM+U3AeqHX9VwmibpuB0z2ne6IjW+LJats6WbOAzsDJWStGhWnaKODCC5SBOAIIQn7YEiot2DDQkzcDUbECUKejnOMUSBvSipOCkbsgMYezVoZG9Jc5Uy026VVfPoEOU0ckCcinSCsVjN1PNWZFftT7pHOqfY2pL+T5QqIlegT+5dvpvyvT9Ui0cWprsGjmmLNqOrcLEuqT0Xt3PxUlaQMMXEKdyguCLvaOTtnU3sSXbs6W6bjU61UrJq7mTbF26/VOdQDdSf0BOzvFz4P6kdlm/Dlcalylj2GPPawj0O68RNUcIEqarRCiAc8YWLcGY/GxHj+kBq5zLOLL814eQft+Jrp 1 − a AAAAC5nicjVLLSgMxFD0dX7W+qi7dDBbBVZmKoMuqG5ct2FaoRWbStB06LzIZoRS/QHBlcetv+Qf1L7yJKajFR8JkTs495yY3iZcEfiod5zVnLSwuLa/kVwtr6xubW8XtnWYaZ4LxBouDWFx7bsoDP+IN6cuAXyeCu6EX8JY3vFDx1h0XqR9HV3KU8E7o9iO/5zNXElU/uy2WnLKjmz0PKgaUYFotLk5xgy5iMGQIwRFBEg7gIqXeRgUOEuI6GBMnCPk6znGPAnkzUnFSuMQOaezTrG3YiOYqZ6rdjFYJ6BPktHFAnph0grBazdbxTGdW7E+5xzqn2tuI/p7JFRIrMSD2L99M+V+fqkWih1Ndg081JZpR1TGTJdOnonZuf6pKUoaEOIW7FBeEmXbOztnWnlTXrs7W1fGpVipWzZnRZnj7tTqPeqjuhJ5A5fuFz4PmUblCuH5cqp6bx5DHHvZxSDd+giouUUNDr/6AJ0ysgfVoTaznD6mVM55dfGnWyzu/MZap cases, δij is equivalent to the identity matrix and it is also equivalent to an (unnormalized) Bell state, well known in quantum information [10]. Similarly, δijk is equivalent to diag(diag(A)) Tr (A) AAAAC5nicjVLLSgMxFD0dX7W+qi7dDBbBVZmKoMuqG5ct2FaoRWbStB06LzIZoRS/QHBlcetv+Qf1L7yJKajFR8JkTs495yY3iZcEfiod5zVnLSwuLa/kVwtr6xubW8XtnWYaZ4LxBouDWFx7bsoDP+IN6cuAXyeCu6EX8JY3vFDx1h0XqR9HV3KU8E7o9iO/5zNXElU/uy2WnLKjmz0PKgaUYFotLk5xgy5iMGQIwRFBEg7gIqXeRgUOEuI6GBMnCPk6znGPAnkzUnFSuMQOaezTrG3YiOYqZ6rdjFYJ6BPktHFAnph0grBazdbxTGdW7E+5xzqn2tuI/p7JFRIrMSD2L99M+V+fqkWih1Ndg081JZpR1TGTJdOnonZuf6pKUoaEOIW7FBeEmXbOztnWnlTXrs7W1fGpVipWzZnRZnj7tTqPeqjuhJ5A5fuFz4PmUblCuH5cqp6bx5DHHvZxSDd+giouUUNDr/6AJ0ysgfVoTaznD6mVM55dfGnWyzu/MZap AAADBnicjVLLSgMxFD0dX7W+qi5dOFgE3chUBF1W3bisYB+gIplprMF5kckIpXQp+C+CK4tbf8M/qH/hTZyCtfhImJlzz73nZG4SN/ZFohznLWdNTE5Nz+RnC3PzC4tLxeWVehKl0uM1L/Ij2XRZwn0R8poSyufNWHIWuD5vuLfHOt+44zIRUXimOjG/DFg7FNfCY4qoq+L6RcDUjQy6LcHava3R6HB7+6pYcnYcM+xxUM5ACdmoRsUBLtBCBA8pAnCEUIR9MCQ0z1GGg5i4S3SJk4SEyXP0UCBtSlWcKhixt/RuU3SesSHF2jMxao9W8emRpLSxSZqI6iRhvZpt8qlx1uxP3l3jqf+tQ1838wqIVbgh9i/dsPK/Ot2LwjUOTA+CeooNo7vzMpfU7Ir+c/tLV4ocYuI0blFeEvaMcrjPttEkpne9t8zkB6ZSszr2stoU779259IM9JnQFSh/P/BxUN/dKRM+3StVjrLLkMcaNrBFJ76PCk5QRY1WuMcjntG3Hqwnq2+9fJZauUyzipFhvX4Ab3KjjA== AAAC93icjVLLSgMxFD0d3+9Rl24Gi6CbMiOCLqtuXCq0VahSMjFtg/MikymU4p8IrhS3fo5/oH/hTRzBBz4SJnNy7jk3uUnCLJK59v2nijM2PjE5NT0zOze/sLjkLq+08rRQXDR5GqXqLGS5iGQimlrqSJxlSrA4jMRpeHVo4qcDoXKZJg09zMRFzHqJ7ErONFEd1z2Pme6reNRQ153tzf2tjlv1a75t3ncQlKCKsh2n7jPOcYkUHAViCCTQhCMw5NTbCOAjI+4CI+IUIWnjAteYJW9BKkEKRuwVjT2atUs2obnJmVs3p1Ui+hQ5PWyQJyWdImxW82y8sJkN+1Pukc1p9jakf1jmionV6BP7l+9d+V+fqUWjiz1bg6SaMsuY6niZpbCnYnbufahKU4aMOIMvKa4Ic+t8P2fPenJbuzlbZuPPVmlYM+eltsDLr9WF1GNzJ/QEgq8X/h20tmsB4ZOdav2gfAzTWMM6NunGd1HHEY7RpBUGuMEd7p2hc+vcOw9vUqdSelbxqTmPr5e8nPU= 2 an (unnormalized) GHZ state [11]. In graphical notation AAAAC5nicjVLLSgMxFD0dX7W+qi7dDBbBVZmKoMuqG5ct2FaoRWbStB06LzIZoRS/QHBlcetv+Qf1L7yJKajFR8JkTs495yY3iZcEfiod5zVnLSwuLa/kVwtr6xubW8XtnWYaZ4LxBouDWFx7bsoDP+IN6cuAXyeCu6EX8JY3vFDx1h0XqR9HV3KU8E7o9iO/5zNXElU/uy2WnLKjmz0PKgaUYFotLk5xgy5iMGQIwRFBEg7gIqXeRgUOEuI6GBMnCPk6znGPAnkzUnFSuMQOaezTrG3YiOYqZ6rdjFYJ6BPktHFAnph0grBazdbxTGdW7E+5xzqn2tuI/p7JFRIrMSD2L99M+V+fqkWih1Ndg081JZpR1TGTJdOnonZuf6pKUoaEOIW7FBeEmXbOztnWnlTXrs7W1fGpVipWzZnRZnj7tTqPeqjuhJ5A5fuFz4PmUblCuH5cqp6bx5DHHvZxSDd+giouUUNDr/6AJ0ysgfVoTaznD6mVM55dfGnWyzu/MZap we indicate such tensors as a small black circle with N wires. In case we have only two wires we can omit the FIG. 3. The Kronecker tensor can extract the diagonal of a black circle (see Fig. 1). Here are a few properties of the matrix A, produce a diagonal matrix from a vector a, erase the off-diagonal elements of A, compute traces and partial
iAAAC6HicjVLLSgMxFD0dX7W+qi7dDBbBVZkpgi6LblxWtA+opcykaY2dFzMZoRQ/QXBlcetf+Qf1L7yJU1CLj4TJnJx7zk1uEjfyRCIt6zVnLCwuLa/kVwtr6xubW8XtnUYSpjHjdRZ6YdxynYR7IuB1KaTHW1HMHd/1eNMdnql4847HiQiDKzmKeMd3BoHoC+ZIoi5Ft9ItlqyypZs5D+wMlJC1Wlic4ho9hGBI4YMjgCTswUFCvQ0bFiLiOhgTFxMSOs5xjwJ5U1JxUjjEDmkc0KydsQHNVc5Euxmt4tEXk9PEAXlC0sWE1Wqmjqc6s2J/yj3WOdXeRvR3s1w+sRI3xP7lmyn/61O1SPRxomsQVFOkGVUdy7Kk+lTUzs1PVUnKEBGncI/iMWGmnbNzNrUn0bWrs3V0fKqVilVzlmlTvP1anUvdV3dCT8D+fuHzoFEp24QvjkrV0+wx5LGHfRzSjR+jinPUUKcVBnjAEybGrfFoTIznD6mRyzy7+NKMl3fcOJd2 2 traces. Note that we are indicating two operations with the
iAAAC6HicjVLLSgMxFD0dX7W+qi7dDBbBVZlRQZdFNy4r2gfUUmbStMbOi5mMUIqfILiyuPWv/IP6F97EKajFR8JkTs495yY3iRt5IpGW9Zoz5uYXFpfyy4WV1bX1jeLmVj0J05jxGgu9MG66TsI9EfCaFNLjzSjmju96vOEOzlS8ccfjRITBlRxGvO07/UD0BHMkUZeic9gplqyypZs5C+wMlJC1alic4BpdhGBI4YMjgCTswUFCvQUbFiLi2hgRFxMSOs5xjwJ5U1JxUjjEDmjs06yVsQHNVc5Euxmt4tEXk9PEHnlC0sWE1Wqmjqc6s2J/yj3SOdXehvR3s1w+sRI3xP7lmyr/61O1SPRwomsQVFOkGVUdy7Kk+lTUzs1PVUnKEBGncJfiMWGmndNzNrUn0bWrs3V0fKKVilVzlmlTvP1anUvdV3dCT8D+fuGzoH5QtglfHJUqp9ljyGMHu9inGz9GBeeookYr9PGAJ4yNW+PRGBvPH1Ijl3m28aUZL+/eupd3 3 iAAAC6HicjVLLSgMxFD0dX7W+qi7dDBbBVZkpgi6LblxWtA+opcykaY2dFzMZoRQ/QXBlcetf+Qf1L7yJU1CLj4TJnJx7zk1uEjfyRCIt6zVnLCwuLa/kVwtr6xubW8XtnUYSpjHjdRZ6YdxynYR7IuB1KaTHW1HMHd/1eNMdnql4847HiQiDKzmKeMd3BoHoC+ZIoi5Ft9ItlqyypZs5D+wMlJC1Wlic4ho9hGBI4YMjgCTswUFCvQ0bFiLiOhgTFxMSOs5xjwJ5U1JxUjjEDmkc0KydsQHNVc5Euxmt4tEXk9PEAXlC0sWE1Wqmjqc6s2J/yj3WOdXeRvR3s1w+sRI3xP7lmyn/61O1SPRxomsQVFOkGVUdy7Kk+lTUzs1PVUnKEBGncI/iMWGmnbNzNrUn0bWrs3V0fKqVilVzlmlTvP1anUvdV3dCT8D+fuHzoFEp24QvjkrV0+wx5LGHfRzSjR+jinPUUKcVBnjAEybGrfFoTIznD6mRyzy7+NKMl3fcOJd2 2 same name diag(·) as their meaning is clear from the context (diag of a matrix yields a vector, and diag of a vector yields a matrix). iAAAC6HicjVLLSgMxFD0dX7W+qi7dDBbBVZkRQZeiG5cVbSvUUmbStMbOi5mMIMVPEFxZ3PpX/kH9C29iCmrxkTCZk3PPuclN4ieByKTjvBasmdm5+YXiYmlpeWV1rby+0cjiPGW8zuIgTi99L+OBiHhdChnwyyTlXugHvOkPTlS8ecvTTMTRhbxLeDv0+pHoCeZJos5Fx+2UK07V0c2eBq4BFZhWi8tjXKGLGAw5QnBEkIQDeMiot+DCQUJcG0PiUkJCxznuUSJvTipOCo/YAY19mrUMG9Fc5cy0m9EqAX0pOW3skCcmXUpYrWbreK4zK/an3EOdU+3tjv6+yRUSK3FN7F++ifK/PlWLRA+HugZBNSWaUdUxkyXXp6J2bn+qSlKGhDiFuxRPCTPtnJyzrT2Zrl2drafjY61UrJozo83x9mt1PvVQ3Qk9Aff7hU+Dxl7VJXy2Xzk6No+hiC1sY5du/ABHOEUNdVqhjwc8YWTdWI/WyHr+kFoF49nEl2a9vAPZtpd1 1 iAAAC6HicjVLLSgMxFD0dX7W+qi7dDBbBVZkpgi6LblxWtA+opcykaY2dFzMZoRQ/QXBlcetf+Qf1L7yJU1CLj4TJnJx7zk1uEjfyRCIt6zVnLCwuLa/kVwtr6xubW8XtnUYSpjHjdRZ6YdxynYR7IuB1KaTHW1HMHd/1eNMdnql4847HiQiDKzmKeMd3BoHoC+ZIoi5Ft9ItlqyypZs5D+wMlJC1Wlic4ho9hGBI4YMjgCTswUFCvQ0bFiLiOhgTFxMSOs5xjwJ5U1JxUjjEDmkc0KydsQHNVc5Euxmt4tEXk9PEAXlC0sWE1Wqmjqc6s2J/yj3WOdXeRvR3s1w+sRI3xP7lmyn/61O1SPRxomsQVFOkGVUdy7Kk+lTUzs1PVUnKEBGncI/iMWGmnbNzNrUn0bWrs3V0fKqVilVzlmlTvP1anUvdV3dCT8D+fuHzoFEp24QvjkrV0+wx5LGHfRzSjR+jinPUUKcVBnjAEybGrfFoTIznD6mRyzy7+NKMl3fcOJd2 2 iAAAC6HicjVLLSgMxFD0dX7W+qi7dDBbBVZkRQZeiG5cVbSvUUmbStMbOi5mMIMVPEFxZ3PpX/kH9C29iCmrxkTCZk3PPuclN4ieByKTjvBasmdm5+YXiYmlpeWV1rby+0cjiPGW8zuIgTi99L+OBiHhdChnwyyTlXugHvOkPTlS8ecvTTMTRhbxLeDv0+pHoCeZJos5Fx+2UK07V0c2eBq4BFZhWi8tjXKGLGAw5QnBEkIQDeMiot+DCQUJcG0PiUkJCxznuUSJvTipOCo/YAY19mrUMG9Fc5cy0m9EqAX0pOW3skCcmXUpYrWbreK4zK/an3EOdU+3tjv6+yRUSK3FN7F++ifK/PlWLRA+HugZBNSWaUdUxkyXXp6J2bn+qSlKGhDiFuxRPCTPtnJyzrT2Zrl2drafjY61UrJozo83x9mt1PvVQ3Qk9Aff7hU+Dxl7VJXy2Xzk6No+hiC1sY5du/ABHOEUNdVqhjwc8YWTdWI/WyHr+kFoF49nEl2a9vAPZtpd1 1
iAAAC6HicjVLLSgMxFD0dX7W+qi7dDBbBVZmRgi6LblxWtA+opcykaY2dFzMZoRQ/QXBlcetf+Qf1L7yJU1CLj4TJnJx7zk1uEjfyRCIt6zVnLCwuLa/kVwtr6xubW8XtnUYSpjHjdRZ6YdxynYR7IuB1KaTHW1HMHd/1eNMdnql4847HiQiDKzmKeMd3BoHoC+ZIoi5Ft9ItlqyypZs5D+wMlJC1Wlic4ho9hGBI4YMjgCTswUFCvQ0bFiLiOhgTFxMSOs5xjwJ5U1JxUjjEDmkc0KydsQHNVc5Euxmt4tEXk9PEAXlC0sWE1Wqmjqc6s2J/yj3WOdXeRvR3s1w+sRI3xP7lmyn/61O1SPRxomsQVFOkGVUdy7Kk+lTUzs1PVUnKEBGncI/iMWGmnbNzNrUn0bWrs3V0fKqVilVzlmlTvP1anUvdV3dCT8D+fuHzoHFUtglfVErV0+wx5LGHfRzSjR+jinPUUKcVBnjAEybGrfFoTIznD6mRyzy7+NKMl3fhPJd4 4 iAAAC6HicjVLLSgMxFD0dX7W+qi7dDBbBVZkRQZeiG5cVbSvUUmbStMbOi5mMIMVPEFxZ3PpX/kH9C29iCmrxkTCZk3PPuclN4ieByKTjvBasmdm5+YXiYmlpeWV1rby+0cjiPGW8zuIgTi99L+OBiHhdChnwyyTlXugHvOkPTlS8ecvTTMTRhbxLeDv0+pHoCeZJos5Fx+2UK07V0c2eBq4BFZhWi8tjXKGLGAw5QnBEkIQDeMiot+DCQUJcG0PiUkJCxznuUSJvTipOCo/YAY19mrUMG9Fc5cy0m9EqAX0pOW3skCcmXUpYrWbreK4zK/an3EOdU+3tjv6+yRUSK3FN7F++ifK/PlWLRA+HugZBNSWaUdUxkyXXp6J2bn+qSlKGhDiFuxRPCTPtnJyzrT2Zrl2drafjY61UrJozo83x9mt1PvVQ3Qk9Aff7hU+Dxl7VJXy2Xzk6No+hiC1sY5du/ABHOEUNdVqhjwc8YWTdWI/WyHr+kFoF49nEl2a9vAPZtpd1 1
iAAAC6HicjVLLSgMxFD0dX7W+qi7dDBbBVZlRQZdFNy4r2gfUUmbStMbOi5mMUIqfILiyuPWv/IP6F97EKajFR8JkTs495yY3iRt5IpGW9Zoz5uYXFpfyy4WV1bX1jeLmVj0J05jxGgu9MG66TsI9EfCaFNLjzSjmju96vOEOzlS8ccfjRITBlRxGvO07/UD0BHMkUZeic9gplqyypZs5C+wMlJC1alic4BpdhGBI4YMjgCTswUFCvQUbFiLi2hgRFxMSOs5xjwJ5U1JxUjjEDmjs06yVsQHNVc5Euxmt4tEXk9PEHnlC0sWE1Wqmjqc6s2J/yj3SOdXehvR3s1w+sRI3xP7lmyr/61O1SPRwomsQVFOkGVUdy7Kk+lTUzs1PVUnKEBGncJfiMWGmndNzNrUn0bWrs3V0fKKVilVzlmlTvP1anUvdV3dCT8D+fuGzoH5QtglfHJUqp9ljyGMHu9inGz9GBeeookYr9PGAJ4yNW+PRGBvPH1Ijl3m28aUZL+/eupd3 3
FIG. 1. The Kronecker tensor is a rank-N Kronecker delta, B. The vectorization tensor which is 1 only if all of the indices have the same value and 0 otherwise. Here the rank-2, rank-3 and rank-4 examples. The vectorization tensor mediates the operation of “se- rializing” multiple indices into one. For example, if we Kronecker tensor. Every D-dimensional Kronecker ten- have two indices i 0,...,I 1 and j 0,...,J 1 , sor, independently of its rank (i.e. the number of wires) we can combine them∈ { into a− single} index∈ { m = i −+ jI} contains D ones and the rest of its values are zeros. This with m 0,...,IJ 1 , and we can always retrieve i ∈ { − } is true also for the rank-1 Kronecker tensor δi, which and j from m if we know I and J. What happens to therefore is the vector with all ones. Similarly, the ten- a matrix Aij that is vectorized is that its columns (or sor product δi1 δi2 . . . δiN is the constant tensor whose all rows, depending on how we choose to perform the con- entries are ones (in this case we can have more than D traction) are concatenated in a vector so that all of the ones, because the tensor product of deltas is not a delta). entries are now indexed by m. The generalized index for- The contraction of any compatible (in terms of index di- mula for N indices i1, i2, . . . , iN with values in the ranges 3
I1,I2,...,IN is The Kronecker and vectorization tensors can be com- posed together and if the dimensions of the wires are N j 1 ! X Y− compatible, they enjoy of the property in Fig. 6. See also m = ij I` (2) Fig. 7 for a an example of a special case. j=1 `=1
= i1 + i2I1 + i3I1I2 + + iN I1I2 ...IN 1 (3) (1) (1) − i i ··· AAAC7nicjVLLSgMxFD0dX7W+qi7dDBahbsqMCLosunFZwT6k1jKTphqcFzMZoZR+heDK4tZf8g/qX3gTp6AWHwmTOTn3nJvcJG7kiURa1mvOmJtfWFzKLxdWVtfWN4qbW40kTGPG6yz0wrjlOgn3RMDrUkiPt6KYO77r8aZ7d6rizXseJyIMLuQg4h3fuQlEXzBHEnUpuvb1sGzvj7rFklWxdDNngZ2BErJWC4sTXKGHEAwpfHAEkIQ9OEiot2HDQkRcB0PiYkJCxzlGKJA3JRUnhUPsHY03NGtnbEBzlTPRbkarePTF5DSxR56QdDFhtZqp46nOrNifcg91TrW3Af3dLJdPrMQtsX/5psr/+lQtEn0c6xoE1RRpRlXHsiypPhW1c/NTVZIyRMQp3KN4TJhp5/ScTe1JdO3qbB0dn2ilYtWcZdoUb79W51L31Z3QE7C/X/gsaBxUbMLnh6XqSfYY8tjBLsp040eo4gw11GkFHw94wtiIjEdjbDx/SI1c5tnGl2a8vANn3ZmJ 1 AAAC7nicjVLLSgMxFD0dX7W+qi7dDBahbsqMCLosunFZwT6k1jKTphqcFzMZoZR+heDK4tZf8g/qX3gTp6AWHwmTOTn3nJvcJG7kiURa1mvOmJtfWFzKLxdWVtfWN4qbW40kTGPG6yz0wrjlOgn3RMDrUkiPt6KYO77r8aZ7d6rizXseJyIMLuQg4h3fuQlEXzBHEnUpuvb1sGzvj7rFklWxdDNngZ2BErJWC4sTXKGHEAwpfHAEkIQ9OEiot2HDQkRcB0PiYkJCxzlGKJA3JRUnhUPsHY03NGtnbEBzlTPRbkarePTF5DSxR56QdDFhtZqp46nOrNifcg91TrW3Af3dLJdPrMQtsX/5psr/+lQtEn0c6xoE1RRpRlXHsiypPhW1c/NTVZIyRMQp3KN4TJhp5/ScTe1JdO3qbB0dn2ilYtWcZdoUb79W51L31Z3QE7C/X/gsaBxUbMLnh6XqSfYY8tjBLsp040eo4gw11GkFHw94wtiIjEdjbDx/SI1c5tnGl2a8vANn3ZmJ 1 We call it vectorization tensor because it flattens all of i(1) M wires i(N) N wires
AAAC7nicjVLLSgMxFD0dX7W+qi7dDBahbsqMCLosunElFexDai0z06jBeTGTEUrpVwiuLG79Jf+g/oU3MQW1+EiYzMm559zkJnFjn6fCsl5zxszs3PxCfrGwtLyyulZc32ikUZZ4rO5FfpS0XCdlPg9ZXXDhs1acMCdwfdZ0745lvHnPkpRH4bnox6wTODchv+aeI4i64F37alA+3R12iyWrYqlmTgNbgxJ0q0XFMS7RQwQPGQIwhBCEfThIqbdhw0JMXAcD4hJCXMUZhiiQNyMVI4VD7B2NNzRrazakucyZKrdHq/j0JeQ0sUOeiHQJYbmaqeKZyizZn3IPVE65tz79XZ0rIFbglti/fBPlf32yFoFrHKoaONUUK0ZW5+ksmToVuXPzU1WCMsTESdyjeELYU87JOZvKk6ra5dk6Kj5WSsnKuae1Gd5+rc6lHsg7oSdgf7/wadDYq9iEz/ZL1SP9GPLYwjbKdOMHqOIENdRphQAPeMLIiI1HY2Q8f0iNnPZs4kszXt4BsNGZpg== the wires that it is attached to into a single one (like a AAAC7nicjVLLSgMxFD0dX7W+qi7dDBahbsqMCLosunEjVLAPqbXMTKMG58VMRiilXyG4srj1l/yD+hfexBTU4iNhMifnnnOTm8SNfZ4Ky3rNGTOzc/ML+cXC0vLK6lpxfaORRlnisboX+VHScp2U+TxkdcGFz1pxwpzA9VnTvTuW8eY9S1IeheeiH7NO4NyE/Jp7jiDqgndPrwZle3fYLZasiqWaOQ1sDUrQrRYVx7hEDxE8ZAjAEEIQ9uEgpd6GDQsxcR0MiEsIcRVnGKJA3oxUjBQOsXc03tCsrdmQ5jJnqtwereLTl5DTxA55ItIlhOVqpopnKrNkf8o9UDnl3vr0d3WugFiBW2L/8k2U//XJWgSucahq4FRTrBhZnaezZOpU5M7NT1UJyhATJ3GP4glhTzkn52wqT6pql2frqPhYKSUr557WZnj7tTqXeiDvhJ6A/f3Cp0Fjr2ITPtsvVY/0Y8hjC9so040foIoT1FCnFQI84AkjIzYejZHx/CE1ctqziS/NeHkHrr2ZpQ== M 1 vector). We indicate it as γi1i2...,m, notice the comma between the indices being vectorized and the new index. =AAAC5nicjVLLSgMxFD0dX7W+qi7dDBbBVZmKoBuh6MZlC7YVapGZadqGzotMRijFLxBcWdz6W/5B/QtvYgpq8ZEwmZNzz7nJTeIlAU+l47zmrIXFpeWV/GphbX1jc6u4vdNM40z4rOHHQSyuPTdlAY9YQ3IZsOtEMDf0Atbyhhcq3rpjIuVxdCVHCeuEbj/iPe67kqj62W2x5JQd3ex5UDGgBNNqcXGKG3QRw0eGEAwRJOEALlLqbVTgICGugzFxghDXcYZ7FMibkYqRwiV2SGOfZm3DRjRXOVPt9mmVgD5BThsH5IlJJwir1Wwdz3Rmxf6Ue6xzqr2N6O+ZXCGxEgNi//LNlP/1qVokejjVNXCqKdGMqs43WTJ9Kmrn9qeqJGVIiFO4S3FB2NfO2Tnb2pPq2tXZujo+1UrFqrlvtBnefq3Oox6qO6EnUPl+4fOgeVSuEK4fl6rn5jHksYd9HNKNn6CKS9TQ0Ks/4AkTa2A9WhPr+UNq5YxnF1+a9fIOtSmWpQ== The indexed definition of the vectorization tensor is N wires M wires ( PN Qj 1 i(N) i(1)
AAAC7nicjVLLSgMxFD0dX7W+qi7dDBahbsqMCLosunElFexDai0z06jBeTGTEUrpVwiuLG79Jf+g/oU3MQW1+EiYzMm559zkJnFjn6fCsl5zxszs3PxCfrGwtLyyulZc32ikUZZ4rO5FfpS0XCdlPg9ZXXDhs1acMCdwfdZ0745lvHnPkpRH4bnox6wTODchv+aeI4i64F37alA+3R12iyWrYqlmTgNbgxJ0q0XFMS7RQwQPGQIwhBCEfThIqbdhw0JMXAcD4hJCXMUZhiiQNyMVI4VD7B2NNzRrazakucyZKrdHq/j0JeQ0sUOeiHQJYbmaqeKZyizZn3IPVE65tz79XZ0rIFbglti/fBPlf32yFoFrHKoaONUUK0ZW5+ksmToVuXPzU1WCMsTESdyjeELYU87JOZvKk6ra5dk6Kj5WSsnKuae1Gd5+rc6lHsg7oSdgf7/wadDYq9iEz/ZL1SP9GPLYwjbKdOMHqOIENdRphQAPeMLIiI1HY2Q8f0iNnPZs4kszXt4BsNGZpg== 1 if m = j=1 ij `=1− I` 1 AAAC7nicjVLLSgMxFD0dX7W+qi7dDBahbsqMCLosunEjVLAPqbXMTKMG58VMRiilXyG4srj1l/yD+hfexBTU4iNhMifnnnOTm8SNfZ4Ky3rNGTOzc/ML+cXC0vLK6lpxfaORRlnisboX+VHScp2U+TxkdcGFz1pxwpzA9VnTvTuW8eY9S1IeheeiH7NO4NyE/Jp7jiDqgndPrwZle3fYLZasiqWaOQ1sDUrQrRYVx7hEDxE8ZAjAEEIQ9uEgpd6GDQsxcR0MiEsIcRVnGKJA3oxUjBQOsXc03tCsrdmQ5jJnqtwereLTl5DTxA55ItIlhOVqpopnKrNkf8o9UDnl3vr0d3WugFiBW2L/8k2U//XJWgSucahq4FRTrBhZnaezZOpU5M7NT1UJyhATJ3GP4glhTzkn52wqT6pql2frqPhYKSUr557WZnj7tTqXeiDvhJ6A/f3Cp0Fjr2ITPtsvVY/0Y8hjC9so040foIoT1FCnFQI84AkjIzYejZHx/CE1ctqziS/NeHkHrr2ZpQ== M γi1i2...iN ,m = (4) 0 otherwise i(N) M wires i(N) N wires
AAAC7nicjVLLSgMxFD0dX7W+qi7dDBahbsqMCLosunGjVLAPqbXMTNMaOi9mMkIp/QrBlcWtv+Qf1L/wJk5BLT4SJnNy7jk3uUns0OWxMIzXjDY3v7C4lF3OrayurW/kN7dqcZBEDqs6gRtEDduKmct9VhVcuKwRRszybJfV7f6pjNfvWRTzwL8Sg5C1PKvn8y53LEHUNW+f3w6LF/ujdr5glAzV9FlgpqCAtFWC/AQ36CCAgwQeGHwIwi4sxNSbMGEgJK6FIXERIa7iDCPkyJuQipHCIrZPY49mzZT1aS5zxsrt0CoufRE5deyRJyBdRFiupqt4ojJL9qfcQ5VT7m1AfzvN5RErcEfsX76p8r8+WYtAF8eqBk41hYqR1TlplkSdity5/qkqQRlC4iTuUDwi7Cjn9Jx15YlV7fJsLRWfKKVk5dxJtQnefq3Opu7JO6EnYH6/8FlQOyiZhC8PC+WT9DFksYNdFOnGj1DGGSqo0goeHvCEsRZqj9pYe/6QapnUs40vTXt5B/exmcI= M AAAC7nicjVLLSgMxFD0dX7W+qi7dDBahbsqMCLosunGjVLAPqbXMTNMaOi9mMkIp/QrBlcWtv+Qf1L/wJk5BLT4SJnNy7jk3uUns0OWxMIzXjDY3v7C4lF3OrayurW/kN7dqcZBEDqs6gRtEDduKmct9VhVcuKwRRszybJfV7f6pjNfvWRTzwL8Sg5C1PKvn8y53LEHUNW+f3w6LF/ujdr5glAzV9FlgpqCAtFWC/AQ36CCAgwQeGHwIwi4sxNSbMGEgJK6FIXERIa7iDCPkyJuQipHCIrZPY49mzZT1aS5zxsrt0CoufRE5deyRJyBdRFiupqt4ojJL9qfcQ5VT7m1AfzvN5RErcEfsX76p8r8+WYtAF8eqBk41hYqR1TlplkSdity5/qkqQRlC4iTuUDwi7Cjn9Jx15YlV7fJsLRWfKKVk5dxJtQnefq3Opu7JO6EnYH6/8FlQOyiZhC8PC+WT9DFksYNdFOnGj1DGGSqo0goeHvCEsRZqj9pYe/6QapnUs40vTXt5B/exmcI= M We draw it as a triangle with N+1 wires: FIG. 6. This identity holds if in the LHS there are N copies of the same vectorization tensor and the Kronecker tensor’s in-
iAAAC6HicjVLLSgMxFD0dX7W+qi7dDBbBVZkRQZeiG5cVbSvUUmbStMbOi5mMIMVPEFxZ3PpX/kH9C29iCmrxkTCZk3PPuclN4ieByKTjvBasmdm5+YXiYmlpeWV1rby+0cjiPGW8zuIgTi99L+OBiHhdChnwyyTlXugHvOkPTlS8ecvTTMTRhbxLeDv0+pHoCeZJos5Fx+2UK07V0c2eBq4BFZhWi8tjXKGLGAw5QnBEkIQDeMiot+DCQUJcG0PiUkJCxznuUSJvTipOCo/YAY19mrUMG9Fc5cy0m9EqAX0pOW3skCcmXUpYrWbreK4zK/an3EOdU+3tjv6+yRUSK3FN7F++ifK/PlWLRA+HugZBNSWaUdUxkyXXp6J2bn+qSlKGhDiFuxRPCTPtnJyzrT2Zrl2drafjY61UrJozo83x9mt1PvVQ3Qk9Aff7hU+Dxl7VJXy2Xzk6No+hiC1sY5du/ABHOEUNdVqhjwc8YWTdWI/WyHr+kFoF49nEl2a9vAPZtpd1 1 dices have a consistent dimension. Notice that we are “trans- posing” the indices. iAAAC6HicjVLLSgMxFD0dX7W+qi7dDBbBVZkpgi6LblxWtA+opcykaY2dFzMZoRQ/QXBlcetf+Qf1L7yJU1CLj4TJnJx7zk1uEjfyRCIt6zVnLCwuLa/kVwtr6xubW8XtnUYSpjHjdRZ6YdxynYR7IuB1KaTHW1HMHd/1eNMdnql4847HiQiDKzmKeMd3BoHoC+ZIoi5Ft9ItlqyypZs5D+wMlJC1Wlic4ho9hGBI4YMjgCTswUFCvQ0bFiLiOhgTFxMSOs5xjwJ5U1JxUjjEDmkc0KydsQHNVc5Euxmt4tEXk9PEAXlC0sWE1Wqmjqc6s2J/yj3WOdXeRvR3s1w+sRI3xP7lmyn/61O1SPRxomsQVFOkGVUdy7Kk+lTUzs1PVUnKEBGncI/iMWGmnbNzNrUn0bWrs3V0fKqVilVzlmlTvP1anUvdV3dCT8D+fuHzoFEp24QvjkrV0+wx5LGHfRzSjR+jinPUUKcVBnjAEybGrfFoTIznD6mRyzy7+NKMl3fcOJd2 2
AAAC5nicjVLLSgMxFD0dX7W+qi7dDBbBVZmKoMuiG5ct2AfUIjPTtA2dF5mMUIpfILiyuPW3/IP6F97EFNTiI2EyJ+eec5ObxEsCnkrHec1ZS8srq2v59cLG5tb2TnF3r5nGmfBZw4+DWLQ9N2UBj1hDchmwdiKYG3oBa3mjSxVv3TGR8ji6luOEdUN3EPE+911JVD28LZacsqObvQgqBpRgWi0uznCDHmL4yBCCIYIkHMBFSr2DChwkxHUxIU4Q4jrOcI8CeTNSMVK4xI5oHNCsY9iI5ipnqt0+rRLQJ8hp44g8MekEYbWareOZzqzYn3JPdE61tzH9PZMrJFZiSOxfvrnyvz5Vi0Qf57oGTjUlmlHV+SZLpk9F7dz+VJWkDAlxCvcoLgj72jk/Z1t7Ul27OltXx2daqVg19402w9uv1XnUQ3Un9AQq3y98ETRPyhXC9dNS9cI8hjwOcIhjuvEzVHGFGhp69Qc8YWoNrUdraj1/SK2c8ezjS7Ne3gEtmJbV
m iAAAC6HicjVLLSgMxFD0dX7W+qi7dDBbBVZlRQZdFNy4r2gfUUmbStMbOi5mMUIqfILiyuPWv/IP6F97EKajFR8JkTs495yY3iRt5IpGW9Zoz5uYXFpfyy4WV1bX1jeLmVj0J05jxGgu9MG66TsI9EfCaFNLjzSjmju96vOEOzlS8ccfjRITBlRxGvO07/UD0BHMkUZeic9gplqyypZs5C+wMlJC1alic4BpdhGBI4YMjgCTswUFCvQUbFiLi2hgRFxMSOs5xjwJ5U1JxUjjEDmjs06yVsQHNVc5Euxmt4tEXk9PEHnlC0sWE1Wqmjqc6s2J/yj3SOdXehvR3s1w+sRI3xP7lmyr/61O1SPRwomsQVFOkGVUdy7Kk+lTUzs1PVUnKEBGncJfiMWGmndNzNrUn0bWrs3V0fKKVilVzlmlTvP1anUvdV3dCT8D+fuGzoH5QtglfHJUqp9ljyGMHu9inGz9GBeeookYr9PGAJ4yNW+PRGBvPH1Ijl3m28aUZL+/eupd3 3
iAAAC6HicjVLLSgMxFD0dX7W+qi7dDBbBVZkRQZdFN66kon1ALWUmTWvsvJjJCKX4CYIri1v/yj+of+FNnIJafCRM5uTcc25yk7iRJxJpWa85Y25+YXEpv1xYWV1b3yhubtWTMI0Zr7HQC+Om6yTcEwGvSSE93oxi7viuxxvu4FTFG3c8TkQYXMlhxNu+0w9ETzBHEnUpOuedYskqW7qZs8DOQAlZq4bFCa7RRQiGFD44AkjCHhwk1FuwYSEiro0RcTEhoeMc9yiQNyUVJ4VD7IDGPs1aGRvQXOVMtJvRKh59MTlN7JEnJF1MWK1m6niqMyv2p9wjnVPtbUh/N8vlEytxQ+xfvqnyvz5Vi0QPx7oGQTVFmlHVsSxLqk9F7dz8VJWkDBFxCncpHhNm2jk9Z1N7El27OltHxydaqVg1Z5k2xduv1bnUfXUn9ATs7xc+C+oHZZvwxWGpcpI9hjx2sIt9uvEjVHCGKmq0Qh8PeMLYuDUejbHx/CE1cplnG1+a8fIOIn+Xkg== N =AAAC5nicjVLLSgMxFD0dX7W+qi7dDBbBVZmKoBuh6MZlC7YVapGZadqGzotMRijFLxBcWdz6W/5B/QtvYgpq8ZEwmZNzz7nJTeIlAU+l47zmrIXFpeWV/GphbX1jc6u4vdNM40z4rOHHQSyuPTdlAY9YQ3IZsOtEMDf0Atbyhhcq3rpjIuVxdCVHCeuEbj/iPe67kqj62W2x5JQd3ex5UDGgBNNqcXGKG3QRw0eGEAwRJOEALlLqbVTgICGugzFxghDXcYZ7FMibkYqRwiV2SGOfZm3DRjRXOVPt9mmVgD5BThsH5IlJJwir1Wwdz3Rmxf6Ue6xzqr2N6O+ZXCGxEgNi//LNlP/1qVokejjVNXCqKdGMqs43WTJ9Kmrn9qeqJGVIiFO4S3FB2NfO2Tnb2pPq2tXZujo+1UrFqrlvtBnefq3Oox6qO6EnUPl+4fOgeVSuEK4fl6rn5jHksYd9HNKNn6CKS9TQ0Ks/4AkTa2A9WhPr+UNq5YxnF1+a9fIOtSmWpQ== FIG. 4. The vectorization tensor serializes multiple indices into a single index in a reversible way (if we keep track of the dimension and position of each wire).
FIG. 7. As an example of the swapping rule in Fig. 6, notice Note that as the vectorization tensor is not symmetric how swapping the Kronecker and vectorization tensors leads (as opposed to the Kronecker tensor), the order of the to twisting wires. wires is crucial. These two tensors enjoy of the following properties: As we will see below, this property is the origin of the bisymmetry property of some of the products that we are =AAAC5nicjVLLSgMxFD0dX7W+qi7dDBbBVZmKoBuh6MZlC7YVapGZadqGzotMRijFLxBcWdz6W/5B/QtvYgpq8ZEwmZNzz7nJTeIlAU+l47zmrIXFpeWV/GphbX1jc6u4vdNM40z4rOHHQSyuPTdlAY9YQ3IZsOtEMDf0Atbyhhcq3rpjIuVxdCVHCeuEbj/iPe67kqj62W2x5JQd3ex5UDGgBNNqcXGKG3QRw0eGEAwRJOEALlLqbVTgICGugzFxghDXcYZ7FMibkYqRwiV2SGOfZm3DRjRXOVPt9mmVgD5BThsH5IlJJwir1Wwdz3Rmxf6Ue6xzqr2N6O+ZXCGxEgNi//LNlP/1qVokejjVNXCqKdGMqs43WTJ9Kmrn9qeqJGVIiFO4S3FB2NfO2Tnb2pPq2tXZujo+1UrFqrlvtBnefq3Oox6qO6EnUPl+4fOgeVSuEK4fl6rn5jHksYd9HNKNn6CKS9TQ0Ks/4AkTa2A9WhPr+UNq5YxnF1+a9fIOtSmWpQ== going to describe.
=AAAC5nicjVLLSgMxFD0dX7W+qi7dDBbBVZmKoBuh6MZlC7YVapGZadqGzotMRijFLxBcWdz6W/5B/QtvYgpq8ZEwmZNzz7nJTeIlAU+l47zmrIXFpeWV/GphbX1jc6u4vdNM40z4rOHHQSyuPTdlAY9YQ3IZsOtEMDf0Atbyhhcq3rpjIuVxdCVHCeuEbj/iPe67kqj62W2x5JQd3ex5UDGgBNNqcXGKG3QRw0eGEAwRJOEALlLqbVTgICGugzFxghDXcYZ7FMibkYqRwiV2SGOfZm3DRjRXOVPt9mmVgD5BThsH5IlJJwir1Wwdz3Rmxf6Ue6xzqr2N6O+ZXCGxEgNi//LNlP/1qVokejjVNXCqKdGMqs43WTJ9Kmrn9qeqJGVIiFO4S3FB2NfO2Tnb2pPq2tXZujo+1UrFqrlvtBnefq3Oox6qO6EnUPl+4fOgeVSuEK4fl6rn5jHksYd9HNKNn6CKS9TQ0Ks/4AkTa2A9WhPr+UNq5YxnF1+a9fIOtSmWpQ== M+N wires N wires M wires III. PRODUCTS
=AAAC5nicjVLLSgMxFD0dX7W+qi7dDBbBVZmKoBuh6MZlC7YVapGZadqGzotMRijFLxBcWdz6W/5B/QtvYgpq8ZEwmZNzz7nJTeIlAU+l47zmrIXFpeWV/GphbX1jc6u4vdNM40z4rOHHQSyuPTdlAY9YQ3IZsOtEMDf0Atbyhhcq3rpjIuVxdCVHCeuEbj/iPe67kqj62W2x5JQd3ex5UDGgBNNqcXGKG3QRw0eGEAwRJOEALlLqbVTgICGugzFxghDXcYZ7FMibkYqRwiV2SGOfZm3DRjRXOVPt9mmVgD5BThsH5IlJJwir1Wwdz3Rmxf6Ue6xzqr2N6O+ZXCGxEgNi//LNlP/1qVokejjVNXCqKdGMqs43WTJ9Kmrn9qeqJGVIiFO4S3FB2NfO2Tnb2pPq2tXZujo+1UrFqrlvtBnefq3Oox6qO6EnUPl+4fOgeVSuEK4fl6rn5jHksYd9HNKNn6CKS9TQ0Ks/4AkTa2A9WhPr+UNq5YxnF1+a9fIOtSmWpQ== We now proceed with the presentation of a few matrix products. Here we summarize the requirements for the dimensions of the matrices/tensors in order for the
=AAAC5nicjVLLSgMxFD0dX7W+qi7dDBbBVZmKoBuh6MZlC7YVapGZadqGzotMRijFLxBcWdz6W/5B/QtvYgpq8ZEwmZNzz7nJTeIlAU+l47zmrIXFpeWV/GphbX1jc6u4vdNM40z4rOHHQSyuPTdlAY9YQ3IZsOtEMDf0Atbyhhcq3rpjIuVxdCVHCeuEbj/iPe67kqj62W2x5JQd3ex5UDGgBNNqcXGKG3QRw0eGEAwRJOEALlLqbVTgICGugzFxghDXcYZ7FMibkYqRwiV2SGOfZm3DRjRXOVPt9mmVgD5BThsH5IlJJwir1Wwdz3Rmxf6Ue6xzqr2N6O+ZXCGxEgNi//LNlP/1qVokejjVNXCqKdGMqs43WTJ9Kmrn9qeqJGVIiFO4S3FB2NfO2Tnb2pPq2tXZujo+1UrFqrlvtBnefq3Oox6qO6EnUPl+4fOgeVSuEK4fl6rn5jHksYd9HNKNn6CKS9TQ0Ks/4AkTa2A9WhPr+UNq5YxnF1+a9fIOtSmWpQ== products to be well-defined: M wires M wires M wires dim(A) dim(B) result FIG. 5. Above: when we contract multiple Kronecker tensors Dot I J J K I K × × × with wires of the same dimension we obtain other Kronecker Tensor I J K L I J K L tensors with the appropriate number of wires. Below: when × × × × × Kronecker I J K L IK JL we contract vectorization tensors, we need to be careful to × × × match the dimensions. Depending on which wires we con- Hadamard I J I J I J × × × tract, we can end up with a high-dimensional identity matrix Kathri-Rao I J K J IK J × × × or with the tensor product of several identity matrices. Tracy-Singh I J K L P Q R S IPKR JQLS × × × × × × × 4
A. Dot product
AAAAC5nicjVLLSgMxFD0dX7W+qi7dDBbBVZmKoMuqG5ct2FaoRWbStB06LzIZoRS/QHBlcetv+Qf1L7yJKajFR8JkTs495yY3iZcEfiod5zVnLSwuLa/kVwtr6xubW8XtnWYaZ4LxBouDWFx7bsoDP+IN6cuAXyeCu6EX8JY3vFDx1h0XqR9HV3KU8E7o9iO/5zNXElU/uy2WnLKjmz0PKgaUYFotLk5xgy5iMGQIwRFBEg7gIqXeRgUOEuI6GBMnCPk6znGPAnkzUnFSuMQOaezTrG3YiOYqZ6rdjFYJ6BPktHFAnph0grBazdbxTGdW7E+5xzqn2tuI/p7JFRIrMSD2L99M+V+fqkWih1Ndg081JZpR1TGTJdOnonZuf6pKUoaEOIW7FBeEmXbOztnWnlTXrs7W1fGpVipWzZnRZnj7tTqPeqjuhJ5A5fuFz4PmUblCuH5cqp6bx5DHHvZxSDd+giouUUNDr/6AJ0ysgfVoTaznD6mVM55dfGnWyzu/MZap The dot product consists in summing over a repeated row-by-column pair. The dot product described with in- BAAAC5nicjVLLSgMxFD0dX/VddelmsAiuylQEXZa6cdmCfUAtMpOm7dB5kckIpfgFgiuLW3/LP6h/4U1MQS0+EiZzcu45N7lJvCTwU+k4rzlraXlldS2/vrG5tb2zW9jbb6ZxJhhvsDiIRdtzUx74EW9IXwa8nQjuhl7AW97oUsVbd1ykfhxdy3HCu6E7iPy+z1xJVL16Wyg6JUc3exGUDSjCtFpcmOEGPcRgyBCCI4IkHMBFSr2DMhwkxHUxIU4Q8nWc4x4b5M1IxUnhEjuicUCzjmEjmqucqXYzWiWgT5DTxjF5YtIJwmo1W8cznVmxP+We6Jxqb2P6eyZXSKzEkNi/fHPlf32qFok+LnQNPtWUaEZVx0yWTJ+K2rn9qSpJGRLiFO5RXBBm2jk/Z1t7Ul27OltXx2daqVg1Z0ab4e3X6jzqoboTegLl7xe+CJqnpTLh+lmxUjWPIY9DHOGEbvwcFVyhhoZe/QFPmFpD69GaWs8fUitnPAf40qyXd8Gzlqo= dices is (AB)ik = AijBjk. In graphical notation we in- dicate it by connecting the wires corresponding to the FIG. 10. The Kronecker product is like a tensor product, index j: except we vectorize the row indices and the column indices. The Kronecker product reduces the total number of wires, but it is reversible, so no information is lost (unlike the Hadamard
AAAC5nicjVLLSgMxFD0dX/VddelmsAiuylQEXZa6cdmCfUAtMpOm7dB5kckIpfgFgiuLW3/LP6h/4U1MQS0+EiZzcu45N7lJvCTwU+k4rzlraXlldS2/vrG5tb2zW9jbb6ZxJhhvsDiIRdtzUx74EW9IXwa8nQjuhl7AW97oUsVbd1ykfhxdy3HCu6E7iPy+z1xJVL16Wyg6JUc3exGUDSjCtFpcmOEGPcRgyBCCI4IkHMBFSr2DMhwkxHUxIU4Q8nWc4x4b5M1IxUnhEjuicUCzjmEjmqucqXYzWiWgT5DTxjF5YtIJwmo1W8cznVmxP+We6Jxqb2P6eyZXSKzEkNi/fHPlf32qFok+LnQNPtWUaEZVx0yWTJ+K2rn9qSpJGRLiFO5RXBBm2jk/Z1t7Ul27OltXx2daqVg1Z0ab4e3X6jzqoboTegLl7xe+CJqnpTLh+lmxUjWPIY9DHOGEbvwcFVyhhoZe/QFPmFpD69GaWs8fUitnPAf40qyXd8Gzlqo= AAAAC5nicjVLLSgMxFD0dX7W+qi7dDBbBVZmKoMuqG5ct2FaoRWbStB06LzIZoRS/QHBlcetv+Qf1L7yJKajFR8JkTs495yY3iZcEfiod5zVnLSwuLa/kVwtr6xubW8XtnWYaZ4LxBouDWFx7bsoDP+IN6cuAXyeCu6EX8JY3vFDx1h0XqR9HV3KU8E7o9iO/5zNXElU/uy2WnLKjmz0PKgaUYFotLk5xgy5iMGQIwRFBEg7gIqXeRgUOEuI6GBMnCPk6znGPAnkzUnFSuMQOaezTrG3YiOYqZ6rdjFYJ6BPktHFAnph0grBazdbxTGdW7E+5xzqn2tuI/p7JFRIrMSD2L99M+V+fqkWih1Ndg081JZpR1TGTJdOnonZuf6pKUoaEOIW7FBeEmXbOztnWnlTXrs7W1fGpVipWzZnRZnj7tTqPeqjuhJ5A5fuFz4PmUblCuH5cqp6bx5DHHvZxSDd+giouUUNDr/6AJ0ysgfVoTaznD6mVM55dfGnWyzu/MZap B product, below).
FIG. 8. The dot product in graphical notation consists of connecting wires directly. Each dot product reduces by 2 the D. Hadamard product total number of initial wires. The Hadamard product is an element-wise multiplica- As two free wires remain, the result is another matrix. tion and it is defined if the two matrices match both of their dimensions exactly: (A B) = A B ◦ ij ij ij
B. Tensor product AAAAC5nicjVLLSgMxFD0dX7W+qi7dDBbBVZmKoMuqG5ct2FaoRWbStB06LzIZoRS/QHBlcetv+Qf1L7yJKajFR8JkTs495yY3iZcEfiod5zVnLSwuLa/kVwtr6xubW8XtnWYaZ4LxBouDWFx7bsoDP+IN6cuAXyeCu6EX8JY3vFDx1h0XqR9HV3KU8E7o9iO/5zNXElU/uy2WnLKjmz0PKgaUYFotLk5xgy5iMGQIwRFBEg7gIqXeRgUOEuI6GBMnCPk6znGPAnkzUnFSuMQOaezTrG3YiOYqZ6rdjFYJ6BPktHFAnph0grBazdbxTGdW7E+5xzqn2tuI/p7JFRIrMSD2L99M+V+fqkWih1Ndg081JZpR1TGTJdOnonZuf6pKUoaEOIW7FBeEmXbOztnWnlTXrs7W1fGpVipWzZnRZnj7tTqPeqjuhJ5A5fuFz4PmUblCuH5cqp6bx5DHHvZxSDd+giouUUNDr/6AJ0ysgfVoTaznD6mVM55dfGnWyzu/MZap
The tensor product preserves all of the index infor- BAAAC5nicjVLLSgMxFD0dX/VddelmsAiuylQEXZa6cdmCfUAtMpOm7dB5kckIpfgFgiuLW3/LP6h/4U1MQS0+EiZzcu45N7lJvCTwU+k4rzlraXlldS2/vrG5tb2zW9jbb6ZxJhhvsDiIRdtzUx74EW9IXwa8nQjuhl7AW97oUsVbd1ykfhxdy3HCu6E7iPy+z1xJVL16Wyg6JUc3exGUDSjCtFpcmOEGPcRgyBCCI4IkHMBFSr2DMhwkxHUxIU4Q8nWc4x4b5M1IxUnhEjuicUCzjmEjmqucqXYzWiWgT5DTxjF5YtIJwmo1W8cznVmxP+We6Jxqb2P6eyZXSKzEkNi/fHPlf32qFok+LnQNPtWUaEZVx0yWTJ+K2rn9qSpJGRLiFO5RXBBm2jk/Z1t7Ul27OltXx2daqVg1Z0ab4e3X6jzqoboTegLl7xe+CJqnpTLh+lmxUjWPIY9DHOGEbvwcFVyhhoZe/QFPmFpD69GaWs8fUitnPAf40qyXd8Gzlqo= mation: (A B)ijk` = AijBk`. To perform the tensor product in graphical⊗ calculus, we simply stack the tensors and we leave their wires untouched: FIG. 11. The Hadamard product is an element-wise product, so we need the two row indices to be the same and the two column indices to be the same, which we achieve by using the Kronecker tensor.
AAAAC5nicjVLLSgMxFD0dX7W+qi7dDBbBVZmKoMuqG5ct2FaoRWbStB06LzIZoRS/QHBlcetv+Qf1L7yJKajFR8JkTs495yY3iZcEfiod5zVnLSwuLa/kVwtr6xubW8XtnWYaZ4LxBouDWFx7bsoDP+IN6cuAXyeCu6EX8JY3vFDx1h0XqR9HV3KU8E7o9iO/5zNXElU/uy2WnLKjmz0PKgaUYFotLk5xgy5iMGQIwRFBEg7gIqXeRgUOEuI6GBMnCPk6znGPAnkzUnFSuMQOaezTrG3YiOYqZ6rdjFYJ6BPktHFAnph0grBazdbxTGdW7E+5xzqn2tuI/p7JFRIrMSD2L99M+V+fqkWih1Ndg081JZpR1TGTJdOnonZuf6pKUoaEOIW7FBeEmXbOztnWnlTXrs7W1fGpVipWzZnRZnj7tTqPeqjuhJ5A5fuFz4PmUblCuH5cqp6bx5DHHvZxSDd+giouUUNDr/6AJ0ysgfVoTaznD6mVM55dfGnWyzu/MZap This is clear also from the fact that we define it in terms
BAAAC5nicjVLLSgMxFD0dX/VddelmsAiuylQEXZa6cdmCfUAtMpOm7dB5kckIpfgFgiuLW3/LP6h/4U1MQS0+EiZzcu45N7lJvCTwU+k4rzlraXlldS2/vrG5tb2zW9jbb6ZxJhhvsDiIRdtzUx74EW9IXwa8nQjuhl7AW97oUsVbd1ykfhxdy3HCu6E7iPy+z1xJVL16Wyg6JUc3exGUDSjCtFpcmOEGPcRgyBCCI4IkHMBFSr2DMhwkxHUxIU4Q8nWc4x4b5M1IxUnhEjuicUCzjmEjmqucqXYzWiWgT5DTxjF5YtIJwmo1W8cznVmxP+We6Jxqb2P6eyZXSKzEkNi/fHPlf32qFok+LnQNPtWUaEZVx0yWTJ+K2rn9qSpJGRLiFO5RXBBm2jk/Z1t7Ul27OltXx2daqVg1Z0ab4e3X6jzqoboTegLl7xe+CJqnpTLh+lmxUjWPIY9DHOGEbvwcFVyhhoZe/QFPmFpD69GaWs8fUitnPAf40qyXd8Gzlqo= of the Kronecker tensor, whose wires all have the same dimension. Note that we presented it for a pair of ma- trices, but obviously the element-wise product is defined FIG. 9. To take the tensor product of two matrices A and B we stack them and preserve all of the indices. The tensor for tensors of any rank. product does not change the total number of wires.
E. Kathri-Rao product
This Khatri-Rao product [15] is useful in data process- ing and in optimizing the solution of inverse problems C. Kronecker product that deal with a diagonal matrix [16]. The definition of the Kathri-Rao product is somewhat cryptic, but in The Kronecker product is what is often conflated with terms of graphical calculus it is very simple. In the stan- the tensor product, especially in the quantum informa- dard description, we have two matrices of dimension I J tion literature. Perhaps the fact that they are usually and K J therefore with the same number of columns,× × both indicated with the simbol contributes to the A = [a0,..., aJ 1] and B = [b0,..., bJ 1]. The Kathri- mixup. To compute the Kronecker⊗ product one begins Rao product is− the matrix with J columns− whose j-th with a tensor product, but then one makes the extra step column is the Kronecker product of columns aj and bj: of vectorizing groups of indices together. For example, (A B) = [a0 b0,..., aJ 1 bJ 1]. the Kronecker product of two matrices is another ma- The definition⊗ in terms−δ ⊗and −γ is (A B) = mn trix: (A B)mn = AijBk`γik,mγj`,n, which means that AijBk`γik,mδj`n which means that we perform a Kro- m 0,...,IK⊗ 1 and n 0,...,JL 1 . Despite necker product on the row idices and a Hadamard prod- having∈ { only two− wires} and not∈ { four, the result− } contains uct on the column indices. There is also a row-version of all of the information contained in the tensor product, the Kathri-Rao product, where the Hadamard product except for the information about which subsystem has is on the left (i.e. on the rows), and the vectorization which dimensionality. If we work with systems of equal on the right (i.e. on the columns). The graphical defini- dimension (such as a set of qubits) this is not a problem. tion makes it trivial to prove that the column- and row- 5
realise at a glance that they are true, without the need
AAAAC5nicjVLLSgMxFD0dX7W+qi7dDBbBVZmKoMuqG5ct2FaoRWbStB06LzIZoRS/QHBlcetv+Qf1L7yJKajFR8JkTs495yY3iZcEfiod5zVnLSwuLa/kVwtr6xubW8XtnWYaZ4LxBouDWFx7bsoDP+IN6cuAXyeCu6EX8JY3vFDx1h0XqR9HV3KU8E7o9iO/5zNXElU/uy2WnLKjmz0PKgaUYFotLk5xgy5iMGQIwRFBEg7gIqXeRgUOEuI6GBMnCPk6znGPAnkzUnFSuMQOaezTrG3YiOYqZ6rdjFYJ6BPktHFAnph0grBazdbxTGdW7E+5xzqn2tuI/p7JFRIrMSD2L99M+V+fqkWih1Ndg081JZpR1TGTJdOnonZuf6pKUoaEOIW7FBeEmXbOztnWnlTXrs7W1fGpVipWzZnRZnj7tTqPeqjuhJ5A5fuFz4PmUblCuH5cqp6bx5DHHvZxSDd+giouUUNDr/6AJ0ysgfVoTaznD6mVM55dfGnWyzu/MZap to dive into lengthy computations. All of the necessary comments are in each figure caption.
BAAAC5nicjVLLSgMxFD0dX/VddelmsAiuylQEXZa6cdmCfUAtMpOm7dB5kckIpfgFgiuLW3/LP6h/4U1MQS0+EiZzcu45N7lJvCTwU+k4rzlraXlldS2/vrG5tb2zW9jbb6ZxJhhvsDiIRdtzUx74EW9IXwa8nQjuhl7AW97oUsVbd1ykfhxdy3HCu6E7iPy+z1xJVL16Wyg6JUc3exGUDSjCtFpcmOEGPcRgyBCCI4IkHMBFSr2DMhwkxHUxIU4Q8nWc4x4b5M1IxUnhEjuicUCzjmEjmqucqXYzWiWgT5DTxjF5YtIJwmo1W8cznVmxP+We6Jxqb2P6eyZXSKzEkNi/fHPlf32qFok+LnQNPtWUaEZVx0yWTJ+K2rn9qSpJGRLiFO5RXBBm2jk/Z1t7Ul27OltXx2daqVg1Z0ab4e3X6jzqoboTegLl7xe+CJqnpTLh+lmxUjWPIY9DHOGEbvwcFVyhhoZe/QFPmFpD69GaWs8fUitnPAf40qyXd8Gzlqo=
AAAC5nicjVLLSgMxFD0dX/VddelmsAiuylQEXZa6cdmCfUAtMpOm7dB5kckIpfgFgiuLW3/LP6h/4U1MQS0+EiZzcu45N7lJvCTwU+k4rzlraXlldS2/vrG5tb2zW9jbb6ZxJhhvsDiIRdtzUx74EW9IXwa8nQjuhl7AW97oUsVbd1ykfhxdy3HCu6E7iPy+z1xJVL16Wyg6JUc3exGUDSjCtFpcmOEGPcRgyBCCI4IkHMBFSr2DMhwkxHUxIU4Q8nWc4x4b5M1IxUnhEjuicUCzjmEjmqucqXYzWiWgT5DTxjF5YtIJwmo1W8cznVmxP+We6Jxqb2P6eyZXSKzEkNi/fHPlf32qFok+LnQNPtWUaEZVx0yWTJ+K2rn9qSpJGRLiFO5RXBBm2jk/Z1t7Ul27OltXx2daqVg1Z0ab4e3X6jzqoboTegLl7xe+CJqnpTLh+lmxUjWPIY9DHOGEbvwcFVyhhoZe/QFPmFpD69GaWs8fUitnPAf40qyXd8Gzlqo= AAAAC5nicjVLLSgMxFD0dX7W+qi7dDBbBVZmKoMuqG5ct2FaoRWbStB06LzIZoRS/QHBlcetv+Qf1L7yJKajFR8JkTs495yY3iZcEfiod5zVnLSwuLa/kVwtr6xubW8XtnWYaZ4LxBouDWFx7bsoDP+IN6cuAXyeCu6EX8JY3vFDx1h0XqR9HV3KU8E7o9iO/5zNXElU/uy2WnLKjmz0PKgaUYFotLk5xgy5iMGQIwRFBEg7gIqXeRgUOEuI6GBMnCPk6znGPAnkzUnFSuMQOaezTrG3YiOYqZ6rdjFYJ6BPktHFAnph0grBazdbxTGdW7E+5xzqn2tuI/p7JFRIrMSD2L99M+V+fqkWih1Ndg081JZpR1TGTJdOnonZuf6pKUoaEOIW7FBeEmXbOztnWnlTXrs7W1fGpVipWzZnRZnj7tTqPeqjuhJ5A5fuFz4PmUblCuH5cqp6bx5DHHvZxSDd+giouUUNDr/6AJ0ysgfVoTaznD6mVM55dfGnWyzu/MZap B FIG. 12. In the Katri-Rao product we perform a Kro- necker product row-wise and a Hadamard product column-
AAAC5nicjVLLSgMxFD0dX7W+qi7dDBbBVZmKoMuiLly2YFuhFplJ03bovMhkhFL8AsGVxa2/5R/Uv/AmpqAWHwmTOTn3nJvcJF4S+Kl0nNectbC4tLySXy2srW9sbhW3d5ppnAnGGywOYnHtuSkP/Ig3pC8Dfp0I7oZewFve8FzFW3dcpH4cXclRwjuh24/8ns9cSVT94rZYcsqObvY8qBhQgmm1uDjFDbqIwZAhBEcESTiAi5R6GxU4SIjrYEycIOTrOMc9CuTNSMVJ4RI7pLFPs7ZhI5qrnKl2M1oloE+Q08YBeWLSCcJqNVvHM51ZsT/lHuucam8j+nsmV0isxIDYv3wz5X99qhaJHk51DT7VlGhGVcdMlkyfitq5/akqSRkS4hTuUlwQZto5O2dbe1JduzpbV8enWqlYNWdGm+Ht1+o86qG6E3oCle8XPg+aR+UK4fpxqXpmHkMee9jHId34Caq4RA0NvfoDnjCxBtajNbGeP6RWznh28aVZL+/Gt5as wise (which is why the number of columns of A and B must CAAAC5nicjVLLSgMxFD0dX/VddelmsAiuylQEXRa7cdmCfUAtMpOm7dB5kckIpfgFgiuLW3/LP6h/4U1MQS0+EiZzcu45N7lJvCTwU+k4rzlraXlldS2/vrG5tb2zW9jbb6ZxJhhvsDiIRdtzUx74EW9IXwa8nQjuhl7AW96oquKtOy5SP46u5Tjh3dAdRH7fZ64kql69LRSdkqObvQjKBhRhWi0uzHCDHmIwZAjBEUESDuAipd5BGQ4S4rqYECcI+TrOcY8N8mak4qRwiR3ROKBZx7ARzVXOVLsZrRLQJ8hp45g8MekEYbWareOZzqzYn3JPdE61tzH9PZMrJFZiSOxfvrnyvz5Vi0QfF7oGn2pKNKOqYyZLpk9F7dz+VJWkDAlxCvcoLggz7Zyfs609qa5dna2r4zOtVKyaM6PN8PZrdR71UN0JPYHy9wtfBM3TUplw/axYuTSPIY9DHOGEbvwcFVyhhoZe/QFPmFpD69GaWs8fUitnPAf40qyXd8Q1lqs= D match). FIG. 14. In this figure we prove the following property of the tensor product: (AB)⊗(CD) = (A⊗C)(B ⊗D). To convince versions of the Kathri-Rao product are linked by a trans- ourselves that this identity holds, we just have to read the T T T diagram from top to bottom or left to right to obtain the two pose operation like so: (A col B) = A row B (see Fig. 18). sides of the identity.
AAAC5nicjVLLSgMxFD0dX7W+qi7dDBbBVZmKoMuqG5ct2FaoRWbStB06LzIZoRS/QHBlcetv+Qf1L7yJKajFR8JkTs495yY3iZcEfiod5zVnLSwuLa/kVwtr6xubW8XtnWYaZ4LxBouDWFx7bsoDP+IN6cuAXyeCu6EX8JY3vFDx1h0XqR9HV3KU8E7o9iO/5zNXElU/uy2WnLKjmz0PKgaUYFotLk5xgy5iMGQIwRFBEg7gIqXeRgUOEuI6GBMnCPk6znGPAnkzUnFSuMQOaezTrG3YiOYqZ6rdjFYJ6BPktHFAnph0grBazdbxTGdW7E+5xzqn2tuI/p7JFRIrMSD2L99M+V+fqkWih1Ndg081JZpR1TGTJdOnonZuf6pKUoaEOIW7FBeEmXbOztnWnlTXrs7W1fGpVipWzZnRZnj7tTqPeqjuhJ5A5fuFz4PmUblCuH5cqp6bx5DHHvZxSDd+giouUUNDr/6AJ0ysgfVoTaznD6mVM55dfGnWyzu/MZap F. Tracy-Singh product A AAAAC5nicjVLLSgMxFD0dX7W+qi7dDBbBVZmKoMuqG5ct2FaoRWbStB06LzIZoRS/QHBlcetv+Qf1L7yJKajFR8JkTs495yY3iZcEfiod5zVnLSwuLa/kVwtr6xubW8XtnWYaZ4LxBouDWFx7bsoDP+IN6cuAXyeCu6EX8JY3vFDx1h0XqR9HV3KU8E7o9iO/5zNXElU/uy2WnLKjmz0PKgaUYFotLk5xgy5iMGQIwRFBEg7gIqXeRgUOEuI6GBMnCPk6znGPAnkzUnFSuMQOaezTrG3YiOYqZ6rdjFYJ6BPktHFAnph0grBazdbxTGdW7E+5xzqn2tuI/p7JFRIrMSD2L99M+V+fqkWih1Ndg081JZpR1TGTJdOnonZuf6pKUoaEOIW7FBeEmXbOztnWnlTXrs7W1fGpVipWzZnRZnj7tTqPeqjuhJ5A5fuFz4PmUblCuH5cqp6bx5DHHvZxSDd+giouUUNDr/6AJ0ysgfVoTaznD6mVM55dfGnWyzu/MZap AAAAC5nicjVLLSgMxFD0dX7W+qi7dDBbBVZmKoMuqG5ct2FaoRWbStB06LzIZoRS/QHBlcetv+Qf1L7yJKajFR8JkTs495yY3iZcEfiod5zVnLSwuLa/kVwtr6xubW8XtnWYaZ4LxBouDWFx7bsoDP+IN6cuAXyeCu6EX8JY3vFDx1h0XqR9HV3KU8E7o9iO/5zNXElU/uy2WnLKjmz0PKgaUYFotLk5xgy5iMGQIwRFBEg7gIqXeRgUOEuI6GBMnCPk6znGPAnkzUnFSuMQOaezTrG3YiOYqZ6rdjFYJ6BPktHFAnph0grBazdbxTGdW7E+5xzqn2tuI/p7JFRIrMSD2L99M+V+fqkWih1Ndg081JZpR1TGTJdOnonZuf6pKUoaEOIW7FBeEmXbOztnWnlTXrs7W1fGpVipWzZnRZnj7tTqPeqjuhJ5A5fuFz4PmUblCuH5cqp6bx5DHHvZxSDd+giouUUNDr/6AJ0ysgfVoTaznD6mVM55dfGnWyzu/MZap
AAAC5nicjVLLSgMxFD0dX/VddelmsAiuylQEXZa6cdmCfUAtMpOm7dB5kckIpfgFgiuLW3/LP6h/4U1MQS0+EiZzcu45N7lJvCTwU+k4rzlraXlldS2/vrG5tb2zW9jbb6ZxJhhvsDiIRdtzUx74EW9IXwa8nQjuhl7AW97oUsVbd1ykfhxdy3HCu6E7iPy+z1xJVL16Wyg6JUc3exGUDSjCtFpcmOEGPcRgyBCCI4IkHMBFSr2DMhwkxHUxIU4Q8nWc4x4b5M1IxUnhEjuicUCzjmEjmqucqXYzWiWgT5DTxjF5YtIJwmo1W8cznVmxP+We6Jxqb2P6eyZXSKzEkNi/fHPlf32qFok+LnQNPtWUaEZVx0yWTJ+K2rn9qSpJGRLiFO5RXBBm2jk/Z1t7Ul27OltXx2daqVg1Z0ab4e3X6jzqoboTegLl7xe+CJqnpTLh+lmxUjWPIY9DHOGEbvwcFVyhhoZe/QFPmFpD69GaWs8fUitnPAf40qyXd8Gzlqo=
AAAC5nicjVLLSgMxFD0dX/VddelmsAiuylQEXRa7cdmCfUAtMpOm7dB5kckIpfgFgiuLW3/LP6h/4U1MQS0+EiZzcu45N7lJvCTwU+k4rzlraXlldS2/vrG5tb2zW9jbb6ZxJhhvsDiIRdtzUx74EW9IXwa8nQjuhl7AW96oquKtOy5SP46u5Tjh3dAdRH7fZ64kql69LRSdkqObvQjKBhRhWi0uzHCDHmIwZAjBEUESDuAipd5BGQ4S4rqYECcI+TrOcY8N8mak4qRwiR3ROKBZx7ARzVXOVLsZrRLQJ8hp45g8MekEYbWareOZzqzYn3JPdE61tzH9PZMrJFZiSOxfvrnyvz5Vi0QfF7oGn2pKNKOqYyZLpk9F7dz+VJWkDAlxCvcoLggz7Zyfs609qa5dna2r4zOtVKyaM6PN8PZrdR71UN0JPYHy9wtfBM3TUplw/axYuTSPIY9DHOGEbvwcFVyhhoZe/QFPmFpD69GaWs8fUitnPAf40qyXd8Q1lqs= B CAAAC5nicjVLLSgMxFD0dX/VddelmsAiuylQEXRa7cdmCfUAtMpOm7dB5kckIpfgFgiuLW3/LP6h/4U1MQS0+EiZzcu45N7lJvCTwU+k4rzlraXlldS2/vrG5tb2zW9jbb6ZxJhhvsDiIRdtzUx74EW9IXwa8nQjuhl7AW96oquKtOy5SP46u5Tjh3dAdRH7fZ64kql69LRSdkqObvQjKBhRhWi0uzHCDHmIwZAjBEUESDuAipd5BGQ4S4rqYECcI+TrOcY8N8mak4qRwiR3ROKBZx7ARzVXOVLsZrRLQJ8hp45g8MekEYbWareOZzqzYn3JPdE61tzH9PZMrJFZiSOxfvrnyvz5Vi0QfF7oGn2pKNKOqYyZLpk9F7dz+VJWkDAlxCvcoLggz7Zyfs609qa5dna2r4zOtVKyaM6PN8PZrdR71UN0JPYHy9wtfBM3TUplw/axYuTSPIY9DHOGEbvwcFVyhhoZe/QFPmFpD69GaWs8fUitnPAf40qyXd8Q1lqs= C =AAAC5nicjVLLSgMxFD0dX7W+qi7dDBbBVZmKoBuh6MZlC7YVapGZadqGzotMRijFLxBcWdz6W/5B/QtvYgpq8ZEwmZNzz7nJTeIlAU+l47zmrIXFpeWV/GphbX1jc6u4vdNM40z4rOHHQSyuPTdlAY9YQ3IZsOtEMDf0Atbyhhcq3rpjIuVxdCVHCeuEbj/iPe67kqj62W2x5JQd3ex5UDGgBNNqcXGKG3QRw0eGEAwRJOEALlLqbVTgICGugzFxghDXcYZ7FMibkYqRwiV2SGOfZm3DRjRXOVPt9mmVgD5BThsH5IlJJwir1Wwdz3Rmxf6Ue6xzqr2N6O+ZXCGxEgNi//LNlP/1qVokejjVNXCqKdGMqs43WTJ9Kmrn9qeqJGVIiFO4S3FB2NfO2Tnb2pPq2tXZujo+1UrFqrlvtBnefq3Oox6qO6EnUPl+4fOgeVSuEK4fl6rn5jHksYd9HNKNn6CKS9TQ0Ks/4AkTa2A9WhPr+UNq5YxnF1+a9fIOtSmWpQ== =AAAC5nicjVLLSgMxFD0dX7W+qi7dDBbBVZmKoBuh6MZlC7YVapGZadqGzotMRijFLxBcWdz6W/5B/QtvYgpq8ZEwmZNzz7nJTeIlAU+l47zmrIXFpeWV/GphbX1jc6u4vdNM40z4rOHHQSyuPTdlAY9YQ3IZsOtEMDf0Atbyhhcq3rpjIuVxdCVHCeuEbj/iPe67kqj62W2x5JQd3ex5UDGgBNNqcXGKG3QRw0eGEAwRJOEALlLqbVTgICGugzFxghDXcYZ7FMibkYqRwiV2SGOfZm3DRjRXOVPt9mmVgD5BThsH5IlJJwir1Wwdz3Rmxf6Ue6xzqr2N6O+ZXCGxEgNi//LNlP/1qVokejjVNXCqKdGMqs43WTJ9Kmrn9qeqJGVIiFO4S3FB2NfO2Tnb2pPq2tXZujo+1UrFqrlvtBnefq3Oox6qO6EnUPl+4fOgeVSuEK4fl6rn5jHksYd9HNKNn6CKS9TQ0Ks/4AkTa2A9WhPr+UNq5YxnF1+a9fIOtSmWpQ==
AAAC5nicjVLLSgMxFD0dX/VddelmsAiuylQEXZa6cdmCfUAtMpOm7dB5kckIpfgFgiuLW3/LP6h/4U1MQS0+EiZzcu45N7lJvCTwU+k4rzlraXlldS2/vrG5tb2zW9jbb6ZxJhhvsDiIRdtzUx74EW9IXwa8nQjuhl7AW97oUsVbd1ykfhxdy3HCu6E7iPy+z1xJVL16Wyg6JUc3exGUDSjCtFpcmOEGPcRgyBCCI4IkHMBFSr2DMhwkxHUxIU4Q8nWc4x4b5M1IxUnhEjuicUCzjmEjmqucqXYzWiWgT5DTxjF5YtIJwmo1W8cznVmxP+We6Jxqb2P6eyZXSKzEkNi/fHPlf32qFok+LnQNPtWUaEZVx0yWTJ+K2rn9qSpJGRLiFO5RXBBm2jk/Z1t7Ul27OltXx2daqVg1Z0ab4e3X6jzqoboTegLl7xe+CJqnpTLh+lmxUjWPIY9DHOGEbvwcFVyhhoZe/QFPmFpD69GaWs8fUitnPAf40qyXd8Gzlqo=
AAAC5nicjVLLSgMxFD0dX/VddelmsAiuylQEXZa6cdmCfUAtMpOm7dB5kckIpfgFgiuLW3/LP6h/4U1MQS0+EiZzcu45N7lJvCTwU+k4rzlraXlldS2/vrG5tb2zW9jbb6ZxJhhvsDiIRdtzUx74EW9IXwa8nQjuhl7AW97oUsVbd1ykfhxdy3HCu6E7iPy+z1xJVL16Wyg6JUc3exGUDSjCtFpcmOEGPcRgyBCCI4IkHMBFSr2DMhwkxHUxIU4Q8nWc4x4b5M1IxUnhEjuicUCzjmEjmqucqXYzWiWgT5DTxjF5YtIJwmo1W8cznVmxP+We6Jxqb2P6eyZXSKzEkNi/fHPlf32qFok+LnQNPtWUaEZVx0yWTJ+K2rn9qSpJGRLiFO5RXBBm2jk/Z1t7Ul27OltXx2daqVg1Z0ab4e3X6jzqoboTegLl7xe+CJqnpTLh+lmxUjWPIY9DHOGEbvwcFVyhhoZe/QFPmFpD69GaWs8fUitnPAf40qyXd8Gzlqo= CAAAC5nicjVLLSgMxFD0dX/VddelmsAiuylQEXRa7cdmCfUAtMpOm7dB5kckIpfgFgiuLW3/LP6h/4U1MQS0+EiZzcu45N7lJvCTwU+k4rzlraXlldS2/vrG5tb2zW9jbb6ZxJhhvsDiIRdtzUx74EW9IXwa8nQjuhl7AW96oquKtOy5SP46u5Tjh3dAdRH7fZ64kql69LRSdkqObvQjKBhRhWi0uzHCDHmIwZAjBEUESDuAipd5BGQ4S4rqYECcI+TrOcY8N8mak4qRwiR3ROKBZx7ARzVXOVLsZrRLQJ8hp45g8MekEYbWareOZzqzYn3JPdE61tzH9PZMrJFZiSOxfvrnyvz5Vi0QfF7oGn2pKNKOqYyZLpk9F7dz+VJWkDAlxCvcoLggz7Zyfs609qa5dna2r4zOtVKyaM6PN8PZrdR71UN0JPYHy9wtfBM3TUplw/axYuTSPIY9DHOGEbvwcFVyhhoZe/QFPmFpD69GaWs8fUitnPAf40qyXd8Q1lqs= B B The Tracy-Singh product [17] is a double Kronecker
AAAC5nicjVLLSgMxFD0dX7W+qi7dDBbBVZmKoMuiLly2YFuhFplJ03bovMhkhFL8AsGVxa2/5R/Uv/AmpqAWHwmTOTn3nJvcJF4S+Kl0nNectbC4tLySXy2srW9sbhW3d5ppnAnGGywOYnHtuSkP/Ig3pC8Dfp0I7oZewFve8FzFW3dcpH4cXclRwjuh24/8ns9cSVT94rZYcsqObvY8qBhQgmm1uDjFDbqIwZAhBEcESTiAi5R6GxU4SIjrYEycIOTrOMc9CuTNSMVJ4RI7pLFPs7ZhI5qrnKl2M1oloE+Q08YBeWLSCcJqNVvHM51ZsT/lHuucam8j+nsmV0isxIDYv3wz5X99qhaJHk51DT7VlGhGVcdMlkyfitq5/akqSRkS4hTuUlwQZto5O2dbe1JduzpbV8enWqlYNWdGm+Ht1+o86qG6E3oCle8XPg+aR+UK4fpxqXpmHkMee9jHId34Caq4RA0NvfoDnjCxBtajNbGeP6RWznh28aVZL+/Gt5as
AAAC5nicjVLLSgMxFD0dX7W+qi7dDBbBVZmKoMuiLly2YFuhFplJ03bovMhkhFL8AsGVxa2/5R/Uv/AmpqAWHwmTOTn3nJvcJF4S+Kl0nNectbC4tLySXy2srW9sbhW3d5ppnAnGGywOYnHtuSkP/Ig3pC8Dfp0I7oZewFve8FzFW3dcpH4cXclRwjuh24/8ns9cSVT94rZYcsqObvY8qBhQgmm1uDjFDbqIwZAhBEcESTiAi5R6GxU4SIjrYEycIOTrOMc9CuTNSMVJ4RI7pLFPs7ZhI5qrnKl2M1oloE+Q08YBeWLSCcJqNVvHM51ZsT/lHuucam8j+nsmV0isxIDYv3wz5X99qhaJHk51DT7VlGhGVcdMlkyfitq5/akqSRkS4hTuUlwQZto5O2dbe1JduzpbV8enWqlYNWdGm+Ht1+o86qG6E3oCle8XPg+aR+UK4fpxqXpmHkMee9jHId34Caq4RA0NvfoDnjCxBtajNbGeP6RWznh28aVZL+/Gt5as D DAAAC5nicjVLLSgMxFD0dX7W+qi7dDBbBVZmKoMuiLly2YFuhFplJ03bovMhkhFL8AsGVxa2/5R/Uv/AmpqAWHwmTOTn3nJvcJF4S+Kl0nNectbC4tLySXy2srW9sbhW3d5ppnAnGGywOYnHtuSkP/Ig3pC8Dfp0I7oZewFve8FzFW3dcpH4cXclRwjuh24/8ns9cSVT94rZYcsqObvY8qBhQgmm1uDjFDbqIwZAhBEcESTiAi5R6GxU4SIjrYEycIOTrOMc9CuTNSMVJ4RI7pLFPs7ZhI5qrnKl2M1oloE+Q08YBeWLSCcJqNVvHM51ZsT/lHuucam8j+nsmV0isxIDYv3wz5X99qhaJHk51DT7VlGhGVcdMlkyfitq5/akqSRkS4hTuUlwQZto5O2dbe1JduzpbV8enWqlYNWdGm+Ht1+o86qG6E3oCle8XPg+aR+UK4fpxqXpmHkMee9jHId34Caq4RA0NvfoDnjCxBtajNbGeP6RWznh28aVZL+/Gt5as D product and it applies to block-matrices: the first is at the level of external blocks and the second is at FIG. 15. In this figure we prove the bisymmetry between the inner (in-block) level. Finally, outer and inner in- Kathri-Rao and Hadamard products: (A B) ◦ (C D) = dices are vectorized to obtain a matrix: (A?B)mn = (A ◦ C) (B ◦ D). To prove this identity we have applied Aijk`Bpqrsγipkr,mγjq`s,n the rule in Fig. 6 to the wires on the left. Note that all of the wires on the right belong to the same rank-5 Kronecker tensor, so it does not matter where they attach to.
AAAAC5nicjVLLSgMxFD0dX7W+qi7dDBbBVZmKoMuqG5ct2FaoRWbStB06LzIZoRS/QHBlcetv+Qf1L7yJKajFR8JkTs495yY3iZcEfiod5zVnLSwuLa/kVwtr6xubW8XtnWYaZ4LxBouDWFx7bsoDP+IN6cuAXyeCu6EX8JY3vFDx1h0XqR9HV3KU8E7o9iO/5zNXElU/uy2WnLKjmz0PKgaUYFotLk5xgy5iMGQIwRFBEg7gIqXeRgUOEuI6GBMnCPk6znGPAnkzUnFSuMQOaezTrG3YiOYqZ6rdjFYJ6BPktHFAnph0grBazdbxTGdW7E+5xzqn2tuI/p7JFRIrMSD2L99M+V+fqkWih1Ndg081JZpR1TGTJdOnonZuf6pKUoaEOIW7FBeEmXbOztnWnlTXrs7W1fGpVipWzZnRZnj7tTqPeqjuhJ5A5fuFz4PmUblCuH5cqp6bx5DHHvZxSDd+giouUUNDr/6AJ0ysgfVoTaznD6mVM55dfGnWyzu/MZap
AAAAC5nicjVLLSgMxFD0dX7W+qi7dDBbBVZmKoMuqG5ct2FaoRWbStB06LzIZoRS/QHBlcetv+Qf1L7yJKajFR8JkTs495yY3iZcEfiod5zVnLSwuLa/kVwtr6xubW8XtnWYaZ4LxBouDWFx7bsoDP+IN6cuAXyeCu6EX8JY3vFDx1h0XqR9HV3KU8E7o9iO/5zNXElU/uy2WnLKjmz0PKgaUYFotLk5xgy5iMGQIwRFBEg7gIqXeRgUOEuI6GBMnCPk6znGPAnkzUnFSuMQOaezTrG3YiOYqZ6rdjFYJ6BPktHFAnph0grBazdbxTGdW7E+5xzqn2tuI/p7JFRIrMSD2L99M+V+fqkWih1Ndg081JZpR1TGTJdOnonZuf6pKUoaEOIW7FBeEmXbOztnWnlTXrs7W1fGpVipWzZnRZnj7tTqPeqjuhJ5A5fuFz4PmUblCuH5cqp6bx5DHHvZxSDd+giouUUNDr/6AJ0ysgfVoTaznD6mVM55dfGnWyzu/MZap AAAAC5nicjVLLSgMxFD0dX7W+qi7dDBbBVZmKoMuqG5ct2FaoRWbStB06LzIZoRS/QHBlcetv+Qf1L7yJKajFR8JkTs495yY3iZcEfiod5zVnLSwuLa/kVwtr6xubW8XtnWYaZ4LxBouDWFx7bsoDP+IN6cuAXyeCu6EX8JY3vFDx1h0XqR9HV3KU8E7o9iO/5zNXElU/uy2WnLKjmz0PKgaUYFotLk5xgy5iMGQIwRFBEg7gIqXeRgUOEuI6GBMnCPk6znGPAnkzUnFSuMQOaezTrG3YiOYqZ6rdjFYJ6BPktHFAnph0grBazdbxTGdW7E+5xzqn2tuI/p7JFRIrMSD2L99M+V+fqkWih1Ndg081JZpR1TGTJdOnonZuf6pKUoaEOIW7FBeEmXbOztnWnlTXrs7W1fGpVipWzZnRZnj7tTqPeqjuhJ5A5fuFz4PmUblCuH5cqp6bx5DHHvZxSDd+giouUUNDr/6AJ0ysgfVoTaznD6mVM55dfGnWyzu/MZap
AAAC5nicjVLLSgMxFD0dX/VddelmsAiuylQEXZa6cdmCfUAtMpOm7dB5kckIpfgFgiuLW3/LP6h/4U1MQS0+EiZzcu45N7lJvCTwU+k4rzlraXlldS2/vrG5tb2zW9jbb6ZxJhhvsDiIRdtzUx74EW9IXwa8nQjuhl7AW97oUsVbd1ykfhxdy3HCu6E7iPy+z1xJVL16Wyg6JUc3exGUDSjCtFpcmOEGPcRgyBCCI4IkHMBFSr2DMhwkxHUxIU4Q8nWc4x4b5M1IxUnhEjuicUCzjmEjmqucqXYzWiWgT5DTxjF5YtIJwmo1W8cznVmxP+We6Jxqb2P6eyZXSKzEkNi/fHPlf32qFok+LnQNPtWUaEZVx0yWTJ+K2rn9qSpJGRLiFO5RXBBm2jk/Z1t7Ul27OltXx2daqVg1Z0ab4e3X6jzqoboTegLl7xe+CJqnpTLh+lmxUjWPIY9DHOGEbvwcFVyhhoZe/QFPmFpD69GaWs8fUitnPAf40qyXd8Gzlqo= B CAAAC5nicjVLLSgMxFD0dX/VddelmsAiuylQEXRa7cdmCfUAtMpOm7dB5kckIpfgFgiuLW3/LP6h/4U1MQS0+EiZzcu45N7lJvCTwU+k4rzlraXlldS2/vrG5tb2zW9jbb6ZxJhhvsDiIRdtzUx74EW9IXwa8nQjuhl7AW96oquKtOy5SP46u5Tjh3dAdRH7fZ64kql69LRSdkqObvQjKBhRhWi0uzHCDHmIwZAjBEUESDuAipd5BGQ4S4rqYECcI+TrOcY8N8mak4qRwiR3ROKBZx7ARzVXOVLsZrRLQJ8hp45g8MekEYbWareOZzqzYn3JPdE61tzH9PZMrJFZiSOxfvrnyvz5Vi0QfF7oGn2pKNKOqYyZLpk9F7dz+VJWkDAlxCvcoLggz7Zyfs609qa5dna2r4zOtVKyaM6PN8PZrdR71UN0JPYHy9wtfBM3TUplw/axYuTSPIY9DHOGEbvwcFVyhhoZe/QFPmFpD69GaWs8fUitnPAf40qyXd8Q1lqs=
AAAC5nicjVLLSgMxFD0dX/VddelmsAiuylQEXZa6cdmCfUAtMpOm7dB5kckIpfgFgiuLW3/LP6h/4U1MQS0+EiZzcu45N7lJvCTwU+k4rzlraXlldS2/vrG5tb2zW9jbb6ZxJhhvsDiIRdtzUx74EW9IXwa8nQjuhl7AW97oUsVbd1ykfhxdy3HCu6E7iPy+z1xJVL16Wyg6JUc3exGUDSjCtFpcmOEGPcRgyBCCI4IkHMBFSr2DMhwkxHUxIU4Q8nWc4x4b5M1IxUnhEjuicUCzjmEjmqucqXYzWiWgT5DTxjF5YtIJwmo1W8cznVmxP+We6Jxqb2P6eyZXSKzEkNi/fHPlf32qFok+LnQNPtWUaEZVx0yWTJ+K2rn9qSpJGRLiFO5RXBBm2jk/Z1t7Ul27OltXx2daqVg1Z0ab4e3X6jzqoboTegLl7xe+CJqnpTLh+lmxUjWPIY9DHOGEbvwcFVyhhoZe/QFPmFpD69GaWs8fUitnPAf40qyXd8Gzlqo= B =AAAC5nicjVLLSgMxFD0dX7W+qi7dDBbBVZmKoBuh6MZlC7YVapGZadqGzotMRijFLxBcWdz6W/5B/QtvYgpq8ZEwmZNzz7nJTeIlAU+l47zmrIXFpeWV/GphbX1jc6u4vdNM40z4rOHHQSyuPTdlAY9YQ3IZsOtEMDf0Atbyhhcq3rpjIuVxdCVHCeuEbj/iPe67kqj62W2x5JQd3ex5UDGgBNNqcXGKG3QRw0eGEAwRJOEALlLqbVTgICGugzFxghDXcYZ7FMibkYqRwiV2SGOfZm3DRjRXOVPt9mmVgD5BThsH5IlJJwir1Wwdz3Rmxf6Ue6xzqr2N6O+ZXCGxEgNi//LNlP/1qVokejjVNXCqKdGMqs43WTJ9Kmrn9qeqJGVIiFO4S3FB2NfO2Tnb2pPq2tXZujo+1UrFqrlvtBnefq3Oox6qO6EnUPl+4fOgeVSuEK4fl6rn5jHksYd9HNKNn6CKS9TQ0Ks/4AkTa2A9WhPr+UNq5YxnF1+a9fIOtSmWpQ==
AAAC5nicjVLLSgMxFD0dX/VddelmsAiuylQEXZa6cdmCfUAtMpOm7dB5kckIpfgFgiuLW3/LP6h/4U1MQS0+EiZzcu45N7lJvCTwU+k4rzlraXlldS2/vrG5tb2zW9jbb6ZxJhhvsDiIRdtzUx74EW9IXwa8nQjuhl7AW97oUsVbd1ykfhxdy3HCu6E7iPy+z1xJVL16Wyg6JUc3exGUDSjCtFpcmOEGPcRgyBCCI4IkHMBFSr2DMhwkxHUxIU4Q8nWc4x4b5M1IxUnhEjuicUCzjmEjmqucqXYzWiWgT5DTxjF5YtIJwmo1W8cznVmxP+We6Jxqb2P6eyZXSKzEkNi/fHPlf32qFok+LnQNPtWUaEZVx0yWTJ+K2rn9qSpJGRLiFO5RXBBm2jk/Z1t7Ul27OltXx2daqVg1Z0ab4e3X6jzqoboTegLl7xe+CJqnpTLh+lmxUjWPIY9DHOGEbvwcFVyhhoZe/QFPmFpD69GaWs8fUitnPAf40qyXd8Gzlqo= CAAAC5nicjVLLSgMxFD0dX/VddelmsAiuylQEXRa7cdmCfUAtMpOm7dB5kckIpfgFgiuLW3/LP6h/4U1MQS0+EiZzcu45N7lJvCTwU+k4rzlraXlldS2/vrG5tb2zW9jbb6ZxJhhvsDiIRdtzUx74EW9IXwa8nQjuhl7AW96oquKtOy5SP46u5Tjh3dAdRH7fZ64kql69LRSdkqObvQjKBhRhWi0uzHCDHmIwZAjBEUESDuAipd5BGQ4S4rqYECcI+TrOcY8N8mak4qRwiR3ROKBZx7ARzVXOVLsZrRLQJ8hp45g8MekEYbWareOZzqzYn3JPdE61tzH9PZMrJFZiSOxfvrnyvz5Vi0QfF7oGn2pKNKOqYyZLpk9F7dz+VJWkDAlxCvcoLggz7Zyfs609qa5dna2r4zOtVKyaM6PN8PZrdR71UN0JPYHy9wtfBM3TUplw/axYuTSPIY9DHOGEbvwcFVyhhoZe/QFPmFpD69GaWs8fUitnPAf40qyXd8Q1lqs= B
AAAC5nicjVLLSgMxFD0dX7W+qi7dDBbBVZmKoMuiLly2YFuhFplJ03bovMhkhFL8AsGVxa2/5R/Uv/AmpqAWHwmTOTn3nJvcJF4S+Kl0nNectbC4tLySXy2srW9sbhW3d5ppnAnGGywOYnHtuSkP/Ig3pC8Dfp0I7oZewFve8FzFW3dcpH4cXclRwjuh24/8ns9cSVT94rZYcsqObvY8qBhQgmm1uDjFDbqIwZAhBEcESTiAi5R6GxU4SIjrYEycIOTrOMc9CuTNSMVJ4RI7pLFPs7ZhI5qrnKl2M1oloE+Q08YBeWLSCcJqNVvHM51ZsT/lHuucam8j+nsmV0isxIDYv3wz5X99qhaJHk51DT7VlGhGVcdMlkyfitq5/akqSRkS4hTuUlwQZto5O2dbe1JduzpbV8enWqlYNWdGm+Ht1+o86qG6E3oCle8XPg+aR+UK4fpxqXpmHkMee9jHId34Caq4RA0NvfoDnjCxBtajNbGeP6RWznh28aVZL+/Gt5as D DAAAC5nicjVLLSgMxFD0dX7W+qi7dDBbBVZmKoMuiLly2YFuhFplJ03bovMhkhFL8AsGVxa2/5R/Uv/AmpqAWHwmTOTn3nJvcJF4S+Kl0nNectbC4tLySXy2srW9sbhW3d5ppnAnGGywOYnHtuSkP/Ig3pC8Dfp0I7oZewFve8FzFW3dcpH4cXclRwjuh24/8ns9cSVT94rZYcsqObvY8qBhQgmm1uDjFDbqIwZAhBEcESTiAi5R6GxU4SIjrYEycIOTrOMc9CuTNSMVJ4RI7pLFPs7ZhI5qrnKl2M1oloE+Q08YBeWLSCcJqNVvHM51ZsT/lHuucam8j+nsmV0isxIDYv3wz5X99qhaJHk51DT7VlGhGVcdMlkyfitq5/akqSRkS4hTuUlwQZto5O2dbe1JduzpbV8enWqlYNWdGm+Ht1+o86qG6E3oCle8XPg+aR+UK4fpxqXpmHkMee9jHId34Caq4RA0NvfoDnjCxBtajNbGeP6RWznh28aVZL+/Gt5as FIG. 13. In the Tracy-Singh product we first combine the indices of the same level (outer with outer and inner with inner) and only afterward we combine the inner/outer indices FIG. 16. In this figure we prove the bisymmetry between with each other. This can be performed in a single step if we Kronecker and Hadamard products: (A ⊗ B) ◦ (C ⊗ D) = are careful with the order. (A ◦ C) ⊗ (B ◦ D). To prove this identity we have applied the rule in Fig. 6 to the wires on both sides.
We leave as a useful exercise for the reader to prove IV. IDENTITIES A B = (A 1)(1 B), Tr(A B) = Tr(A)Tr(B) and (A⊗ B)T = ⊗AT B⊗T . ⊗ ⊗ ⊗ Thanks to our graphical notation we can easily prove many identities that connect the various products to each A. Identities involving vectorization other. Several are in the form
f(g(A, B), g(C,D)) = g(f(A, C), f(B,D)) (5) Vectorization (as introduced above through the tensor γ) is the procedure of turning a high-rank tensor into a where f and g can be some combinations of the prod- column or row vector by stacking its entries in some or- ucts that we have seen above, and A, B, C, D are suit- der. For notational simplicity, when we stack the columns ably sized matrices or tensors. Such property is called of the matrix A into a column vector we will indicate the bysimmetry (notice that B and C are swapped) and it operation with col(A), and when we concatenate the rows is due to the fact that f and g are homomorphisms that of A into a row vector we will indicate it with row(A). preserve each other [18]. The two are related by col(A)T = row(AT ), as stacking The proofs consist in visual representations of the the columns of a matrix and then transposing the result- statement that we want to prove. Note that often just ing vector is the same as transposing the matrix and then writing the identities in graphical notation is sufficient to concatenating the rows. 6
T
AAAC6nicjVLLTgIxFD2ML8QX6tLNRGLiigzGRJeoG5eYMECCaGZKwYZ5ZaZjQgjfYOJK4taf8g/wL7ytQ6ISH22mc3ruObe9bd3IE4m0rNecsbC4tLySXy2srW9sbhW3dxpJmMaM2yz0wrjlOgn3RMBtKaTHW1HMHd/1eNMdXKh4857HiQiDuhxGvOM7/UD0BHMkUfbZzag+vi2WrLKlmzkPKhkoIWu1sDjFNboIwZDCB0cASdiDg4R6GxVYiIjrYERcTEjoOMcYBfKmpOKkcIgd0NinWTtjA5qrnIl2M1rFoy8mp4kD8oSkiwmr1UwdT3Vmxf6Ue6Rzqr0N6e9muXxiJe6I/cs3U/7Xp2qR6OFU1yCopkgzqjqWZUn1qaidm5+qkpQhIk7hLsVjwkw7Z+dsak+ia1dn6+j4VCsVq+Ys06Z4+7U6l7qv7oSeQOX7hc+DxlG5QvjquFQ9zx5DHnvYxyHd+AmquEQNNq0g8IAnTAzPeDQmxvOH1Mhlnl18acbLO4bMmHs=
AAAC5nicjVLLSgMxFD0dX7W+qi7dDBbBVZmKoMuqG5ct2FaoRWbStB06LzIZoRS/QHBlcetv+Qf1L7yJKajFR8JkTs495yY3iZcEfiod5zVnLSwuLa/kVwtr6xubW8XtnWYaZ4LxBouDWFx7bsoDP+IN6cuAXyeCu6EX8JY3vFDx1h0XqR9HV3KU8E7o9iO/5zNXElU/uy2WnLKjmz0PKgaUYFotLk5xgy5iMGQIwRFBEg7gIqXeRgUOEuI6GBMnCPk6znGPAnkzUnFSuMQOaezTrG3YiOYqZ6rdjFYJ6BPktHFAnph0grBazdbxTGdW7E+5xzqn2tuI/p7JFRIrMSD2L99M+V+fqkWih1Ndg081JZpR1TGTJdOnonZuf6pKUoaEOIW7FBeEmXbOztnWnlTXrs7W1fGpVipWzZnRZnj7tTqPeqjuhJ5A5fuFz4PmUblCuH5cqp6bx5DHHvZxSDd+giouUUNDr/6AJ0ysgfVoTaznD6mVM55dfGnWyzu/MZap A A CAAAC5nicjVLLSgMxFD0dX/VddelmsAiuylQEXRa7cdmCfUAtMpOm7dB5kckIpfgFgiuLW3/LP6h/4U1MQS0+EiZzcu45N7lJvCTwU+k4rzlraXlldS2/vrG5tb2zW9jbb6ZxJhhvsDiIRdtzUx74EW9IXwa8nQjuhl7AW96oquKtOy5SP46u5Tjh3dAdRH7fZ64kql69LRSdkqObvQjKBhRhWi0uzHCDHmIwZAjBEUESDuAipd5BGQ4S4rqYECcI+TrOcY8N8mak4qRwiR3ROKBZx7ARzVXOVLsZrRLQJ8hp45g8MekEYbWareOZzqzYn3JPdE61tzH9PZMrJFZiSOxfvrnyvz5Vi0QfF7oGn2pKNKOqYyZLpk9F7dz+VJWkDAlxCvcoLggz7Zyfs609qa5dna2r4zOtVKyaM6PN8PZrdR71UN0JPYHy9wtfBM3TUplw/axYuTSPIY9DHOGEbvwcFVyhhoZe/QFPmFpD69GaWs8fUitnPAf40qyXd8Q1lqs= =AAAC5nicjVLLSgMxFD0dX7W+qi7dDBbBVZmKoBuh6MZlC7YVapGZadqGzotMRijFLxBcWdz6W/5B/QtvYgpq8ZEwmZNzz7nJTeIlAU+l47zmrIXFpeWV/GphbX1jc6u4vdNM40z4rOHHQSyuPTdlAY9YQ3IZsOtEMDf0Atbyhhcq3rpjIuVxdCVHCeuEbj/iPe67kqj62W2x5JQd3ex5UDGgBNNqcXGKG3QRw0eGEAwRJOEALlLqbVTgICGugzFxghDXcYZ7FMibkYqRwiV2SGOfZm3DRjRXOVPt9mmVgD5BThsH5IlJJwir1Wwdz3Rmxf6Ue6xzqr2N6O+ZXCGxEgNi//LNlP/1qVokejjVNXCqKdGMqs43WTJ9Kmrn9qeqJGVIiFO4S3FB2NfO2Tnb2pPq2tXZujo+1UrFqrlvtBnefq3Oox6qO6EnUPl+4fOgeVSuEK4fl6rn5jHksYd9HNKNn6CKS9TQ0Ks/4AkTa2A9WhPr+UNq5YxnF1+a9fIOtSmWpQ==
AAAAC5nicjVLLSgMxFD0dX7W+qi7dDBbBVZmKoMuqG5ct2FaoRWbStB06LzIZoRS/QHBlcetv+Qf1L7yJKajFR8JkTs495yY3iZcEfiod5zVnLSwuLa/kVwtr6xubW8XtnWYaZ4LxBouDWFx7bsoDP+IN6cuAXyeCu6EX8JY3vFDx1h0XqR9HV3KU8E7o9iO/5zNXElU/uy2WnLKjmz0PKgaUYFotLk5xgy5iMGQIwRFBEg7gIqXeRgUOEuI6GBMnCPk6znGPAnkzUnFSuMQOaezTrG3YiOYqZ6rdjFYJ6BPktHFAnph0grBazdbxTGdW7E+5xzqn2tuI/p7JFRIrMSD2L99M+V+fqkWih1Ndg081JZpR1TGTJdOnonZuf6pKUoaEOIW7FBeEmXbOztnWnlTXrs7W1fGpVipWzZnRZnj7tTqPeqjuhJ5A5fuFz4PmUblCuH5cqp6bx5DHHvZxSDd+giouUUNDr/6AJ0ysgfVoTaznD6mVM55dfGnWyzu/MZap
AAAC5nicjVLLSgMxFD0dX7W+qi7dDBbBVZmKoMuiLly2YFuhFplJ03bovMhkhFL8AsGVxa2/5R/Uv/AmpqAWHwmTOTn3nJvcJF4S+Kl0nNectbC4tLySXy2srW9sbhW3d5ppnAnGGywOYnHtuSkP/Ig3pC8Dfp0I7oZewFve8FzFW3dcpH4cXclRwjuh24/8ns9cSVT94rZYcsqObvY8qBhQgmm1uDjFDbqIwZAhBEcESTiAi5R6GxU4SIjrYEycIOTrOMc9CuTNSMVJ4RI7pLFPs7ZhI5qrnKl2M1oloE+Q08YBeWLSCcJqNVvHM51ZsT/lHuucam8j+nsmV0isxIDYv3wz5X99qhaJHk51DT7VlGhGVcdMlkyfitq5/akqSRkS4hTuUlwQZto5O2dbe1JduzpbV8enWqlYNWdGm+Ht1+o86qG6E3oCle8XPg+aR+UK4fpxqXpmHkMee9jHId34Caq4RA0NvfoDnjCxBtajNbGeP6RWznh28aVZL+/Gt5as BAAAC5nicjVLLSgMxFD0dX/VddelmsAiuylQEXZa6cdmCfUAtMpOm7dB5kckIpfgFgiuLW3/LP6h/4U1MQS0+EiZzcu45N7lJvCTwU+k4rzlraXlldS2/vrG5tb2zW9jbb6ZxJhhvsDiIRdtzUx74EW9IXwa8nQjuhl7AW97oUsVbd1ykfhxdy3HCu6E7iPy+z1xJVL16Wyg6JUc3exGUDSjCtFpcmOEGPcRgyBCCI4IkHMBFSr2DMhwkxHUxIU4Q8nWc4x4b5M1IxUnhEjuicUCzjmEjmqucqXYzWiWgT5DTxjF5YtIJwmo1W8cznVmxP+We6Jxqb2P6eyZXSKzEkNi/fHPlf32qFok+LnQNPtWUaEZVx0yWTJ+K2rn9qSpJGRLiFO5RXBBm2jk/Z1t7Ul27OltXx2daqVg1Z0ab4e3X6jzqoboTegLl7xe+CJqnpTLh+lmxUjWPIY9DHOGEbvwcFVyhhoZe/QFPmFpD69GaWs8fUitnPAf40qyXd8Gzlqo= D T AAAAC6nicjVLLTgIxFD2ML8QX6tLNRGLiigzGRJeoG5eYMECCaGZKwYZ5ZaZjQgjfYOJK4taf8g/wL7ytQ6ISH22mc3ruObe9bd3IE4m0rNecsbC4tLySXy2srW9sbhW3dxpJmMaM2yz0wrjlOgn3RMBtKaTHW1HMHd/1eNMdXKh4857HiQiDuhxGvOM7/UD0BHMkUfbZzag+vi2WrLKlmzkPKhkoIWu1sDjFNboIwZDCB0cASdiDg4R6GxVYiIjrYERcTEjoOMcYBfKmpOKkcIgd0NinWTtjA5qrnIl2M1rFoy8mp4kD8oSkiwmr1UwdT3Vmxf6Ue6Rzqr0N6e9muXxiJe6I/cs3U/7Xp2qR6OFU1yCopkgzqjqWZUn1qaidm5+qkpQhIk7hLsVjwkw7Z+dsak+ia1dn6+j4VCsVq+Ys06Z4+7U6l7qv7oSeQOX7hc+DxlG5QvjquFQ9zx5DHnvYxyHd+AmquEQNNq0g8IAnTAzPeDQmxvOH1Mhlnl18acbLO4bMmHs=
AAAC5nicjVLLSgMxFD0dX7W+qi7dDBbBVZmKoBuh6MZlC7YVapGZadqGzotMRijFLxBcWdz6W/5B/QtvYgpq8ZEwmZNzz7nJTeIlAU+l47zmrIXFpeWV/GphbX1jc6u4vdNM40z4rOHHQSyuPTdlAY9YQ3IZsOtEMDf0Atbyhhcq3rpjIuVxdCVHCeuEbj/iPe67kqj62W2x5JQd3ex5UDGgBNNqcXGKG3QRw0eGEAwRJOEALlLqbVTgICGugzFxghDXcYZ7FMibkYqRwiV2SGOfZm3DRjRXOVPt9mmVgD5BThsH5IlJJwir1Wwdz3Rmxf6Ue6xzqr2N6O+ZXCGxEgNi//LNlP/1qVokejjVNXCqKdGMqs43WTJ9Kmrn9qeqJGVIiFO4S3FB2NfO2Tnb2pPq2tXZujo+1UrFqrlvtBnefq3Oox6qO6EnUPl+4fOgeVSuEK4fl6rn5jHksYd9HNKNn6CKS9TQ0Ks/4AkTa2A9WhPr+UNq5YxnF1+a9fIOtSmWpQ== = =AAAC5nicjVLLSgMxFD0dX7W+qi7dDBbBVZmKoBuh6MZlC7YVapGZadqGzotMRijFLxBcWdz6W/5B/QtvYgpq8ZEwmZNzz7nJTeIlAU+l47zmrIXFpeWV/GphbX1jc6u4vdNM40z4rOHHQSyuPTdlAY9YQ3IZsOtEMDf0Atbyhhcq3rpjIuVxdCVHCeuEbj/iPe67kqj62W2x5JQd3ex5UDGgBNNqcXGKG3QRw0eGEAwRJOEALlLqbVTgICGugzFxghDXcYZ7FMibkYqRwiV2SGOfZm3DRjRXOVPt9mmVgD5BThsH5IlJJwir1Wwdz3Rmxf6Ue6xzqr2N6O+ZXCGxEgNi//LNlP/1qVokejjVNXCqKdGMqs43WTJ9Kmrn9qeqJGVIiFO4S3FB2NfO2Tnb2pPq2tXZujo+1UrFqrlvtBnefq3Oox6qO6EnUPl+4fOgeVSuEK4fl6rn5jHksYd9HNKNn6CKS9TQ0Ks/4AkTa2A9WhPr+UNq5YxnF1+a9fIOtSmWpQ== AAAAC5nicjVLLSgMxFD0dX7W+qi7dDBbBVZmKoMuqG5ct2FaoRWbStB06LzIZoRS/QHBlcetv+Qf1L7yJKajFR8JkTs495yY3iZcEfiod5zVnLSwuLa/kVwtr6xubW8XtnWYaZ4LxBouDWFx7bsoDP+IN6cuAXyeCu6EX8JY3vFDx1h0XqR9HV3KU8E7o9iO/5zNXElU/uy2WnLKjmz0PKgaUYFotLk5xgy5iMGQIwRFBEg7gIqXeRgUOEuI6GBMnCPk6znGPAnkzUnFSuMQOaezTrG3YiOYqZ6rdjFYJ6BPktHFAnph0grBazdbxTGdW7E+5xzqn2tuI/p7JFRIrMSD2L99M+V+fqkWih1Ndg081JZpR1TGTJdOnonZuf6pKUoaEOIW7FBeEmXbOztnWnlTXrs7W1fGpVipWzZnRZnj7tTqPeqjuhJ5A5fuFz4PmUblCuH5cqp6bx5DHHvZxSDd+giouUUNDr/6AJ0ysgfVoTaznD6mVM55dfGnWyzu/MZap
AAAC5nicjVLLSgMxFD0dX7W+qi7dDBbBVZmKoMuqG5ct2FaoRWbStB06LzIZoRS/QHBlcetv+Qf1L7yJKajFR8JkTs495yY3iZcEfiod5zVnLSwuLa/kVwtr6xubW8XtnWYaZ4LxBouDWFx7bsoDP+IN6cuAXyeCu6EX8JY3vFDx1h0XqR9HV3KU8E7o9iO/5zNXElU/uy2WnLKjmz0PKgaUYFotLk5xgy5iMGQIwRFBEg7gIqXeRgUOEuI6GBMnCPk6znGPAnkzUnFSuMQOaezTrG3YiOYqZ6rdjFYJ6BPktHFAnph0grBazdbxTGdW7E+5xzqn2tuI/p7JFRIrMSD2L99M+V+fqkWih1Ndg081JZpR1TGTJdOnonZuf6pKUoaEOIW7FBeEmXbOztnWnlTXrs7W1fGpVipWzZnRZnj7tTqPeqjuhJ5A5fuFz4PmUblCuH5cqp6bx5DHHvZxSDd+giouUUNDr/6AJ0ysgfVoTaznD6mVM55dfGnWyzu/MZap A CAAAC5nicjVLLSgMxFD0dX/VddelmsAiuylQEXRa7cdmCfUAtMpOm7dB5kckIpfgFgiuLW3/LP6h/4U1MQS0+EiZzcu45N7lJvCTwU+k4rzlraXlldS2/vrG5tb2zW9jbb6ZxJhhvsDiIRdtzUx74EW9IXwa8nQjuhl7AW96oquKtOy5SP46u5Tjh3dAdRH7fZ64kql69LRSdkqObvQjKBhRhWi0uzHCDHmIwZAjBEUESDuAipd5BGQ4S4rqYECcI+TrOcY8N8mak4qRwiR3ROKBZx7ARzVXOVLsZrRLQJ8hp45g8MekEYbWareOZzqzYn3JPdE61tzH9PZMrJFZiSOxfvrnyvz5Vi0QfF7oGn2pKNKOqYyZLpk9F7dz+VJWkDAlxCvcoLggz7Zyfs609qa5dna2r4zOtVKyaM6PN8PZrdR71UN0JPYHy9wtfBM3TUplw/axYuTSPIY9DHOGEbvwcFVyhhoZe/QFPmFpD69GaWs8fUitnPAf40qyXd8Q1lqs= FIG. 19. A block vector (top) and a vectorized matrix (bot- tom). We denote both with col(A), as it is usually clear from the context which we are referring to.
AAAC5nicjVLLSgMxFD0dX7W+qi7dDBbBVZmKoMuiLly2YFuhFplJ03bovMhkhFL8AsGVxa2/5R/Uv/AmpqAWHwmTOTn3nJvcJF4S+Kl0nNectbC4tLySXy2srW9sbhW3d5ppnAnGGywOYnHtuSkP/Ig3pC8Dfp0I7oZewFve8FzFW3dcpH4cXclRwjuh24/8ns9cSVT94rZYcsqObvY8qBhQgmm1uDjFDbqIwZAhBEcESTiAi5R6GxU4SIjrYEycIOTrOMc9CuTNSMVJ4RI7pLFPs7ZhI5qrnKl2M1oloE+Q08YBeWLSCcJqNVvHM51ZsT/lHuucam8j+nsmV0isxIDYv3wz5X99qhaJHk51DT7VlGhGVcdMlkyfitq5/akqSRkS4hTuUlwQZto5O2dbe1JduzpbV8enWqlYNWdGm+Ht1+o86qG6E3oCle8XPg+aR+UK4fpxqXpmHkMee9jHId34Caq4RA0NvfoDnjCxBtajNbGeP6RWznh28aVZL+/Gt5as BAAAC5nicjVLLSgMxFD0dX/VddelmsAiuylQEXZa6cdmCfUAtMpOm7dB5kckIpfgFgiuLW3/LP6h/4U1MQS0+EiZzcu45N7lJvCTwU+k4rzlraXlldS2/vrG5tb2zW9jbb6ZxJhhvsDiIRdtzUx74EW9IXwa8nQjuhl7AW97oUsVbd1ykfhxdy3HCu6E7iPy+z1xJVL16Wyg6JUc3exGUDSjCtFpcmOEGPcRgyBCCI4IkHMBFSr2DMhwkxHUxIU4Q8nWc4x4b5M1IxUnhEjuicUCzjmEjmqucqXYzWiWgT5DTxjF5YtIJwmo1W8cznVmxP+We6Jxqb2P6eyZXSKzEkNi/fHPlf32qFok+LnQNPtWUaEZVx0yWTJ+K2rn9qSpJGRLiFO5RXBBm2jk/Z1t7Ul27OltXx2daqVg1Z0ab4e3X6jzqoboTegLl7xe+CJqnpTLh+lmxUjWPIY9DHOGEbvwcFVyhhoZe/QFPmFpD69GaWs8fUitnPAf40qyXd8Gzlqo= D
=AAAC5nicjVLLSgMxFD0dX7W+qi7dDBbBVZmKoBuh6MZlC7YVapGZadqGzotMRijFLxBcWdz6W/5B/QtvYgpq8ZEwmZNzz7nJTeIlAU+l47zmrIXFpeWV/GphbX1jc6u4vdNM40z4rOHHQSyuPTdlAY9YQ3IZsOtEMDf0Atbyhhcq3rpjIuVxdCVHCeuEbj/iPe67kqj62W2x5JQd3ex5UDGgBNNqcXGKG3QRw0eGEAwRJOEALlLqbVTgICGugzFxghDXcYZ7FMibkYqRwiV2SGOfZm3DRjRXOVPt9mmVgD5BThsH5IlJJwir1Wwdz3Rmxf6Ue6xzqr2N6O+ZXCGxEgNi//LNlP/1qVokejjVNXCqKdGMqs43WTJ9Kmrn9qeqJGVIiFO4S3FB2NfO2Tnb2pPq2tXZujo+1UrFqrlvtBnefq3Oox6qO6EnUPl+4fOgeVSuEK4fl6rn5jHksYd9HNKNn6CKS9TQ0Ks/4AkTa2A9WhPr+UNq5YxnF1+a9fIOtSmWpQ== the vectorization of B as block-vectorization, and then we use the identity at the bottom of Fig. 5. For the case in which B is diagonal we can use the first rule in Fig. 3.
AAAC5nicjVLLSgMxFD0dX7W+qi7dDBbBVZmKoMuqG5ct2FaoRWbStB06LzIZoRS/QHBlcetv+Qf1L7yJKajFR8JkTs495yY3iZcEfiod5zVnLSwuLa/kVwtr6xubW8XtnWYaZ4LxBouDWFx7bsoDP+IN6cuAXyeCu6EX8JY3vFDx1h0XqR9HV3KU8E7o9iO/5zNXElU/uy2WnLKjmz0PKgaUYFotLk5xgy5iMGQIwRFBEg7gIqXeRgUOEuI6GBMnCPk6znGPAnkzUnFSuMQOaezTrG3YiOYqZ6rdjFYJ6BPktHFAnph0grBazdbxTGdW7E+5xzqn2tuI/p7JFRIrMSD2L99M+V+fqkWih1Ndg081JZpR1TGTJdOnonZuf6pKUoaEOIW7FBeEmXbOztnWnlTXrs7W1fGpVipWzZnRZnj7tTqPeqjuhJ5A5fuFz4PmUblCuH5cqp6bx5DHHvZxSDd+giouUUNDr/6AJ0ysgfVoTaznD6mVM55dfGnWyzu/MZap A CAAAC5nicjVLLSgMxFD0dX/VddelmsAiuylQEXRa7cdmCfUAtMpOm7dB5kckIpfgFgiuLW3/LP6h/4U1MQS0+EiZzcu45N7lJvCTwU+k4rzlraXlldS2/vrG5tb2zW9jbb6ZxJhhvsDiIRdtzUx74EW9IXwa8nQjuhl7AW96oquKtOy5SP46u5Tjh3dAdRH7fZ64kql69LRSdkqObvQjKBhRhWi0uzHCDHmIwZAjBEUESDuAipd5BGQ4S4rqYECcI+TrOcY8N8mak4qRwiR3ROKBZx7ARzVXOVLsZrRLQJ8hp45g8MekEYbWareOZzqzYn3JPdE61tzH9PZMrJFZiSOxfvrnyvz5Vi0QfF7oGn2pKNKOqYyZLpk9F7dz+VJWkDAlxCvcoLggz7Zyfs609qa5dna2r4zOtVKyaM6PN8PZrdR71UN0JPYHy9wtfBM3TUplw/axYuTSPIY9DHOGEbvwcFVyhhoZe/QFPmFpD69GaWs8fUitnPAf40qyXd8Q1lqs=
col(ABC) = (CT A)col(B)[B not diagonal] (6)
AAAC5nicjVLLSgMxFD0dX7W+qi7dDBbBVZmKoMuiLly2YFuhFplJ03bovMhkhFL8AsGVxa2/5R/Uv/AmpqAWHwmTOTn3nJvcJF4S+Kl0nNectbC4tLySXy2srW9sbhW3d5ppnAnGGywOYnHtuSkP/Ig3pC8Dfp0I7oZewFve8FzFW3dcpH4cXclRwjuh24/8ns9cSVT94rZYcsqObvY8qBhQgmm1uDjFDbqIwZAhBEcESTiAi5R6GxU4SIjrYEycIOTrOMc9CuTNSMVJ4RI7pLFPs7ZhI5qrnKl2M1oloE+Q08YBeWLSCcJqNVvHM51ZsT/lHuucam8j+nsmV0isxIDYv3wz5X99qhaJHk51DT7VlGhGVcdMlkyfitq5/akqSRkS4hTuUlwQZto5O2dbe1JduzpbV8enWqlYNWdGm+Ht1+o86qG6E3oCle8XPg+aR+UK4fpxqXpmHkMee9jHId34Caq4RA0NvfoDnjCxBtajNbGeP6RWznh28aVZL+/Gt5as BAAAC5nicjVLLSgMxFD0dX/VddelmsAiuylQEXZa6cdmCfUAtMpOm7dB5kckIpfgFgiuLW3/LP6h/4U1MQS0+EiZzcu45N7lJvCTwU+k4rzlraXlldS2/vrG5tb2zW9jbb6ZxJhhvsDiIRdtzUx74EW9IXwa8nQjuhl7AW97oUsVbd1ykfhxdy3HCu6E7iPy+z1xJVL16Wyg6JUc3exGUDSjCtFpcmOEGPcRgyBCCI4IkHMBFSr2DMhwkxHUxIU4Q8nWc4x4b5M1IxUnhEjuicUCzjmEjmqucqXYzWiWgT5DTxjF5YtIJwmo1W8cznVmxP+We6Jxqb2P6eyZXSKzEkNi/fHPlf32qFok+LnQNPtWUaEZVx0yWTJ+K2rn9qSpJGRLiFO5RXBBm2jk/Z1t7Ul27OltXx2daqVg1Z0ab4e3X6jzqoboTegLl7xe+CJqnpTLh+lmxUjWPIY9DHOGEbvwcFVyhhoZe/QFPmFpD69GaWs8fUitnPAf40qyXd8Gzlqo= D ⊗ col(ABC) = (CT A)diag(B)[B diagonal] (7)
FIG. 17. In this figure we prove (A?B)(C?D) = (AC)?(BD). Note that in the first equation we use a Kronecker prod- To prove this identity we have applied the rule at the bottom uct (not a tensor product). of Fig. 5.
T
CAAAC6HicjVLLSgMxFD0dX7W+qi7dDBbBVZkRQZfFblxW7AtqlZlpWsfOi0xGKMVPEFxZ3PpX/kH9C29iBLX4SJjMybnn3OQmcZPAT4VlveSMufmFxaX8cmFldW19o7i51UzjjHus4cVBzNuuk7LAj1hD+CJg7YQzJ3QD1nKHVRlv3TKe+nFUF6OEdUNnEPl933MEUefVy/pVsWSVLdXMWWBrUIJutbg4xQV6iOEhQwiGCIJwAAcp9Q5sWEiI62JMHCfkqzjDHQrkzUjFSOEQO6RxQLOOZiOay5ypcnu0SkAfJ6eJPfLEpOOE5Wqmimcqs2R/yj1WOeXeRvR3da6QWIFrYv/yfSj/65O1CPRxrGrwqaZEMbI6T2fJ1KnInZufqhKUISFO4h7FOWFPOT/O2VSeVNUuz9ZR8alSSlbOPa3N8PprdS71UN4JPQH7+4XPguZB2SZ8dliqnOjHkMcOdrFPN36ECk5RQ4NWGOAej5gYN8aDMTGe3qVGTnu28aUZz2/PYZdx T
CAAAC6HicjVLLSgMxFD0dX7W+qi7dDBbBVZkRQZfFblxW7AtqlZlpWsfOi0xGKMVPEFxZ3PpX/kH9C29iBLX4SJjMybnn3OQmcZPAT4VlveSMufmFxaX8cmFldW19o7i51UzjjHus4cVBzNuuk7LAj1hD+CJg7YQzJ3QD1nKHVRlv3TKe+nFUF6OEdUNnEPl933MEUefVy/pVsWSVLdXMWWBrUIJutbg4xQV6iOEhQwiGCIJwAAcp9Q5sWEiI62JMHCfkqzjDHQrkzUjFSOEQO6RxQLOOZiOay5ypcnu0SkAfJ6eJPfLEpOOE5Wqmimcqs2R/yj1WOeXeRvR3da6QWIFrYv/yfSj/65O1CPRxrGrwqaZEMbI6T2fJ1KnInZufqhKUISFO4h7FOWFPOT/O2VSeVNUuz9ZR8alSSlbOPa3N8PprdS71UN4JPQH7+4XPguZB2SZ8dliqnOjHkMcOdrFPN36ECk5RQ4NWGOAej5gYN8aDMTGe3qVGTnu28aUZz2/PYZdx
AAAC5nicjVLLSgMxFD0dX7W+qi7dDBbBVZmKoBuh6MZlC7YVapGZadqGzotMRijFLxBcWdz6W/5B/QtvYgpq8ZEwmZNzz7nJTeIlAU+l47zmrIXFpeWV/GphbX1jc6u4vdNM40z4rOHHQSyuPTdlAY9YQ3IZsOtEMDf0Atbyhhcq3rpjIuVxdCVHCeuEbj/iPe67kqj62W2x5JQd3ex5UDGgBNNqcXGKG3QRw0eGEAwRJOEALlLqbVTgICGugzFxghDXcYZ7FMibkYqRwiV2SGOfZm3DRjRXOVPt9mmVgD5BThsH5IlJJwir1Wwdz3Rmxf6Ue6xzqr2N6O+ZXCGxEgNi//LNlP/1qVokejjVNXCqKdGMqs43WTJ9Kmrn9qeqJGVIiFO4S3FB2NfO2Tnb2pPq2tXZujo+1UrFqrlvtBnefq3Oox6qO6EnUPl+4fOgeVSuEK4fl6rn5jHksYd9HNKNn6CKS9TQ0Ks/4AkTa2A9WhPr+UNq5YxnF1+a9fIOtSmWpQ==
AAAC5nicjVLLSgMxFD0dX/VddelmsAiuylQEXZa6cdmCfUAtMpOm7dB5kckIpfgFgiuLW3/LP6h/4U1MQS0+EiZzcu45N7lJvCTwU+k4rzlraXlldS2/vrG5tb2zW9jbb6ZxJhhvsDiIRdtzUx74EW9IXwa8nQjuhl7AW97oUsVbd1ykfhxdy3HCu6E7iPy+z1xJVL16Wyg6JUc3exGUDSjCtFpcmOEGPcRgyBCCI4IkHMBFSr2DMhwkxHUxIU4Q8nWc4x4b5M1IxUnhEjuicUCzjmEjmqucqXYzWiWgT5DTxjF5YtIJwmo1W8cznVmxP+We6Jxqb2P6eyZXSKzEkNi/fHPlf32qFok+LnQNPtWUaEZVx0yWTJ+K2rn9qSpJGRLiFO5RXBBm2jk/Z1t7Ul27OltXx2daqVg1Z0ab4e3X6jzqoboTegLl7xe+CJqnpTLh+lmxUjWPIY9DHOGEbvwcFVyhhoZe/QFPmFpD69GaWs8fUitnPAf40qyXd8Gzlqo=
AAAC5nicjVLLSgMxFD0dX7W+qi7dDBbBVZmKoBuh6MZlC7YVapGZadqGzotMRijFLxBcWdz6W/5B/QtvYgpq8ZEwmZNzz7nJTeIlAU+l47zmrIXFpeWV/GphbX1jc6u4vdNM40z4rOHHQSyuPTdlAY9YQ3IZsOtEMDf0Atbyhhcq3rpjIuVxdCVHCeuEbj/iPe67kqj62W2x5JQd3ex5UDGgBNNqcXGKG3QRw0eGEAwRJOEALlLqbVTgICGugzFxghDXcYZ7FMibkYqRwiV2SGOfZm3DRjRXOVPt9mmVgD5BThsH5IlJJwir1Wwdz3Rmxf6Ue6xzqr2N6O+ZXCGxEgNi//LNlP/1qVokejjVNXCqKdGMqs43WTJ9Kmrn9qeqJGVIiFO4S3FB2NfO2Tnb2pPq2tXZujo+1UrFqrlvtBnefq3Oox6qO6EnUPl+4fOgeVSuEK4fl6rn5jHksYd9HNKNn6CKS9TQ0Ks/4AkTa2A9WhPr+UNq5YxnF1+a9fIOtSmWpQ==
AAAC5nicjVLLSgMxFD0dX/VddelmsAiuylQEXZa6cdmCfUAtMpOm7dB5kckIpfgFgiuLW3/LP6h/4U1MQS0+EiZzcu45N7lJvCTwU+k4rzlraXlldS2/vrG5tb2zW9jbb6ZxJhhvsDiIRdtzUx74EW9IXwa8nQjuhl7AW97oUsVbd1ykfhxdy3HCu6E7iPy+z1xJVL16Wyg6JUc3exGUDSjCtFpcmOEGPcRgyBCCI4IkHMBFSr2DMhwkxHUxIU4Q8nWc4x4b5M1IxUnhEjuicUCzjmEjmqucqXYzWiWgT5DTxjF5YtIJwmo1W8cznVmxP+We6Jxqb2P6eyZXSKzEkNi/fHPlf32qFok+LnQNPtWUaEZVx0yWTJ+K2rn9qSpJGRLiFO5RXBBm2jk/Z1t7Ul27OltXx2daqVg1Z0ab4e3X6jzqoboTegLl7xe+CJqnpTLh+lmxUjWPIY9DHOGEbvwcFVyhhoZe/QFPmFpD69GaWs8fUitnPAf40qyXd8Gzlqo= = B T AAAC5nicjVLLSgMxFD0dX7W+qi7dDBbBVZmKoMuqG5ct2FaoRWbStB06LzIZoRS/QHBlcetv+Qf1L7yJKajFR8JkTs495yY3iZcEfiod5zVnLSwuLa/kVwtr6xubW8XtnWYaZ4LxBouDWFx7bsoDP+IN6cuAXyeCu6EX8JY3vFDx1h0XqR9HV3KU8E7o9iO/5zNXElU/uy2WnLKjmz0PKgaUYFotLk5xgy5iMGQIwRFBEg7gIqXeRgUOEuI6GBMnCPk6znGPAnkzUnFSuMQOaezTrG3YiOYqZ6rdjFYJ6BPktHFAnph0grBazdbxTGdW7E+5xzqn2tuI/p7JFRIrMSD2L99M+V+fqkWih1Ndg081JZpR1TGTJdOnonZuf6pKUoaEOIW7FBeEmXbOztnWnlTXrs7W1fGpVipWzZnRZnj7tTqPeqjuhJ5A5fuFz4PmUblCuH5cqp6bx5DHHvZxSDd+giouUUNDr/6AJ0ysgfVoTaznD6mVM55dfGnWyzu/MZap
AAAC5nicjVLLSgMxFD0dX/VddelmsAiuylQEXZa6cdmCfUAtMpOm7dB5kckIpfgFgiuLW3/LP6h/4U1MQS0+EiZzcu45N7lJvCTwU+k4rzlraXlldS2/vrG5tb2zW9jbb6ZxJhhvsDiIRdtzUx74EW9IXwa8nQjuhl7AW97oUsVbd1ykfhxdy3HCu6E7iPy+z1xJVL16Wyg6JUc3exGUDSjCtFpcmOEGPcRgyBCCI4IkHMBFSr2DMhwkxHUxIU4Q8nWc4x4b5M1IxUnhEjuicUCzjmEjmqucqXYzWiWgT5DTxjF5YtIJwmo1W8cznVmxP+We6Jxqb2P6eyZXSKzEkNi/fHPlf32qFok+LnQNPtWUaEZVx0yWTJ+K2rn9qSpJGRLiFO5RXBBm2jk/Z1t7Ul27OltXx2daqVg1Z0ab4e3X6jzqoboTegLl7xe+CJqnpTLh+lmxUjWPIY9DHOGEbvwcFVyhhoZe/QFPmFpD69GaWs8fUitnPAf40qyXd8Gzlqo=
AAAC5nicjVLLSgMxFD0dX/VddelmsAiuylQEXRa7cdmCfUAtMpOm7dB5kckIpfgFgiuLW3/LP6h/4U1MQS0+EiZzcu45N7lJvCTwU+k4rzlraXlldS2/vrG5tb2zW9jbb6ZxJhhvsDiIRdtzUx74EW9IXwa8nQjuhl7AW96oquKtOy5SP46u5Tjh3dAdRH7fZ64kql69LRSdkqObvQjKBhRhWi0uzHCDHmIwZAjBEUESDuAipd5BGQ4S4rqYECcI+TrOcY8N8mak4qRwiR3ROKBZx7ARzVXOVLsZrRLQJ8hp45g8MekEYbWareOZzqzYn3JPdE61tzH9PZMrJFZiSOxfvrnyvz5Vi0QfF7oGn2pKNKOqYyZLpk9F7dz+VJWkDAlxCvcoLggz7Zyfs609qa5dna2r4zOtVKyaM6PN8PZrdR71UN0JPYHy9wtfBM3TUplw/axYuTSPIY9DHOGEbvwcFVyhhoZe/QFPmFpD69GaWs8fUitnPAf40qyXd8Q1lqs= AAAC5nicjVLLSgMxFD0dX7W+qi7dDBbBVZmKoMuqG5ct2FaoRWbStB06LzIZoRS/QHBlcetv+Qf1L7yJKajFR8JkTs495yY3iZcEfiod5zVnLSwuLa/kVwtr6xubW8XtnWYaZ4LxBouDWFx7bsoDP+IN6cuAXyeCu6EX8JY3vFDx1h0XqR9HV3KU8E7o9iO/5zNXElU/uy2WnLKjmz0PKgaUYFotLk5xgy5iMGQIwRFBEg7gIqXeRgUOEuI6GBMnCPk6znGPAnkzUnFSuMQOaezTrG3YiOYqZ6rdjFYJ6BPktHFAnph0grBazdbxTGdW7E+5xzqn2tuI/p7JFRIrMSD2L99M+V+fqkWih1Ndg081JZpR1TGTJdOnonZuf6pKUoaEOIW7FBeEmXbOztnWnlTXrs7W1fGpVipWzZnRZnj7tTqPeqjuhJ5A5fuFz4PmUblCuH5cqp6bx5DHHvZxSDd+giouUUNDr/6AJ0ysgfVoTaznD6mVM55dfGnWyzu/MZap A B AAAC6HicjVLLSgMxFD0dX7W+qi7dDBbBVZkRQZdVNy4r9gW1ykya1tF5kckIpfgJgiuLW//KP6h/4U2cglp8JEzm5NxzbnKTuLHvJdKyXnPGzOzc/EJ+sbC0vLK6VlzfaCRRKhivs8iPRMt1Eu57Ia9LT/q8FQvuBK7Pm+7tiYo377hIvCisyUHMO4HTD72exxxJ1PnRZe2qWLLKlm7mNLAzUELWqlFxjAt0EYEhRQCOEJKwDwcJ9TZsWIiJ62BInCDk6TjHPQrkTUnFSeEQe0tjn2btjA1prnIm2s1oFZ8+QU4TO+SJSCcIq9VMHU91ZsX+lHuoc6q9DejvZrkCYiWuif3LN1H+16dqkejhUNfgUU2xZlR1LMuS6lNROzc/VSUpQ0ycwl2KC8JMOyfnbGpPomtXZ+vo+FgrFavmLNOmePu1Opd6oO6EnoD9/cKnQWOvbBM+2y9VjrPHkMcWtrFLN36ACk5RRZ1W6OMBTxgZN8ajMTKeP6RGLvNs4kszXt4BylmXbw== C = AAAC5nicjVLLSgMxFD0dX/VddelmsAiuylQEXRa7cdmCfUAtMpOm7dB5kckIpfgFgiuLW3/LP6h/4U1MQS0+EiZzcu45N7lJvCTwU+k4rzlraXlldS2/vrG5tb2zW9jbb6ZxJhhvsDiIRdtzUx74EW9IXwa8nQjuhl7AW96oquKtOy5SP46u5Tjh3dAdRH7fZ64kql69LRSdkqObvQjKBhRhWi0uzHCDHmIwZAjBEUESDuAipd5BGQ4S4rqYECcI+TrOcY8N8mak4qRwiR3ROKBZx7ARzVXOVLsZrRLQJ8hp45g8MekEYbWareOZzqzYn3JPdE61tzH9PZMrJFZiSOxfvrnyvz5Vi0QfF7oGn2pKNKOqYyZLpk9F7dz+VJWkDAlxCvcoLggz7Zyfs609qa5dna2r4zOtVKyaM6PN8PZrdR71UN0JPYHy9wtfBM3TUplw/axYuTSPIY9DHOGEbvwcFVyhhoZe/QFPmFpD69GaWs8fUitnPAf40qyXd8Q1lqs= B
AAAC5nicjVLLSgMxFD0dX7W+qi7dDBbBVZmKoMuqG5ct2FaoRWbStB06LzIZoRS/QHBlcetv+Qf1L7yJKajFR8JkTs495yY3iZcEfiod5zVnLSwuLa/kVwtr6xubW8XtnWYaZ4LxBouDWFx7bsoDP+IN6cuAXyeCu6EX8JY3vFDx1h0XqR9HV3KU8E7o9iO/5zNXElU/uy2WnLKjmz0PKgaUYFotLk5xgy5iMGQIwRFBEg7gIqXeRgUOEuI6GBMnCPk6znGPAnkzUnFSuMQOaezTrG3YiOYqZ6rdjFYJ6BPktHFAnph0grBazdbxTGdW7E+5xzqn2tuI/p7JFRIrMSD2L99M+V+fqkWih1Ndg081JZpR1TGTJdOnonZuf6pKUoaEOIW7FBeEmXbOztnWnlTXrs7W1fGpVipWzZnRZnj7tTqPeqjuhJ5A5fuFz4PmUblCuH5cqp6bx5DHHvZxSDd+giouUUNDr/6AJ0ysgfVoTaznD6mVM55dfGnWyzu/MZap A CAAAC5nicjVLLSgMxFD0dX/VddelmsAiuylQEXRa7cdmCfUAtMpOm7dB5kckIpfgFgiuLW3/LP6h/4U1MQS0+EiZzcu45N7lJvCTwU+k4rzlraXlldS2/vrG5tb2zW9jbb6ZxJhhvsDiIRdtzUx74EW9IXwa8nQjuhl7AW96oquKtOy5SP46u5Tjh3dAdRH7fZ64kql69LRSdkqObvQjKBhRhWi0uzHCDHmIwZAjBEUESDuAipd5BGQ4S4rqYECcI+TrOcY8N8mak4qRwiR3ROKBZx7ARzVXOVLsZrRLQJ8hp45g8MekEYbWareOZzqzYn3JPdE61tzH9PZMrJFZiSOxfvrnyvz5Vi0QfF7oGn2pKNKOqYyZLpk9F7dz+VJWkDAlxCvcoLggz7Zyfs609qa5dna2r4zOtVKyaM6PN8PZrdR71UN0JPYHy9wtfBM3TUplw/axYuTSPIY9DHOGEbvwcFVyhhoZe/QFPmFpD69GaWs8fUitnPAf40qyXd8Q1lqs= A C A AAAAC5nicjVLLSgMxFD0dX7W+qi7dDBbBVZmKoMuqG5ct2FaoRWbStB06LzIZoRS/QHBlcetv+Qf1L7yJKajFR8JkTs495yY3iZcEfiod5zVnLSwuLa/kVwtr6xubW8XtnWYaZ4LxBouDWFx7bsoDP+IN6cuAXyeCu6EX8JY3vFDx1h0XqR9HV3KU8E7o9iO/5zNXElU/uy2WnLKjmz0PKgaUYFotLk5xgy5iMGQIwRFBEg7gIqXeRgUOEuI6GBMnCPk6znGPAnkzUnFSuMQOaezTrG3YiOYqZ6rdjFYJ6BPktHFAnph0grBazdbxTGdW7E+5xzqn2tuI/p7JFRIrMSD2L99M+V+fqkWih1Ndg081JZpR1TGTJdOnonZuf6pKUoaEOIW7FBeEmXbOztnWnlTXrs7W1fGpVipWzZnRZnj7tTqPeqjuhJ5A5fuFz4PmUblCuH5cqp6bx5DHHvZxSDd+giouUUNDr/6AJ0ysgfVoTaznD6mVM55dfGnWyzu/MZap =AAAC5nicjVLLSgMxFD0dX7W+qi7dDBbBVZmKoBuh6MZlC7YVapGZadqGzotMRijFLxBcWdz6W/5B/QtvYgpq8ZEwmZNzz7nJTeIlAU+l47zmrIXFpeWV/GphbX1jc6u4vdNM40z4rOHHQSyuPTdlAY9YQ3IZsOtEMDf0Atbyhhcq3rpjIuVxdCVHCeuEbj/iPe67kqj62W2x5JQd3ex5UDGgBNNqcXGKG3QRw0eGEAwRJOEALlLqbVTgICGugzFxghDXcYZ7FMibkYqRwiV2SGOfZm3DRjRXOVPt9mmVgD5BThsH5IlJJwir1Wwdz3Rmxf6Ue6xzqr2N6O+ZXCGxEgNi//LNlP/1qVokejjVNXCqKdGMqs43WTJ9Kmrn9qeqJGVIiFO4S3FB2NfO2Tnb2pPq2tXZujo+1UrFqrlvtBnefq3Oox6qO6EnUPl+4fOgeVSuEK4fl6rn5jHksYd9HNKNn6CKS9TQ0Ks/4AkTa2A9WhPr+UNq5YxnF1+a9fIOtSmWpQ== T
AAAC5nicjVLLSgMxFD0dX7W+qi7dDBbBVZmKoMuiLly2YFuhFplJ03bovMhkhFL8AsGVxa2/5R/Uv/AmpqAWHwmTOTn3nJvcJF4S+Kl0nNectbC4tLySXy2srW9sbhW3d5ppnAnGGywOYnHtuSkP/Ig3pC8Dfp0I7oZewFve8FzFW3dcpH4cXclRwjuh24/8ns9cSVT94rZYcsqObvY8qBhQgmm1uDjFDbqIwZAhBEcESTiAi5R6GxU4SIjrYEycIOTrOMc9CuTNSMVJ4RI7pLFPs7ZhI5qrnKl2M1oloE+Q08YBeWLSCcJqNVvHM51ZsT/lHuucam8j+nsmV0isxIDYv3wz5X99qhaJHk51DT7VlGhGVcdMlkyfitq5/akqSRkS4hTuUlwQZto5O2dbe1JduzpbV8enWqlYNWdGm+Ht1+o86qG6E3oCle8XPg+aR+UK4fpxqXpmHkMee9jHId34Caq4RA0NvfoDnjCxBtajNbGeP6RWznh28aVZL+/Gt5as
AAAC5nicjVLLSgMxFD0dX/VddelmsAiuylQEXZa6cdmCfUAtMpOm7dB5kckIpfgFgiuLW3/LP6h/4U1MQS0+EiZzcu45N7lJvCTwU+k4rzlraXlldS2/vrG5tb2zW9jbb6ZxJhhvsDiIRdtzUx74EW9IXwa8nQjuhl7AW97oUsVbd1ykfhxdy3HCu6E7iPy+z1xJVL16Wyg6JUc3exGUDSjCtFpcmOEGPcRgyBCCI4IkHMBFSr2DMhwkxHUxIU4Q8nWc4x4b5M1IxUnhEjuicUCzjmEjmqucqXYzWiWgT5DTxjF5YtIJwmo1W8cznVmxP+We6Jxqb2P6eyZXSKzEkNi/fHPlf32qFok+LnQNPtWUaEZVx0yWTJ+K2rn9qSpJGRLiFO5RXBBm2jk/Z1t7Ul27OltXx2daqVg1Z0ab4e3X6jzqoboTegLl7xe+CJqnpTLh+lmxUjWPIY9DHOGEbvwcFVyhhoZe/QFPmFpD69GaWs8fUitnPAf40qyXd8Gzlqo=
AAAC5nicjVLLSgMxFD0dX7W+qi7dDBbBVZmKoMuiLly2YFuhFplJ03bovMhkhFL8AsGVxa2/5R/Uv/AmpqAWHwmTOTn3nJvcJF4S+Kl0nNectbC4tLySXy2srW9sbhW3d5ppnAnGGywOYnHtuSkP/Ig3pC8Dfp0I7oZewFve8FzFW3dcpH4cXclRwjuh24/8ns9cSVT94rZYcsqObvY8qBhQgmm1uDjFDbqIwZAhBEcESTiAi5R6GxU4SIjrYEycIOTrOMc9CuTNSMVJ4RI7pLFPs7ZhI5qrnKl2M1oloE+Q08YBeWLSCcJqNVvHM51ZsT/lHuucam8j+nsmV0isxIDYv3wz5X99qhaJHk51DT7VlGhGVcdMlkyfitq5/akqSRkS4hTuUlwQZto5O2dbe1JduzpbV8enWqlYNWdGm+Ht1+o86qG6E3oCle8XPg+aR+UK4fpxqXpmHkMee9jHId34Caq4RA0NvfoDnjCxBtajNbGeP6RWznh28aVZL+/Gt5as B D BAAAC6HicjVLLSgMxFD0dX7W+qi7dDBbBVZkRQZelblxW7AtqlZlpWmPnxUxGKMVPEFxZ3PpX/kH9C29iBLX4SJjMybnn3OQmcWOfp8KyXnLG3PzC4lJ+ubCyura+UdzcaqZRlnis4UV+lLRdJ2U+D1lDcOGzdpwwJ3B91nKHJzLeumVJyqOwLkYx6wbOIOR97jmCqPPqZf2qWLLKlmrmLLA1KEG3WlSc4gI9RPCQIQBDCEHYh4OUegc2LMTEdTEmLiHEVZzhDgXyZqRipHCIHdI4oFlHsyHNZc5UuT1axacvIaeJPfJEpEsIy9VMFc9UZsn+lHuscsq9jejv6lwBsQLXxP7l+1D+1ydrEejjWNXAqaZYMbI6T2fJ1KnInZufqhKUISZO4h7FE8Kecn6cs6k8qapdnq2j4lOllKyce1qb4fXX6lzqgbwTegL29wufBc2Dsk347LBUqerHkMcOdrFPN36ECk5RQ4NWGOAej5gYN8aDMTGe3qVGTnu28aUZz2/M3Zdw D T
CAAAC6HicjVLLSgMxFD0dX7W+qi7dDBbBVZkRQZfFblxW7AtqlZlpWsfOi0xGKMVPEFxZ3PpX/kH9C29iBLX4SJjMybnn3OQmcZPAT4VlveSMufmFxaX8cmFldW19o7i51UzjjHus4cVBzNuuk7LAj1hD+CJg7YQzJ3QD1nKHVRlv3TKe+nFUF6OEdUNnEPl933MEUefVy/pVsWSVLdXMWWBrUIJutbg4xQV6iOEhQwiGCIJwAAcp9Q5sWEiI62JMHCfkqzjDHQrkzUjFSOEQO6RxQLOOZiOay5ypcnu0SkAfJ6eJPfLEpOOE5Wqmimcqs2R/yj1WOeXeRvR3da6QWIFrYv/yfSj/65O1CPRxrGrwqaZEMbI6T2fJ1KnInZufqhKUISFO4h7FOWFPOT/O2VSeVNUuz9ZR8alSSlbOPa3N8PprdS71UN4JPQH7+4XPguZB2SZ8dliqnOjHkMcOdrFPN36ECk5RQ4NWGOAej5gYN8aDMTGe3qVGTnu28aUZz2/PYZdx
BAAAC5nicjVLLSgMxFD0dX/VddelmsAiuylQEXZa6cdmCfUAtMpOm7dB5kckIpfgFgiuLW3/LP6h/4U1MQS0+EiZzcu45N7lJvCTwU+k4rzlraXlldS2/vrG5tb2zW9jbb6ZxJhhvsDiIRdtzUx74EW9IXwa8nQjuhl7AW97oUsVbd1ykfhxdy3HCu6E7iPy+z1xJVL16Wyg6JUc3exGUDSjCtFpcmOEGPcRgyBCCI4IkHMBFSr2DMhwkxHUxIU4Q8nWc4x4b5M1IxUnhEjuicUCzjmEjmqucqXYzWiWgT5DTxjF5YtIJwmo1W8cznVmxP+We6Jxqb2P6eyZXSKzEkNi/fHPlf32qFok+LnQNPtWUaEZVx0yWTJ+K2rn9qSpJGRLiFO5RXBBm2jk/Z1t7Ul27OltXx2daqVg1Z0ab4e3X6jzqoboTegLl7xe+CJqnpTLh+lmxUjWPIY9DHOGEbvwcFVyhhoZe/QFPmFpD69GaWs8fUitnPAf40qyXd8Gzlqo=
AAAAC5nicjVLLSgMxFD0dX7W+qi7dDBbBVZmKoMuqG5ct2FaoRWbStB06LzIZoRS/QHBlcetv+Qf1L7yJKajFR8JkTs495yY3iZcEfiod5zVnLSwuLa/kVwtr6xubW8XtnWYaZ4LxBouDWFx7bsoDP+IN6cuAXyeCu6EX8JY3vFDx1h0XqR9HV3KU8E7o9iO/5zNXElU/uy2WnLKjmz0PKgaUYFotLk5xgy5iMGQIwRFBEg7gIqXeRgUOEuI6GBMnCPk6znGPAnkzUnFSuMQOaezTrG3YiOYqZ6rdjFYJ6BPktHFAnph0grBazdbxTGdW7E+5xzqn2tuI/p7JFRIrMSD2L99M+V+fqkWih1Ndg081JZpR1TGTJdOnonZuf6pKUoaEOIW7FBeEmXbOztnWnlTXrs7W1fGpVipWzZnRZnj7tTqPeqjuhJ5A5fuFz4PmUblCuH5cqp6bx5DHHvZxSDd+giouUUNDr/6AJ0ysgfVoTaznD6mVM55dfGnWyzu/MZap FIG. 18. In this figure we prove the transpose property of (if B is diagonal) the Kathri-Rao product: (A B)T (C D) = AT C ◦ BT D. This is not the bisymmetry property because the Kathri-Rao FIG. 20. This type of manipulation is found for example in product does not appear in the right hand side, but rather optimization problems [19]. In this proof we used identities the Hadamard product. Note that in the first part of the from Fig. 3 and Fig. 5. LHS we have swapped row and column indices to express the transpose. Then we have used the identity at the bottom of Fig. 5.
V. NEW IDENTITIES The vectorization of an M N matrix is a column or × row vector with MN elements. Note that we could ei- To show the flexibility of graphical notation, we now ther truly form a vector that has one single index, or we prove in Fig. 21 and Fig. 22 two new (to the best of the could reinterpret both indices of A as column (or both as author’s knowledge) identities. The first identity is row) indices, in which case we would have a “block vec- tor” with the first index indexing N blocks of dimension col(A B)row(C D) = (col(A)row(C)) ? (col(B)row(D)) M 1 (or M blocks of dimension 1 N) and a sec- ⊗ ⊗ (8) ond× index indexing the elements within× each block. In any case, given that the vectorization tensor is reversible, which involves vectorization, the Kronecker product and conflating vectorization and block-vectorization is just as the Tracy-Singh product. acceptable as conflating Kronecker and tensor products. The second identity is
One can come up with several interesting identities in- col(A B)row(C D) = (col(A)row(C)) (col(B)row(D)) volving vectorization. We begin with an identity involv- ◦ ◦ ◦ (9) ing the vectorization of a product of three matrices and the Kathri-Rao product or the Kronecker product, de- involving Hadamard product and vectorizations. pending on whether or not the matrix B is diagonal. For Notice that the proof in Fig. 22 can be separated into the case in which B is not diagonal, we first interpret two parts (the left and the right parts), which yield the 7
order to swap the Kronecker and vectorization tensors we
AAAC5nicjVLLSgMxFD0dX7W+qi7dDBbBVZmKoMuqG5ct2FaoRWbStB06LzIZoRS/QHBlcetv+Qf1L7yJKajFR8JkTs495yY3iZcEfiod5zVnLSwuLa/kVwtr6xubW8XtnWYaZ4LxBouDWFx7bsoDP+IN6cuAXyeCu6EX8JY3vFDx1h0XqR9HV3KU8E7o9iO/5zNXElU/uy2WnLKjmz0PKgaUYFotLk5xgy5iMGQIwRFBEg7gIqXeRgUOEuI6GBMnCPk6znGPAnkzUnFSuMQOaezTrG3YiOYqZ6rdjFYJ6BPktHFAnph0grBazdbxTGdW7E+5xzqn2tuI/p7JFRIrMSD2L99M+V+fqkWih1Ndg081JZpR1TGTJdOnonZuf6pKUoaEOIW7FBeEmXbOztnWnlTXrs7W1fGpVipWzZnRZnj7tTqPeqjuhJ5A5fuFz4PmUblCuH5cqp6bx5DHHvZxSDd+giouUUNDr/6AJ0ysgfVoTaznD6mVM55dfGnWyzu/MZap A CAAAC5nicjVLLSgMxFD0dX/VddelmsAiuylQEXRa7cdmCfUAtMpOm7dB5kckIpfgFgiuLW3/LP6h/4U1MQS0+EiZzcu45N7lJvCTwU+k4rzlraXlldS2/vrG5tb2zW9jbb6ZxJhhvsDiIRdtzUx74EW9IXwa8nQjuhl7AW96oquKtOy5SP46u5Tjh3dAdRH7fZ64kql69LRSdkqObvQjKBhRhWi0uzHCDHmIwZAjBEUESDuAipd5BGQ4S4rqYECcI+TrOcY8N8mak4qRwiR3ROKBZx7ARzVXOVLsZrRLQJ8hp45g8MekEYbWareOZzqzYn3JPdE61tzH9PZMrJFZiSOxfvrnyvz5Vi0QfF7oGn2pKNKOqYyZLpk9F7dz+VJWkDAlxCvcoLggz7Zyfs609qa5dna2r4zOtVKyaM6PN8PZrdR71UN0JPYHy9wtfBM3TUplw/axYuTSPIY9DHOGEbvwcFVyhhoZe/QFPmFpD69GaWs8fUitnPAf40qyXd8Q1lqs= need to twist the central wires, which is why we obtain the Hadamard product of distinct vectorized matrices.
AAAC5nicjVLLSgMxFD0dX/VddelmsAiuylQEXZa6cdmCfUAtMpOm7dB5kckIpfgFgiuLW3/LP6h/4U1MQS0+EiZzcu45N7lJvCTwU+k4rzlraXlldS2/vrG5tb2zW9jbb6ZxJhhvsDiIRdtzUx74EW9IXwa8nQjuhl7AW97oUsVbd1ykfhxdy3HCu6E7iPy+z1xJVL16Wyg6JUc3exGUDSjCtFpcmOEGPcRgyBCCI4IkHMBFSr2DMhwkxHUxIU4Q8nWc4x4b5M1IxUnhEjuicUCzjmEjmqucqXYzWiWgT5DTxjF5YtIJwmo1W8cznVmxP+We6Jxqb2P6eyZXSKzEkNi/fHPlf32qFok+LnQNPtWUaEZVx0yWTJ+K2rn9qSpJGRLiFO5RXBBm2jk/Z1t7Ul27OltXx2daqVg1Z0ab4e3X6jzqoboTegLl7xe+CJqnpTLh+lmxUjWPIY9DHOGEbvwcFVyhhoZe/QFPmFpD69GaWs8fUitnPAf40qyXd8Gzlqo= B DAAAC5nicjVLLSgMxFD0dX7W+qi7dDBbBVZmKoMuiLly2YFuhFplJ03bovMhkhFL8AsGVxa2/5R/Uv/AmpqAWHwmTOTn3nJvcJF4S+Kl0nNectbC4tLySXy2srW9sbhW3d5ppnAnGGywOYnHtuSkP/Ig3pC8Dfp0I7oZewFve8FzFW3dcpH4cXclRwjuh24/8ns9cSVT94rZYcsqObvY8qBhQgmm1uDjFDbqIwZAhBEcESTiAi5R6GxU4SIjrYEycIOTrOMc9CuTNSMVJ4RI7pLFPs7ZhI5qrnKl2M1oloE+Q08YBeWLSCcJqNVvHM51ZsT/lHuucam8j+nsmV0isxIDYv3wz5X99qhaJHk51DT7VlGhGVcdMlkyfitq5/akqSRkS4hTuUlwQZto5O2dbe1JduzpbV8enWqlYNWdGm+Ht1+o86qG6E3oCle8XPg+aR+UK4fpxqXpmHkMee9jHId34Caq4RA0NvfoDnjCxBtajNbGeP6RWznh28aVZL+/Gt5as
=AAAC5nicjVLLSgMxFD0dX7W+qi7dDBbBVZmKoBuh6MZlC7YVapGZadqGzotMRijFLxBcWdz6W/5B/QtvYgpq8ZEwmZNzz7nJTeIlAU+l47zmrIXFpeWV/GphbX1jc6u4vdNM40z4rOHHQSyuPTdlAY9YQ3IZsOtEMDf0Atbyhhcq3rpjIuVxdCVHCeuEbj/iPe67kqj62W2x5JQd3ex5UDGgBNNqcXGKG3QRw0eGEAwRJOEALlLqbVTgICGugzFxghDXcYZ7FMibkYqRwiV2SGOfZm3DRjRXOVPt9mmVgD5BThsH5IlJJwir1Wwdz3Rmxf6Ue6xzqr2N6O+ZXCGxEgNi//LNlP/1qVokejjVNXCqKdGMqs43WTJ9Kmrn9qeqJGVIiFO4S3FB2NfO2Tnb2pPq2tXZujo+1UrFqrlvtBnefq3Oox6qO6EnUPl+4fOgeVSuEK4fl6rn5jHksYd9HNKNn6CKS9TQ0Ks/4AkTa2A9WhPr+UNq5YxnF1+a9fIOtSmWpQ== VI. CONVOLUTIONS
AAAC5nicjVLLSgMxFD0dX7W+qi7dDBbBVZmKoMuqG5ct2FaoRWbStB06LzIZoRS/QHBlcetv+Qf1L7yJKajFR8JkTs495yY3iZcEfiod5zVnLSwuLa/kVwtr6xubW8XtnWYaZ4LxBouDWFx7bsoDP+IN6cuAXyeCu6EX8JY3vFDx1h0XqR9HV3KU8E7o9iO/5zNXElU/uy2WnLKjmz0PKgaUYFotLk5xgy5iMGQIwRFBEg7gIqXeRgUOEuI6GBMnCPk6znGPAnkzUnFSuMQOaezTrG3YiOYqZ6rdjFYJ6BPktHFAnph0grBazdbxTGdW7E+5xzqn2tuI/p7JFRIrMSD2L99M+V+fqkWih1Ndg081JZpR1TGTJdOnonZuf6pKUoaEOIW7FBeEmXbOztnWnlTXrs7W1fGpVipWzZnRZnj7tTqPeqjuhJ5A5fuFz4PmUblCuH5cqp6bx5DHHvZxSDd+giouUUNDr/6AJ0ysgfVoTaznD6mVM55dfGnWyzu/MZap We now turn to the graphical description of con- A CAAAC5nicjVLLSgMxFD0dX/VddelmsAiuylQEXRa7cdmCfUAtMpOm7dB5kckIpfgFgiuLW3/LP6h/4U1MQS0+EiZzcu45N7lJvCTwU+k4rzlraXlldS2/vrG5tb2zW9jbb6ZxJhhvsDiIRdtzUx74EW9IXwa8nQjuhl7AW96oquKtOy5SP46u5Tjh3dAdRH7fZ64kql69LRSdkqObvQjKBhRhWi0uzHCDHmIwZAjBEUESDuAipd5BGQ4S4rqYECcI+TrOcY8N8mak4qRwiR3ROKBZx7ARzVXOVLsZrRLQJ8hp45g8MekEYbWareOZzqzYn3JPdE61tzH9PZMrJFZiSOxfvrnyvz5Vi0QfF7oGn2pKNKOqYyZLpk9F7dz+VJWkDAlxCvcoLggz7Zyfs609qa5dna2r4zOtVKyaM6PN8PZrdR71UN0JPYHy9wtfBM3TUplw/axYuTSPIY9DHOGEbvwcFVyhhoZe/QFPmFpD69GaWs8fUitnPAf40qyXd8Q1lqs= volutions. The mechanisms of convolution and cross- correlation can be nicely translated to graphical calculus
AAAC5nicjVLLSgMxFD0dX7W+qi7dDBbBVZmKoMuiLly2YFuhFplJ03bovMhkhFL8AsGVxa2/5R/Uv/AmpqAWHwmTOTn3nJvcJF4S+Kl0nNectbC4tLySXy2srW9sbhW3d5ppnAnGGywOYnHtuSkP/Ig3pC8Dfp0I7oZewFve8FzFW3dcpH4cXclRwjuh24/8ns9cSVT94rZYcsqObvY8qBhQgmm1uDjFDbqIwZAhBEcESTiAi5R6GxU4SIjrYEycIOTrOMc9CuTNSMVJ4RI7pLFPs7ZhI5qrnKl2M1oloE+Q08YBeWLSCcJqNVvHM51ZsT/lHuucam8j+nsmV0isxIDYv3wz5X99qhaJHk51DT7VlGhGVcdMlkyfitq5/akqSRkS4hTuUlwQZto5O2dbe1JduzpbV8enWqlYNWdGm+Ht1+o86qG6E3oCle8XPg+aR+UK4fpxqXpmHkMee9jHId34Caq4RA0NvfoDnjCxBtajNbGeP6RWznh28aVZL+/Gt5as BAAAC5nicjVLLSgMxFD0dX/VddelmsAiuylQEXZa6cdmCfUAtMpOm7dB5kckIpfgFgiuLW3/LP6h/4U1MQS0+EiZzcu45N7lJvCTwU+k4rzlraXlldS2/vrG5tb2zW9jbb6ZxJhhvsDiIRdtzUx74EW9IXwa8nQjuhl7AW97oUsVbd1ykfhxdy3HCu6E7iPy+z1xJVL16Wyg6JUc3exGUDSjCtFpcmOEGPcRgyBCCI4IkHMBFSr2DMhwkxHUxIU4Q8nWc4x4b5M1IxUnhEjuicUCzjmEjmqucqXYzWiWgT5DTxjF5YtIJwmo1W8cznVmxP+We6Jxqb2P6eyZXSKzEkNi/fHPlf32qFok+LnQNPtWUaEZVx0yWTJ+K2rn9qSpJGRLiFO5RXBBm2jk/Z1t7Ul27OltXx2daqVg1Z0ab4e3X6jzqoboTegLl7xe+CJqnpTLh+lmxUjWPIY9DHOGEbvwcFVyhhoZe/QFPmFpD69GaWs8fUitnPAf40qyXd8Gzlqo= D notation, and one can notice several interesting features visually. In particular, we will see that convolution is FIG. 21. In this proof of the identity col(A ⊗ B)row(C ⊗ represented just like the products that we have described D) = (col(A)row(C)) ? (col(B)row(D)) we simply drag the in the previous sections, and that the (discrete) convo- inner triangles outwards and we notice that we recover the lution theorem is a special case of a more general tensor definition of the Tracy-Singh product. contraction rule that is entirely independent from the tensors being convolved.
AAAC5nicjVLLSgMxFD0dX7W+qi7dDBbBVZmKoMuqG5ct2FaoRWbStB06LzIZoRS/QHBlcetv+Qf1L7yJKajFR8JkTs495yY3iZcEfiod5zVnLSwuLa/kVwtr6xubW8XtnWYaZ4LxBouDWFx7bsoDP+IN6cuAXyeCu6EX8JY3vFDx1h0XqR9HV3KU8E7o9iO/5zNXElU/uy2WnLKjmz0PKgaUYFotLk5xgy5iMGQIwRFBEg7gIqXeRgUOEuI6GBMnCPk6znGPAnkzUnFSuMQOaezTrG3YiOYqZ6rdjFYJ6BPktHFAnph0grBazdbxTGdW7E+5xzqn2tuI/p7JFRIrMSD2L99M+V+fqkWih1Ndg081JZpR1TGTJdOnonZuf6pKUoaEOIW7FBeEmXbOztnWnlTXrs7W1fGpVipWzZnRZnj7tTqPeqjuhJ5A5fuFz4PmUblCuH5cqp6bx5DHHvZxSDd+giouUUNDr/6AJ0ysgfVoTaznD6mVM55dfGnWyzu/MZap A CAAAC5nicjVLLSgMxFD0dX/VddelmsAiuylQEXRa7cdmCfUAtMpOm7dB5kckIpfgFgiuLW3/LP6h/4U1MQS0+EiZzcu45N7lJvCTwU+k4rzlraXlldS2/vrG5tb2zW9jbb6ZxJhhvsDiIRdtzUx74EW9IXwa8nQjuhl7AW96oquKtOy5SP46u5Tjh3dAdRH7fZ64kql69LRSdkqObvQjKBhRhWi0uzHCDHmIwZAjBEUESDuAipd5BGQ4S4rqYECcI+TrOcY8N8mak4qRwiR3ROKBZx7ARzVXOVLsZrRLQJ8hp45g8MekEYbWareOZzqzYn3JPdE61tzH9PZMrJFZiSOxfvrnyvz5Vi0QfF7oGn2pKNKOqYyZLpk9F7dz+VJWkDAlxCvcoLggz7Zyfs609qa5dna2r4zOtVKyaM6PN8PZrdR71UN0JPYHy9wtfBM3TUplw/axYuTSPIY9DHOGEbvwcFVyhhoZe/QFPmFpD69GaWs8fUitnPAf40qyXd8Q1lqs= A. The convolution tensor
AAAC5nicjVLLSgMxFD0dX/VddelmsAiuylQEXZa6cdmCfUAtMpOm7dB5kckIpfgFgiuLW3/LP6h/4U1MQS0+EiZzcu45N7lJvCTwU+k4rzlraXlldS2/vrG5tb2zW9jbb6ZxJhhvsDiIRdtzUx74EW9IXwa8nQjuhl7AW97oUsVbd1ykfhxdy3HCu6E7iPy+z1xJVL16Wyg6JUc3exGUDSjCtFpcmOEGPcRgyBCCI4IkHMBFSr2DMhwkxHUxIU4Q8nWc4x4b5M1IxUnhEjuicUCzjmEjmqucqXYzWiWgT5DTxjF5YtIJwmo1W8cznVmxP+We6Jxqb2P6eyZXSKzEkNi/fHPlf32qFok+LnQNPtWUaEZVx0yWTJ+K2rn9qSpJGRLiFO5RXBBm2jk/Z1t7Ul27OltXx2daqVg1Z0ab4e3X6jzqoboTegLl7xe+CJqnpTLh+lmxUjWPIY9DHOGEbvwcFVyhhoZe/QFPmFpD69GaWs8fUitnPAf40qyXd8Gzlqo= B DAAAC5nicjVLLSgMxFD0dX7W+qi7dDBbBVZmKoMuiLly2YFuhFplJ03bovMhkhFL8AsGVxa2/5R/Uv/AmpqAWHwmTOTn3nJvcJF4S+Kl0nNectbC4tLySXy2srW9sbhW3d5ppnAnGGywOYnHtuSkP/Ig3pC8Dfp0I7oZewFve8FzFW3dcpH4cXclRwjuh24/8ns9cSVT94rZYcsqObvY8qBhQgmm1uDjFDbqIwZAhBEcESTiAi5R6GxU4SIjrYEycIOTrOMc9CuTNSMVJ4RI7pLFPs7ZhI5qrnKl2M1oloE+Q08YBeWLSCcJqNVvHM51ZsT/lHuucam8j+nsmV0isxIDYv3wz5X99qhaJHk51DT7VlGhGVcdMlkyfitq5/akqSRkS4hTuUlwQZto5O2dbe1JduzpbV8enWqlYNWdGm+Ht1+o86qG6E3oCle8XPg+aR+UK4fpxqXpmHkMee9jHId34Caq4RA0NvfoDnjCxBtajNbGeP6RWznh28aVZL+/Gt5as
=AAAC5nicjVLLSgMxFD0dX7W+qi7dDBbBVZmKoBuh6MZlC7YVapGZadqGzotMRijFLxBcWdz6W/5B/QtvYgpq8ZEwmZNzz7nJTeIlAU+l47zmrIXFpeWV/GphbX1jc6u4vdNM40z4rOHHQSyuPTdlAY9YQ3IZsOtEMDf0Atbyhhcq3rpjIuVxdCVHCeuEbj/iPe67kqj62W2x5JQd3ex5UDGgBNNqcXGKG3QRw0eGEAwRJOEALlLqbVTgICGugzFxghDXcYZ7FMibkYqRwiV2SGOfZm3DRjRXOVPt9mmVgD5BThsH5IlJJwir1Wwdz3Rmxf6Ue6xzqr2N6O+ZXCGxEgNi//LNlP/1qVokejjVNXCqKdGMqs43WTJ9Kmrn9qeqJGVIiFO4S3FB2NfO2Tnb2pPq2tXZujo+1UrFqrlvtBnefq3Oox6qO6EnUPl+4fOgeVSuEK4fl6rn5jHksYd9HNKNn6CKS9TQ0Ks/4AkTa2A9WhPr+UNq5YxnF1+a9fIOtSmWpQ== We now define a general convolution tensor that can implement several different types of convolution. The convolution tensor is a family of D-dimensional, rank-3
AAAC5nicjVLLSgMxFD0dX7W+qi7dDBbBVZmKoMuqG5ct2FaoRWbStB06LzIZoRS/QHBlcetv+Qf1L7yJKajFR8JkTs495yY3iZcEfiod5zVnLSwuLa/kVwtr6xubW8XtnWYaZ4LxBouDWFx7bsoDP+IN6cuAXyeCu6EX8JY3vFDx1h0XqR9HV3KU8E7o9iO/5zNXElU/uy2WnLKjmz0PKgaUYFotLk5xgy5iMGQIwRFBEg7gIqXeRgUOEuI6GBMnCPk6znGPAnkzUnFSuMQOaezTrG3YiOYqZ6rdjFYJ6BPktHFAnph0grBazdbxTGdW7E+5xzqn2tuI/p7JFRIrMSD2L99M+V+fqkWih1Ndg081JZpR1TGTJdOnonZuf6pKUoaEOIW7FBeEmXbOztnWnlTXrs7W1fGpVipWzZnRZnj7tTqPeqjuhJ5A5fuFz4PmUblCuH5cqp6bx5DHHvZxSDd+giouUUNDr/6AJ0ysgfVoTaznD6mVM55dfGnWyzu/MZap A CAAAC5nicjVLLSgMxFD0dX/VddelmsAiuylQEXRa7cdmCfUAtMpOm7dB5kckIpfgFgiuLW3/LP6h/4U1MQS0+EiZzcu45N7lJvCTwU+k4rzlraXlldS2/vrG5tb2zW9jbb6ZxJhhvsDiIRdtzUx74EW9IXwa8nQjuhl7AW96oquKtOy5SP46u5Tjh3dAdRH7fZ64kql69LRSdkqObvQjKBhRhWi0uzHCDHmIwZAjBEUESDuAipd5BGQ4S4rqYECcI+TrOcY8N8mak4qRwiR3ROKBZx7ARzVXOVLsZrRLQJ8hp45g8MekEYbWareOZzqzYn3JPdE61tzH9PZMrJFZiSOxfvrnyvz5Vi0QfF7oGn2pKNKOqYyZLpk9F7dz+VJWkDAlxCvcoLggz7Zyfs609qa5dna2r4zOtVKyaM6PN8PZrdR71UN0JPYHy9wtfBM3TUplw/axYuTSPIY9DHOGEbvwcFVyhhoZe/QFPmFpD69GaWs8fUitnPAf40qyXd8Q1lqs= tensors defined as ( ( ) 1 if i j k = 0 mod D
AAAC5nicjVLLSgMxFD0dX/VddelmsAiuylQEXZa6cdmCfUAtMpOm7dB5kckIpfgFgiuLW3/LP6h/4U1MQS0+EiZzcu45N7lJvCTwU+k4rzlraXlldS2/vrG5tb2zW9jbb6ZxJhhvsDiIRdtzUx74EW9IXwa8nQjuhl7AW97oUsVbd1ykfhxdy3HCu6E7iPy+z1xJVL16Wyg6JUc3exGUDSjCtFpcmOEGPcRgyBCCI4IkHMBFSr2DMhwkxHUxIU4Q8nWc4x4b5M1IxUnhEjuicUCzjmEjmqucqXYzWiWgT5DTxjF5YtIJwmo1W8cznVmxP+We6Jxqb2P6eyZXSKzEkNi/fHPlf32qFok+LnQNPtWUaEZVx0yWTJ+K2rn9qSpJGRLiFO5RXBBm2jk/Z1t7Ul27OltXx2daqVg1Z0ab4e3X6jzqoboTegLl7xe+CJqnpTLh+lmxUjWPIY9DHOGEbvwcFVyhhoZe/QFPmFpD69GaWs8fUitnPAf40qyXd8Gzlqo= B DAAAC5nicjVLLSgMxFD0dX7W+qi7dDBbBVZmKoMuiLly2YFuhFplJ03bovMhkhFL8AsGVxa2/5R/Uv/AmpqAWHwmTOTn3nJvcJF4S+Kl0nNectbC4tLySXy2srW9sbhW3d5ppnAnGGywOYnHtuSkP/Ig3pC8Dfp0I7oZewFve8FzFW3dcpH4cXclRwjuh24/8ns9cSVT94rZYcsqObvY8qBhQgmm1uDjFDbqIwZAhBEcESTiAi5R6GxU4SIjrYEycIOTrOMc9CuTNSMVJ4RI7pLFPs7ZhI5qrnKl2M1oloE+Q08YBeWLSCcJqNVvHM51ZsT/lHuucam8j+nsmV0isxIDYv3wz5X99qhaJHk51DT7VlGhGVcdMlkyfitq5/akqSRkS4hTuUlwQZto5O2dbe1JduzpbV8enWqlYNWdGm+Ht1+o86qG6E3oCle8XPg+aR+UK4fpxqXpmHkMee9jHId34Caq4RA0NvfoDnjCxBtajNbGeP6RWznh28aVZL+/Gt5as χ ±±± = ± ± ± (12) ijk 0 otherwise
=AAAC5nicjVLLSgMxFD0dX7W+qi7dDBbBVZmKoBuh6MZlC7YVapGZadqGzotMRijFLxBcWdz6W/5B/QtvYgpq8ZEwmZNzz7nJTeIlAU+l47zmrIXFpeWV/GphbX1jc6u4vdNM40z4rOHHQSyuPTdlAY9YQ3IZsOtEMDf0Atbyhhcq3rpjIuVxdCVHCeuEbj/iPe67kqj62W2x5JQd3ex5UDGgBNNqcXGKG3QRw0eGEAwRJOEALlLqbVTgICGugzFxghDXcYZ7FMibkYqRwiV2SGOfZm3DRjRXOVPt9mmVgD5BThsH5IlJJwir1Wwdz3Rmxf6Ue6xzqr2N6O+ZXCGxEgNi//LNlP/1qVokejjVNXCqKdGMqs43WTJ9Kmrn9qeqJGVIiFO4S3FB2NfO2Tnb2pPq2tXZujo+1UrFqrlvtBnefq3Oox6qO6EnUPl+4fOgeVSuEK4fl6rn5jHksYd9HNKNn6CKS9TQ0Ks/4AkTa2A9WhPr+UNq5YxnF1+a9fIOtSmWpQ== where the signature ( ) determines four possibilities (under the + symmetry).± ± ± In graphical notation, we ↔ −
AAAC5nicjVLLSgMxFD0dX7W+qi7dDBbBVZmKoMuqG5ct2FaoRWbStB06LzIZoRS/QHBlcetv+Qf1L7yJKajFR8JkTs495yY3iZcEfiod5zVnLSwuLa/kVwtr6xubW8XtnWYaZ4LxBouDWFx7bsoDP+IN6cuAXyeCu6EX8JY3vFDx1h0XqR9HV3KU8E7o9iO/5zNXElU/uy2WnLKjmz0PKgaUYFotLk5xgy5iMGQIwRFBEg7gIqXeRgUOEuI6GBMnCPk6znGPAnkzUnFSuMQOaezTrG3YiOYqZ6rdjFYJ6BPktHFAnph0grBazdbxTGdW7E+5xzqn2tuI/p7JFRIrMSD2L99M+V+fqkWih1Ndg081JZpR1TGTJdOnonZuf6pKUoaEOIW7FBeEmXbOztnWnlTXrs7W1fGpVipWzZnRZnj7tTqPeqjuhJ5A5fuFz4PmUblCuH5cqp6bx5DHHvZxSDd+giouUUNDr/6AJ0ysgfVoTaznD6mVM55dfGnWyzu/MZap A CAAAC5nicjVLLSgMxFD0dX/VddelmsAiuylQEXRa7cdmCfUAtMpOm7dB5kckIpfgFgiuLW3/LP6h/4U1MQS0+EiZzcu45N7lJvCTwU+k4rzlraXlldS2/vrG5tb2zW9jbb6ZxJhhvsDiIRdtzUx74EW9IXwa8nQjuhl7AW96oquKtOy5SP46u5Tjh3dAdRH7fZ64kql69LRSdkqObvQjKBhRhWi0uzHCDHmIwZAjBEUESDuAipd5BGQ4S4rqYECcI+TrOcY8N8mak4qRwiR3ROKBZx7ARzVXOVLsZrRLQJ8hp45g8MekEYbWareOZzqzYn3JPdE61tzH9PZMrJFZiSOxfvrnyvz5Vi0QfF7oGn2pKNKOqYyZLpk9F7dz+VJWkDAlxCvcoLggz7Zyfs609qa5dna2r4zOtVKyaM6PN8PZrdR71UN0JPYHy9wtfBM3TUplw/axYuTSPIY9DHOGEbvwcFVyhhoZe/QFPmFpD69GaWs8fUitnPAf40qyXd8Q1lqs= define the convolution tensor as
AAAC5nicjVLLSgMxFD0dX7W+qi7dDBbBVZmKoMuiG5ct2AfUIjNp2obOi0xGKMUvEFxZ3Ppb/kH9C2/iFNTiI2EyJ+eec5ObxIt9kSjHec1ZS8srq2v59cLG5tb2TnF3r5lEqWS8wSI/km3PTbgvQt5QQvm8HUvuBp7PW97oUsdbd1wmIgqv1Tjm3cAdhKIvmKuIqovbYskpO6bZi6CSgRKyVouKM9yghwgMKQJwhFCEfbhIqHdQgYOYuC4mxElCwsQ57lEgb0oqTgqX2BGNA5p1Mjakuc6ZGDejVXz6JDltHJEnIp0krFezTTw1mTX7U+6Jyan3Nqa/l+UKiFUYEvuXb678r0/XotDHualBUE2xYXR1LMuSmlPRO7c/VaUoQ0ycxj2KS8LMOOfnbBtPYmrXZ+ua+MwoNavnLNOmePu1Oo96oO+EnkDl+4UvguZJuUK4flqqXmSPIY8DHOKYbvwMVVyhhoZZ/QFPmFpD69GaWs8fUiuXefbxpVkv7yOQltE=
AAAC5nicjVLLSgMxFD0dX7W+qi7dDBbBVZmKoMuiLly2YFuhFplJ03bovMhkhFL8AsGVxa2/5R/Uv/AmpqAWHwmTOTn3nJvcJF4S+Kl0nNectbC4tLySXy2srW9sbhW3d5ppnAnGGywOYnHtuSkP/Ig3pC8Dfp0I7oZewFve8FzFW3dcpH4cXclRwjuh24/8ns9cSVT94rZYcsqObvY8qBhQgmm1uDjFDbqIwZAhBEcESTiAi5R6GxU4SIjrYEycIOTrOMc9CuTNSMVJ4RI7pLFPs7ZhI5qrnKl2M1oloE+Q08YBeWLSCcJqNVvHM51ZsT/lHuucam8j+nsmV0isxIDYv3wz5X99qhaJHk51DT7VlGhGVcdMlkyfitq5/akqSRkS4hTuUlwQZto5O2dbe1JduzpbV8enWqlYNWdGm+Ht1+o86qG6E3oCle8XPg+aR+UK4fpxqXpmHkMee9jHId34Caq4RA0NvfoDnjCxBtajNbGeP6RWznh28aVZL+/Gt5as i BAAAC5nicjVLLSgMxFD0dX/VddelmsAiuylQEXZa6cdmCfUAtMpOm7dB5kckIpfgFgiuLW3/LP6h/4U1MQS0+EiZzcu45N7lJvCTwU+k4rzlraXlldS2/vrG5tb2zW9jbb6ZxJhhvsDiIRdtzUx74EW9IXwa8nQjuhl7AW97oUsVbd1ykfhxdy3HCu6E7iPy+z1xJVL16Wyg6JUc3exGUDSjCtFpcmOEGPcRgyBCCI4IkHMBFSr2DMhwkxHUxIU4Q8nWc4x4b5M1IxUnhEjuicUCzjmEjmqucqXYzWiWgT5DTxjF5YtIJwmo1W8cznVmxP+We6Jxqb2P6eyZXSKzEkNi/fHPlf32qFok+LnQNPtWUaEZVx0yWTJ+K2rn9qSpJGRLiFO5RXBBm2jk/Z1t7Ul27OltXx2daqVg1Z0ab4e3X6jzqoboTegLl7xe+CJqnpTLh+lmxUjWPIY9DHOGEbvwcFVyhhoZe/QFPmFpD69GaWs8fUitnPAf40qyXd8Gzlqo= D
FIG. 22. In this proof of the identity col(A ◦ B)row(C ◦ D) = (col(A)row(C))◦(col(B)row(D)), we swap the Kronecker and ±AAAC6HicjVLLSgMxFD0dX7W+qi7dDBbBVZmKoMuiG5cV7QNakZk0rbHzIpMRSvETBFcWt/6Vf1D/wps4BbX4SJjMybnn3OQm8WJfJMpxXnPW3PzC4lJ+ubCyura+UdzcaiRRKhmvs8iPZMtzE+6LkNeVUD5vxZK7gefzpjc41fHmHZeJiMJLNYz5VeD2Q9ETzFVEXXTi4LpYcsqOafYsqGSghKzVouIEHXQRgSFFAI4QirAPFwn1NipwEBN3hRFxkpAwcY57FMibkoqTwiV2QGOfZu2MDWmucybGzWgVnz5JTht75IlIJwnr1WwTT01mzf6Ue2Ry6r0N6e9luQJiFW6I/cs3Vf7Xp2tR6OHY1CCoptgwujqWZUnNqeid25+qUpQhJk7jLsUlYWac03O2jScxteuzdU18YpSa1XOWaVO8/VqdRz3Qd0JPoPL9wmdB46BcIXx+WKqeZI8hjx3sYp9u/AhVnKGGOq3QxwOeMLZurUdrbD1/SK1c5tnGl2a9vAN6PJe1
AAAC5nicjVLLSgMxFD0dX7W+qi7dDBbBVZmKoMuiG5ct2AfUIjNp2obOi0xGKMUvEFxZ3Ppb/kH9C2/iFNTiI2EyJ+eec5ObxIt9kSjHec1ZS8srq2v59cLG5tb2TnF3r5lEqWS8wSI/km3PTbgvQt5QQvm8HUvuBp7PW97oUsdbd1wmIgqv1Tjm3cAdhKIvmKuIqo9uiyWn7JhmL4JKBkrIWi0qznCDHiIwpAjAEUIR9uEiod5BBQ5i4rqYECcJCRPnuEeBvCmpOClcYkc0DmjWydiQ5jpnYtyMVvHpk+S0cUSeiHSSsF7NNvHUZNbsT7knJqfe25j+XpYrIFZhSOxfvrnyvz5di0If56YGQTXFhtHVsSxLak5F79z+VJWiDDFxGvcoLgkz45yfs208ialdn61r4jOj1Kyes0yb4u3X6jzqgb4TegKV7xe+CJon5Qrh+mmpepE9hjwOcIhjuvEzVHGFGhpm9Qc8YWoNrUdraj1/SK1c5tnHl2a9vAMolJbT vectorization tensors, which implies twisting the inner wires k AAAC6HicjVLLSgMxFD0dX7W+qi7dDBbBVZmKoMuiG5cV7QNakZk0rbHzIpMRSvETBFcWt/6Vf1D/wps4BbX4SJjMybnn3OQm8WJfJMpxXnPW3PzC4lJ+ubCyura+UdzcaiRRKhmvs8iPZMtzE+6LkNeVUD5vxZK7gefzpjc41fHmHZeJiMJLNYz5VeD2Q9ETzFVEXXTi4LpYcsqOafYsqGSghKzVouIEHXQRgSFFAI4QirAPFwn1NipwEBN3hRFxkpAwcY57FMibkoqTwiV2QGOfZu2MDWmucybGzWgVnz5JTht75IlIJwnr1WwTT01mzf6Ue2Ry6r0N6e9luQJiFW6I/cs3Vf7Xp2tR6OHY1CCoptgwujqWZUnNqeid25+qUpQhJk7jLsUlYWac03O2jScxteuzdU18YpSa1XOWaVO8/VqdRz3Qd0JPoPL9wmdB46BcIXx+WKqeZI8hjx3sYp9u/AhVnKGGOq3QxwOeMLZurUdrbD1/SK1c5tnGl2a9vAN6PJe1 ± and leaves us with a neat Hadamard product. Note that the AAAC6HicjVLLSgMxFD0dX7W+qi7dDBbBVZmKoMuiG5cV7QNakZk0rbHzIpMRSvETBFcWt/6Vf1D/wps4BbX4SJjMybnn3OQm8WJfJMpxXnPW3PzC4lJ+ubCyura+UdzcaiRRKhmvs8iPZMtzE+6LkNeVUD5vxZK7gefzpjc41fHmHZeJiMJLNYz5VeD2Q9ETzFVEXXTi4LpYcsqOafYsqGSghKzVouIEHXQRgSFFAI4QirAPFwn1NipwEBN3hRFxkpAwcY57FMibkoqTwiV2QGOfZu2MDWmucybGzWgVnz5JTht75IlIJwnr1WwTT01mzf6Ue2Ry6r0N6e9luQJiFW6I/cs3Vf7Xp2tR6OHY1CCoptgwujqWZUnNqeid25+qUpQhJk7jLsUlYWac03O2jScxteuzdU18YpSa1XOWaVO8/VqdRz3Qd0JPoPL9wmdB46BcIXx+WKqeZI8hjx3sYp9u/AhVnKGGOq3QxwOeMLZurUdrbD1/SK1c5tnGl2a9vAN6PJe1 two parts (left and right), if considered in isolation, form two ± identities on their own: col(A ◦ B) = col(A) ◦ col(B) and row(C ◦ D) = row(C) ◦ row(D).
AAAC5nicjVLLSgMxFD0dX7W+qi7dDBbBVZmKoMuiG5ct2AdUkZk0bWPnRSYjlOIXCK4sbv0t/6D+hTdxCmrxkTCZk3PPuclN4sW+SJTjvOashcWl5ZX8amFtfWNzq7i900yiVDLeYJEfybbnJtwXIW8ooXzejiV3A8/nLW94ruOtOy4TEYWXahTz68Dth6InmKuIqt/eFEtO2THNngeVDJSQtVpUnOIKXURgSBGAI4Qi7MNFQr2DChzExF1jTJwkJEyc4x4F8qak4qRwiR3S2KdZJ2NDmuuciXEzWsWnT5LTxgF5ItJJwno128RTk1mzP+Uem5x6byP6e1mugFiFAbF/+WbK//p0LQo9nJoaBNUUG0ZXx7IsqTkVvXP7U1WKMsTEadyluCTMjHN2zrbxJKZ2fbauiU+NUrN6zjJtirdfq/OoB/pO6AlUvl/4PGgelSuE68el6ln2GPLYwz4O6cZPUMUFamiY1R/whIk1sB6tifX8IbVymWcXX5r18g4mEpbS j
FIG. 23. The convolution tensor depends on three signs, identities which determine which combination of indices yields a nonzero value. col(A B) = col(A) col(B) (10) ◦ ◦ row(C D) = row(C) row(D) (11) ◦ ◦ Depending on the signature, this tensor implements ( ) where the Hadamard product of two vectors is still the different convolutions between two vectors (a b)k±±± = ( ) ∗ element-wise product. This can be used to write the χijk±±± aibj, or more explicitly: second identity in yet a different way by reading the last diagram left to right rather than top to bottom. D 1 − As you look at the figures, notice how simple it is to ( ) X (a b)k±±± = a ib ( k i) mod D (13) ∗ ± ± ∓ −∓ observe differences and similarities between proofs: the i=0 structure of the diagrams in Fig. 21 and 22 is almost identical, with the exception that in the second proof, in Such definitions include the conventional discrete con- 8
If we discretize the functions into D-dimensional vec-
aAAAC73icjVLLSgMxFD0dX7W+qi7dDBbBVZkRQZdFNy4r2Ae2Iplp2g6dl5mMUEr/QnBlcesn+Qf1L7yJU1CLj4TJnJx7zk1uEif2vURa1mvOWFhcWl7JrxbW1jc2t4rbO/UkSoXLa27kR6LpsIT7Xshr0pM+b8aCs8DxecMZnKt4456LxIvCKzmM+U3AeqHX9VwmibpuB0z2ne6IjW+LJats6WbOAzsDJWStGhWnaKODCC5SBOAIIQn7YEiot2DDQkzcDUbECUKejnOMUSBvSipOCkbsgMYezVoZG9Jc5Uy026VVfPoEOU0ckCcinSCsVjN1PNWZFftT7pHOqfY2pL+T5QqIlegT+5dvpvyvT9Ui0cWprsGjmmLNqOrcLEuqT0Xt3PxUlaQMMXEKdyguCLvaOTtnU3sSXbs6W6bjU61UrJq7mTbF26/VOdQDdSf0BOzvFz4P6kdlm/Dlcalylj2GPPawj0O68RNUcIEqarRCiAc8YWLcGY/GxHj+kBq5zLOLL814eQft+Jrp tors, the Fourier transform becomes a matrix multi- plication by F = 1 exp( 2πi mn) and the prod- mn √D − D uct becomes the Hadamard (element-wise) product (see bAAAC73icjVLLSgMxFD0dX7W+qi7dDBbBVZkRQZdFNy4r2Ae2IpNp2obOy5mMUEr/QnBlcesn+Qf1L7yJU1CLj4TJnJx7zk1uEhZ5IpGW9ZozFhaXllfyq4W19Y3NreL2Tj0J09jlNTf0wrjJnIR7IuA1KaTHm1HMHZ95vMEG5yreuOdxIsLgSg4jfuM7vUB0hetIoq7bviP7rDti49tiySpbupnzwM5ACVmrhsUp2ugghIsUPjgCSMIeHCTUW7BhISLuBiPiYkJCxznGKJA3JRUnhUPsgMYezVoZG9Bc5Uy026VVPPpicpo4IE9IupiwWs3U8VRnVuxPuUc6p9rbkP4sy+UTK9En9i/fTPlfn6pFootTXYOgmiLNqOrcLEuqT0Xt3PxUlaQMEXEKdygeE3a1c3bOpvYkunZ1to6OT7VSsWruZtoUb79Wx6j76k7oCdjfL3we1I/KNuHL41LlLHsMeexhH4d04yeo4AJV1GiFAA94wsS4Mx6NifH8ITVymWcXX5rx8g7we5rq Fig. 26).
FIG. 24. The convolution of two vectors can be expressed
AAAC73icjVLLSgMxFD0dX7W+qi7dDBbBVZkRQZdFNy4r2Ae2Iplp2g6dl5mMUEr/QnBlcesn+Qf1L7yJU1CLj4TJnJx7zk1uEif2vURa1mvOWFhcWl7JrxbW1jc2t4rbO/UkSoXLa27kR6LpsIT7Xshr0pM+b8aCs8DxecMZnKt4456LxIvCKzmM+U3AeqHX9VwmibpuB0z2ne6IjW+LJats6WbOAzsDJWStGhWnaKODCC5SBOAIIQn7YEiot2DDQkzcDUbECUKejnOMUSBvSipOCkbsgMYezVoZG9Jc5Uy026VVfPoEOU0ckCcinSCsVjN1PNWZFftT7pHOqfY2pL+T5QqIlegT+5dvpvyvT9Ui0cWprsGjmmLNqOrcLEuqT0Xt3PxUlaQMMXEKdyguCLvaOTtnU3sSXbs6W6bjU61UrJq7mTbF26/VOdQDdSf0BOzvFz4P6kdlm/Dlcalylj2GPPawj0O68RNUcIEqarRCiAc8YWLcGY/GxHj+kBq5zLOLL814eQft+Jrp AAAC73icjVLLSgMxFD0dX7W+qi7dDBbBVZkRQZdFNy4r2Ae2Iplp2g6dl5mMUEr/QnBlcesn+Qf1L7yJU1CLj4TJnJx7zk1uEif2vURa1mvOWFhcWl7JrxbW1jc2t4rbO/UkSoXLa27kR6LpsIT7Xshr0pM+b8aCs8DxecMZnKt4456LxIvCKzmM+U3AeqHX9VwmibpuB0z2ne6IjW+LJats6WbOAzsDJWStGhWnaKODCC5SBOAIIQn7YEiot2DDQkzcDUbECUKejnOMUSBvSipOCkbsgMYezVoZG9Jc5Uy026VVfPoEOU0ckCcinSCsVjN1PNWZFftT7pHOqfY2pL+T5QqIlegT+5dvpvyvT9Ui0cWprsGjmmLNqOrcLEuqT0Xt3PxUlaQMMXEKdyguCLvaOTtnU3sSXbs6W6bjU61UrJq7mTbF26/VOdQDdSf0BOzvFz4P6kdlm/Dlcalylj2GPPawj0O68RNUcIEqarRCiAc8YWLcGY/GxHj+kBq5zLOLL814eQft+Jrp as a product. AAAC8HicjVLLSgMxFD0d3++qSzeDRXAlMyLosiiISwVrC22RTJrq0HmRyQil9DMEVxa3/pF/oH/hTUxBLT4SZubk3Htu5uQmyKIwV573UnKmpmdm5+YXFpeWV1bXyusbV3laSC5qPI1S2QhYLqIwETUVqkg0MilYHESiHvROdLx+J2Qepsml6meiHbObJOyGnCmimq2YqVvOosHp8Lpc8fY8M9xJ4FtQgR3nafkVLXSQgqNADIEEinAEhpxmEz48ZMS1MSBOEgpNXGCIRdIWlCUogxHbo/cNrZqWTWita+ZGzWmXiB5JShc7pEkpTxLWu7kmXpjKmv2p9sDU1P/Wp29ga8XEKtwS+5dunPlfnfai0MWR8RCSp8ww2h23VQpzKvrP3U+uFFXIiNO4Q3FJmBvl+Jxdo8mNd322zMRfTaZm9Zrb3AJvv7oLaMa6J3QF/O8NnwRX+3s+4YuDSvXYXoZ5bGEbu9TxQ1RxhnPUTE/u8YiRI50HZ+Q8faQ6JavZxJfhPL8D2P2bQA== a a
AAAC5nicjVLLSgMxFD0dX7W+qi7dDBbBVZmKoBuh6MZlC7YVapGZadqGzotMRijFLxBcWdz6W/5B/QtvYgpq8ZEwmZNzz7nJTeIlAU+l47zmrIXFpeWV/GphbX1jc6u4vdNM40z4rOHHQSyuPTdlAY9YQ3IZsOtEMDf0Atbyhhcq3rpjIuVxdCVHCeuEbj/iPe67kqj62W2x5JQd3ex5UDGgBNNqcXGKG3QRw0eGEAwRJOEALlLqbVTgICGugzFxghDXcYZ7FMibkYqRwiV2SGOfZm3DRjRXOVPt9mmVgD5BThsH5IlJJwir1Wwdz3Rmxf6Ue6xzqr2N6O+ZXCGxEgNi//LNlP/1qVokejjVNXCqKdGMqs43WTJ9Kmrn9qeqJGVIiFO4S3FB2NfO2Tnb2pPq2tXZujo+1UrFqrlvtBnefq3Oox6qO6EnUPl+4fOgeVSuEK4fl6rn5jHksYd9HNKNn6CKS9TQ0Ks/4AkTa2A9WhPr+UNq5YxnF1+a9fIOtSmWpQ==
F = AAAC8HicjVLLSgMxFD0d3++qSzeDRXAlMyLosiiISwVrC22RTJrq0HmRyQil9DMEVxa3/pF/oH/hTUxBLT4SZubk3Htu5uQmyKIwV573UnKmpmdm5+YXFpeWV1bXyusbV3laSC5qPI1S2QhYLqIwETUVqkg0MilYHESiHvROdLx+J2Qepsml6meiHbObJOyGnCmimq2YqVvOosHp8Lpc8fY8M9xJ4FtQgR3nafkVLXSQgqNADIEEinAEhpxmEz48ZMS1MSBOEgpNXGCIRdIWlCUogxHbo/cNrZqWTWita+ZGzWmXiB5JShc7pEkpTxLWu7kmXpjKmv2p9sDU1P/Wp29ga8XEKtwS+5dunPlfnfai0MWR8RCSp8ww2h23VQpzKvrP3U+uFFXIiNO4Q3FJmBvl+Jxdo8mNd322zMRfTaZm9Zrb3AJvv7oLaMa6J3QF/O8NnwRX+3s+4YuDSvXYXoZ5bGEbu9TxQ1RxhnPUTE/u8YiRI50HZ+Q8faQ6JavZxJfhPL8D2P2bQA==
AAAC8HicjVLLSgMxFD0d3++qSzeDRXAlMyLosiiISwVrC22RTJrq0HmRyQil9DMEVxa3/pF/oH/hTUxBLT4SZubk3Htu5uQmyKIwV573UnKmpmdm5+YXFpeWV1bXyusbV3laSC5qPI1S2QhYLqIwETUVqkg0MilYHESiHvROdLx+J2Qepsml6meiHbObJOyGnCmimq2YqVvOosHp8Lpc8fY8M9xJ4FtQgR3nafkVLXSQgqNADIEEinAEhpxmEz48ZMS1MSBOEgpNXGCIRdIWlCUogxHbo/cNrZqWTWita+ZGzWmXiB5JShc7pEkpTxLWu7kmXpjKmv2p9sDU1P/Wp29ga8XEKtwS+5dunPlfnfai0MWR8RCSp8ww2h23VQpzKvrP3U+uFFXIiNO4Q3FJmBvl+Jxdo8mNd322zMRfTaZm9Zrb3AJvv7oLaMa6J3QF/O8NnwRX+3s+4YuDSvXYXoZ5bGEbu9TxQ1RxhnPUTE/u8YiRI50HZ+Q8faQ6JavZxJfhPL8D2P2bQA== bAAAC73icjVLLSgMxFD0dX7W+qi7dDBbBVZkRQZdFNy4r2Ae2IpNp2obOy5mMUEr/QnBlcesn+Qf1L7yJU1CLj4TJnJx7zk1uEhZ5IpGW9ZozFhaXllfyq4W19Y3NreL2Tj0J09jlNTf0wrjJnIR7IuA1KaTHm1HMHZ95vMEG5yreuOdxIsLgSg4jfuM7vUB0hetIoq7bviP7rDti49tiySpbupnzwM5ACVmrhsUp2ugghIsUPjgCSMIeHCTUW7BhISLuBiPiYkJCxznGKJA3JRUnhUPsgMYezVoZG9Bc5Uy026VVPPpicpo4IE9IupiwWs3U8VRnVuxPuUc6p9rbkP4sy+UTK9En9i/fTPlfn6pFootTXYOgmiLNqOrcLEuqT0Xt3PxUlaQMEXEKdygeE3a1c3bOpvYkunZ1to6OT7VSsWruZtoUb79Wx6j76k7oCdjfL3we1I/KNuHL41LlLHsMeexhH4d04yeo4AJV1GiFAA94wsS4Mx6NifH8ITVymWcXX5rx8g7we5rq F bAAAC73icjVLLSgMxFD0dX7W+qi7dDBbBVZkRQZdFNy4r2Ae2IpNp2obOy5mMUEr/QnBlcesn+Qf1L7yJU1CLj4TJnJx7zk1uEhZ5IpGW9ZozFhaXllfyq4W19Y3NreL2Tj0J09jlNTf0wrjJnIR7IuA1KaTHm1HMHZ95vMEG5yreuOdxIsLgSg4jfuM7vUB0hetIoq7bviP7rDti49tiySpbupnzwM5ACVmrhsUp2ugghIsUPjgCSMIeHCTUW7BhISLuBiPiYkJCxznGKJA3JRUnhUPsgMYezVoZG9Bc5Uy026VVPPpicpo4IE9IupiwWs3U8VRnVuxPuUc6p9rbkP4sy+UTK9En9i/fTPlfn6pFootTXYOgmiLNqOrcLEuqT0Xt3PxUlaQMEXEKdygeE3a1c3bOpvYkunZ1to6OT7VSsWruZtoUb79Wx6j76k7oCdjfL3we1I/KNuHL41LlLHsMeexhH4d04yeo4AJV1GiFAA94wsS4Mx6NifH8ITVymWcXX5rx8g7we5rq F volution (+ + ) and the cross-correlation (+ ): − − − FIG. 26. The standard convolution theorem where the con- D 1 volution tensor has signature (+ + −) (++ ) X− (a b)k − = aib(k i) mod D (14) ∗ − i=0 As the convolution theorem must hold regardless of the D 1 (+ ) X− vectors being convolved, it must be a consequence of the (a b) −− = a b (15) ∗ k i (k+i) mod D interplay between Fourier matrices and the convolution i=0 tensor itself: when we contract each index of the con- volution tensor by a Fourier matrix, we obtain the Kro- Note that the cross-correlation (usually in its non- necker tensor, which is what we need to implement the circular version) is what is usually referred to as “convo- Hadamard (entry-wise) product. However, depending on lution” in the context of convolutional neural networks. the three signs of the convolution tensor, the convolu- It is straightforward to extend this definition to higher tion theorem takes a slightly different form: indices that rank convolutions, e.g. for matrices (where we intend all have a plus sign are contracted by the Fourier matrix index algebra to be modulo the dimension of the index), and indices that have a minus sign are contracted by the the standard definition is: inverse Fourier matrix (or vice versa, under the + ↔ − Dx 1 Dy 1 symmetry). See Fig. 27 for an example. X− X− (A B)mn = Ai,jBm i,n j (16) ∗ − − i=0 j=0 †
AAAC+nicjVLLSgMxFD0dX/U92qWbwSK4kqkIuiwK4rKC1UJbSyaN7dB5kckIpfotgivFrT/jH9S/8CZOwQc+EiZzcu45N7lJvCTwU+W6LwVranpmdq44v7C4tLyyaq+tn6dxJrmo8ziIZcNjqQj8SNSVrwLRSKRgoReIC29wpOMX10KmfhydqWEi2iHrRf6Vz5kiqmOXWiFTfc6C0fHtZavLej0hO3bZ3XFNc76DSg7KyFsttsdooYsYHBlCCERQhAMwpNSbqMBFQlwbI+IkId/EBW6xQN6MVIIUjNgBjT2aNXM2ornOmRo3p1UC+iQ5HWyRJyadJKxXc0w8M5k1+1Pukcmp9zakv5fnColV6BP7l2+i/K9P16JwhQNTg081JYbR1fE8S2ZORe/c+VCVogwJcRp3KS4Jc+OcnLNjPKmpXZ8tM/GxUWpWz3muzfD6a3Ue9VDfCT2BytcL/w7Od3cqhE/3ytXD/DEUsYFNbNON76OKE9RQpxWGuMMDHq0b6956tJ7epVYh95TwqVnPb67wnuU= which corresponds to the following graphical diagram AAAC5nicjVLLSgMxFD0dX7W+qi7dDBbBjWUqgi6Lbly2YFuhFpmZpm3ovMhkhFL8AsGVxa2/5R/Uv/AmpqAWHwmTOTn3nJvcJF4S8FQ6zmvOWlhcWl7JrxbW1jc2t4rbO800zoTPGn4cxOLac1MW8Ig1JJcBu04Ec0MvYC1veKHirTsmUh5HV3KUsE7o9iPe474riaof3RZLTtnRzZ4HFQNKMK0WF6e4QRcxfGQIwRBBEg7gIqXeRgUOEuI6GBMnCHEdZ7hHgbwZqRgpXGKHNPZp1jZsRHOVM9Vun1YJ6BPktHFAnph0grBazdbxTGdW7E+5xzqn2tuI/p7JFRIrMSD2L99M+V+fqkWihzNdA6eaEs2o6nyTJdOnonZuf6pKUoaEOIW7FBeEfe2cnbOtPamuXZ2tq+NTrVSsmvtGm+Ht1+o86qG6E3oCle8XPg+ax+UK4fpJqXpuHkMee9jHId34Kaq4RA0NvfoDnjCxBtajNbGeP6RWznh28aVZL++NCZaV
AAAC5nicjVLLSgMxFD0dX7W+qi7dDBbBjWUqgi6Lbly2YFuhFpmZpm3ovMhkhFL8AsGVxa2/5R/Uv/AmpqAWHwmTOTn3nJvcJF4S8FQ6zmvOWlhcWl7JrxbW1jc2t4rbO800zoTPGn4cxOLac1MW8Ig1JJcBu04Ec0MvYC1veKHirTsmUh5HV3KUsE7o9iPe474riaof3RZLTtnRzZ4HFQNKMK0WF6e4QRcxfGQIwRBBEg7gIqXeRgUOEuI6GBMnCHEdZ7hHgbwZqRgpXGKHNPZp1jZsRHOVM9Vun1YJ6BPktHFAnph0grBazdbxTGdW7E+5xzqn2tuI/p7JFRIrMSD2L99M+V+fqkWihzNdA6eaEs2o6nyTJdOnonZuf6pKUoaEOIW7FBeEfe2cnbOtPamuXZ2tq+NTrVSsmvtGm+Ht1+o86qG6E3oCle8XPg+ax+UK4fpJqXpuHkMee9jHId34Kaq4RA0NvfoDnjCxBtajNbGeP6RWznh28aVZL++NCZaV where the convolutions have signature (+ + ): F =AAAC5nicjVLLSgMxFD0dX7W+qi7dDBbBVZmKoBuh6MZlC7YVapGZadqGzotMRijFLxBcWdz6W/5B/QtvYgpq8ZEwmZNzz7nJTeIlAU+l47zmrIXFpeWV/GphbX1jc6u4vdNM40z4rOHHQSyuPTdlAY9YQ3IZsOtEMDf0Atbyhhcq3rpjIuVxdCVHCeuEbj/iPe67kqj62W2x5JQd3ex5UDGgBNNqcXGKG3QRw0eGEAwRJOEALlLqbVTgICGugzFxghDXcYZ7FMibkYqRwiV2SGOfZm3DRjRXOVPt9mmVgD5BThsH5IlJJwir1Wwdz3Rmxf6Ue6xzqr2N6O+ZXCGxEgNi//LNlP/1qVokejjVNXCqKdGMqs43WTJ9Kmrn9qeqJGVIiFO4S3FB2NfO2Tnb2pPq2tXZujo+1UrFqrlvtBnefq3Oox6qO6EnUPl+4fOgeVSuEK4fl6rn5jHksYd9HNKNn6CKS9TQ0Ks/4AkTa2A9WhPr+UNq5YxnF1+a9fIOtSmWpQ== AAAC+nicjVLLSgMxFD0dX/U92qWbwSK4kqkIuiwK4rKC1UJbSyaN7dB5kckIpfotgivFrT/jH9S/8CZOwQc+EiZzcu45N7lJvCTwU+W6LwVranpmdq44v7C4tLyyaq+tn6dxJrmo8ziIZcNjqQj8SNSVrwLRSKRgoReIC29wpOMX10KmfhydqWEi2iHrRf6Vz5kiqmOXWiFTfc6C0fHtZavLej0hO3bZ3XFNc76DSg7KyFsttsdooYsYHBlCCERQhAMwpNSbqMBFQlwbI+IkId/EBW6xQN6MVIIUjNgBjT2aNXM2ornOmRo3p1UC+iQ5HWyRJyadJKxXc0w8M5k1+1Pukcmp9zakv5fnColV6BP7l2+i/K9P16JwhQNTg081JYbR1fE8S2ZORe/c+VCVogwJcRp3KS4Jc+OcnLNjPKmpXZ8tM/GxUWpWz3muzfD6a3Ue9VDfCT2BytcL/w7Od3cqhE/3ytXD/DEUsYFNbNON76OKE9RQpxWGuMMDHq0b6956tJ7epVYh95TwqVnPb67wnuU= − † +AAAC5nicjVLLSgMxFD0dX7W+qi7dDBZBEMpUBF0W3bhswbZCLTIzTdvQeZHJCKX4BYIri1t/yz+of+FNTEEtPhImc3LuOTe5Sbwk4Kl0nNectbC4tLySXy2srW9sbhW3d5ppnAmfNfw4iMW156Ys4BFrSC4Ddp0I5oZewFre8ELFW3dMpDyOruQoYZ3Q7Ue8x31XElU/ui2WnLKjmz0PKgaUYFotLk5xgy5i+MgQgiGCJBzARUq9jQocJMR1MCZOEOI6znCPAnkzUjFSuMQOaezTrG3YiOYqZ6rdPq0S0CfIaeOAPDHpBGG1mq3jmc6s2J9yj3VOtbcR/T2TKyRWYkDsX76Z8r8+VYtED2e6Bk41JZpR1fkmS6ZPRe3c/lSVpAwJcQp3KS4I+9o5O2dbe1JduzpbV8enWqlYNfeNNsPbr9V51EN1J/QEKt8vfB40j8sVwvWTUvXcPIY89rCPQ7rxU1RxiRoaevUHPGFiDaxHa2I9f0itnPHs4kuzXt4BiAWWkw==
F AAAC8HicjVLLSgMxFD0d3++qSzeDRXAlMyLosiiISwVrC22RTJrq0HmRyQil9DMEVxa3/pF/oH/hTUxBLT4SZubk3Htu5uQmyKIwV573UnKmpmdm5+YXFpeWV1bXyusbV3laSC5qPI1S2QhYLqIwETUVqkg0MilYHESiHvROdLx+J2Qepsml6meiHbObJOyGnCmimq2YqVvOosHp8Lpc8fY8M9xJ4FtQgR3nafkVLXSQgqNADIEEinAEhpxmEz48ZMS1MSBOEgpNXGCIRdIWlCUogxHbo/cNrZqWTWita+ZGzWmXiB5JShc7pEkpTxLWu7kmXpjKmv2p9sDU1P/Wp29ga8XEKtwS+5dunPlfnfai0MWR8RCSp8ww2h23VQpzKvrP3U+uFFXIiNO4Q3FJmBvl+Jxdo8mNd322zMRfTaZm9Zrb3AJvv7oLaMa6J3QF/O8NnwRX+3s+4YuDSvXYXoZ5bGEbu9TxQ1RxhnPUTE/u8YiRI50HZ+Q8faQ6JavZxJfhPL8D2P2bQA== F AAAAC5nicjVLLSgMxFD0dX7W+qi7dDBbBVZmKoMuqG5ct2FaoRWbStB06LzIZoRS/QHBlcetv+Qf1L7yJKajFR8JkTs495yY3iZcEfiod5zVnLSwuLa/kVwtr6xubW8XtnWYaZ4LxBouDWFx7bsoDP+IN6cuAXyeCu6EX8JY3vFDx1h0XqR9HV3KU8E7o9iO/5zNXElU/uy2WnLKjmz0PKgaUYFotLk5xgy5iMGQIwRFBEg7gIqXeRgUOEuI6GBMnCPk6znGPAnkzUnFSuMQOaezTrG3YiOYqZ6rdjFYJ6BPktHFAnph0grBazdbxTGdW7E+5xzqn2tuI/p7JFRIrMSD2L99M+V+fqkWih1Ndg081JZpR1TGTJdOnonZuf6pKUoaEOIW7FBeEmXbOztnWnlTXrs7W1fGpVipWzZnRZnj7tTqPeqjuhJ5A5fuFz4PmUblCuH5cqp6bx5DHHvZxSDd+giouUUNDr/6AJ0ysgfVoTaznD6mVM55dfGnWyzu/MZap
FIG. 27. The convolution tensor in this example satisfies the BAAAC5nicjVLLSgMxFD0dX/VddelmsAiuylQEXZa6cdmCfUAtMpOm7dB5kckIpfgFgiuLW3/LP6h/4U1MQS0+EiZzcu45N7lJvCTwU+k4rzlraXlldS2/vrG5tb2zW9jbb6ZxJhhvsDiIRdtzUx74EW9IXwa8nQjuhl7AW97oUsVbd1ykfhxdy3HCu6E7iPy+z1xJVL16Wyg6JUc3exGUDSjCtFpcmOEGPcRgyBCCI4IkHMBFSr2DMhwkxHUxIU4Q8nWc4x4b5M1IxUnhEjuicUCzjmEjmqucqXYzWiWgT5DTxjF5YtIJwmo1W8cznVmxP+We6Jxqb2P6eyZXSKzEkNi/fHPlf32qFok+LnQNPtWUaEZVx0yWTJ+K2rn9qSpJGRLiFO5RXBBm2jk/Z1t7Ul27OltXx2daqVg1Z0ab4e3X6jzqoboTegLl7xe+CJqnpTLh+lmxUjWPIY9DHOGEbvwcFVyhhoZe/QFPmFpD69GaWs8fUitnPAf40qyXd8Gzlqo= generalized convolution theorem where indices with opposite (±±±) sign in χijk get contracted by opposite Fourier transforms FIG. 25. The convolution of two matrices A and B can be and always yield the Kronecker tensor. expressed as a suitable tensor contraction and therefore has the same visual representation as a matrix product. (notice that we could also apply convolutions of different signature to different sets of indices) VII. TENSOR CONTRACTIONS IN PYTHON And one can continue in a similar fashion, for an in- creasing number of indices. Interestingly, even if a con- Python’s numerical math library numpy contains a x y function called “einsum” (short for Einstein summation volution kernel is not separable (e.g. Kij = Ki Kj ), the “process” is, which means that rank-N convolution6 convention) [20]. Learning how to use einsum can be is computable by contracting two indices at a time. quite challenging without a good grasp of the rules of tensor contraction. This is where graphical calculus can help. B. The convolution theorem There are two ways to use einsum: the first is to define all the contraction rules in the form of a string (passed as first argument) that specifies what happens to the indices The convolution theorem states that under the Fourier of the various tensors passed as subsequent arguments transform, products become convolutions and vice versa: (one can pass several tensors, not just one or two). For example, if we want to perform the matrix-matrix multi- f(\x)g(x) = fb(k) g(k) (17) ∗ b plication AijBjk, the string would read "ij,jk -> ik", 9 where we can specify in which order we want the leftover formula in the language of einsum. This is a foolproof indices. A tensor product AijBkl would correspond to way of implementing tensor contractions in python. the string "ij,kl", a partial trace Aiijk to "iijk" and so on: VIII. CONCLUSION AdotB = einsum("ij,jk->ik",A,B) AtensorB = einsum("ij,kl",A,B) In this paper we have seen two of the many poten- tr1A = einsum("iijk",A) tial applications of graphical calculus. The message that At the time of writing (March 2019), there are some I wish to convey is that graphical calculus is a powerful functionalities that are still missing from einsum, such companion to the student and to the practitioner of linear as the possibility of creating a diagonal matrix from the algebra. As innumerable topics in science are based on vector of the diagonal, which would correspond to the linear algebra, graphical calculus can become an invalu- string "i -> ii". able ally that strengthens our understanding and makes The second way to use einsum does not require a string us reach further with less effort. The fact that visual rep- with the index specifications. Rather, the arguments resentations are much simpler to grasp and manipulate of einsum alternate between tensors and lists of inte- than conventional mathematical expressions, makes it a ger numbers that identify their indices. When an index wonderful tool also for audiences who lack a knowledge repeats it is summed over. The matrix multiplication, of university-level algebra. tensor product and partial trace in the paragraph above Finally, I would like to point out that these sort of sym- would be bolic manipulations lend themselves very well to gamifi- cation. I believe that it would be valuable to explore computer and smartphone applications that can teach AdotB = einsum(A,[0,1],B,[1,2],[0,2]) linear algebra (even rather complex topics) through the AtensorB = einsum(A,[0,1],B,[2,3]) gamified manipulation of graphical objects. tr1A = einsum(A,[0,0,1,2]) IX. ACKNOWLEDGEMENTS It is of great help to draw by hand the required network of tensor contractions graphically, then identify the wires I thank Jacob Biamonte for several useful conversa- by number or by letter (thus wires that contract have tions and Electra Eleftheriadou for her support and help- a single number or letter) and finally simply copy the ful feedback on this manuscript.
[1] David Rose. Universal design for learning. Journal of physics, pages 95–172. Springer, 2010. Special Education Technology, 15(3):45–49, 2000. [10] Christopher J Wood, Jacob D Biamonte, and David G [2] Yann LeCun, Bernhard Boser, John S Denker, Don- Cory. Tensor networks and graphical calculus for open nie Henderson, Richard E Howard, Wayne Hubbard, quantum systems. arXiv preprint arXiv:1111.6950, 2011. and Lawrence D Jackel. Backpropagation applied to [11] Jacob Biamonte. Charged string tensor networks. handwritten zip code recognition. Neural computation, Proceedings of the National Academy of Sciences, 1(4):541–551, 1989. 114(10):2447–2449, 2017. [3] Roger Penrose. Applications of negative dimensional ten- [12] Arthur Jaffe, Zhengwei Liu, and Alex Wozniakowski. sors. Combinatorial mathematics and its applications, Holographic software for quantum networks. Science 1:221–244, 1971. China Mathematics, 61(4):593–626, 2018. [4] Yves Lafont. Towards an algebraic theory of boolean [13] Miriam Backens. The zx-calculus is complete for sta- circuits. Journal of Pure and Applied Algebra, 184(2- bilizer quantum mechanics. New Journal of Physics, 3):257–310, 2003. 16(9):093021, 2014. [5] Samson Abramsky and Bob Coecke. A categorical seman- [14] Emmanuel Jeandel, Simon Perdrix, and Renaud Vilmart. tics of quantum protocols. In Logic in computer science, A complete axiomatisation of the zx-calculus for clifford+ 2004. Proceedings of the 19th Annual IEEE Symposium t quantum mechanics. In Proceedings of the 33rd Annual on, pages 415–425. IEEE, 2004. ACM/IEEE Symposium on Logic in Computer Science, [6] Bob Coecke. Quantum picturalism. Contemporary pages 559–568. ACM, 2018. physics, 51(1):59–83, 2010. [15] CG Khatri and C Radhakrishna Rao. Solutions to some [7] Robert B Griffiths, Shengjun Wu, Li Yu, and Scott M functional equations and their applications to characteri- Cohen. Atemporal diagrams for quantum circuits. Phys- zation of probability distributions. Sankhy¯a:The Indian ical Review A, 73(5):052309, 2006. Journal of Statistics, Series A, pages 167–180, 1968. [8] Peter Selinger. A survey of graphical languages for [16] Xian Zhang, ZP Yang, and CG Cao. Inequalities involv- monoidal categories. In New structures for physics, pages ing khatri-rao products of positive semidefinite matrices. 289–355. Springer, 2010. Applied Mathematics E-Notes, 2:117–124, 2002. [9] John Baez and Mike Stay. Physics, topology, logic and [17] Derrick S Tracy and Rana P Singh. A new matrix prod- computation: a rosetta stone. In New structures for uct and its applications in partitioned matrix differenti- 10
ation. Statistica Neerlandica, 26(4):143–157, 1972. [19] Hongli Yang and Guoping He. Some properties of ma- [18] J´anosAcz´el.On mean values. Bulletin of the American trix product and its applications in nonnegative tensor Mathematical Society, 54(4):392–400, 1948. decomposition. Journal of Information and Computing Science, 3(4):269–280, 2008. [20] Travis E Oliphant. A guide to NumPy, volume 1. Trelgol Publishing USA, 2006.