PRELIMINARY VERSION: DO NOT CITE The AAAI Digital Library will contain the published version some time after the conference Dual Quaternion Knowledge Graph Embeddings Zongsheng Cao 1,2, Qianqian Xu 3,*, Zhiyong Yang 1,2, Xiaochun Cao 1,2,6, Qingming Huang 3,4,5,6,* 1 State Key Laboratory of Information Security, Institute of Information Engineering, CAS, Beijing, China 2 School of Cyber Security, University of Chinese Academy of Sciences, Beijing, China 3 Key Laboratory of Intelligent Information Processing, Institute of Computing Technology, CAS, Beijing, China 4 School of Computer Science and Technology, University of Chinese Academy of Sciences, Beijing, China 5 Key Laboratory of Big Data Mining and Knowledge Management, Chinese Academy of Sciences, Beijing, China 6 Peng Cheng Laboratory, Shenzhen, China fcaozongsheng,yangzhiyong,[email protected], [email protected], [email protected] Abstract In this paper, we study the problem of learning representations of entities and relations in the knowledge graph for the link pre- diction task. Our idea is based on the observation that the vast majority of the related work only models the relation as a sin- gle geometric operation such as translation or rotation, which limits the representation power of the underlying models and makes it harder to match the complicated relations existed in real-world datasets. To embrace a richer set of relational information, we propose a new method called dual quaternion Figure 1: An illustration of that DualE is a unified framework knowledge graph embeddings (DualE), which introduces dual of translation family and rotation family. quaternions into knowledge graph embeddings. Specifically, a dual quaternion behaves like a “complex quaternion” with its could be traced back to TransE (Bordes et al. 2013). A re- real and imaginary part all being quaternary. The core of DualE lies a specific design of dual-quaternion-based multiplication, markable trait of the models belonging to this family is that which universally models relations as the compositions of a they provide a natural way to represent the hierarchical rela- series of translation and rotation operations. The major merits tions which are extremely common in KGs. And its variants of DualE are three-fold: 1) it is the first unified framework such as (Wang et al. 2014; Lin et al. 2015) can model the embracing both rotation-based and translation-based models multiple relations pattern. Meanwhile, rotation family refers in 3D space, 2) it expands the embedding space to the dual to the models in which the relations are modeled as rota- quaternion space with a more intuitive physical and geometric tion operations, which typically includes (Sun et al. 2019; interpretation, 3) it satisfies the key patterns and the multiple Zhang et al. 2019). As what is claimed in the representative relations pattern of relational representation learning. Exper- work RotatE (Sun et al. 2019), rotation can model all the imental results on four real-world datasets demonstrate the three fundamental patterns of relations in KG, i.e., symme- effectiveness of our DualE method. try/antisymmetry, inversion, and composition. A thorough discussion of these methods is delayed to the next section. Introduction With the efforts of the translation family and rotation fam- Knowledge Graph (KG) represents a collection of interlinked ily models, we have witnessed great success of KG-based descriptions of entities, namely, real-world objects, events, applications. Nonetheless, a single translation or rotation is situations, or abstract concepts. During the past decade, KG not always a better way to represent relations. For example, has been proven to be an indispensable build-block for a wide translation family cannot model all the three fundamental spectrum of applications ranging from question answering, patterns (Sun et al. 2019); rotation family models have little knowledge inference to natural language processing. To effec- effect on hierarchical relations and multiple relations pattern. tively integrate KG into downstream AI applications, a key See the motivation part for more detailed analysis on this step, known as Knowledge Graph Embedding (KGE), then issue. As such, we find that the strengths of translation fam- comes into play as a powerful tool which encodes entities and ily and rotation family are essentially complementary. This relations of the graph into low-dimensional representations. brings about the question that is there a way to unify both translation and rotation in one framework? We could roughly divide the vast majority of KGE method- s into two general families based on how relations in KG are Fortunately, we can seek out a solution by means of a formulated. As the name suggests, translation family refers number field named dual quaternion, which is proposed by to the models which regard relations as translations, which William Kingdom Clifford (Clifford 1871). The dual quater- nion has an intuitive geometric and physical interpretation: Copyright c 2021, Association for the Advancement of Artificial (1) It can represent both translation and rotation. (2) It pro- Intelligence (www.aaai.org). All rights reserved. vides an elegant way of solving a range of problems that are otherwise complex, such as rigid transformations. with only one rotating surface. As a remarkable property, In this paper, we propose a new method called dual quater- RotatE is the first model to unify symmetry/antisymmetry, nion knowledge graph embeddings (DualE). As a key instru- inversion, and composition patterns for KGE. This suggests ment, we introduce the dual quaternion as the number field that rotation operations in the complex space have a strong of the embedding space. Specifically, embeddings in a dual potential to empower a universal knowledge representation. quaternion space are vectors in the hypercomplex space Hd This is why we name this direction after rotation. Most recent- formed as a+b, where a and b are two quaternions represent- ly, QuatE (Zhang et al. 2019) extends the complex space into ing the real and dual part of the vector, respectively. It could the quaternion space with two rotating surfaces. However, be proved that with a proper definition of dual-quaternion- rotation family cannot model the hierarchical structure, nor based multiplication, we can represent relations in the KGs can it model multiple relations between two entities simulta- as compositions of translation and rotation operations. As neously. shown in Figure 1, this allows us to unify the previous studies KGE with Deep Learning. Recently there are some models in the translation family and rotation family. using neural networks to produce KG embeddings with re- To summarize, our contributions are as follows: 1) We pro- markable effects. For instance, R-GCN (Schlichtkrull et al. vide theoretical analyses on why both rotation and translation 2018), ConvE (Dettmers et al. 2017) and ConvKB (Nguyen operations are necessary for knowledge graph embedding. 2) et al. 2018), KBGAT (Nathani et al. 2019), A2N (Bansal et al. We propose a new framework called DualE to effectively in- 2019). A downside of these methods is that the geometric tegrate rotation and translation operations in KG embedding meaning is not clear or the transformation is single. methods. 3) We conduct a series of theoretical and empirical analyses to show the strength of DualE against some of the Problem Setup SOTA methods. Problem Definition Related Work In the KGE problem, we define the head entity, the relation and the tail entity as h, r and t, respectively. Now we define Recall that we divide the majority of KGE methods rough- r ly into two families according to the way they manipulate a triplet as (h; r; t), which represents h ! t, i.e., the head relations. In this section, we take a further step to provide a entity h is related to tail entity t by a relation r. Then a detailed review of the methods in each family. knowledge graph could be represented by means of a set Translation Family. This family includes TransE (Bordes of specific triplets (h; r; t) ⊆ V × R × V, where V and et al. 2013) and its variants. The common trait of these meth- R are entity and relation sets, respectively. Practically, it ods is that they all model relations as translations between is impossible to collect a complete knowledge graph with heads and tails in the embedding space. TransE (Bordes et al. pure human efforts. The link prediction task then comes into 2013) is the first model that realizes this assumption based on play as a powerful application to automatically infer missing the principle head + relation ≈ tail. TransH (Wang et al. links in a knowledge graph. From a practical perspective, our 2014), TransR (Lin et al. 2015), TransD (Ji et al. 2015), and ultimate goal is then to develop effective knowledge graph TransA (Xiao et al. 2015) then improve this idea with dif- embedding algorithms for the link prediction task. In this way, ferent projection strategies. TransG (Xiao, Huang, and Zhu our algorithm will simultaneously produce proper embedding 2016) and KG2E (He et al. 2015) further inject probabilis- of the nodes/relations as well as the confidence score of a tic principles into this framework by considering Bayesian given triplet (h; r; t). nonparametric Gaussian mixture model and Gaussian distri- Motivation bution covariance, respectively. TranSparse (Ji et al. 2016) provides adaptive sparsity to the transfer matrices in search Before introducing the motivation, we give definitions of the of a solution to heterogeneity and imbalance issues of KGs. key patterns (symmetry/antisymmetry, inversion, composi- Last but definitely not the least, a recent work called TorusE tion) and multiple relations pattern first. (Ebisu and Ichise 2018) adopts torus to avoid forcing embed- Definition 1 A relation r is symmetric (antisymmetric) ding to be on a sphere. In one word, translation family models if 8x; y provide a simple yet effective way to achieve better results than previous complex models such as (Bordes et al.