The Thirty-Fifth AAAI Conference on Artificial Intelligence (AAAI-21) Dual Quaternion Knowledge Graph Embeddings Zongsheng Cao 1,2, Qianqian Xu 3,*, Zhiyong Yang 1,2, Xiaochun Cao 1,2,6, Qingming Huang 3,4,5,6,* 1 State Key Laboratory of Information Security, Institute of Information Engineering, CAS, Beijing, China 2 School of Cyber Security, University of Chinese Academy of Sciences, Beijing, China 3 Key Laboratory of Intelligent Information Processing, Institute of Computing Technology, CAS, Beijing, China 4 School of Computer Science and Technology, University of Chinese Academy of Sciences, Beijing, China 5 Key Laboratory of Big Data Mining and Knowledge Management, Chinese Academy of Sciences, Beijing, China 6 Peng Cheng Laboratory, Shenzhen, China fcaozongsheng,yangzhiyong,[email protected], [email protected], [email protected] Abstract In this paper, we study the problem of learning representations of entities and relations in the knowledge graph for the link pre- diction task. Our idea is based on the observation that the vast majority of the related work only models the relation as a sin- gle geometric operation such as translation or rotation, which limits the representation power of the underlying models and makes it harder to match the complicated relations existed in real-world datasets. To embrace a richer set of relational information, we propose a new method called dual quaternion Figure 1: An illustration of that DualE is a unified framework knowledge graph embeddings (DualE), which introduces dual of translation family and rotation family. quaternions into knowledge graph embeddings. Specifically, a dual quaternion behaves like a “complex quaternion” with its real and imaginary part all being quaternary. The core of DualE lies a specific design of dual-quaternion-based multiplication, could be traced back to TransE (Bordes et al. 2013). A re- which universally models relations as the compositions of a markable trait of the models belonging to this family is that series of translation and rotation operations. The major merits they provide a natural way to represent the hierarchical rela- of DualE are three-fold: 1) it is the first unified framework tions which are extremely common in KGs. And its variants embracing both rotation-based and translation-based models such as (Wang et al. 2014; Lin et al. 2015) can model the in 3D space, 2) it expands the embedding space to the dual multiple relations pattern. Meanwhile, rotation family refers quaternion space with a more intuitive physical and geometric to the models in which the relations are modeled as rota- interpretation, 3) it satisfies the key patterns and the multiple tion operations, which typically includes (Sun et al. 2019; relations pattern of relational representation learning. Exper- Zhang et al. 2019). As what is claimed in the representative imental results on four real-world datasets demonstrate the work RotatE (Sun et al. 2019), rotation can model all the effectiveness of our DualE method. three fundamental patterns of relations in KG, i.e., symme- try/antisymmetry, inversion, and composition. A thorough Introduction discussion of these methods is delayed to the next section. Knowledge Graph (KG) represents a collection of interlinked With the efforts of the translation family and rotation fam- descriptions of entities, namely, real-world objects, events, ily models, we have witnessed great success of KG-based situations, or abstract concepts. During the past decade, KG applications. Nonetheless, a single translation or rotation is has been proven to be an indispensable build-block for a wide not always a better way to represent relations. For example, spectrum of applications ranging from question answering, translation family cannot model all the three fundamental knowledge inference to natural language processing. To effec- patterns (Sun et al. 2019); rotation family models have little tively integrate KG into downstream AI applications, a key effect on hierarchical relations and multiple relations pattern. step, known as Knowledge Graph Embedding (KGE), then See the motivation part for more detailed analysis on this comes into play as a powerful tool which encodes entities and issue. As such, we find that the strengths of translation fam- relations of the graph into low-dimensional representations. ily and rotation family are essentially complementary. This brings about the question that is there a way to unify both We could roughly divide the vast majority of KGE method- translation and rotation in one framework? s into two general families based on how relations in KG are formulated. As the name suggests, translation family refers Fortunately, we can seek out a solution by means of a to the models which regard relations as translations, which number field named dual quaternion, which is proposed by William Kingdom Clifford (Clifford 1871). The dual quater- Copyright c 2021, Association for the Advancement of Artificial nion has an intuitive geometric and physical interpretation: Intelligence (www.aaai.org). All rights reserved. (1) It can represent both translation and rotation. (2) It pro- 6894 vides an elegant way of solving a range of problems that are tail entity through rotational operation in the complex space otherwise complex, such as rigid transformations. with only one rotating surface. As a remarkable property, In this paper, we propose a new method called dual quater- RotatE is the first model to unify symmetry/antisymmetry, nion knowledge graph embeddings (DualE). As a key instru- inversion, and composition patterns for KGE. This suggests ment, we introduce the dual quaternion as the number field that rotation operations in the complex space have a strong of the embedding space. Specifically, embeddings in a dual potential to empower a universal knowledge representation. quaternion space are vectors in the hypercomplex space Hd This is why we name this direction after rotation. Most recent- formed as a+b, where a and b are two quaternions represent- ly, QuatE (Zhang et al. 2019) extends the complex space into ing the real and dual part of the vector, respectively. It could the quaternion space with two rotating surfaces. However, be proved that with a proper definition of dual-quaternion- rotation family cannot model the hierarchical structure, nor based multiplication, we can represent relations in the KGs can it model multiple relations between two entities simulta- as compositions of translation and rotation operations. As neously. shown in Figure 1, this allows us to unify the previous studies KGE with Deep Learning. Recently there are some models in the translation family and rotation family. using neural networks to produce KG embeddings with re- To summarize, our contributions are as follows: 1) We pro- markable effects. For instance, R-GCN (Schlichtkrull et al. vide theoretical analyses on why both rotation and translation 2018), ConvE (Dettmers et al. 2017) and ConvKB (Nguyen operations are necessary for knowledge graph embedding. 2) et al. 2018), KBGAT (Nathani et al. 2019), A2N (Bansal et al. We propose a new framework called DualE to effectively in- 2019). A downside of these methods is that the geometric tegrate rotation and translation operations in KG embedding meaning is not clear or the transformation is single. methods. 3) We conduct a series of theoretical and empirical analyses to show the strength of DualE against some of the Problem Setup SOTA methods. Problem Definition Related Work In the KGE problem, we define the head entity, the relation and the tail entity as h, r and t, respectively. Now we define Recall that we divide the majority of KGE methods rough- r ly into two families according to the way they manipulate a triplet as (h; r; t), which represents h ! t, i.e., the head relations. In this section, we take a further step to provide a entity h is related to tail entity t by a relation r. Then a detailed review of the methods in each family. knowledge graph could be represented by means of a set Translation Family. This family includes TransE (Bordes of specific triplets (h; r; t) ⊆ V × R × V, where V and et al. 2013) and its variants. The common trait of these meth- R are entity and relation sets, respectively. Practically, it ods is that they all model relations as translations between is impossible to collect a complete knowledge graph with heads and tails in the embedding space. TransE (Bordes et al. pure human efforts. The link prediction task then comes into 2013) is the first model that realizes this assumption based on play as a powerful application to automatically infer missing the principle head + relation ≈ tail. TransH (Wang et al. links in a knowledge graph. From a practical perspective, our 2014), TransR (Lin et al. 2015), TransD (Ji et al. 2015), and ultimate goal is then to develop effective knowledge graph TransA (Xiao et al. 2015) then improve this idea with dif- embedding algorithms for the link prediction task. In this way, ferent projection strategies. TransG (Xiao, Huang, and Zhu our algorithm will simultaneously produce proper embedding 2016) and KG2E (He et al. 2015) further inject probabilis- of the nodes/relations as well as the confidence score of a tic principles into this framework by considering Bayesian given triplet (h; r; t). nonparametric Gaussian mixture model and Gaussian distri- bution covariance, respectively. TranSparse (Ji et al. 2016) Motivation provides adaptive sparsity to the transfer matrices in search Before introducing the motivation, we give definitions of of a solution to heterogeneity and imbalance issues of KGs. the key patterns (see Appendix for the definitions of sym- Last but definitely not the least, a recent work called TorusE metry/antisymmetry, inversion, composition patterns) and (Ebisu and Ichise 2018) adopts torus to avoid forcing embed- multiple relations pattern first. ding to be on a sphere. In one word, translation family models Definition 1 Relations ri are multiple if 8i 2 f0; ··· ; mg, provide a simple yet effective way to achieve better results (h; ri; t) can hold in KGs simultaneously.