Clusternet: Deep Hierarchical Cluster Network with Rigorously Rotation-Invariant Representation for Point Cloud Analysis

ClusterNet: Deep Hierarchical Cluster Network with Rigorously Rotation-Invariant Representation for Point Cloud Analysis

Chao Chen1 Guanbin Li1∗ Ruijia Xu1 Tianshui Chen1,2 Meng Wang3 Liang Lin1,2 1Sun Yat-sen University 2DarkMatter AI Research 3Hefei University of Technology [email protected], [email protected], [email protected] [email protected], [email protected], [email protected]

Abstract group, a 3D object possesses rotated clones in infinite atti- tudes, thus a machine learning model is obliged to extract Current neural networks for 3D object recognition are features from an extremely huge input space. For exam- vulnerable to 3D rotation. Existing works mostly rely on ple, in 3D object classification task, the category label of massive amounts of rotation-augmented data to alleviate an object is invariant against arbitrary rotation transforma- the problem, which lacks solid guarantee of the 3D rotation tion in majority situations. However, from the perspective invariance. In this paper, we address the issue by introduc- of a classification model, an object and its rotated clone are ing a novel point cloud representation that can be mathe- distinct in input metric space, hence the model, such as neu- matically proved rigorously rotation-invariant, i.e., identi- ral network based methods, should have enough capacity to cal point clouds in different orientations are unified as a learn rotation invariance from data and then approximate a unique and consistent representation. Moreover, the pro- complex function that maps identical objects in infinite atti- posed representation is conditional information-lossless, tudes to similar features in feature metric space. because it retains all necessary information of point cloud To alleviate the curse of rotation, a straightforward except for orientation information. In addition, the pro- method is to design a model with high capacity, such as a posed representation is complementary with existing net- deep neural network with considerable layers, and feed the work architectures for point cloud and fundamentally im- model with great amounts of rotation-augmented data [1] proves their robustness against rotation transformation. Fi- based on a well-designed augmentation pipeline. Although nally, we propose a deep hierarchical cluster network called data augmentation is effective to some extent, it is computa- ClusterNet to better adapt to the proposed representation. tionally expensive in training phase and lacks solid guaran- We employ hierarchical clustering to explore and exploit tee of rotation robustness. [11, 18] apply spatial transformer the geometric structure of point cloud, which is embed- network [5] to canonicalize the input data before feature ex- ded in a hierarchical structure tree. Extensive experimen- traction, which improves the rotation-robustness of model tal results have shown that our proposed method greatly but still inherits all the defects of the data augmentation. outperforms the state-of-the-arts in rotation robustness on [16] proposes a rotation-equivariant network for 3D point rotation-augmented 3D object classification benchmarks. clouds using a special convolutional operation with local rotation invariance as a basic block. The method attempts to equip the neural network with rotation-symmetry. How- 1. Introduction ever, it is hard to guarantee the capacity of such network to Rotation transformation is natural and common in 3D satisfy all rotation-equivariant constraints in each layer. world, however, it gives rise to an intractable challenge for We address the issue by introducing a novel Rigorous 3D recognization. Theoretically, since SO(3)1 is an infinite Rotation-Invariant (RRI) representation of point cloud. Identical objects in different orientations are unified as a ∗Corresponding author is Guanbin Li. This work was supported in part by the State Key Development Program under Grant 2016YFB1001004, consistent representation, which implies that the input space in part by the National Natural Science Foundation of China under Grant is heavily reduced and the 3D recognization tasks become No.61602533 and No.61702565, in part by the Fundamental Research much easier. It can be mathematically proved that the pro- Funds for the Central Universities under Grant No.18lgpy63. This work posed representation is rigorously rotation-invariant, and was also sponsored by SenseTime Research Fund. 13D rotation group, denoted as SO(3), contains all rotation transforma- information-lossless under a mild condition. Given any tions in R3 under the operation of composition. data point in point cloud and a non-collinear neighbor ar-

4994 bitrarily, the whole point cloud can be restored intactly with the RRI representation, even if the point cloud is under an 0.4 0.2

Z unknown orientation. In other words, the RRI representa- 0.0 tion maintains all necessary information of point cloud ex- − 0.2 cept for the volatile orientation information which is associ- − 0.4 ated with specific rotation transformation. Furthermore, the 0.2 0.1 -1.0 -0.8 -0.6 -0.4 0.0 Y -0.2 0.0 0.2 0.4 0.6 -0.1 RRI representation is flexible to be plugged into the cur- X rent neural architectures and endows them with rigorous rotation invariance. The major difference between rotation- Figure 1: The left figure is a dendrogram of a point cloud equivariant network and our proposed method is that the learned by hierarchical clustering. The right figure shows former embeds the invariance property as a priori into neu- partition of the point cloud of plane in a merge-level re- ral network, but the latter separates the rotation invariance maining 8 clusters. from neural network and directly cut down the orientation- redundancy of input space. to suffer from loss of resolution and high computational Moreover, we propose a deep hierarchical network expense during transformation and subsequent processing. called ClusterNet to better adapt to our new representation. In order to escape from the limit of volumetric grid, some Specifically, we employ unsupervised hierarchical cluster- methods partition the R3 space by the traditional data struc- ing to learn the underlying geometric structure of point tures, such as k-d trees [6] and octrees [14, 17], to allevi- cloud. As a result, we can obtain a hierarchical structure ate the issues. PointNet [11] is the most pioneering work tree and then employ it to guide hierarchical features learn- that takes point cloud as input and applies MLPs and max ing. Similar to CNNs, ClusterNet extracts features corre- pooling to construct a universal approximator with permu- sponding with small clusters, which learns fine-grained pat- tation invariance. Since the lack of sensing capability for terns of point cloud; the smaller cluster features are then local information, a variety of hierarchical neural networks aggregated as larger cluster features capturing higher-level for point cloud, such as PointNet++ [13] and DGCNN [18], information. The process of embedding is repeated along are proposed to progressively abstract features along a hi- the hierarchical structure tree from bottom to top until we erarchical structure designed in a heuristic way. Recently, achieve the global features of the whole point cloud. Chen et al. [?] proposed to leverages nonlinear Radial Basis We summarize our major contributions as follows: 6 Function (RBF) convolution as basis feature extractor for 1. We propose a new point cloud representation that sat- robust point cloud representation. As far as we known, the isfies, both theoretically and empirically, rotation in- existing methods merely design the hierarchical structure variance and information preservation; by priori knowledge and none of them have made effort to explore the geometric structure underlying the point cloud, 2. The proposed representation is complementary with which is prone to cause lower capacity of the hierarchical the existing neural network frameworks and funda- neural network. mentally improves their robustness against rotation Hierarchical Clustering. In the area of unsupervised transformation; learning, hierarchical clustering [10] is a classical method to build a hierarchy (also called dendrogram) of clusters. It 3. We further introduce a novel deep hierarchical network generally consists of agglomerative type and divisive type. called ClusterNet to better adapt to our new repre- The first one considers all data points as the smallest cluster sentation. Combing the novel point cloud representa- and merges the two closest ones with respect to a particular tion and the elaborate ClusterNet, our method achieves distance metric and a linkage criteria from bottom to top, state-of-the-art robustness in standard 3D classification and the latter performs in an opposite direction. A typical benchmarks. linkage criteria is ward linkage minimizing the total within- 2. Related Work cluster variance, which can remedy the degeneration case of uneven cluster sizes. Furthermore, the point cloud in low Deep Learning for 3D Objects. In general, the develop- dimensional space, such as R3, is quite suitable for hier- ment of deep learning for 3D object is closely related to the archical clustering. A dendrogram and a partition of point progress of representation form of 3D object from geomet- cloud is shown in Figure 1. ric regular data to irregular one. For the conventional CNNs, Rotation-Equivariant Network for 3D Objects. Point- it is intractable to handle the geometric irregular data, such Net [11] solves the permutation invariance problem of point as meshes and point clouds. Thus, previous literatures strive cloud by a symmetric pooling operator, which remarkably to transform such data into voxel representations [9, 12, 20] reduces the N! cases (given a point cloud with N points) of or multi-images (views) [15, 20]. However, it is inevitable permutation into merely one case. However, rotation invari-

4995 ance is a more challenging problem needed to be solved, Definition 2 (RRI k-ary Operator). A rigorously rotation- since SO(3) is infinite. Previous works have attempted to invariant (RRI) k-ary operator is an operator G : upgrade the existing neural networks with the property of R3 × R3 ×···× R3 7→ Rn such that rotation-equivariance [7, 19]. For example, [16] designs k a special convolutional operation with local rotation in- | {z } variance and applies it as basic block to build a rotation- G(Rx1,Rx2,...,Rxk)= G(x1, x2,..., xk) , equivariant network. Besides, [2] proposes a method that holds for all x1, x2,..., x ∈ R3 and all rotation mapping transforms the 3D voxel data into spherical representation k R ∈ SO(3). and then employs a spherical convolution operator to extract rotation-equivariant features. However, it is unavoidable to Apparently, the vector norm k·k2 is a unary RRI operator suffer from loss of information as there is no bijection be- from R3 to R. At the same time, it can be shown that the in- tween R3 and S2. 2 ner product of two arbitrary points in S is rotation-invariant, because rotation transformation is orthonormal: 3. Approach T T hRx,Ryi =(Rx) (Ry)= x y = hx, yi , (2) 3.1. Rotation•Invariant Representation in R3 holds for ∀x, y ∈ R3. Thus inner product is an RRI bi- A point cloud with N data points is often expressed as a nary operator from R3 × R3 to R. Note that hx, yi = point set S = {(x ,y ,z ) | x ,y ,z ∈ R}N−1 in Cartesian i i i i i i i=0 kxk kyk cos θ holds when x, y ∈ R3, the formulas coordinate system. In another experssion, it can be repre- 2 2 xy (1,2) imply that the relative angle θ between any two sented as S ∈ RN×3 in a matrix form. In terms of the point xy points x, y ∈ S is a rotation-invariant quantity. cloud in RN×3, rotation transformation is a linear mapping Similarly, it can be proved that for any point p ∈ S\{0}, in correspondence with a 3 × 3 real orthogonal matrix. if T is an orthogonal projection operator of R3 onto a In order to precisely describe the rigorous rotation invari- p plane L past the origin and p is orthogonal to L, then the ance, we conduct a definition as below. inner product of two arbitrary points in Tp(S) is rotation- Definition 1 (RRI Mapping). If N,D ∈ N+, a rigor- invariant. The proof is given as below: ously rotation-invariant (RRI) mapping is a set mapping hT (Rx), T (Ry)i F : RN×3 7→ RN×D such that Rp Rp T T T = Rx − ((Rx) Rn) · Rn Ry − ((Ry) Rn) · Rn F(S)= F(R(S)) T T T = x − x n · n y − y n · n RN×3 holds for all point set S ∈ and all rotation mapping = hTp(x), Tp(y)i , R ∈ SO(3). Then F(S) is called as a rigorously rotation- (3) invariant representation of S. R3 p where x, y ∈ , R ∈ SO(3) and n = kpk . Hence the composite operator G (x, y, p)= hT (x), T (y)i is an The definition introduces an RRI mapping that not only 1 p p RRI ternary operator. maintains rotation invariance but also rigorously preserves Furthermore, according to the property of cross product, the cardinality of output point set as same as the cardinality it can be shown that the composite operator G (x, y, p) = of the input one, i.e., the input set with N points should be 2 hT (x) ×T (y), pi is also an RRI ternary operator. The mapped to output set with N features. p p R3 N−1 proof is as below. For example, given a point set S = {pi | pi ∈ }i=0 , it is obvious that kpik2 is rotation-invariant since the rota- hTRp(Rx) ×TRp(Ry),Rpi tion invariance of 2-norm: = hRTp(x) × RTp(y),Rpi 2 2 3 −1 T kRxk = kxk , ∀x ∈ R . (1) = h(det R)(R ) Tp(x) ×Tp(y) ,Rpi (4) 2 2 = hR Tp(x) ×Tp(y) ,Rpi Hence the row-wise vector norm k·k2 can be defined as an RRI mapping from RN×3 to RN×1. = hTp(x) ×Tp(y), pi. Since rotation transformation has the property of pre- Consequently, four rotation-invariant operators have serving the relative positional relationships among several been found in the previous discussion, and we can make points, a definition is conducted to describe the rotation- use of them to construct a rotation-invariant representation invariance of k-ary operator as follow. and the construction method is just an RRI mapping. 2Two-dimensional sphere, denoted as S2, is the surface of a completely In order to introduce the proposed representation, we round ball in R3. need to construct a K-nearest neighbor (K-NN) graph G =

4996 $" Theorem 1. The mapping defined by (6) is a rigorously $"# %" rotation-invariant mapping and the representation (5) is rigorously rotation-invariant. !"# %"# !"& $"& Proof. Firstly, the computation method in (6) exactly de- '"& N×3 N×(3K+1) %"& fines a set of mappings F : R 7→ R . K '"# Note that the -NN neighborhood of arbitrary point x ∈ S is uniquely determined by kx − yk2 with respect to all point y ∈ S, which is proved rotation-invariant by (1), so searching K-NN of point x ∈ S is a rotation-invariant operation. Besides, since rotation transformation has no influ- ence to the permutation of point cloud, we can obtain consistent order of K-NN by stable sort algorithm that main- Figure 2: The diagram illustrates each elements in RRI rep- tains the relative order of points with equal distance. resentation (cf. formula (5)) by a trivial case in which we builds 2-NN graph on three points. As the result of (1,2), it is obvious that ri, rik and θik are rotation-invariant. On the basis of formulas (3,4), sin ψj (S, E) on point set S, where E = {(x, y) ∈ S × S | y is and cos ψj are both of rotation-invariance, hence ψj and φik one of the K-NN of x}. are also rotation-invariant. According to the K-NN graph, we can employ the RRI Therefore formulas (6) define an RRI mapping and the operators to capture the relative positional patterns underly- representation defined by (5) is an RRI representation. ing the K-NN neighborhood of each point in S, and benefit from the property of rotation invariance at the same time. Specifically, given a K-NN graph G on point set S, the However, RRI mapping probably loses some essential proposed representation of each point p ∈ S is i information from the original data because the pursuit of

(ri, (ri1,θi1,φi1), (ri2,θi2,φi2),..., (riK ,θiK ,φiK )) , rigorous rotation invariance may result in lower capacity of (5) the RRI representation. For example, the 2-norm k·k2 is where indeed an RRI mapping as the discussion of formula (1), whereas it only captures the distance information of the r = kp k , i i 2 points in S and totally discards the relative positional pat- rik = kpikk2 (pik is one of the K-NN of pi with id k) , tern of them. pi pik θik = arccos h , i , It is remarkable to point out that the proposed represen- ri rik tation not only satisﬁes the property of rigorous rotation φik = ψj∗ , min{ψj | 1 ≤ j ≤ K, j 6= k,ψj ≥ 0} , invariance but also preserves necessary information which

ψj = atan2(sin ψj, cos ψj) , helps to reconstruct the original point cloud on a weak condition as stated in the following theorem. Tpi (pik) Tpi (pij) pi sin ψj = h × , i , kTpi (pik)k2 kTpi (pij)k2 ri Theorem 2. Given a K-NN graph G = (S, E) on point Tpi (pik) Tpi (pij) cos ψj = h , i . set S, if G is a strongly connected graph, then for ∀R ∈ kTpi (pik)k2 kTpi (pij)k2 (6) SO(3), given Cartesian coordinates of a nonzero point and one of its non-collinear K-NN neighbor, the Cartesian co- Note that for a given point pi and one of its k-nearest ordinates of R(S) can be determined by the RRI represen- neighbor pik, if we apply pi as normal vector, then ψj rep- tation deﬁned by (5). resents the relative angle between Tpi (pik) and Tpi (pij) according to the right-hand rule, thereupon φik is the relative angle between Tpi (pik) and its rotation-nearest point Proof. Given the Cartesian coordinates of arbitrary point ∗ 0 Tpi (pij ) in anti-clockwise direction. The representation pi ∈ R(S)\{ } and one of its non-collinear K-NN neigh- (5) has intuitive geometric meaning which is illustrated in bor pik, we can obtain the 2-norm of them and their relative Figure 2. The function atan2(·, ·) in formula (6) is a special positional information, such as θik and φik. With the rep- arctan(·) choosing the quadrant correctly. resentation (5), we will show that the coordinate of another On the foundation of the four RRI operators, the pro- K-NN neighbor pij∗ , which is the rotation-nearest point posed representation in formula (5) is rigorously rotation- of pik in anti-clockwise direction after applying orthogonal invariant as the claim of the following theorem. projection Tpi , can be uniquely determined by the following

4997 equation system, 3.2.1 RRI Representation Processing

2 hpij∗ , pij∗ i = rij∗ We can reformulate the proposed representation (5) of each point p ∈ S as hp , p ∗ i = r r ∗ cos θ ∗ i i ij i ij ij (7) p , p ∗ t t ∗ ψ ∗ hTpi ( ik) Tpi ( ij )i = ik ij cos j ((ri,ri1,θi1,φi1), (ri,ri2,θi2,φi2),..., (ri,riK ,θiK ,φiK )). hT (p ) ×T (p ∗ ), p i = r t t ∗ sin ψ ∗ , pi ik pi ij i i ik ij j Ti1 Ti2 TiK | {z } | {z } | {z (10)} where ψ ∗ = φ ,t = kT (p )k . The quantities in the j ik ik pi ik 2 In other words, we summarize the RRI information between right hand side of the equation system (7) are all known in point p and its K nearest neighbors as (T ,T ,...,T ) the representation (5). In other words, the unique unknown i i1 i2 iK to characterize point pi. Hence the new representation of a variable is p ∗ . On the foundation of the rotation invariance ij point cloud S ∈ RN×3 is a tensor T ∈ RN×K×4. Since revealed from Theorem 1, it is apparent that there exists at the local neighborhood pattern of p is probably embed- least one solution for the equation system (7) because it has i ded in its K nearest neighbors, the proposed representation a solution for the original point cloud S. takes advantages of the property and captures the local pat- Suppose that the solution set of the equation system (7) tern in the K-NN neighborhood by an RRI and conditional contains at least two different solutions p ∗ and p ∗ , then ij ij information-lossless mechanism. the equation system (7) would imply that e Since the RRI representation of point pi can be regarded hpi , pij∗ − pij∗ i = 0 as a mini point cloud (Ti1,Ti2,...,TiK ), and PointNet is a

∗ ∗ universal continuous set function approximator, we can ap- hTpi (pik) , pij − peij i = 0 (8) ply PointNet as a basic block to learn a representation of the hT (p ) × (p ∗ − p ∗ )e, p i = 0 pi ik ij ij i mini point cloud and extract local features to characterize

∗ e ∗ Since both Tpi (pik) and pij − pij are in the plane L = the K-NN neighborhood. In other words, we can transform R3 ∗ ∗ the RRI representation, an N × K × 4 tensor, into a N × D {x ∈ | x ⊥ pi}, Tpi (pik) × (epij − pij )= αpi holds R for some α ∈ \{0}. However, it woulde imply that tensor of neighborhood features by means of PointNet as the following formula. ∗ ∗ hTpi (pik) × (pij − pij ) , pii = αhpi , pii = 0 . (9) ′ e pi = max fΘ(Tik) , (11) Since α 6= 0 and pi 6= 0, the equation (9) causes a con- 1≤k≤K tradiction. Thus, the solution set of (7) contains a unique solution, i.e., given points pi and pik, the Cartesian coor- where fΘ(·) is a multi-layer perceptron network with pa- dinate of pij∗ can be uniquely determined by the equation rameters Θ shared with all output features. In other words, ′ system (7). we extract a feature pi corresponding to the original point Similarly, we can solve the coordinate of the next neigh- pi ∈ S. bor which is rotation-nearest from pij∗ in anti-clockwise In the view of DGCNN, the formula (11) is a special case of the EdgeConv. However, we utilize an RRI repre- direction after applying orthogonal projection Tpi . The process is repeated until all the K-NN neighbors of pi are resentation to describe the relationship between a point and constructed intactly. its K-NN neighbors while DGCNN only uses the differ- Since the graph G = (S, E) is strongly connected, two ence vector pi − pik concatenated with pi, both of them arbitrary points a, b ∈ S are connected by at least one path vary with rotation transformation. (x0, x1,..., xn) between them, where n is the path length, and x0 = a, xn = b and (xi, xi+1) ∈E holds for 0 ≤ i ≤ 3.2.2 Hierarchical Clustering Tree n − 1. Therefore, starting from the K-NN neighborhood Since point could embeds in low dimensional space R3 of pi, we can restore the coordinate of arbitrary point in S step-by-step along a path with ﬁnite length. equipped with Euclidean metric, hierarchical clustering is an appropriate method to analyze the hierarchical structure of point cloud. With the support of hierarchical clustering, we can learn a hierarchical clustering tree which illustrates 3.2. Hierarchical Clustering based ClusterNet the arrangement of partition and the relationships between We propose a hierarchical clustering based neural net- different clusters. work, called ClusterNet, to learn a hierarchical structure Speciﬁcally, we employ the agglomerative hierarchical tree for the instruction of hierarchical feature representation clustering with ward-linkage criteria to learn the hierarchi- of point clouds. With the assistance of unsupervised learn- cal structures of point cloud. The ward-linkage criteria ing, we can explore and exploit distribution information of minimizes the total within-cluster variance, which tends to point cloud with regard to the hierarchical structure tree. partition the point cloud into several clusters with similar

4998 { by iiae olcldfuino iia etrsi h feature the in features space. similar of diffusion nonlocal cilitates E⊆ rp ic h rp safce yasmlrt arxof matrix similarity dynamic a The by layer. previous affected from is features graph the since graph h oe takes model The ut-ae ecpin(l)wt h ubro ae ern endas defined neurons layer of number the with (mlp) perception multi-layer point. each of embedding local into K xrcsheacia lse etrsuigtrecutrasrcinmdls n vnulyotisagoa etr fthe of feature for global scores a classification obtains generate eventually to and used modules, is abstraction which cluster cloud, three point using whole features cluster hierarchical extracts R apn ooti h R ersnaino on lu n hnageaetepitfaue in features point the aggregate then and cloud point of representation RRI the obtain to mapping RRI onsaecniee ssmldpit n h etr of feature its the from aggregated and is points point sampled each [18], as all DGCNN considered i.e., are sampling, by dense points proposed by PointNet++ first the improves is which layer EdgeConv The Feature Cluster for EdgeConv 3.2.3 way. aggregate efficient and more extract a tree to in how clustering features network hierarchical neural the the In instruct apply to can re- tree. we cluster clustering words, the hierarchical other to the according in revealed cluster higher- lationships larger a of into aggregated feature further level and are features clusters local small the then from structures geometric fine-grained concerning function objective variance. minimum optimal the ward’s an to respect make with will partition method clustering clustering hierarchical hierarchical in tree, merge-level particular a aggregation On feature sizes. for guidance a as structure hierarchical leverages it Then cluster. each features. within super-cluster obtain to aggregation iue3: Figure x naetnihosaedtrie yadynamic a by determined are neighbors -nearest 1 pcfial,gvna given Specifically, iia oCN,Cutre erstelclfaue of features local the learns ClusterNet CNNs, to Similar P , Point&Cloud x × 2 , . . . , P N x 3 N x 3 oe architecture Model n hnteotu fEgCn a eobtained be can EdgeConv of output the then and RRI mapping x x RRI i ′ N (64) max = ⊂ } N j :( onsa nu,apisteRImdl oetatrgrul oainivratfaue o ahpoint, each for features rotation-invariant rigorously extract to module RRI the applies input, as points i,k R F RRI )

∈E N x 3 ecncntuta construct can we , N x K x 4 F f (64) dmninlpitset point -dimensional (64,64,128)(32) Θ mlp mlp mlp tcnit fa R oue he lse btato oue n h atcasﬁrmodule. classiﬁer last the and modules abstraction cluster three module, RRI an of consists it : Abstraction( ( x Cluster( i (64) (64) (64) K ... shared − lse btato module Abstraction Cluster naetnihos The neighbors. -nearest x k ,

x N x K x 64 K i Pool Max ) N rp fa- graph -NN . K 32 x 128 N graph -NN N x 64 K (256,256)(8) Abstraction( P Cluster( -NN (12)

= M x D mlp(% 4999 KH h oueetat defaue fec u-lse using sub-cluster each of features edge extracts module the : ihrlvlfauet hrceietecluster the characterize to feature higher-level ihDCNwihetat etrsfralpit nthe in points all cloud. for point features compared original extracts GPU which of DGCNN usage compu- memory with total the the and reduces expense aggregation tational Besides, of of data. corruptions aggregation terms mild input against proposed In robustness improve the can downsampling, signal. method of maximum the property in- maintain the the downsample and both data they put since CNNs, in methods ing root the from obtained tree. be the can of cloud node point fea- whole global the the ﬁnally of and ture top hierarchical to the bottom from hierarchical learn tree the clustering to along passing feasible cluster is each cluster- of it representation hierarchical Therefore, the in tree. super-cluster ing sub-clusters a how into records merged which features are index sub-cluster cluster the to to function according pooling we particular, max In apply super-cluster. for can a of method that into aggregation sub- features aggregate cluster to novel relationship the a utilizes which propose cloud, point we relationship contains clusters, tree of clustering hierarchical the Since Cluster within Aggregation 3.2.4 pnigto sponding R in ec,i eapyEgCn to EdgeConv apply we if Hence, tion. of NN 1 , D D % h rpsdageainmto ssmlrt h pool- the to similar is method aggregation proposed The ifrn rmDCN h nu fEgCn saset a is EdgeConv of input the DGCNN, from Different where , graph 2 8 x 256 dmninlcutrfeatures cluster -dimensional ,..., EdgeConv c categories. Cluster(Abstraction % (512,1024)(1) ( ( Abstraction( { ) % n a 1 Cluster( K block , 1 stenme fcutr napriua parti- particular a in clusters of number the is % a ,

naetcutr of clusters -nearest M x K x 2 ,..., 2 Pool Max a , . . . , R module RRI % global ( )(C) M x n } 1024 features n hnapisneighborhood applies then and within Max mlp((512,256,c) h R oueemploys module RRI the : (C) pooling( cluster c C i ilb grgtda a as aggregated be will = K C h etrscorre- features the , { N neighborhood -NN c 1 output classification ,

c C x 2 c , . . . , i . c scores c n ∈ } Method Input (size) z/z z/SO(3) SO(3)/SO(3) SO(3)/SO(3)∗ PointNet (without STN) [11] pc (1024 × 3) 88.5 14.4 70.5 72.5 PointNet++ (MSG without STN) [13] pc+normal (5000 × 6) 91.9 16.0 74.7 78.5 SO-Net (without STN) [8] pc+normal (5000 × 6) 93.4 19.6 78.1 81.4 DGCNN (without STN) [18] pc (1024 × 3) 91.2 16.2 75.3 76.4 PointNet [11] pc (1024 × 3) 89.2 16.4 75.5 76.4 PointNet++ (MSG) [13] pc+normal (5000 × 6) 91.8 18.4 77.4 79.3 SO-Net [8] pc+normal (5000 × 6) 92.6 21.1 80.2 81.9 DGCNN [18] pc (1024 × 3) 92.2 20.6 81.1 82.0 Spherical CNN [2] voxel (2 × 642) 88.9 76.9 86.9 86.9 Ours pc (1024 × 3) 87.1 87.1 87.1 87.1

Table 1: Comparison of Rotation Robustness on rotation-augmented benchmark.

3.2.5 Permutation Invariance of ClusterNet solve the rotation matrix R corresponding to the Euler axis e and the rotation angle θ, Trivial K-nearest neighbor searching is not permutation- invariant, since the K-nearest neighbors will become un- T R = I3 cos θ + (1 − cos θ)ee +[e]× sin θ, stable when there exists some neighbors with exactly the same 2-norm. In such degeneration case, the result of K- 0 −e3 e2 (13) ,   NN searching is inevitably affected by the order of input [e]× e3 0 −e1 . points. However, we can modify the method of K-NN −e2 e1 0  searching to avoid such degeneration. Specifically, if pk As [4] stated, Fibonacci lattice is a mathematical idealiza- is the k-th nearest neighbor of p, then we consider the set tion of natural patterns with optimal packing, where the of k-nearest neighbors of p as {q ∈ S | kqk2 ≤ kpkk2}. area represented by each point is almost identical. Owing On the foundation of the modified K-NN searching and the to the favorable property, we sample the Fibonacci lattice permutation-symmetric aggregation, it is obvious that the (points) from unit sphere surface as Euler axes and then proposed ClusterNet is of permutation invariance. uniformly sample the rotation angle in the space [0, 2π). 4. Experiments We choose such sampling method to generate Euler axes and rotation angles, and then solve the rotation matrix by In this section, we propose a novel benchmark to evalu- the formulas (13). Consequently, we obtained a rotaion- ate the rotation robustness, on which we compare the pro- sampling method that can sample rotation transformations posed method with the state-of-the-art methods to empir- from SO(3) uniformly. ically validate the effectiveness of the RRI representation In terms of dataset, we choose ModelNet40 [20], a and ClusterNet. Furthermore, we conduct an experiment to widely-used 3D object classification dataset, as our basic validate the superiority of ClusterNet over other architec- dataset. ModelNet40 dataset consists of 12,311 CAD mod- tures to learn deep hierarchical embeddings from the RRI els from 40 manmade object categories, in which 9,843 is representation. Finally, we evaluate the effect of the unique used for training and 2,468 is used for testing. Since each hyperparameter K in the RRI representation if we construct CAD model in ModelNet40 is composed of many mesh a K-nearest neighbor (K-NN) graph on point cloud. faces, we sample point cloud from them uniformly with respect to face area and then shift and normalize each point 4.1. Benchmarks cloud into [−1, 1]3 with centroid on the origin. We employ We design a new benchmark to fairly evaluate the rota- the sampling method to generate 500 Euler axes and 60 ro- tion robustness of a model. Since the majority of objects tation angles for each Euler axes, i.e., 30,000 rotation trans- in the original dataset are in a fixed postures, we are re- formations are sampled uniformly from SO(3) to augment quired to conduct rotation augmentation to enrich the test the test set. As a result, we obtain a rotation-augmented test set. Considering SO(3) is infinite, it is infeasible to cover set with 74,040,000 point clouds in total as the benchmark all the postures thoroughly, so we uniformly sample a rea- dataset. We employ the augmented test set to evaluate the sonable amount of rotation transformations from SO(3). rotation robustness of each model. According to Euler’s rotation theorem [3], any rotation 4.2. Comparison of Rotation Robustness can be represented by a Euler axis and a rotation angle. The Euler axis is a three-dimensional unit vector and the rotation We compare the proposed method with the state-of-the- angle is a scalar. We can employ the following formulas to art approaches on the benchmark for rotation-robustness

5000 Method Accuracy (%) Time (h) test set. RRI+PointNet 85.9 8.5 RRI+DGCNN 86.4 12 4.3. Ablation Analysis RRI+ClusterNet (8, 1) 86.6 9 4.3.1 Analysis of Architecture Design RRI+ClusterNet (32, 1) 86.8 10.5 RRI+ClusterNet (32, 8, 1) 87.1 9.5 Since the proposed RRI representation can be processed to be compatible with many architectures dealing with Table 2: Analysis of Architecture Design point cloud data, we enhance PointNet and DGCNN with the RRI representation, and Table 2 shows that Cluster- Net outperforms both the enhanced PointNet and the en- evaluation. The results are summarized in Table 1 with hanced DGCNN by a large margin on the foundation of the four comparison modes: (1) both training set and test set same RRI representation. As is illustrated in Section 3.2, are augmented by azimuthal rotation (z/z); (2) training DGCNN is a special case of ClusterNet without cluster ag- with azimuthal rotation and testing with arbitrary rotation gregation, thus Table 2 shows that the aggregation within (z/SO(3)); (3) both training and testing with arbitrary ro- cluster along hierarchy indeed facilitates the hierarchical tation (SO(3)/SO(3)); (4) conditions are almost as same as features learning and then extracts more discriminative fea- (3), but test the model with multi-rotation voting strategy tures for 3D recognization. (SO(3)/SO(3)∗). In order to make the comparison more comprehensive, we make use of the following methods to improve the rotation-robustness of existing methods. 4.3.2 Effectiveness of K in RRI Representation Rotation-augmentation is applied to the training set using two sampling strategies respectively. The first strategy K 40 50 60 70 80 90 only samples azimuthal rotations for augmentation, i.e., we Acc. (%) 85.6 86.4 86.8 87.0 87.1 87.1 merely use z-axis as Euler axis. While the second one samples all rotations from SO(3). In a particular epoch, we ro- Table 3: Effectiveness of K in RRI Representation tate each object using the sampled rotation transformation so that the model might improve rotation robustness from In terms of the proposed RRI representation, K is the the objects under different orientations. We can use multi- unique hyperparameter, which controls the connectivity of rotation voting strategy to boost the robustness of model. the graph G, thus we analyze the effectiveness of different Specially, we feed the model with test set in several orienta- K in RRI representation. As shown in Table 3, the archi- tion and then sum up the confidence scores as a total one to tecture of ClusterNet is robust to diverse values of K even determine the classification result. Variants of spatial trans- when K is too small to satisfy the connectivity condition in former network [5] are used to alleviate the problem caused Theorem 2. For example, when K = 40, there exists nearly by rotation transformation. For example, both PointNet and 25% of the point clouds not satisfying strongly connected DGCNN employ spatial transformation module to learn a condition, however, it still achieves comparable classification accuracy. When K is gradually increased to over 70, 3 × 3 rotation matrix which transforms point cloud into the canonical space. accuracies of the model remain stable. Table 1 consists of four groups of methods. The first 5. Conclusion group from the top of the table consists of four models without using spatial transformer network (STN), while the In this paper, we step forward to enhance the rotation methods in the second group are equipped with STN. In robustness of 3D object recognization model. Specifically, the third group, we choose a representative method based we introduce a novel RRI representation to assign a unique on rotation-equivariant network, spherical CNN[2], to com- and consistent data form for any identical object in infinite pare with our proposed method. As shown in Table 1, the attitude. We theoretically and empirically demonstrate that widely used augmentation using azimuthal rotations suf- the representation is rigorously rotation-invariant and con- fers from a sharp decline on the rotation-augmented test ditional information-lossless. Besides, our representation set. Furthermore, it illustrates that rotation-augmentation is complementary with prevailing 3D recognition architec- and STN can improve the rotation robustness of models but ture and improves their rotation robustness. Finally, we fur- still have a large margin with our proposed method without ther design a deep hierarchical network called ClusterNet the demand of any data augmentation. Although the spheri- to better adapt to RRI representation. Extensive experimen- cal CNN is rotation-equivalent, it is also dependent with ro- tal evaluation on augmented test split set from widely-used tation augmentation and its performance is sensitive to the 3D classification benchmark demonstrates the superiority of strategy of augmentation. Besides, our proposed method our novel RRI representation as well as the elaborate Clus- also outperforms spherical CNN on the rotation-augmented terNet.

5001 References for 3d shape recognition. In Proceedings of the IEEE international conference on computer vision, pages 945–953, [1] Etienne Barnard and David Casasent. Invariance and neural 2015. nets. IEEE Transactions on Neural Networks, 2(5):498–508, [16] Nathaniel Thomas, Tess Smidt, Steven Kearnes, Lusann 1991. Yang, Li Li, Kai Kohlhoff, and Patrick Riley. Tensor field [2] Carlos Esteves, Christine Allen-Blanchette, Ameesh Maka- networks: Rotation-and translation-equivariant neural net- dia, and Kostas Daniilidis. Learning so (3) equivariant repre- works for 3d point clouds. arXiv preprint arXiv:1802.08219, sentations with spherical cnns. In Proceedings of the Euro- 2018. pean Conference on Computer Vision (ECCV), pages 52–68, [17] Peng-Shuai Wang, Yang Liu, Yu-Xiao Guo, Chun-Yu Sun, 2018. and Xin Tong. O-cnn: Octree-based convolutional neu- [3] L Euler. General formulas for the translation of arbitrary ral networks for 3d shape analysis. ACM Transactions on rigid bodies. Novi Commentarii academiae scientiarum Graphics (TOG), 36(4):72, 2017. Petropolitanae, 20(1776):189–207, 1790. [18] Yue Wang, Yongbin Sun, Ziwei Liu, Sanjay E Sarma, [4] Alvaro´ Gonzalez.´ Measurement of areas on a sphere using Michael M Bronstein, and Justin M Solomon. Dynamic fibonacci and latitude–longitude lattices. Mathematical Geo- graph cnn for learning on point clouds. arXiv preprint sciences, 42(1):49, 2010. arXiv:1801.07829, 2018. [5] Max Jaderberg, Karen Simonyan, Andrew Zisserman, et al. [19] Maurice Weiler, Mario Geiger, Max Welling, Wouter Spatial transformer networks. In Advances in neural infor- Boomsma, and Taco Cohen. 3d steerable cnns: Learn- mation processing systems, pages 2017–2025, 2015. ing rotationally equivariant features in volumetric data. In [6] Roman Klokov and Victor Lempitsky. Escape from cells: Advances in Neural Information Processing Systems, pages Deep kd-networks for the recognition of 3d point cloud mod- 10402–10413, 2018. els. In Computer Vision (ICCV), 2017 IEEE International [20] Zhirong Wu, Shuran Song, Aditya Khosla, Fisher Yu, Lin- Conference on, pages 863–872. IEEE, 2017. guang Zhang, Xiaoou Tang, and Jianxiong Xiao. 3d [7] Risi Kondor, Zhen Lin, and Shubhendu Trivedi. Cleb- shapenets: A deep representation for volumetric shapes. In sch–gordan nets: a fully fourier space spherical convo- Proceedings of the IEEE conference on computer vision and lutional neural network. In S. Bengio, H. Wallach, H. pattern recognition, pages 1912–1920, 2015. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, editors, Advances in Neural Information Processing Systems 31, pages 10117–10126. Curran Associates, Inc., 2018. [8] Jiaxin Li, Ben M Chen, and Gim Hee Lee. So-net: Self- organizing network for point cloud analysis. In Proceed- ings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 9397–9406, 2018. [9] Daniel Maturana and Sebastian Scherer. Voxnet: A 3d convolutional neural network for real-time object recognition. In Intelligent Robots and Systems (IROS), 2015 IEEE/RSJ International Conference on, pages 922–928. IEEE, 2015. [10] Daniel Mullner.¨ Modern hierarchical, agglomerative clustering algorithms. arXiv preprint arXiv:1109.2378, 2011. [11] Charles R Qi, Hao Su, Kaichun Mo, and Leonidas J Guibas. Pointnet: Deep learning on point sets for 3d classification and segmentation. Proc. Computer Vision and Pattern Recognition (CVPR), IEEE, 1(2):4, 2017. [12] Charles R Qi, Hao Su, Matthias Nießner, Angela Dai, Mengyuan Yan, and Leonidas J Guibas. Volumetric and multi-view cnns for object classification on 3d data. In Pro- ceedings of the IEEE conference on computer vision and pattern recognition, pages 5648–5656, 2016. [13] Charles Ruizhongtai Qi, Li Yi, Hao Su, and Leonidas J Guibas. Pointnet++: Deep hierarchical feature learning on point sets in a metric space. In Advances in Neural Informa- tion Processing Systems, pages 5099–5108, 2017. [14] Gernot Riegler, Ali Osman Ulusoy, and Andreas Geiger. Octnet: Learning deep 3d representations at high resolutions. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, volume 3, 2017. [15] Hang Su, Subhransu Maji, Evangelos Kalogerakis, and Erik Learned-Miller. Multi-view convolutional neural networks

5002