1 Non-Rigid Point Set Registration Networks Lingjing Wang, Jianchun Chen, Xiang Li and Yi Fang Abstract—Point set registration is defined as a process to (e.g. Euclidean distance loss) between the transformed source determine the spatial transformation from the source point set point set and the target one. Previous efforts [5, 7, 15] have to the target one. Existing methods often iteratively search for achieved great success in point set registration through the de- the optimal geometric transformation to register a given pair of point sets, driven by minimizing a predefined alignment loss velopment of a variety of optimization algorithms and distance function. In contrast, the proposed point registration neural metrics as summarized in [15]. However these methods are network (PR-Net) actively learns the registration pattern as a often not designed to handle the real-time point set registration parametric function from a training dataset, consequently predict or to deal with a large volume dataset. This limitation is the desired geometric transformation to align a pair of point mainly contributed by the fact that, for each given pair of point sets. PR-Net can transfer the learned knowledge (i.e. registration pattern) from registering training pairs to testing ones without sets, the iterative method needs to start over a new iterative additional iterative optimization. Specifically, in this paper, we optimization process even for the trivial similar cases. This develop novel techniques to learn shape descriptors from point observation suggests that the existing efforts are mainly con- sets that help formulate a clear correlation between source and centrated on the stand-alone development of the optimization target point sets. With the defined correlation, PR-Net tends strategies rather than the techniques to smartly transferring the to predict the transformation so that the source and target point sets can be statistically aligned, which in turn leads to registration pattern acquired from aligning one pair to another. an optimal spatial geometric registration. PR-Net achieves robust This triggers the motivation to develop our proposed PR-Net and superior performance for non-rigid registration of point sets, with the hope to actively learn the registration pattern from even in presence of Gaussian noise, outliers, and missing points, a set of training data, consequently, to adaptively utilize that but requires much less time for registering large number of pairs. knowledge to directly predict the geometric transformation for More importantly, for a new pair of point sets, PR-Net is able to directly predict the desired transformation using the learned a new pair of unseen point sets. As a result, PR-Net is capable model without repetitive iterative optimization routine. Our code of handling the real-time point set registration or a large is available at https://github.com/Lingjing324/PR-Net. volume datasets with a similar pattern. To better understand the point set registrations, we briefly review related works as follows. I. INTRODUCTION A. Background B. Related Works Over past decades, point set matching and registration is one of the most important computer vision tasks [1]–[8, 28], Iterative registration methods. Current mainstream point set serving a widespread applications such as stereo matching, registration methods focus on the development of optimiza- medical image registration, large-scale 3D reconstruction, 3D tion algorithms to estimate the rigid or non-rigid geometric point cloud matching, semantic segmentation and so on [9]– transformations in an iterative routine. With the assumption [14]. The point set registration is mathematically defined as that a pair of point sets are related by a rigid transformation, a process to determine the spatial geometric transformations a registration approach is to estimate the best translation (i.e. rigid and non-rigid transformation) that can optimally and rotation parameters in the iterative search routine aiming register the source point set to the target one. The desired to minimize a distance metric between two sets of points. arXiv:1904.01428v1 [cs.GR] 2 Apr 2019 registration algorithm can find both rigid (i.e. rotation, reflec- One of the most popular methods for rigid registration, the tion, and shifting) and non-rigid (i.e. dilation and stretching) Iterative Closest Point (ICP) algorithm [11], was proposed to transformations, as well as being robust to outliers, Gaussian handle point set registration with least-squares estimation of point drift, data incompleteness and so on. transformation parameters. ICP starts with an initial estimation To formulate the problem of point set registration, existing of rigid transformation, followed by iteratively refining the methods [5, 7] often iteratively search the optimal geometric transformation by alternately choosing corresponding points transformation to register two sets of points, driven by min- from the point sets as estimate transformation parameters. The imizing a predefined alignment loss function. The alignment ICP algorithm is reported to be vulnerable to the selection of loss is usually pre-defined as a certain type of distance metric corresponding points for initial transformation estimation, and also incapable of dealing with non-rigid transformation. L.Wang is with MMVC Lab, the Department of Mathematics, To accommodate the deformation (e.g. morphing, articula- New York University, New York, NY, 30332 USA e-mail: tion) between a pair of point sets, many efforts were spent in [email protected]. J.Chen is with the MMVC Lab, New York University, New York, NY, 30332 USA e-mail: [email protected]. the development of algorithms to address the challenges of a X.Li is with the MMVC Lab, New York University, New York, NY, 30332 non-rigid transformation. Chui and Rangarajan [16] proposed USA e-mail: [email protected]. Y.Fang is with MMVC Lab, Dept. a robust method to model non-rigid transformation named of ECE, NYU Abu Dhabi, UAE. Dept. of ECE, NYU Tandon School of Engineering, USA. USA e-mail: [email protected]. as thin-plate spline [17]. They proposed TPS-RSM algorithm Corresponding author. Email: [email protected] with penalization on second order derivatives to optimize 2 the parameters of the desired transformation. Ma et al. [18] C. Our Solution: Point Set Registration Neural Network (PR- introduced a L2E estimator for non-rigid registration for Net) handling significant scale changes and rotations. In addition, Different from image data with a regular grid, point cloud Myronenko et al. [28] proposed non-parametric coherence data is often recorded in an irregular and disordered format. point drift (CPD) algorithm which leverages Gaussian mixture Learning the point set registration requires the deep neural likelihood and penalizes derivatives of all orders of the velocity networks to be applicable to irregular point cloud data. In field to enforce velocity coherence so that centroids of source addition, unlike the image containing rich texture and color point set move coherently to target point set. They reported information, the point cloud is solely represented with geo- that their algorithm can be easily extended to N-dimensional metric information (i.e. coordinates, curvature, normal). This space compared to TPS-RSM algorithm. Ma et al. [7] proposed suggests that a learning-based solution for point set registration a non-parametric vector field consensus algorithm to establish needs to address two main technical challenges: 1) robust the robust correspondence between two sets of points. Their learning of both local and global geometric feature from experimental result demonstrated that the proposed method point clouds and 2) robust learning of the transformation from is quite robust to outliers. In [1], the authors emphasized the well-defined correlation measure between pairwise geometric importance to preserve local and global structures for non-rigid feature sets. Therefore, the proposed PR-Net investigates two point set registration. Wang et al. [19] proposed path following major research problems: 1) the design of the techniques for strategy for graph matching in order to improve the compu- point cloud learning by introducing a novel reference operator tation efficency. Zhou et al. [20] proposed a fast alternating to enable formulating the correlation measure on arbitrary- minimization algorithm for multi-image matching. Existing structured data, and 2) the development of learning paradigm methods have achieved great success for both rigid and non- for the geometric transformation learning from pairwise fea- rigid point set registration over past decades. However, they ture sets. are mainly concentrated on the stand-alone development of the optimization strategies for point set registration rather than Figure 1 illustrates the pipeline of the proposed PR- the techniques to learn the registration process as a pattern. In Net which is composed of three main components. The this paper, the deficiency of these current algorithms drives first component is “learning shape descriptor tensor”. us to develop a learning-based registration paradigm that is In this component, the proposed grid-reference structure able to actively learn the knowledge about how to register two is developed to enable feature learning and formulate point sets, consequently, to adaptively utilize those knowledge the correlation relationship on arbitrary-structured data. to directly predict the geometric transformation without the The second component