RPM-Net: Recurrent Prediction of Motion and Parts from Point Cloud

ACM Transactions on Graphics (Proceedings of SIGGRAPH ASIA 2019)

Zihao Yan1†    Ruizhen Hu1†    Xingguang Yan1    Luanmin Chen1    Oliver van Kaick2    Hao Zhang   Hui Huang1*

1Shenzhen University    2Carleton University    3Simon Fraser University

(†Joint first authors)

Fig. 1. Given an unsegmented, possibly partial, point cloud shape, our deep recurrent neural network, RPM-Net, simultaneously hallucinates a motion sequence (via point-wise displacements) and infers a motion-based segmentation of the shape into, possibly multiple, moveable parts. RPM-Net predicts a non-trivial motion for the umbrella and multi-part motions for both the cabinet (drawer sliding and door rotating) and the office chair (seat moving up and wheels rotating). The umbrella and cabinet are synthetic scans while the office chair is a single-view scan acquired with a Kinect sensor. Input to RPM-Net was downsampled to 2,048 points.
Fig. 9. Motion prediction results on real scans, where the moving parts are shown with different colors and the reference part is colored gray.


We introduce RPM-Net, a deep learning-based approach which simultaneously infers movable parts and hallucinates their motions from a single, unsegmented, and possibly partial, 3D point cloud shape. RPM-Net is a novel Recurrent Neural Network (RNN), composed of an encoder-decoder pair with interleaved Long Short-Term Memory (LSTM) components, which together predict a temporal sequence of pointwise displacements for the input shape. At the same time, the displacements allow the network to learn moveable parts, resulting in a motion-based shape segmentation. Recursive applications of RPM-Net on the obtained parts can predict finer-level part motions, resulting in a hierarchical object segmentation. Furthermore, we develop a separate network to estimate part mobilities, e.g., per part motion parameters, from the segmented motion sequence. Both networks learn the deep predictive models from a training set that exemplifies a variety of mobilities for diverse objects. We show results of simultaneous motion and part predictions from synthetic and real scans of 3D objects exhibiting a variety of part mobilities, possibly involving multiple moveable parts.

Fig. 3. The architecture of the motion hallucination network RPM-Net. Given an input point cloud P, the network predicts displacement maps {D} along with the segmentation S of the point cloud, which together provide the final segmented motion sequence {Pt}. The network is composed of encoders (SA), decoders (FP), LSTM units, fully-connected layers (FC), and special operations denoted with the pink circles.

Fig. 5. Motion prediction results on shapes with a single moving part. We observe how our method can be applied to a variety of shapes with diverse mobilities, including both complete point clouds and partial scans. For each input cloud, we show the first four frames of the predicted motion, along with the predicted transformation axis drawn as a green line, and moving and reference parts colored red and gray, respectively. We observe how RPM-Net can predict the correct motion sequences for different inputs and estimate the corresponding part mobility parameters.

Fig. 6. Motion prediction results on shapes with multiple moving parts. (a) Input point cloud, (b) Motion hallucinated with our method (four frames), (c) Segmentation with predicted motion axis for each moving part.
Fig. 7. Non-trivial motion prediction results: motion hallucinated for two bows (both synthetic) and two balances (one synthetic and one a real scan), which cannot be described with a simple transformation.

Fig. 13. Visual comparison of our method to previous works (Shape2Motion (S2M) [Wang et al. 2019] and SGPN+BL [Wang et al. 2018]) on example shapes from our dataset.
Fig. 14. Visual comparison of our method to Shape2Motion (S2M) [Wang et al. 2019] on example shapes from their dataset.

Data & Code

Note that the DATA and CODE are free for Research and Education Use ONLY. 

Please cite our paper (add the bibtex below) if you use any part of our ALGORITHM, CODE, DATA or RESULTS in any publication.

Link: https://github.com/Salingo/RPM-Net


We thank the reviewers for their valuable comments. This work was supported in parts by National 973 Program (2015CB352501), NSFC (61872250, 61602311, 61861130365), GD Science and Technology Program (2015A030312015), LHTD (20170003), Shenzhen Innovation Program (JCYJ20170302153208613), NSERC (611370, 2015-05407), and the National Engineering Laboratory for Big Data System Computing Technology.

title = {
RPM-Net: Recurrent Prediction of Motion and Parts from Point Cloud},
author = {Zihao Yan and Ruizhen Hu and Xingguang Yan and Luanmin Chen and Oliver van Kaick and Hao Zhang and Hui Huang},
journal = {ACM Transactions on Graphics (Proceedings of SIGGRAPH ASIA 2019)},
volume = {38},
number = {6},
pages = {240:1--240:15},  
year = {2019},

Downloads (faster for people in China)

Downloads (faster for people in other places)