SAGNet: Structure-aware Generative Network for 3D-Shape Modeling

ACM Transactions on Graphics (Proceedings of SIGGRAPH 2019)


Zhijie Wu1            Xiang Wang1            Di Lin1            Dani Lischinski2            Daniel Cohen-Or1,3            Hui Huang1*

1Shenzhen University         2Hebrew University of Jerusalem      3Tel-Aviv University    





Fig. 1. Our generative model jointly analyzes the structure and geometry of shapes, encoding them into a single latent code. The highlighted triplets above demonstrate that, in this joint latent space, pairs of nearby points represent models that are close to each other in both geometry and structure, while stepping away from the pair introduces differences in either geometry, or structure, or both. Note that, all the shapes shown here are voxel-based, and the bounding boxes of their parts are hidden on purpose for a clearer visualization.


Abstract

We present SAGNet, a structure-aware generative model for 3D shapes. Given a set of segmented objects of a certain class, the geometry of their parts and the pairwise relationships between them (the structure) are jointly learned and embedded in a latent space by an autoencoder. The encoder intertwines the geometry and structure features into a single latent code, while the decoder disentangles the features and reconstructs the geometry and structure of the 3D model. Our autoencoder consists of two branches, one for the structure and one for the geometry. The key idea is that during the analysis, the two branches exchange information between them, thereby learning the dependencies between structure and geometry and encoding two augmented features, which are then fused into a single latent code. This explicit intertwining of information enables separately controlling the geometry and the structure of the generated models. We evaluate the performance of our method and conduct an ablation study. We explicitly show that encoding of shapes accounts for both similarities in structure and geometry. A variety of quality results generated by SAGNet are presented.



Fig. 2. Overview of SAGNet. Given 3D shapes as training data, the network has traditional 3D convolutional and fully-connected layers to extract visual features for shape parts. The network is equipped with GRU-based encoder and attention component, which jointly analyzes the geometry and structural information of shapes. All the information are provided for the 2-way VAE, which offers the generative power to the network. Our network eventually decodes the geometry and structural information to generate 3D shapes.


Fig. 6. The generation results of G2L [Wang et al. 2018] (top row), GRASS [Li et al. 2017] (middle row) and SAGNet(bottom row). Compared to SAGNet, G2L often generates coarser and less-structured shapes while the voxel grids of GRASS may lose geometric details and thus are less visually appealing.


Fig. 7. For each generated sample in the top row, we retrieve the 3-nearest neighbors in the training data. It may be seen that the generated shapes are original.


Fig. 9. We measure the symmetry scores for legs of chairs generated by SAGNet, or baselines, or existing models. Along the horizontal axis, we set different thresholds for the scores. Along the vertical axis, we provide the percentage of shapes, which have smaller scores than the given thresholds.


Fig. 10. We measure the centroid-to-plane distances for each airplane generated by SAGNet, or baselines, or existing models. The centroid-to-plane distances are computed using the fore- and back-wings of airplanes. Along the horizontal axis, we provide different thresholds for the distances. Along the vertical axis, we provide the percentage of shapes, which have smaller distances than the given thresholds.


Fig. 11. Cavity analysis and comparison on the synthetic tenon-mortise joints. We present two generated joint shapes by SAGNet, G2L and GRASS models, respectively, in the right. In the left, along the horizontal axis, we provide different thresholds for the fitting accuracy R. Along the vertical axis, we provide the percentage of shapes, which have better fitting accuracy than the given thresholds.




Fig. 12. Visual comparison of geometry-to-structure mapping results.
Fig. 13. Visual comparison of structure-to-geometry mapping results.




Data & Code

To reference our ALGORITHM, CODE, DATA or RESULTS in any publication, Please include the bibtex below.
Linkļ¼šhttps://github.com/zhijieW-94/SAGNet




Acknowledgement

We thank the reviewers for their valuable comments. This work was supported in parts by National 973 Program (2015CB352501), NSFC (61761146002, 61861130365, 61702338), Guangdong Science and Technology Program (2015A030312015), Shenzhen Innovation Program (KQJSCX20170727101233642), LHTD (20170003), ISF (2366/16), ISF-NSFC Joint Research Program (2472/17), and the National Engineering Laboratory for Big Data System Computing Technology.


Bibtex
@article{SAGnet19,
title = {
SAGNet: Structure-aware Generative Network for 3D-Shape Modeling},
author = {Zhijie Wu and Xiang Wang and Di Lin and Dani Lischinski and Daniel Cohen-Or and Hui Huang},
journal = {ACM Transactions on Graphics (Proceedings of SIGGRAPH 2019)},
volume = {38},
number = {4},
pages = {91:1--91:14},  
year = {2019},

Downloads(faster for people in China)

Downloads(faster for people in other places)