ACM Transactions on Graphics (Proceedings of SIGGRAPH 2019)
Zhijie Wu^{1} Xiang Wang^{1} Di Lin^{1} Dani Lischinski^{2} Daniel CohenOr^{1,3} Hui Huang^{1*}
^{1}Shenzhen University ^{2}Hebrew University of Jerusalem ^{3}Tel Aviv University
Fig. 1. Our generative model jointly analyzes the structure and geometry of shapes, encoding them into a single latent code. The highlighted triplets above demonstrate that, in this joint latent space, pairs of nearby points represent models that are close to each other in both geometry and structure, while stepping away from the pair introduces differences in either geometry, or structure, or both. Note that, all the shapes shown here are voxelbased, and the bounding boxes of their parts are hidden on purpose for a clearer visualization.
Abstract
We present SAGNet, a structureaware generative model for 3D shapes. Given a set of segmented objects of a certain class, the geometry of their parts and the pairwise relationships between them (the structure) are jointly learned and embedded in a latent space by an autoencoder. The encoder intertwines the geometry and structure features into a single latent code, while the decoder disentangles the features and reconstructs the geometry and structure of the 3D model. Our autoencoder consists of two branches, one for the structure and one for the geometry. The key idea is that during the analysis, the two branches exchange information between them, thereby learning the dependencies between structure and geometry and encoding two augmented features, which are then fused into a single latent code. This explicit intertwining of information enables separately controlling the geometry and the structure of the generated models. We evaluate the performance of our method and conduct an ablation study. We explicitly show that encoding of shapes accounts for both similarities in structure and geometry. A variety of quality results generated by SAGNet are presented.
Fig. 2. Overview of SAGNet. Given 3D shapes as training data, the network has traditional 3D convolutional and fullyconnected layers to extract visual features for shape parts. The network is equipped with GRUbased encoder and attention component, which jointly analyzes the geometry and structural information of shapes. All the information are provided for the 2way VAE, which offers the generative power to the network. Our network eventually decodes the geometry and structural information to generate 3D shapes.
Fig. 6. The generation results of G2L [Wang et al. 2018] (top row), GRASS [Li et al. 2017] (middle row) and SAGNet(bottom row). Compared to SAGNet, G2L often generates coarser and lessstructured shapes while the voxel grids of GRASS may lose geometric details and thus are less visually appealing.
Fig. 7. For each generated sample in the top row, we retrieve the 3nearest neighbors in the training data. It may be seen that the generated shapes are original.
Fig. 9. We measure the symmetry scores for legs of chairs generated by SAGNet, or baselines, or existing models. Along the horizontal axis, we set different thresholds for the scores. Along the vertical axis, we provide the percentage of shapes, which have smaller scores than the given thresholds.
Fig. 10. We measure the centroidtoplane distances for each airplane generated by SAGNet, or baselines, or existing models. The centroidtoplane distances are computed using the fore and backwings of airplanes. Along the horizontal axis, we provide different thresholds for the distances. Along the vertical axis, we provide the percentage of shapes, which have smaller distances than the given thresholds.
Fig. 11. Cavity analysis and comparison on the synthetic tenonmortise joints. We present two generated joint shapes by SAGNet, G2L and GRASS models, respectively, in the right. In the left, along the horizontal axis, we provide different thresholds for the fitting accuracy R. Along the vertical axis, we provide the percentage of shapes, which have better fitting accuracy than the given thresholds.


Fig. 12. Visual comparison of geometrytostructure mapping results. 
Fig. 13. Visual comparison of structuretogeometry mapping results. 
Data & Code
Note that the DATA and CODE are free for Research and Education Use ONLY.
Please cite our paper (add the bibtex below) if you use any part of our ALGORITHM, CODE, DATA or RESULTS in any publication.
We thank the reviewers for their valuable comments. This work was supported in parts by National 973 Program (2015CB352501), NSFC (61761146002, 61861130365, 61702338), Guangdong Science and Technology Program (2015A030312015), Shenzhen Innovation Program (KQJSCX20170727101233642), LHTD (20170003), ISF (2366/16), ISFNSFC Joint Research Program (2472/17), and the National Engineering Laboratory for Big Data System Computing Technology.
Bibtex
@article{SAGnet19,
title = {SAGNet: Structureaware Generative Network for 3DShape Modeling},
author = {Zhijie Wu and Xiang Wang and Di Lin and Dani Lischinski and Daniel CohenOr and Hui Huang},
journal = {ACM Transactions on Graphics (Proceedings of SIGGRAPH 2019)},
volume = {38},
number = {4},
pages = {91:191:14},
year = {2019},
}