TreePartNet: Neural Decomposition of Point Clouds for 3D Tree Reconstruction

ACM Transactions on Graphics (Proceedings of SIGGRAPH ASIA 2021)


Yanchao Liu1,2,3    Jianwei Guo1,3    Bedrich Benes4    Oliver Deussen5    Xiaopeng Zhang1,3    Hui Huang2*

1University of Chinese Academy of Sciences    2Shenzhen University    3NLPR, Institute of Automation, CAS    4Purdue University    5University of Konstanz




Fig. 1. Starting from a noisy and incomplete set of unstructured points from a raw scan (a), we propose TreePartNet to find branching structures and create a cylindrical decomposition (b). The geometry of the tree is represented by generalized cylinders and smooth branching points (c). Eventually, textures and leaves can be added to enhance visual appeal (d). In (e), we show a rendered version from a different viewpoint.


Abstract

We present TreePartNet, a neural network aimed at reconstructing tree geometry from point clouds obtained by scanning real trees. Our key idea is to learn a natural neural decomposition exploiting the assumption that a tree comprises locally cylindrical shapes. In particular, reconstruction is a two-step process. First, two networks are used to detect priors from the point clouds. One detects semantic branching points, and the other network is trained to learn a cylindrical representation of the branches. In the second step, we apply a neural merging module to reduce the cylindrical representation to a final set of generalized cylinders combined by branches. We demonstrate results of reconstructing realistic tree geometry for a variety of input models and with varying input point quality, e.g., noise, outliers, and incompleteness. We evaluate our approach extensively by using data from both synthetic and real trees and comparing it with alternative methods.



Fig. 3. Network architecture for neural decomposition: The top branch of our network (indicated by green arrows) represents the semantic segmentation module, which learns the multi-scale per-point features to detect junction parts. The other two branches (indicated by orange and blue arrows) are the fine clustering module and pairwise affinity module. The former concatenates local contextual features with point-wise feature vectors to decompose the input into a set of local cylindrical patches, while the latter module merges the patches by learning an affinity matrix.



Fig. 6. Evaluation on two synthetic examples from our test dataset, where we show the step-by-step results of our decomposition and reconstruction process. For each example, from left to right, we show the input point cloud, junction detection, initial clusters, merged clusters, extracted skeleton, and our final reconstructed, textured models.



Fig. 7. Several results of reconstructed trees from real data. From left to right: reference photo, input point cloud, our foliage segmentation, and branch decomposition, reconstructed model, and rendering result adding leaves and textures. The input point clouds in (a)-(c) are obtained by using multi-view stereo reconstruction from 64, 66, 40 images, respectively.



Fig. 11. Visual comparison of different clustering methods. The ground-truth labeling (GT) is provided in the second column. Our clustering reveals the branching structure clearer than other methods while still having the right granularity.



Fig. 15. Reconstruction comparison on three real scanned point clouds. From left to right: input point cloud, reconstruction results, and error maps of different methods.



Fig. 16. Comparison of reconstruction results for real trees with dense foliage. The detailed views show the reconstruction quality in local regions.



Data & Code

Note that the DATA and CODE are free for Research and Education Use ONLY. 

Please cite our paper (add the bibtex below) if you use any part of our ALGORITHM, CODE, DATA or RESULTS in any publication.

Link: https://github.com/marktube/TreePartNet


Acknowledgements

We thank Zhihao Liu for providing the implementation of selforganizing tree models. This work was funded in parts by National Key R&D Program (2018YFB2100602), NSFC (62172416, 61802406, U2001206), Guangdong Talent Program (2019JC05X328, 00201509), DEGP Key Project (2018KZDXM058), Shenzhen Science and Technology Program (RCJC20200714114435012, JCYJ20210324120213036, JCYJ20180507182222355), NSF (10001387), Foundation for Food and Agriculture Research (602757), National Engineering Laboratory for Big Data System Computing Technology, and Guangdong Laboratory of Artificial Intelligence and Digital Economy (SZ).


Bibtex

@article{TreePartNet21,

title={TreePartNet: Neural Decomposition of Point Clouds for 3D Tree Reconstruction},

author={Yanchao Liu and Jianwei Guo and Bedrich Benes and Oliver Deussen and Xiaopeng Zhang and Hui Huang},

journal={ACM Transactions on Graphics (Proceedings of SIGGRAPH ASIA)},

volume={40},

number={6},

pages={232:1--232:16},

year={2021},

}



Downloads (faster for people in China)

Downloads (faster for people in other places)