Photo-to-Shape Material Transfer for Diverse Structures

ACM Transactions on Graphics (Proceedings of SIGGRAPH 2022)


Ruizhen Hu1    Xiangyu Su1    Xiangkai Chen1    Oliver van Kaick2    Hui Huang1*

1Shenzhen University    2Carleton University


Fig. 1. A sample of PhotoShapes (shapes with realistic relightable materials) created with our material assignment method by transferring materials from photo exemplars to 3D shapes with diverse structures, where the input exemplars are the photos in-the-wild on the left.


Abstract

We introduce a method for assigning photorealistic relightable materials to 3D shapes in an automatic manner. Our method takes as input a photo exemplar of a real object and a 3D object with segmentation, and uses the exemplar to guide the assignment of materials to the parts of the shape, so that the appearance of the resulting shape is as similar as possible to the exemplar. To accomplish this goal, our method combines an image translation neural network with a material assignment neural network. The image translation network translates the color from the exemplar to a projection of the 3D shape and the part segmentation from the projection to the exemplar. Then, the material prediction network assigns materials from a collection of realistic materials to the projected parts, based on the translated images and perceptual similarity of the materials.


Fig. 2. Overview of our method for assigning photorealistic relightable materials to 3D shapes based on photo exemplars. Given a 3D shape with segmentation and photo exemplar as input, we first project the 3D shape from a similar view as the exemplar. Next, an image translation network translates the color from the exemplar to the projection and the part segmentation from the projection to the exemplar. Then, a material prediction network assigns materials to the projected parts based on the translated images. Note that the joint translation of segmentation and color enables us to better ensure the consistency in the material assignment step. The result of the method is a 3D shape that can be rendered from different viewpoints with realistic materials.


Fig. 6. Results of our method for transferring materials from exemplars to 3D shapes. We show results generated from different combinations of photo exemplars (given in the top row) and shapes (given in the remaining rows). Note the realism of the assigned materials, the resemblance of the results to the exemplars, and the diversity of structures in the shapes and exemplars.


Fig. 7. Sample of results showing the steps of our method: Input exemplar and segmentation (left), segmentation and color translation (middle), and material transfer result (right). Rows 1-2: shapes with similar number of parts but with different structure; Rows 3-4: target shapes with fewer parts; Rows 5-6: target shapes with more parts.


Fig. 9. Example results where the material of different exemplars (columns) is transferred to the same shape (rows). For better comparison, we rotate the 3D shape to the same view as the exemplar.




Fig. 15. Results for other categories of shapes to demonstrate the generality of our method.




Data & Code

Note that the DATA and CODE are free for Research and Education Use ONLY. 

Please cite our paper (add the bibtex below) if you use any part of our ALGORITHM, CODE, DATA or RESULTS in any publication.

Codehttps://github.com/XiangyuSu611/TMT


Acknowledgements

We thank the anonymous reviewers for their valuable comments. This work was supported in parts by NSFC (61872250, U2001206, U21B2023, 62161146005), GD Talent Plan (2019JC05X328), GD Natural Science Foundation (2021B1515020085), DEGP Key Project (2018KZDXM058, 2020SFKC059), Shenzhen Science and Technology Program (RCYX20210609103121030, RCJC20200714114435012, JCYJ20210324120213036), the Natural Sciences and Engineering Research Council of Canada (NSERC), and Guangdong Laboratory of Artificial Intelligence and Digital Economy (SZ).


Bibtex

@article{TMT,

title = {Photo-to-Shape Material Transfer for Diverse Structures},

author = {Ruizhen Hu and Xiangyu Su and Xiangkai Chen and Oliver van Kaick and Hui Huang},

journal = {ACM Transactions on Graphics (Proceedings of SIGGRAPH)},

volume = {39},

number = {6},

pages = {113:1--113:14},

year = {2022},

}



Downloads (faster for people in China)

Downloads (faster for people in other places)