PointGrow: Autoregressively Learned Point Cloud Generation with Self-Attention

Yongbin Sun1      Yue Wang1      Ziwei Liu2      Joshua E. Siegel1      Sanjay E. Sarma1
1. Massachusetts Institute of Technology      2. The Chinese University of Hong Kong     
Winter Conference on Applications of Computer Vision (WACV) 2020, Oral Presentation

Abstract


A point cloud is an agile 3D representation, efficiently modeling an object's surface geometry. However, these surface-centric properties also pose challenges on designing tools to recognize and synthesize point clouds. This work presents a novel autoregressive model, PointGrow, which generates realistic point cloud samples from scratch or conditioned on given semantic contexts. Our model operates recurrently, with each point sampled according to a conditional distribution given its previously-generated points. Since point cloud object shapes are typically encoded by long-range interpoint dependencies, we augment our model with dedicated self-attention modules to capture these relations. Extensive evaluation demonstrates that PointGrow achieves satisfying performance on both unconditional and conditional point cloud generation tasks, with respect to fidelity, diversity and semantic preservation. Further, conditional PointGrow learns a smooth manifold of given image conditions where 3D shape interpolation and arithmetic calculation can be performed inside.

Public Video



Materials


Code and Models


Citation

@inproceedings{sun2018pointgrow,
title={PointGrow: Autoregressively Learned Point Cloud Generation with Self-Attention},
author={Sun, Yongbin and Wang, Yue and Liu, Ziwei and Siegel, Joshua E and Sarma, Sanjay E},
booktitle={Winter Conference on Applications of Computer Vision},
year={2020}
}