3D-aware Conditional Image Synthesis
| Kangle Deng | Gengshan Yang | Deva Ramanan | Jun-Yan Zhu |
Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2023)
We propose pix2pix3D, a 3D-aware conditional generative model for controllable photorealistic image synthesis. Given a 2D label map, such as a segmentation or edge map, our model learns to synthesize a corresponding image from different viewpoints. To enable explicit 3D user control, we extend conditional generative models with neural radiance fields. Given widely-available posed monocular image and label map pairs, our model learns to assign a label to every 3D point in addition to color and density, which enables it to render the image and pixel-aligned label map simultaneously. Finally, we build an interactive system that allows users to edit the label map from different viewpoints and generate outputs accordingly.
Kangle Deng, Gengshan Yang, Deva Ramanan, Jun-Yan Zhu (2023). 3D-aware Conditional Image Synthesis. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).
@article{deng2023pix2pix3d,
title = {3D-aware Conditional Image Synthesis},
author = {Kangle Deng and Gengshan Yang and Deva Ramanan and Jun-Yan Zhu},
booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
year = {2023},
}
