PAT3D: Physics-Augmented Text-to-3D Scene Generation

Guying Lin	Kemeng Huang	Michael Liu	Ruihan Gao	Hanke Chen	Lyuhao Chen	Beijia Lu	Taku Komura	Yuan Liu	Jun-Yan Zhu	Minchen Li

The Fourteenth International Conference on Learning Representations (ICLR) (2026)

We introduce PAT3D, the first physics-augmented text-to-3D scene generation framework that integrates vision–language models with physics-based simulation to produce physically plausible, simulation-ready, and intersection-free 3D scenes. Given a text prompt, PAT3D generates 3D objects, infers their spatial relations, and organizes them into a hierarchical scene tree, which is then converted into initial conditions for simulation. A differentiable rigid-body simulator ensures realistic object interactions under gravity, driving the scene toward static equilibrium without interpenetrations. To further enhance scene quality, we introduce a simulation-in-the-loop optimization procedure that guarantees physical stability and non-intersection, while improving semantic consistency with the input prompt. Experiments demonstrate that PAT3D substantially outperforms prior approaches in physical plausibility, semantic consistency, and visual quality. Beyond high-quality generation, PAT3D uniquely enables simulation-ready 3D scenes for downstream tasks such as scene editing and robotic manipulation.

Guying Lin, Kemeng Huang, Michael Liu, Ruihan Gao, Hanke Chen, Lyuhao Chen, Beijia Lu, Taku Komura, Yuan Liu, Jun-Yan Zhu, Minchen Li (2026). PAT3D: Physics-Augmented Text-to-3D Scene Generation. The Fourteenth International Conference on Learning Representations (ICLR).

@article{lin2026pat3d,
author = {Guying Lin and Kemeng Huang and Michael Liu and Ruihan Gao and Hanke Chen and Lyuhao Chen and Beijia Lu and Taku Komura and Yuan Liu and Jun-Yan Zhu and Minchen Li},
title = {PAT3D: Physics-Augmented Text-to-3D Scene Generation},
year = {2026},
booktitle={The Fourteenth International Conference on Learning Representations (ICLR)},
url = {https://simulation-intelligence.github.io/PAT3D/},
}