Residual path integrals for re-rendering

EGSR 2024 (journal), best-paper award!

We consider the problem of efficiently rendering versions of a given an initial rendering (a), with object movement or material editing (b-e). We devise sampling techniques to focus on the light-transport paths affected by the scene changes and a method for incremental re-rendering faster than path tracing from scratch (PT) or computing correlated-path differences (Correlated PT). Our approach renders a residual image (f) between the original and the edited scene. We show equal-time comparisons with baselines for edits 3 and 4, including final images for the edited scenes and residual images. Numbers shown are MSE x 1000.

Abstract

Conventional rendering techniques are primarily designed and optimized for single-frame rendering. In practical applications, such as scene editing and animation rendering, users frequently encounter scenes where only a small portion is modified between consecutive frames. In this paper, we develop a novel approach to incremental re-rendering of scenes with dynamic objects, where only a small part of a scene moves from one frame to the next. We formulate the difference (or residual) in the image between two frames as a (correlated) light-transport integral which we call the residual path integral. Efficient numerical solution of this integral then involves (1) devising importance sampling strategies to focus on paths with non-zero residual-transport contributions and (2) choosing appropriate mappings between the native path spaces of the two frames. We introduce a set of path importance sampling strategies that trace from the moving object(s) which are the sources of residual energy. We explore path mapping strategies that generalize those from gradient-domain path tracing to our importance sampling techniques specially for dynamic scenes. Additionally, our formulation can be applied to material editing as a simpler special case. We demonstrate speed-ups over previous correlated sampling of path differences and over rendering the new frame independently. Our formulation brings new insights into the re-rendering problem and paves the way for devising new types of sampling techniques and path mappings with different trade-offs.

Downloads and links

BibTeX reference

@article{Xu:2023:ResidualPathIntegral,
  author = {Bing Xu and Tzu-Mao Li and Iliyan Georgiev and Trevor Hedstrom and Ravi Ramamoorthi},
  title = {Residual path integrals for re-rendering},
  journal = {Computer Graphics Forum (Proceedings of EGSR)},
  year = {2024},
  number = {4},
  volume = {43}
}