VOiLA: Vectorized Online Planning with Learned Diffusion Models for POMDP Agents
Marcus Hoerger, Rishikesh Joshi, Rahul Shome, Ian Manchester, Hanna Kurniawati
VOiLA (Vectorized Online planning wIth Learned diffusion model for POMDP Agents) is a framework for online planning under uncertainty that combines learned, task-agnostic POMDP world models with the massively parallel Vectorized Online POMDP Planner (VOPP). It enables robots to plan in belief space without requiring manually specified transition and observation models.
Partially Observable Markov Decision Processes (POMDPs) provide a principled framework for reasoning about uncertain action outcomes and noisy, incomplete observations. However, constructing faithful POMDP models for complex robots is difficult. Reinforcement learning (RL) avoids the need for explicit models, but often requires substantial task-specific training data and can struggle when the environment changes. VOiLA combines the strengths of both approaches: it learns reusable models from offline simulation data, then uses those models to compute actions online, adapting its decisions to the robot's current belief and task.
Across three evaluation problems, VOiLA achieves competitive or better results than leading RL-based POMDP methods while using substantially less training data. In a physical robot experiment, models trained entirely in simulation enabled a Unitree Go2 quadruped to complete its target-finding task in 10 out of 10 trials, without additional real-world model training.
Key Idea: Learn the World Model Offline, Plan Online
Instead of learning a task-specific policy, VOiLA learns the components of a POMDP model from trajectories collected by a task-independent uniform policy in simulation. The learned models can subsequently be used for different objectives that share the same underlying transition and observation models. VOiLA learns three components that support both online planning and particle-based belief updates:
- Transition sampler $f_T(s,a,\omega_T)$: samples possible next states given the current state and action, capturing uncertainty in robot dynamics.
- Observation sampler $f_Z(s',a,\omega_Z)$: generates possible observations conditioned on the next state and action. For high-dimensional observations, such as RGB images, sampling takes place in a compact learned latent space.
- Observation-likelihood model $\ell(s',a,o)$: provides relative observation likelihoods used to weight particles during belief updates. It is trained with a contrastive, InfoNCE-style objective.
Efficient Generative Models through Diffusion Distillation
Complex robot dynamics and observation distributions can be high-dimensional and multimodal, making them difficult to represent with simple parametric models. VOiLA first trains conditional diffusion models to capture these distributions. However, generating samples through iterative diffusion inference is too expensive for online POMDP planning, which may require thousands of forward simulations per planning step.
To overcome this bottleneck, VOiLA distills diffusion samplers into compact feedforward neural networks. These generators produce stochastic next-state and observation samples in a single forward pass, making them suitable for massively batched inference. In our benchmarks, the distilled transition and observation samplers achieved approximately 105× and 67× the sampling throughput of their diffusion counterparts, respectively (batch size 1,024), while closely approximating their outputs.
Online Belief-Space Planning with VOPP
The learned models are integrated into VOPP, a fully vectorized online POMDP planner. During planning, VOPP uses the learned transition and observation samplers to simulate candidate action sequences in parallel, expanding a belief tree and updating action preferences using batched tensor operations. VOiLA also extends VOPP with vectorized progressive widening to support continuous observation spaces.
After executing the selected action, VOiLA incorporates the new observation through a Sequential Importance Resampling (SIR) particle filter. The learned transition sampler generates proposal particles, and the learned likelihood model determines their relative weights. Planning then resumes from the updated belief.
Benchmark Problems
We evaluated VOiLA on three planning-under-uncertainty problems:
- FloorPositioning — a robot must reach the correct goal in an environment with two possible floors, using noisy distance-to-wall measurements to infer its location.
- LeggedNavigation — an ANYmal-C quadruped navigates a cluttered 20×20m environment using onboard 32×32 RGB observations. Two internal walls have five possible configurations; models are trained only in configurations 1 and 2 and tested on all five.
- TargetFinding — a physical Unitree Go2 quadruped searches for a target asset hidden behind one of two movable boxes, using lidar observations while reasoning about uncertainty in its pose and obstacle and target locations.
Results
We compared VOiLA against DreamerV3, Recurrent Soft Actor Critic for Discrete Actions (RSAC-D), and Visual Tree Search (VTS). The experiments used comparable total offline training-time budgets for the principal comparisons, while also evaluating RL baselines trained with substantially more data. VOiLA's learned models were trained on 50,000 simulation transitions per problem.
- FloorPositioning: VOiLA achieved a 100% success rate across 100 trials, using 50K training transitions compared with 900K for DreamerV3, 1.8M for RSAC-D, and 32M for VTS.
- LeggedNavigation (trained configurations): VOiLA achieved 94% and 100% success in environments 1 and 2, respectively, using 50K transitions. DreamerV3 and RSAC-D used 600K transitions in the comparable-training-time experiments.
- Generalization to unseen environments: VOiLA achieved success rates of 98%, 84%, and 88% in environments 3, 4, and 5, respectively. Both DreamerV3 and RSAC-D achieved 0% in environment 3, even when trained with 2.6M transitions.
Real-World Deployment
For TargetFinding, VOiLA was deployed on a Unitree Go2 in a 4.9×3.6m room. The robot had to identify which of two locations contained the target asset while navigating around obstacles whose positions were uncertain. Its lidar measurements were processed into 36-dimensional range observations, using the same preprocessing in simulation and on the robot. The target and movable obstacles were varied across test runs.
The learned models were trained entirely in simulation using just 50,000 transitions and transferred directly to the physical robot, without any additional real-world training. As the robot moved and gathered observations, its particle belief over robot pose, target position, and obstacle locations evolved, supporting information-gathering actions and collision-aware navigation. VOiLA successfully located and reached the target in 10 out of 10 physical robot trials.