Efficient Diffusion Policy Fine Tuning with Latent Noise Representation Bridging (LARBRID)
Yuanchu Liang, Yue Yang, An-Chi He, Hanna Kurniawati
LARBRID (Latent Representation Bridging for Diffusion) fine-tunes pre-trained diffusion and flow policies with reinforcement learning by learning only the distribution of their initial noise. The pre-trained model stays frozen, so it can be treated as a black box, and no gradients need to be backpropagated through the denoising chain.
Pre-trained diffusion and flow policies have become the backbone of many robotic foundation models, but they often fail on tasks that are not well covered by their training data. Fine-tuning them with RL is a natural fix, yet existing approaches are computationally expensive, sample inefficient and lack theoretical guarantees. Steering the initial noise of a deterministic sampler (e.g. DDIM or flow) is a cheap alternative, but it suffers from the noise aliasing problem: many noises can produce the same action and similar noises can produce different actions, so there is no natural link between the value of a noise and the value of the action it generates.
LARBRID resolves this by endowing the environment with a linear MDP structure. A frozen generative policy acting in a linear MDP induces a Latent Noise Linear MDP (LNL-MDP) over the noise space, whose representation is simply the environment representation pulled back through the pre-trained policy. State-action and state-noise pairs therefore share a single representation space, and the resulting Bridge Equation aligns action-space and noise-space Q-values exactly, removing noise aliasing without separate critic networks.
Key Ideas
- Latent Noise Linear MDP. When the environment MDP is linear, the latent noise MDP induced by a frozen diffusion/flow policy is also linear, with features obtained by pulling the environment features back through the pre-trained policy.
- The Bridge Equation. One shared representation gives the same Q-value for a noise and the action it generates, so a single critic serves both the action and noise spaces and noise aliasing disappears.
- Near-optimal fine-tuning guarantees. Treating rollouts of the pre-trained model as a virtual offline dataset places LNL-MDP fine-tuning within the FineTuneRL framework, which gives near-optimal (minimax up to horizon factors) bounds on the number of online episodes needed once the representation is known.
- Practical representation learning. LARBRID learns the shared representation from scratch with diffusion spectral representation, and then optimises a Gaussian noise policy with Soft Actor-Critic on top of it.
- Flexible across settings. Since fine-tuning in the noise space and in the action space are two sides of the same coin, LARBRID has simplified variants, including LARBRID-I, which works directly in the action space for offline RL and avoids large critic ensembles.
Results
We evaluate LARBRID in online, offline and offline-to-online fine-tuning settings across Gymnasium, Robomimic and OGBench locomotion and manipulation tasks (including visual-input tasks), and when fine-tuning the 3.3B-parameter π0 vision-language-action model on Aloha and Libero tasks. LARBRID gives up to 2-3× performance increases over state-of-the-art baselines. For example, it improves on DSRL by roughly 70% in the Robomimic Transport task, and LARBRID-I yields a 3× improvement on the OGBench large humanoid maze in the offline setting.
Videos
Rollouts on OGBench tasks before and after fine-tuning. Each row is one task: the pre-trained policy is shown on the left and the fine-tuned policy on the right.