Efficient Diffusion Policy Fine Tuning with Latent Noise Representation Bridging (LARBRID)

Yuanchu Liang, Yue Yang, An-Chi He, Hanna Kurniawati

NeurIPS 2026

LARBRID (Latent Representation Bridging for Diffusion) fine-tunes pre-trained diffusion and flow policies with reinforcement learning by learning only the distribution of their initial noise. The pre-trained model stays frozen, so it can be treated as a black box, and no gradients need to be backpropagated through the denoising chain.

Pre-trained diffusion and flow policies have become the backbone of many robotic foundation models, but they often fail on tasks that are not well covered by their training data. Fine-tuning them with RL is a natural fix, yet existing approaches are computationally expensive, sample inefficient and lack theoretical guarantees. Steering the initial noise of a deterministic sampler (e.g. DDIM or flow) is a cheap alternative, but it suffers from the noise aliasing problem: many noises can produce the same action and similar noises can produce different actions, so there is no natural link between the value of a noise and the value of the action it generates.

LARBRID resolves this by endowing the environment with a linear MDP structure. A frozen generative policy acting in a linear MDP induces a Latent Noise Linear MDP (LNL-MDP) over the noise space, whose representation is simply the environment representation pulled back through the pre-trained policy. State-action and state-noise pairs therefore share a single representation space, and the resulting Bridge Equation aligns action-space and noise-space Q-values exactly, removing noise aliasing without separate critic networks.

(a) Noise aliasing causes ambiguous associations between action values and noise values. (b) The LNL-MDP maps the state-action and state-noise spaces into a shared representation space, where the Q-values are automatically aligned by the Bridge Equation.

Key Ideas

  • Latent Noise Linear MDP. When the environment MDP is linear, the latent noise MDP induced by a frozen diffusion/flow policy is also linear, with features obtained by pulling the environment features back through the pre-trained policy.
  • The Bridge Equation. One shared representation gives the same Q-value for a noise and the action it generates, so a single critic serves both the action and noise spaces and noise aliasing disappears.
  • Near-optimal fine-tuning guarantees. Treating rollouts of the pre-trained model as a virtual offline dataset places LNL-MDP fine-tuning within the FineTuneRL framework, which gives near-optimal (minimax up to horizon factors) bounds on the number of online episodes needed once the representation is known.
  • Practical representation learning. LARBRID learns the shared representation from scratch with diffusion spectral representation, and then optimises a Gaussian noise policy with Soft Actor-Critic on top of it.
  • Flexible across settings. Since fine-tuning in the noise space and in the action space are two sides of the same coin, LARBRID has simplified variants, including LARBRID-I, which works directly in the action space for offline RL and avoids large critic ensembles.

Results

We evaluate LARBRID in online, offline and offline-to-online fine-tuning settings across Gymnasium, Robomimic and OGBench locomotion and manipulation tasks (including visual-input tasks), and when fine-tuning the 3.3B-parameter π0 vision-language-action model on Aloha and Libero tasks. LARBRID gives up to 2-3× performance increases over state-of-the-art baselines. For example, it improves on DSRL by roughly 70% in the Robomimic Transport task, and LARBRID-I yields a 3× improvement on the OGBench large humanoid maze in the offline setting.

Online fine-tuning of pre-trained DDIM policies on Gymnasium control tasks, averaged over 4 random seeds.

Videos

Rollouts on OGBench tasks before and after fine-tuning. Each row is one task: the pre-trained policy is shown on the left and the fine-tuned policy on the right.

Pre-trained
Fine-tuned
Ant Soccer
Cube
Scene
Puzzle

Code

LARBRID

References

  1. Efficient Diffusion Policy Fine Tuning with Latent Noise Representation Bridging
    Yuanchu Liang, Yue Yang, An-Chi He, and Hanna Kurniawati
    In Advances in Neural Information Processing Systems (NeurIPS), 2026