Meta-RL with Bayesian Linear Task Models (GLiBRL)

Jingyang You, Hanna Kurniawati

NeurIPS 2026

GLiBRL (Generalised Linear Models in Deep Bayesian RL with Learnable Basis Functions) combines learned neural features with Bayesian linear reward and transition models. Closed-form Bayesian updates infer the task from new interactions, and a belief-conditioned policy uses this inference to adapt its behaviour.

Standard reinforcement learning algorithms typically do not account for variations in transition and reward dynamics, so they generalise poorly to unseen tasks. Bayesian Reinforcement Learning (BRL), a special case of Meta-RL, addresses this by maintaining distributions over the transition and reward models and continually performing Bayesian inference on them. Recent deep BRL methods, such as PEARL and VariBAD, learn these models with neural networks. However, applying neural networks directly to the joint space of data and task parameters makes exact inference intractable, so these methods resort to variational inference and the Evidence Lower Bound (ELBO). This brings high-variance Monte Carlo estimates, amortisation gaps and posterior collapse, which often prevent the agent from extracting meaningful, distinctive task representations.

GLiBRL takes a different route. It enforces a generalised linear relation between task parameters and data features computed by learnable non-linear basis functions. This structure admits conjugate Bayesian inference: exact, sequential posterior updates over both the task parameters and the model noise, together with a closed-form marginal likelihood that removes variational inference entirely.

Overview of GLiBRL. At each timestep, the belief is updated to the exact posterior over the task parameters, and the belief-conditioned policy selects the next action.

Key Ideas

  • Exact, sequential posterior updates. Transition and reward models are linear in learned features, with a Normal-Wishart prior over task parameters and model noise. The posterior is updated in closed form after every interaction, with no sampling or amortised encoder.
  • ELBO-free model learning. The basis functions are trained by directly maximising the exact marginal log-likelihood of the observed data, avoiding the approximation error of variational objectives.
  • Inference over model noise. Unlike ALPaCA and CAMeLiD, which assume known noise, GLiBRL infers the noise exactly. This reduces prediction errors on unseen tasks for both the transition and reward models.
  • Permutation invariance. The exact Bayesian update does not depend on the order of the data, so GLiBRL plugs into both off-policy (SAC) and on-policy (PPO) algorithms.
  • Analytically traceable task representations. An exact kernel identity relates distances between learned task representations to signed-measure kernel discrepancies over the task contexts.

Training Loop

GLiBRL is an online algorithm that alternates between data collection and learning. During data collection, a batch of tasks is sampled from the task distribution. At each timestep, GLiBRL (1) updates the current belief to the exact posterior over the task's transition and reward parameters, (2) samples an action from the belief-conditioned policy, and (3) executes it to observe the next state and reward. Once enough data has been collected, GLiBRL updates the policy network with standard RL losses and the basis functions with the exact marginal log-likelihood.

Benchmark Problems

We evaluate GLiBRL on locomotion and manipulation benchmarks, focusing on zero-shot test-time performance: the agent receives no task-specific observations before evaluation and adapts only by updating its posterior online, with no gradient updates at test time.

  • MuJoCo locomotion: HalfCheetahDir, AntDir and HalfCheetahVel. The agent must infer from rewards whether to run forward or backward, or at which target velocity.
  • MetaWorld ML10 / ML45: 10 or 45 training manipulation tasks, with 5 held-out test tasks where both the transition and reward functions are unseen. This is one of the most challenging Meta-RL benchmarks.

Results

GLiBRL was compared against eight representative and recent Meta-RL methods across the three main families: gradient-based (MAML), black-box (RL2, AMAGO, TrMRL, ECET) and task-inference / deep BRL (PEARL, VariBAD, SDVT). GLiBRL achieves the highest aggregate zero-shot test performance on both the MuJoCo locomotion and the MetaWorld ML10/45 manipulation benchmarks.

Test-time return and success rate in all benchmarks. Curves report the interquartile mean (IQM), and shaded regions denote 95% confidence intervals. In GLiBRL, $D_T=16$ and $D_R=256$ for MetaWorld tasks, and $D_R=8$ for MuJoCo tasks, which do not require inference in transition. In MuJoCo tasks, PEARL and GLiBRL are run with SAC for $10^6$ steps and the other baselines with PPO for $10^7$ steps.

Code

GLiBRL

References

  1. Meta-RL with Bayesian Linear Task Models
    Jingyang You and Hanna Kurniawati
    In Advances in Neural Information Processing Systems, 2026