Learning Loco-Manipulation From SMPC Demonstrations With Sparse Offline-to-Online RL
Overview video: real-world loco-manipulation skills learned from purely sparse rewards and deployed on an arm-equipped Spot quadruped and a Unitree G1 humanoid.
Abstract
Integrating locomotion and manipulation is essential for robot autonomy, but scaling standard reinforcement learning (RL) to complex tasks is severely bottlenecked by the slow, manual process of dense reward shaping. To bypass this limitation, we leverage sample-based model predictive control (SMPC) entirely in simulation as an automated, rapidly tunable expert to generate massive offline datasets. Because this data solves the fundamental exploration problem, we can train an off-policy RL agent using purely sparse task rewards, drastically reducing the time required to learn new skills and eliminating the need for manual tuning. Integrating this high-level agent with a low-level dynamic stability controller yields more optimal behaviors that strictly align with true task objectives, ultimately allowing the learned policies to surpass the original optimal control teacher. We validate the robustness of this sim-to-real framework by successfully deploying complex loco-manipulation skills across different morphologies, including an arm-equipped Spot quadruped and a G1 humanoid.
The Issue with Current RL Approaches
Combining locomotion and manipulation is one of the capabilities we most want from autonomous robots, but learning it from scratch with RL is hard. The main reason is that scaling RL to complex tasks leans heavily on dense reward shaping to guide exploration, and every change to that reward means another full training run before you know whether it helped. That makes tuning slow.
Take an arm-equipped quadruped rolling a heavy tire. The tire is too heavy to drag, so the robot has to keep stepping to stay balanced while pushing it forward with the arm. Hand-designing a dense reward for that kind of coordination means working through a brittle, tightly constrained optimization problem, and each adjustment costs another training cycle to check.
Going Sparse
What if we drop reward shaping altogether? We use a strictly sparse task reward: zero at the goal, a one-time penalty if the robot crashes, and −1 otherwise. There is nothing to tune and no surrogate terms to skew the behavior, so the agent optimizes the true objective and can find strategies better than anything we would have designed by hand.
The catch is exploration. Optimizing a sparse reward in the high-dimensional, continuous action space of a legged, arm-equipped robot is extremely hard: from a random start the agent almost never reaches the goal, so learning never gets going. Recent works have accelerated training with offline-to-online RL, where we bootstrap the agent with examples of successful trajectories to get it past this initial phase.
But that just moves the problem. Now we need a source of expert demonstrations, and the real question is where to get them without falling back into long tuning cycles.
Where Do We Get The Data From?
The usual options don't fit well. Human teleoperation and motion retargeting don't transfer to non-humanoid robots like an arm-equipped quadruped, and most planners come with their own slow tuning. Our answer is sample-based model predictive control (SMPC), used purely as a data generator in simulation. SMPC is driven by dense cost functions, which sounds like the tuning problem all over again, with one important difference: it needs no training. You can adjust its costs interactively and watch the behavior change in near real-time, so a new behavior takes minutes to tune.
On a single RTX 5090, our SMPC runs at about 0.5× real-time, which is enough for this interactive tuning. Once the costs are set, we scale up data collection with a massively parallel, tiled GPU pipeline: it broadcasts one task across many environments, rolls out sampled action splines, and keeps the best trajectories. That gives us roughly a million samples per hour, enough to bootstrap a task in about four GPU hours.
For a sampling-based MPC implementation, see judo [1], an open-source, hackable sampling-based MPC toolbox.
Show extended figure
Summary: we swap a slow, train-in-the-loop reward search for a fast, interactive one, then lean on parallelism to turn a tuned controller into a large offline dataset.
Training Loco-Manipulation Policies With Sparse Rewards
With the expert data ready, we train an off-policy agent (a modified FastTD3) in an offline-to-online setup. Early on, half of every replay-buffer batch comes from the expert dataset, which gives the critics successful transitions to learn from before the agent has found any on its own. A curriculum then phases the expert data out once the agent passes a 10% success rate, handing the rest of training to pure online RL.
The result shows up clearly in training: across every task, policies reach near-100% success in simulation with no reward shaping, while the same setup without expert data fails to learn at all. And because nothing skews the objective, the learned policies don't just match the teacher, they beat it, finishing tasks faster (by over 50% on some) with 11–45% lower variance.
Note: the improvement over the teacher is local. Because the agent is bootstrapped from SMPC demonstrations, it optimizes in the vicinity of the dataset distribution rather than finding globally optimal strategies that are completely separate from it.
Deploying Sparse Policies
Sparse rewards push policies to the physical limits of the robot, which tends to make them aggressive: good for performance, but risky on real hardware. To keep them deployable, the high-level policy outputs action deltas (Δv, Δqarm, Δh, Δp) relative to the current command instead of absolute targets. This lets us cap the maximum accelerations and speeds directly in the action space, smoothing the motion and bounding how aggressive the policy can get.
Below it, a frozen low-level whole-body controller (trained with ReLIC [2]) tracks the commanded arm motion and base velocity while constantly adjusting the legs to stay balanced, so the high-level policy only has to worry about the task. For sim-to-real, we randomize object mass, size, and friction and use an asymmetric actor-critic, where the actor sees noisy observations and the critic sees the exact simulator state. Together, these let all five tasks transfer reliably to real Spot and G1 hardware without any platform-specific reward engineering.
What Matters When Using SMPC Data?
SMPC data is convenient, but it comes with some caveats. We ran a set of ablations to see what actually matters when bootstrapping sparse-reward RL from a planner. Use the tabs below to explore each factor.
Amount of data. The dataset size you need grows with task complexity. Simple tasks like navigation barely notice it, while our hardest task (tire rolling) needs around four million samples to converge.
Data quality. We reduce quality by using fewer sampling environments during SMPC collection. Most tasks are surprisingly robust to this, but high-coordination tasks like tire rolling clearly suffer: the quality you need is closely tied to the precision the task demands.
Multimodality. SMPC is naturally multi-modal: it might kick, push, or step into the tire. A uni-modal policy can't capture that, and training on multi-modal data fails outright, even when the demonstrations themselves have a high success rate. Enforcing a single mode in the data is essential.
Expert ratio. How much of the early replay buffer should be expert data? A 50% ratio works well across tasks. Most are insensitive to the exact value thanks to the phase-out curriculum, but tire rolling benefits from more expert data and collapses below 50%.
Phase-out. Expert data is essential to start, but harmful if kept too long. Phasing it out once the policy reaches a ~10% success rate matters: higher thresholds slow convergence, and keeping the data indefinitely degrades training on nearly every task except the simplest.
Bounded critics. Our sparse reward has known minimum and maximum returns, so we clamp the critic's output to the minimum and maximum Q-values theoretically possible under γ. This stabilizes training and widens the range of hyperparameters that still work; unbounded critics lead to sharp performance degradation spikes at aggressive learning rates.
BibTeX
@misc{schuck2026learninglocomanipulationsmpcdemonstrations,
title={Learning Loco-Manipulation From SMPC Demonstrations With Sparse Offline-to-Online RL},
author={Martin Schuck and Maks Sorokin and Simone Manni and Duy Ta and Angela P. Schoellig and Marco Hutter and Simon Le Cleac'H and Jan Brüdigam},
year={2026},
eprint={2608.12063},
archivePrefix={arXiv},
primaryClass={cs.RO},
url={https://arxiv.org/abs/2608.12063},
}