Despite recent advances in policy learning, long-horizon manipulation remains difficult because learned policies must avoid compounding errors while preserving future feasibility. While search-based planning can explicitly reason over such future consequences, it quickly becomes expensive in high-dimensional continuous action spaces. This raises a central question:
How can long-horizon reasoning from search be transferred into policies in continuous action spaces?
To this end, we propose DISCO: Distilled Tree Search over Continuous Actions. DISCO uses progressive-widening MCTS in physics simulation to generate long-horizon plans. During search, a learned policy proposes continuous actions, while a learned value function scores actions by their expected progress toward task completion. Using GPU-parallelized simulation, DISCO periodically distills MCTS-generated plans into the policy-value model, improving the action proposals and value estimates used in subsequent searches. Finally, the learned policy value model can be directly deployed without the need for simulator-based search at test time. Across simulated and real-world rearrangement tasks, DISCO out performs behavior cloning, reinforcement learning, and search-based baselines.
DISCO uses Monte-Carlo Tree Search with a learned policy and value function. It uses progressive widening to explore the continuous state-action space, and iteratively distills the discovered long-horizon reasoning into the policy and value models. Each step of the algorithm is visualized below.
Pick a task and watch DISCO's MCTS grow its search tree, one iteration at a time. Press play, or drag the timeline to jump to any iteration.
If you would like to play the DISCO Game first, looking at this visualization would be cheating
Direct rollouts of the distilled policy: no tree search at test time.
Pack every object into a shelf of random size without anything falling out. DISCO may nudge objects already inside to make room.
Stack objects of very different shapes into one stable tower, without knocking any off.
One task, more than one way to solve it. Because DISCO distills many plans found by search, its policy keeps several solution strategies instead of averaging them away.
Transferred sim-to-real with no fine-tuning. DISCO against the 3DFA baseline, trained on the same sub-task demos, over three trials each.
Why 3DFA? It is the same policy architecture DISCO distills into, so the comparison isolates the effect of DISCO's search-and-distill self-improvement loop.
The real table-bussing point cloud with DISCO's grasp pose for the orange cup (magenta) and candidate placements in the tray, coloured by their predicted Q-value. Placements that leave space for the plates score highest.
@InProceedings{saha2026disco,
title = {Long-Horizon Multi-Object Rearrangement via Distilled Tree Search over Continuous Actions},
author = {Saha, Kallol and Hong, Yining and Li, Amber and Choi, Yejin and Likhachev, Maxim and Held, David},
booktitle = {Proceedings of The 10th Conference on Robot Learning},
year = {2026},
}
This material is based upon work supported by the Toyota Research Institute, ONR MURI N00014-24-1-2748, and ONR MURI N00014-26-1-2024. This work was also supported by IITP funded by the Korean Government (MSIT) (No. RS-2024-00457882, National AI Research Lab Project). Additionally, we thank Yishu Li for feedback on the writing, figures and framing of this paper, and for coming up with the acronym "DISCO". We thank Sriram Krishna for the idea behind the DISCO game. We also thank Sriram Krishna, Itamar Mishani, Nayesha Gandotra, Pratik Bhowal and Mino Nakura for helpful discussions and feedback on this paper. We are grateful to Haotian Zhan and Vera Xiao for their help with real-world camera calibration. David Held holds concurrent appointments at CMU and as an Amazon Scholar. This paper describes work performed at CMU and is not associated with Amazon.