DISCO Distilled Tree Search over Continuous Actions

CoRL 2026 Best Paper Finalist ๐Ÿ† Top 0.01%

Catch our oral talk on Tue, Nov 10 at 11โ€“11:30 AM at Hilton Austin!
Poster: Nov 10, 5:00โ€“6:30 PM | JW Marriott Austin (Session P2, Poster #176)

Abstract

Despite recent advances in policy learning, long-horizon manipulation remains difficult because learned policies must avoid compounding errors while preserving future feasibility. While search-based planning can explicitly reason over such future consequences, it quickly becomes expensive in high-dimensional continuous action spaces. This raises a central question:

How can long-horizon reasoning from search be transferred into policies in continuous action spaces?

To this end, we propose DISCO: Distilled Tree Search over Continuous Actions. DISCO uses progressive-widening MCTS in physics simulation to generate long-horizon plans. During search, a learned policy proposes continuous actions, while a learned value function scores actions by their expected progress toward task completion. Using GPU-parallelized simulation, DISCO periodically distills MCTS-generated plans into the policy-value model, improving the action proposals and value estimates used in subsequent searches. Finally, the learned policy value model can be directly deployed without the need for simulator-based search at test time. Across simulated and real-world rearrangement tasks, DISCO out performs behavior cloning, reinforcement learning, and search-based baselines.

How DISCO Works

DISCO uses Monte-Carlo Tree Search with a learned policy and value function. It uses progressive widening to explore the continuous state-action space, and iteratively distills the discovered long-horizon reasoning into the policy and value models. Each step of the algorithm is visualized below.

Watch DISCO in Action (during training)

Pick a task and watch DISCO's MCTS grow its search tree, one iteration at a time. Press play, or drag the timeline to jump to any iteration.

If you would like to play the DISCO Game first, looking at this visualization would be cheating

DISCO during Execution

Direct rollouts of the distilled policy: no tree search at test time.

Simulation Tasks

Pack every object into a shelf of random size without anything falling out. DISCO may nudge objects already inside to make room.

  • 2โ€“7 objects
  • Budget β = 270,000 simulator calls
  • 50 training ยท 15 test scenes
  • Pre-trained on 1,776 single-step demos
  • Reward: fraction of objects in the shelf

Multi-Modality

One task, more than one way to solve it. Because DISCO distills many plans found by search, its policy keeps several solution strategies instead of averaging them away.

Object stacking: two different stable towers built by DISCO's policy.

Real World Execution

Transferred sim-to-real with no fine-tuning. DISCO against the 3DFA baseline, trained on the same sub-task demos, over three trials each.

Why 3DFA? It is the same policy architecture DISCO distills into, so the comparison isolates the effect of DISCO's search-and-distill self-improvement loop.

Inside DISCO's value function

The real table-bussing point cloud with DISCO's grasp pose for the orange cup (magenta) and candidate placements in the tray, coloured by their predicted Q-value. Placements that leave space for the plates score highest.

Quantitative Results

Reward and success against simulation budget for shelf packing and object stacking
Simulation budget

More search, better plans, fastest for DISCO

  • DISCO leads at every simulation budget.
  • Best final success: 100% packing, 98% stacking (next best: 88%, 92%).
  • Stacking is near-perfect after only a few hundred simulator calls.
DISCO (ours)DISCO w/o PWDISCO w/o policy RL-PDMPPIBeam search
Paper, Figure 4 (left)
Reward and success against number of objects for shelf packing and object stacking
Task complexity

Holds up as tasks get longer

  • DISCO keeps 100% success at every object count, up to 7 packed and 6 stacked.
  • Every baseline falls to 0% success at the largest size.
DISCO (ours)DISCO w/o PWDISCO w/o policy RL-PDMPPIBeam search
Paper, Figure 4 (right)
Held-out shelf packing success across distillation rounds
Iterative distillation

Each round of distillation compounds

  • The policy alone climbs from 13.3% to 46.7% held-out success over two rounds.
  • The distilled value function adds 20 to 47 points on top: 80% after round two.
Paper, Figure 7
Constrained packing success of DISCO and test-time planners
Constrained packing

Planner-level results, without planning

  • Rolled out directly, with no search at test time: 85.9% success.
  • On par with test-time planners that use relational abstractions (SPOT 84.7%, P2P 80.7%).
Paper, Figure 5

BibTeX

@InProceedings{saha2026disco,
  title     = {Long-Horizon Multi-Object Rearrangement via Distilled Tree Search over Continuous Actions},
  author    = {Saha, Kallol and Hong, Yining and Li, Amber and Choi, Yejin and Likhachev, Maxim and Held, David},
  booktitle = {Proceedings of The 10th Conference on Robot Learning},
  year      = {2026},
}

Acknowledgments

This material is based upon work supported by the Toyota Research Institute, ONR MURI N00014-24-1-2748, and ONR MURI N00014-26-1-2024. This work was also supported by IITP funded by the Korean Government (MSIT) (No. RS-2024-00457882, National AI Research Lab Project). Additionally, we thank Yishu Li for feedback on the writing, figures and framing of this paper, and for coming up with the acronym "DISCO". We thank Sriram Krishna for the idea behind the DISCO game. We also thank Sriram Krishna, Itamar Mishani, Nayesha Gandotra, Pratik Bhowal and Mino Nakura for helpful discussions and feedback on this paper. We are grateful to Haotian Zhan and Vera Xiao for their help with real-world camera calibration. David Held holds concurrent appointments at CMU and as an Amazon Scholar. This paper describes work performed at CMU and is not associated with Amazon.