Reinforcement learning outperforms a classical pipeline on a route-based service task

Collection as a coupled control problem

Route-based collection is a coupled navigation and manipulation problem. At each house, the vehicle must bring a curbside arm within grasping range and complete the transfer before continuing along the route. The approach determines whether the bin can be grasped. The timing of arm extension and stowage affects both service time and departure. Overall performance therefore depends on coordination between vehicle and arm control.

Our baseline uses a conventional modular design. Separate components perform path tracking, select a stopping point, and execute a fixed arm sequence. Additional logic manages the transitions between components. This approach is reliable within the benchmark, but each handoff imposes a constraint on the sequence of actions. For example, the arm remains stationary until the vehicle stops even though the simulated dynamics permit both systems to move at once.

We instead formulate the full task as a single sequential decision problem. One policy observes the state of both systems and controls them together. Its outputs include acceleration and steering as well as commands for all three arm axes. Reinforcement learning permits these actions to be optimized against a route-level objective without prescribing subsystem handoffs. The experiment asks whether this unified policy can improve on the modular system that generated its initial demonstrations. Both systems operate under the same task specification.

Methodological context

The first methodological influence is Robust Autonomy Emerges from Self-Play ^[1]. That work demonstrates how large-scale simulated experience and PPO can produce robust driving behavior under a task-level objective. TrashBench does not use self-play and does not test general driving capability. It adopts the narrower premise that scalable simulation can optimize long-horizon vehicle behavior without prescribing each action sequence.

The training procedure is more directly influenced by Human-like Autonomy Emerges from Self-Play and a Pinch of Human Data ^[2]. That method fits a behavioral-cloning anchor from demonstration data and regularizes PPO toward the anchor on states visited by the learned policy. We adapt this pattern to a single-agent service task. Our demonstrations come from a hand-coded pipeline rather than human driving logs. The cloned network also initializes the learned policy, and the regularization weight decays to zero during training. We do not claim this training strategy as a new algorithm. The contribution is the route-based benchmark, the unified control formulation, and the resulting comparison with the demonstration pipeline.

TrashBench v0

TrashBench is a simulated benchmark for route-based collection. The v0 environment contains a rectangular street loop measuring 120 by 60 meters. Eight houses are distributed along the loop. Each house has one curbside bin whose position is randomized for every episode. The simulated vehicle matches the geometry of our platform and has a 4.21-meter wheelbase. A kinematic bicycle model governs its motion. The curbside arm has three degrees of freedom: a telescoping boom, a carriage that follows a J-shaped track to the dump point, and a gripper. Physics advances at 20 Hz. The policy issues commands at 5 Hz.

Two design constraints separate task specification from policy behavior. The first concerns the boundary between physics and control. The simulator defines the mechanics of grasping, collision, and dumping. A valid grasp requires the gripper to close at ground level while a bin is inside a 0.28-meter window. The policy determines the action sequence without scripted subroutines or ordering hints. It is not instructed to stop before reaching.

The second constraint limits the policy to state that could be available on a physical vehicle. Its observation vector contains 23 values. Some describe vehicle motion and arm configuration. The remaining values encode nearby bins in vehicle-relative coordinates as well as route progress and distance to the next planned stop. The policy has no overhead map or privileged access to simulator state.

The reward function approximates the operational objective of route completion. It assigns positive value to completed service and negative value to elapsed time. The time penalty increases with the number of bins that remain unserviced. A knocked bin, a dropped bin, or departure from the pavement ends the episode. The terminal cost includes the time remaining on the clock. Under this formulation, productive service has a higher return than idling. Idling has a higher return than failure. This ordering holds for any discount factor.

Evaluation

The hand-written baseline combines pure-pursuit path tracking with a computed stopping point for each bin and a fixed arm sequence. We tuned the system until it serviced every route it attempted. The baseline then generated 160 demonstrations for behavior cloning. The cloned network provided the initial policy and served as a regularization reference during early training. We subsequently optimized the policy with PPO for 4.8 billion simulated steps. This required approximately 77 minutes on one GPU. The influence of the regularization reference decayed to zero, so the final policy was optimized solely against the benchmark objective.

The evaluation protocol comprises 100 route instances generated from seeds that were excluded from training. Every instance retains the same street topology but samples a new curbside position for each bin. The held-out seeds ensure that these exact bin configurations were not used during training. The protocol therefore measures generalization to new bin placements on a fixed route. It does not measure generalization to new road layouts. The final policy executes each instance deterministically. Every policy in the table is evaluated on the same set of instances.

The learned policy reduced mean route time by 38.4 seconds relative to the hand-coded baseline, an improvement of 28.9%. Mean service time per house fell by 0.63 seconds, or 14.8%. Off-pavement driving decreased from 17.7% of driving time to 0.7%. The learned policy reached the time limit on one route. The baseline completed all 100 routes. Neither policy damaged a bin during evaluation.

Emergent properties of joint control

Trajectory inspection shows coordinated behaviors that are absent from the scripted baseline. The learned policy begins braking later, extends the boom during its approach, and accelerates while the arm stows. None of these behaviors was specified as a subroutine or demonstrated by the baseline. They emerged from optimizing the complete task against a route-level objective. We use emergent in this limited sense. It describes unscripted coordination within the evaluated task, not a general capability outside the benchmark distribution.

The replays below reconstruct the trajectories produced by both policies on the same held-out route instance. Blue bins await service. Green bins have been serviced. These examples illustrate behavior but do not replace the aggregate evaluation above.

Training procedure

Training begins with behavior cloning from 160 demonstrations generated by the scripted baseline. The cloned policy provides both the initial parameters and a regularization reference on policy-visited states. The regularization weight decays to zero over training, after which the learned policy is unconstrained by the demonstrator. Standard PPO ^[3] then optimizes the unmodified benchmark reward. Training uses a vectorized simulator with 16,384 parallel environments. On one L40S GPU, the simulator processes two million steps per second. The full run accumulates 4.8 billion simulated steps in 77 minutes. This corresponds to approximately 7.5 years of continuous simulated operation. A staged curriculum increases the task from one house to two, then three, and finally eight. Deterministic success controls advancement between stages. Performance under exploration noise is not used for promotion.

Several failure modes informed the final task formulation. Early policies remained stationary because the immediate cost of motion outweighed rewards beyond the effective credit-assignment horizon. Another policy learned to terminate episodes by collision after an incorrectly calibrated penalty made failure less costly than route completion. Behavior cloning also failed when the demonstrator braked in response to bins beyond the policy's 30-meter sensing horizon. We addressed these failures by revising the task formulation. These revisions affected the observations, terminal costs, and information available to the demonstrator. We did not add scripted behaviors to the learned policy.

Speed and reliability improved at different stages of training. For most of the run, checkpoints tended to be fast or reliable rather than both. This gap closed late in training as exploration noise decreased. We attribute the pattern to safety margins learned under stochastic actions. During deterministic evaluation, those margins initially appeared as excess caution. The training and evaluation action distributions converged as exploration noise approached zero. The final checkpoint then achieved the best combination of speed and reliability observed during the run.

Scope and limitations

These results are specific to TrashBench v0 and should not be interpreted as evidence of real-world vehicle performance. This version uses a kinematic vehicle model and low-dimensional state observations. Evaluation is limited to one route topology with bin positions sampled from a defined distribution. TrashBench v0 does not model perception error or detailed hardware dynamics. It also omits weather, traffic, and pedestrians. Within this scope, joint policy optimization outperforms the modular baseline on route time and off-pavement driving. This report evaluates one trained checkpoint across variation in bin placement. It does not measure variation across independent training runs. Transfer and safety on physical vehicles remain to be demonstrated.

Why we do this

To put it plainly, we want the future to be better than the present. Waste collection is essential physical work. It also presents ongoing safety, staffing, and cost challenges. Our objective is to develop autonomous systems that reduce human exposure to hazardous tasks and improve the reliability of the service. Learned policies provide a way to improve system behavior as operating conditions and equipment change. They do not require every response to be specified in advance. TrashBench v0 offers a deliberately limited test of that premise. The present result is initial evidence from simulation rather than a conclusion about deployment.

We intend this work to be reproducible. The planned public release will include the benchmark specification, environments, and baselines. It will also include training and evaluation code, the exact configuration used for this result, evaluation seeds, and the final checkpoint. Subsequent TrashBench versions will use procedurally varied route topologies and place bins on both sides of the street. Later versions will also introduce parked and moving traffic, degraded sensing, and multi-vehicle coordination. We will publish new results here as the benchmark develops.

References

1. Cusumano-Towner, M., Hafner, D., Hertzberg, A., Huval, B., Petrenko, A., Vinitsky, E., Wijmans, E., Killian, T., Bowers, S., Sener, O., Krähenbühl, P., and Koltun, V. Robust Autonomy Emerges from Self-Play. 2025. arXiv:2502.03349.
2. Cornelisse, D., Hunt, J., Zhang, Z., Doulazmi, W., Joseph, K., Fernández Fisac, J., and Vinitsky, E. Human-like Autonomy Emerges from Self-Play and a Pinch of Human Data. 2026. arXiv:2606.19370.
3. Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. Proximal Policy Optimization Algorithms. 2017. arXiv:1707.06347.