From Hand-Coded Rules to VLAs: What Robot Learning Measured, and What It Missed

Every stage of robot learning has been organised around a number measured somewhere other than where the robot has to work. Repeatability on a fixtured line. Trajectory error against a demonstration. Task success in a simulator. Position on a leaderboard. This is a history of methods and a history of proxies, and the gap between them is the part of the field we work on: a model scoring 95% in simulation can drop to 60% in production, and one lighting change can cost 5-10% accuracy.

Rules, and the environments built around them

Early robots ran explicit rules through modular pipelines, sensing, planning and control each in its own stage. They worked in tightly constrained settings, arms repeating one motion for years, and failed when the environment moved a centimetre. The benchmark of that era was honest, though: repeatability was measured under the conditions the robot would run in for its whole service life, so nothing was asked to transfer.

Rodney Brooks' subsumption architecture was the reaction to brittle top-down planning, layering simple, robust behaviours on each other instead of one monolithic planner. Brooks' deeper argument was about testing. Intelligence had to be shown in a physical environment rather than a symbolic model of one, and that claim has aged better than the systems built to prove it.

Teaching by demonstration

Programming every variation by hand does not scale, so researchers moved to Learning from Demonstration (LfD), using teleoperation and kinaesthetic teaching with trajectory regression to fit what was captured. Dynamical Movement Primitives (DMPs) made the result reusable, encoding a trajectory as a stable attractor-based pattern that can be modulated towards a new goal and combined with others, which is why dexterous manipulation leaned so heavily on them.

The first real proxy arrived with the evaluation. Scoring an LfD system means measuring how closely the reproduced trajectory matches the demonstration, and a robot can track a demonstration tightly and still fail because the object moved between teaching and running. Trajectory error tells you about the controller, not the task.

Pixels straight to motor commands

The next shift was to stop engineering the middle. End-to-end deep visuomotor policies mapped camera images directly to control outputs, the perception stack learned rather than specified.

That worked, and it cost something. Once perception lives inside the policy you cannot test it on its own. The old pipeline was brittle but legible: when a grasp failed you could ask whether the pose estimate was wrong or the planner was. An end-to-end policy gives you a success rate and nothing else.

Learning in simulation, and what randomisation admits

Reinforcement learning improves a policy by trial and error, but on hardware the trials are slow and expensive and the errors break things. Simulation solved throughput: episodes run in parallel by the thousand, and failure costs nothing.

OpenAI's robot hand solving a Rubik's Cube is the landmark result, combining massive simulated RL experience with automatic domain randomisation (ADR) to produce dexterous behaviour that transferred to real hardware. Be precise about what ADR is, though. Randomising visuals and physics across a wide distribution is not a bridge to reality; it is a hedge against not knowing which simulator parameters are wrong, because we cannot model contact and lighting well enough to trust any single setting. A score produced that way describes a distribution of imagined worlds. Your factory is one specific world.

Video of people doing things

Learning everyday tasks needs data of everyday tasks. EPIC-KITCHENS captured unscripted first-person kitchen activity; Ego4D scaled the idea to a far larger corpus of people handling objects in natural settings. Both were built to study human activity, and that purpose shows in what they contain. They record people who already know the task, with no account of the forces involved and nothing about what a robot should do after it drops something. The failures that decide a deployment are not what these corpora were collected to capture.

Transformers, and the leaderboard era

Transformers suit robot data for the reason they suit language: long-range dependencies over sequences, and few assumptions about what the tokens mean. Images, actions and instructions go into one stream. RT-1 showed the payoff, mapping camera images and natural-language instructions to actions across many tasks in one policy rather than one per task. Add language grounding of the SayCan kind, where a language model turns an instruction into subtask plans, and you have the modern Vision Language Action stack.

You also have leaderboards. Our stake in the argument is sharpest here, because we run evaluations for a living. Open-weight VLAs that match the proprietary models on public benchmarks fall apart zero-shot on real tasks they were never trained for. The ranking is not fraudulent. It measures the benchmark.

What a VLA system is actually made of

Underneath the acronym, a working system is an assembly of older parts. Behavioural cloning does most of the heavy lifting, supervised learning from demonstrations to predict the next action. Offline RL improves a policy on logged experience without fresh rollouts on hardware. Representation learning, usually self-supervised, gives the visual encoder embeddings that survive a change of scene. Language grounding turns an instruction into goals or subtask plans. Domain randomisation and system identification attack sim–real mismatch from opposite ends, widening the training distribution or measuring the real robot to narrow it. Hierarchical control keeps the generalist away from the torques: the VLA proposes a goal, a low-level controller executes it. Nothing there is new. What changed is that one model now sits over all of it and gets scored as a unit, which is convenient for publishing and unhelpful for diagnosis.

What still breaks

• VLA outputs still need mapping onto torques by safety-aware controllers, rebuilt per platform.
• Imitation copies unsafe human behaviour as faithfully as the useful kind, so deployed policies need constraints and verification over the top.
• Human data is vast and still misses the long tail; few-shot adaptation to a new site is hard.
• Policies overfit to the kinematics and sensors they trained on, and moving between morphologies is a research problem.
• Nobody can say why a VLA chose a given sequence, and published experiments are hard to reproduce.

What we would fix first

The consensus roadmap is scale plus modularity: larger multimodal pretraining for stronger priors, shared simulation environments for reproducible benchmarking, hierarchical systems pairing generalist front-ends with reliable low-level controllers, and fine-tuning on small amounts of real data guided by system identification and verification.

We would reorder it. Evaluation comes first, because everything else on that list is judged by it. Better simulators are worth building once we know how far their scores sit from real success rates, and the way to learn that is to run the same checkpoint in both and publish the difference, per-episode video included. A pass/fail number tells you a policy broke, not that it broke because the tray was reflective.

One lighting change costs 5-10% accuracy. Until an evaluation surfaces that before a cell is bolted to a floor, a leaderboard position is a claim about a simulator rather than about a factory.