Sim-to-Real Reinforcement Learning: Why a Policy Breaks Outside the Simulator

A policy that scores perfectly in simulation is not a policy that works in the real world. It is a policy that solved the simulator.

That distinction sounds pedantic until you watch a robot that mastered a simulated assembly task fumble the same part on a real table. The gap is not a tuning problem. It is a distribution-shift problem, and until you treat it as one, you will keep chasing simulator fidelity when you should be questioning what your simulated scores actually measure.

The Simulator Is a Different MDP, Not a Cheaper Copy

Here is the formal frame that changes how you think about sim-to-real transfer: the simulator and the real world are two different Markov decision processes. They share a reward function but disagree on everything else that matters.

Let the real world be the MDP M = (S, A, P, R) and the simulator be M_sim = (S, A, P_sim, R). The state space, action space, and reward function are identical by design. The transition function is not. When you train a policy in M_sim, you are optimizing for value under the simulator's dynamics. When you deploy it, you are asking it to maximize value under P, the real transition function. Nothing about the training objective guarantees that the policy you found for P_sim is even reasonable under P.

Treat this shared-MDP equation as a useful baseline abstraction, not a complete description. Practical transfer often changes more than the transition function. Observation interfaces differ between rendered pixels and camera feeds. Action semantics shift when the simulator applies torque instantly but real motors have latency. Safety wrappers, termination conditions, and even the operational success measure may need separate scrutiny. The formal frame isolates the core problem; the taxonomy below shows where the abstraction stops being exact.

This is why the reality gap is not one thing. It is a family of mismatches, and each one breaks a policy differently:

• Dynamics mismatch: wrong physical parameters—inertia, friction, mass—even when the simulator's equations are correct. This is the system-identification problem.
• Observation mismatch: rendered images differ from real sensor data in texture, lighting, and noise. A vision policy trained on synthetic pixels can fail even with perfect dynamics.
• Actuation mismatch: the simulator's action model is cleaner than real motors, which have latency, dead zones, and noise.
• Contact and deformable-object gaps: contact-rich and deformable dynamics are computationally expensive to simulate accurately, so simulators approximate them in ways that change the task.

A high simulated score only proves one thing: the policy solves the simulator's MDP. It says nothing about whether the policy will transfer. This extends the held-out evaluation idea from generalization work to the case where the test distribution is the physical world—but with a sharper consequence. A held-out simulated environment still shares the simulator's dynamics. The real world does not.

Where the Reality Gap Actually Comes From

When a policy fails on the real system, the first question is not "how do I make the simulator more realistic?" It is "which mismatch broke this policy?" The fix differs depending on the answer.

Dynamics mismatch is the classic case. The simulator's equations of motion might be correct, but the parameters feeding them are estimates. Mass, center of gravity, friction coefficients, joint damping—each one is a guess. When the guess is wrong, the policy learns a control strategy tuned to the wrong physics. A manipulation policy that learned to grip with a certain force profile will fail if the real object is heavier or slicker than the simulated one.

Observation mismatch breaks vision policies even when the dynamics are perfect. If the policy maps pixels to actions, it learns features from synthetic images: specific textures, lighting conditions, camera noise patterns. Real camera images live in a different visual distribution. The policy may detect edges and shapes correctly but fail to generalize from rendered textures to real ones. This is why semantic label maps often transfer better than raw pixels—the gap between simulated and real label maps is smaller than the gap between simulated and real textures.

Actuation mismatch is subtler because it is invisible in simulation. Real motors have latency between command and response. They have noise, backlash, and saturation. The simulator's action model applies torque instantly and exactly. A policy that learned to exploit that precision will produce jittery or unstable behavior on hardware that cannot keep up.

Contact and deformable-object gaps are the hardest category because they are computational, not just parametric. Accurate contact simulation requires fine time steps and sophisticated solvers. Deformable objects need finite element methods that are too expensive for RL training loops. Simulators approximate, and the approximations create behaviors that only work in simulation.

There is also a category that surprises people: the policy may exploit simulator artifacts. Training sometimes produces remarkably dynamic and creative motions that look impressive in simulation. These behaviors often emerge from exploiting quirks of the simulated dynamics—a slightly too-forgiving contact model, an unrealistic friction cone, a physics bug. They do not transfer because they rely on the simulator being wrong in a specific way.

Name the dominant mismatch before choosing a strategy. If the problem is visual, dynamics randomization will not help. If the problem is contact, better rendering will not help.

Domain Randomization: Making the Real World Look Like Just Another Variation

The dominant strategy for crossing the reality gap is domain randomization. The core idea is simple: with enough variability in the simulator, the real world may appear to the model as just another variation from the training distribution.

There are two flavors. Dynamics randomization varies physical parameters—mass, friction, delays, action noise—across training episodes. Visual randomization varies rendered features—textures, lighting, camera noise. The policy never sees a fixed simulator. It sees a distribution of simulators, and it must learn behavior that works across all of them.

The simplest instantiation is injecting noise into actions or sensors. A stronger variant trains an adversary that perturbs the agent's actions during learning, forcing the policy to be robust to disturbances it did not choose.

Domain randomization does not remove the reality gap. It widens the training distribution so the real world falls inside it. That distinction matters because it reveals the tradeoff. Too little randomization leaves the policy brittle—it memorizes one simulator. Too much randomization can make the task too hard to learn, or worse, force the policy toward conservative behavior that avoids the randomization's worst cases rather than solving the task skillfully.

The art is choosing what to randomize and how much. Randomize the parameters that are uncertain and that matter for your task. Do not randomize everything indiscriminately; you will drown the learning signal.

A Transfer Workflow That Tests Behavior, Not Scores

A practical sim-to-real workflow separates two questions that beginners conflate:

1. Learning progress: does the policy improve in simulation?
2. Transfer readiness: does it behave correctly in reality?

The first question is about optimization. The second is about generalization across MDPs. A policy can answer yes to the first and no to the second. The simulated learning curve tells you the optimizer is working. It tells you nothing about whether the policy will survive contact with the real world.

A five-step workflow catches transfer failure before it becomes expensive. Each step produces evidence that shapes the next decision.

Step 1: Identify the dominant mismatch. Is the task vision-driven? Observation mismatch is the primary risk. Is it contact-rich manipulation? Dynamics and contact gaps dominate. Is it a high-speed control task? Actuation mismatch is the suspect. The dominant mismatch determines everything downstream.

Step 2: Choose the bridging strategy by what you can measure and afford. This decision is not a default; it is a cost-and-observability calculation.

• Calibrate when the uncertain parameters can be measured directly. Real-to-sim tuning uses real-world data to adjust simulator parameters, shrinking the gap rather than widening the training distribution. This works when you can instrument the system and the mismatch is parametric, not structural.
• Randomize when uncertainty is broad but bounded. If you cannot measure the exact friction coefficient but know it lives in a plausible range, randomization is the right tool.
• Abstract to higher-level actions when low-level dynamics are the bottleneck. If the policy outputs waypoints rather than torques, a low-level controller handles the dynamics that are hard to simulate. This trades control precision for transferability.
• Learn directly in the real world when interaction is safe, cheap, and fast enough. You remove the gap entirely by never introducing it.

Step 3: Define behavioral success criteria for the real system. Do not use simulator score thresholds. Define what correct behavior looks like on the real hardware: the object is grasped and held for N seconds, the robot reaches the target within tolerance, the system recovers from a specific disturbance. These criteria are what you will actually test.

Step 4: Run a narrow real-world probe early. Before investing in full deployment, test the policy on a small, safe subset of the task. This is the cheapest transfer failure detector you have. If the policy cannot perform the basic behavior on real hardware, no amount of simulated fine-tuning will fix it.

Step 5: Diagnose probe failures with a mismatch matrix, then iterate. When the probe fails, resist the urge to randomize everything and retrain. A real-world failure is evidence about what the simulator gets wrong. Map the observed behavior to a hypothesis, then update the simulator or the randomization distribution accordingly.

Consider a grasp-and-hold task. The probe fails in three different ways, and each points to a different cause:

The same outcome—a failed grasp—has different causes. The matrix forces you to inspect the failure trace before changing the training setup. If you randomize everything after the first failure, you will not know which randomization dimension fixed the problem or whether the fix was luck.

When the task involves humans or other actors that are genuinely hard to model, consider an iterative sim-to-real loop. The approach alternates between training in simulation and deploying in the real world, using each real deployment to improve the model of human behavior and the policy. This matters when the bottleneck is not physics but the unpredictability of the other agent.

When Sim-to-Real Is the Wrong Tool

Sim-to-real is a robustness strategy, not a universal requirement. It pays off when real-world data collection is expensive, unsafe, or slow. When those conditions do not hold, it can be overkill.

If the real system is cheap and safe to interact with, direct real-world RL may beat sim-to-real entirely. It removes the gap by never introducing it. You trade the simulator's sample efficiency for the reality of the actual task.

If the task depends on dynamics that are genuinely hard to simulate—deformable objects, tight human interaction—the simulator may be too inaccurate to justify the transfer effort. You would be training on a model that is wrong in ways you cannot fix with parameter randomization.

If the policy only needs to work in one narrow, well-characterized setting, system identification plus a calibrated simulator may be simpler than broad domain randomization. You do not need robustness to a wide distribution of simulators if you know exactly what the real system looks like.

The decision rule: use sim-to-real when the real world is expensive to interact with and the simulator captures enough of the task's essential dynamics. Otherwise, consider whether you are adding a transfer problem instead of solving one.

From Simulated Scores to Deployment Evidence

Simulation is a screening and diagnostic tool. It tells you whether the optimizer is learning, which mismatch is most likely to break the policy, and whether a randomization strategy produces behavior robust enough to justify a hardware test. It does not tell you whether the policy is ready to deploy. Real behavior under realistic conditions is the final criterion.

If you want to see this in action, run a small transfer experiment: train a baseline policy in a fixed simulator, train a domain-randomized policy in the same simulator with randomized parameters, and test both on a real or high-fidelity target. Compare their behavior, not their simulated scores. The baseline will likely look better in simulation. The randomized policy will likely survive contact with reality. That contrast is the whole lesson.

When the probe fails, resist the urge to randomize everything and retrain. A real-world failure is evidence about what the simulator gets wrong. Map the observed behavior to a hypothesis, then update the simulator or the randomization distribution accordingly.