Run a Soft Actor-Critic Experiment for Continuous Control

Your SAC implementation is probably correct. That is exactly why the failed run is so frustrating: the losses move, the entropy term behaves, and the agent still stands still. The bug is usually not in the algorithm. It is in the contract between the policy and the environment.

This is a diagnostic experiment, not another derivation. We will build a minimal runnable loop, log the signals that actually predict success or failure, and learn to read a broken run instead of restarting it and hoping.

What This Experiment Is Meant to Prove

The target is a small continuous-control task with a low-dimensional observation and action space, chosen so a full run finishes in minutes rather than hours. A pendulum-style swing-up task is the right size: one continuous action, a handful of observation dimensions, and a reward signal you can reason about without a plot.

I am assuming you already know the SAC loop. In one breath: the actor is a tanh-squashed Gaussian, the critic is a twin Q-network with a target copy, the Bellman backup is entropy-augmented, and the temperature is tuned automatically toward a target entropy. If any of those pieces are fuzzy, the mechanism is what matters here, not the notation — the diagnostics below will expose whether each piece is doing its job.

Dependencies and assumptions:

• A standard RL environment library for the task, and a deep-learning framework for the networks.
• Fixed seeds for the environment, action sampling, replay-buffer sampling, and network initialization.
• A single-machine run on CPU or GPU. No distributed setup, no vectorized environments.

Success criteria. A passing run reaches a stable return band across seeds, entropy settles near the target rather than collapsing or exploding, and critic loss stays bounded. A run that hits those three is a run you can trust enough to modify.

What this does not prove. It is not a benchmark comparison against TD3 or PPO. A single seed is not evidence of anything. And a result on one small task does not transfer to high-dimensional or sparse-reward domains without re-testing.

Minimal Runnable SAC Loop

Structure the loop as four visible stages, in this order:

1. Collect transitions into the replay buffer.
2. Sample minibatches.
3. Update the two critics against the entropy-augmented target.
4. Update the actor and the temperature.

Most silent bugs live in what is updated when, not in the loss formula. Keep the update order and the target-network soft update explicit in the code, not buried in a helper.

for step in range(total_steps):
    if step < warmup_steps:
        action = env.action_space.sample()
    else:
        action = actor.sample(obs)

    next_obs, reward, done, _ = env.step(action)
    buffer.add(obs, action, reward, next_obs, done)
    obs = next_obs if not done else env.reset()

    if step >= warmup_steps and step % update_every == 0:
        batch = buffer.sample(batch_size)

        # 1. Critic update against the entropy-augmented target.
        with torch.no_grad():
            next_action, next_logp = actor.sample(batch.next_obs)
            q_target = torch.min(
                target_q1(batch.next_obs, next_action),
                target_q2(batch.next_obs, next_action),
            ) - alpha * next_logp
            y = batch.reward + gamma * (1 - batch.done) * q_target

        critic_loss = mse(q1(batch.obs, batch.action), y) \
                    + mse(q2(batch.obs, batch.action), y)

        # 2. Actor update: maximize Q minus temperature-weighted log-prob.
        action, logp = actor.sample(batch.obs)
        actor_loss = (alpha * logp - torch.min(q1(batch.obs, action),
                                               q2(batch.obs, action))).mean()

        # 3. Temperature update toward target entropy.
        alpha_loss = -(alpha * (logp + target_entropy).detach()).mean()

        # 4. Soft-update the target networks.
        soft_update(target_q1, q1, tau)
        soft_update(target_q2, q2, tau)

Log per-update scalars from the first step: critic losses, actor loss, mean Q estimate, policy entropy, temperature, and mean absolute action magnitude. Those six numbers are the entire diagnostic surface for this experiment.

Seed everything that matters — environment, action sampling, replay-buffer sampling, and network initialization. An unseeded run is not a result; it is a coincidence.

The knobs that change run length: warm-up steps before learning starts, updates per environment step, and buffer size. Warm-up matters because the critics need data before the actor can trust them. Updates per environment step controls how much gradient work happens per unit of experience.

Reading the Actor, Critic, and Entropy Signals

The traces are evidence about the mechanism, not decoration. Read them as claims the system is making about itself.

The gap between the two critics is a disagreement signal. A small, stable gap means both critics agree on the value landscape. A persistent large gap means one of them is chasing a target the other rejects — often a sign that target smoothing is too aggressive or the reward scale is pushing Q values into a range the networks cannot track.

Policy entropy should drift toward the target entropy rather than collapse. The temperature trace tells you whether automatic tuning is doing its job: if entropy is below target and temperature is not rising, the tuning loop is broken, not the policy.

The classic false positive is mean Q rising while episode return stays flat. That is a critic that has learned to be optimistic, not a policy that has learned to act. Do not celebrate a rising Q curve until the return curve agrees with it.

Action magnitude and saturation rate tell you whether the squashed policy is actually using its range or pinning to the bounds. A policy that always outputs near the limits is not exploring; it is stuck against a wall.

Action Scaling: The Failure That Looks Like a Learning Failure

This is the most common practical break in continuous-control SAC, and it is the one that wastes the most time because every other signal looks fine.

The policy emits a squashed action in a normalized range — typically [-1, 1] after the tanh. The environment expects a specific bounded range, which may be different. If that rescaling is missing, wrong, or applied twice, the agent explores a region the task never rewards.

The symptom pattern is distinctive: entropy looks healthy, critic loss looks healthy, returns stay at the random-policy floor, and action magnitudes cluster at one extreme. Every diagnostic says the algorithm is working. The agent is just acting in the wrong units.

Contrast that with a genuine learning failure:

• Entropy collapsing early points at temperature tuning or an actor loss that is over-weighting Q.
• Critic loss diverging points at target-network updates, reward scale, or learning rate.
• Returns rising then collapsing points at overestimation catching up with the policy.

Each of those has a different cause. Action scaling has none of those signatures — which is exactly why it is easy to miss.

Check the boundary conditions explicitly. What happens at the action limits? Does the environment clip, wrap, or reject out-of-range actions? A silent clip at the boundary will hide the bug: the agent thinks it is exploring, the environment quietly truncates, and the reward signal never reflects the policy's actual output.

Common mistake: Applying the rescaling inside the actor's sample() method and again in the environment wrapper. The action gets squashed twice, the effective range shrinks, and the policy can never reach the task's extremes.

A short verification step catches this before training. Feed a fixed observation through the actor and confirm the resulting action lands inside the environment's declared bounds:

obs = env.reset()
action = actor.sample(obs, deterministic=True)
print("action:", action)
print("env bounds:", env.action_space.low, env.action_space.high)
assert env.action_space.contains(action), "action outside environment bounds"

Run this once. If it fails, you have found your bug in thirty seconds instead of thirty thousand steps.

Evaluating Across Seeds, Not Episodes

A single run is not a result. SAC variance across seeds is often larger than the difference between algorithms, so a protocol that reports one seed is reporting noise.

Run at least three seeds and plot the spread, not just the mean. Evaluate with a deterministic or mean action on a fixed evaluation schedule, and keep evaluation separate from the training-time stochastic policy. Mixing the two makes the return curve unreadable.

Report the return band and the number of environment steps to reach it. Sample efficiency is the claim SAC is usually making, so the step count matters as much as the final return.

Distinguish a run that is slow from a run that is broken. A slow run still trends upward across seeds; a broken one does not. If all three seeds are flat, treat that as a strong signal to check the action contract and the critic-actor relationship before assuming a hyperparameter problem. If one seed is flat and two are climbing, you have variance — which is a different problem with a different fix.

Keep the evaluation protocol fixed when you change hyperparameters, or you will be comparing noise to noise.

One Modification Worth Running

Turn the experiment into a controlled comparison so you learn something causal rather than just getting a working script.

Pick one variable and change only that. Good candidates: reward scale, target entropy, or the number of gradient steps per environment step.

Predict the outcome before running. For example: raising the reward scale without adjusting temperature should push the policy toward exploitation and lower entropy, because the reward term now dominates the entropy term in the actor loss.

Compare the diagnostic traces, not just the final return. The interesting result is usually in how the curves changed shape — did entropy settle lower? Did the critic gap widen? Did the return band tighten or spread?

Record what surprised you. A prediction that failed is the most useful output of the experiment, because it means your mental model of the mechanism was wrong somewhere specific.

Note the boundary: conclusions from one small task do not transfer to high-dimensional or sparse-reward domains without re-testing. The mechanism generalizes; the hyperparameters do not.

What to Do When a Run Fails

When a SAC run fails, check action scaling and the critic-actor signal relationship before touching hyperparameters. Those two checks catch the majority of failures that look like learning problems but are not.

Treat the diagnostic traces as the primary evidence. A reproducible run with visible signals is worth more than a lucky one, because you can debug it, modify it, and trust the result.

The next practical step is to extend the same instrumentation to a harder continuous-control task — more action dimensions, a longer horizon — and watch which signals break first. Or compare against a deterministic actor-critic baseline on the same task and see whether the entropy term is earning its keep. Either way, keep the logging. The signals are the experiment.