Parameterized Policies in Reinforcement Learning: From Tables to Networks

A policy network is not a bigger table. That is the mental model to dismantle before you touch policy-gradient code, because it quietly teaches you the wrong lesson about what changed.

A table stores one answer per state. A network stores a shared recipe that must produce answers for situations it has never seen. The difference is not storage size. It is the difference between memorization and generalization—and that shift changes what your policy can do, where it breaks, and how you debug it.

The Table That Stopped Scaling

In tabular reinforcement learning, a policy is a lookup table. Each state has an entry, and that entry says which action to take or how likely each action is. Simple. Exact. And completely stuck once the world stops being small enough to enumerate.

Consider what happens when states become high-dimensional images, sensor readings, or continuous positions. A table needs an entry for every possible observation. That is not a scaling problem. It is a wall. You cannot store one row per pixel configuration, and you cannot visit enough states to fill the table even if you could.

The reframe is simple: instead of storing an answer per state, store a shared recipe that computes an answer from any observation. That recipe is a parameterized policy—a function controlled by a set of adjustable parameters, usually the weights and biases of a neural network. Feed it an observation, and it produces an action or a distribution over actions.

Think of the difference as a phone book versus a rule. A phone book lists every person you might call. A rule reads the situation—who you need, where they are, what channel works—and derives the number. The phone book fails the moment you meet someone new. The rule handles novelty because it was never about memorizing individuals in the first place.

One clarification before we go further: I am using state when the observation fully describes the environment, and observation when referring to the actual input available to the policy. In many real systems the policy only sees partial information. The network consumes whatever input it is given, so keep that distinction in mind as we trace the mechanism.

What the Parameters Actually Represent

Here is where beginners get fuzzy: the parameters are not the policy's memory in the table sense. They are knobs that shape the transformation from observation to output.

A policy network takes an observation, pushes it through layers of weighted sums and nonlinearities, and produces an output. The weights and biases determine how that transformation behaves. Adjust them, and the same observation produces a different output. The policy's behavior changes because the transformation changed.

The critical property is parameter sharing. One set of weights is applied to every observation the agent encounters. There is no per-state storage. The same transformation that turns state A into a good action must also turn state B, which the agent has never seen, into a sensible action. That is why the policy can generalize: similar observations flow through the same learned transformation, so they produce similar outputs.

A table's memory is per-state. A network's memory is a compressed, shared rule. The table can be exactly right about one state and completely ignorant about its neighbor. The network cannot afford that luxury—it has one set of knobs, and those knobs must work everywhere.

Discrete Actions: Turning Observations into Probabilities

For a finite set of actions, the network's output layer produces one score per action. Those scores are then normalized into a probability distribution, typically through a softmax operation. The result is a stochastic policy: a distribution over actions that the agent samples from.

This connects directly to the stochastic policies you have already met. The distribution is what the agent samples from, which means the policy can explore while still favoring better actions. If the network assigns 0.7 probability to "move left" and 0.3 to "move right," the agent will usually move left but sometimes move right. That randomness is the exploration mechanism.

Notice what the network does not do: it does not pick the action. It outputs probabilities. The sampling decision stays outside the network, in the agent's action-selection step. This separation matters more than it seems, because it means the same observation always flows through the same weights, yet the agent can still behave differently across visits. The distribution is deterministic given the observation; the sampled action is not.

Continuous Actions: When a Probability Table Has No Meaning

Discrete actions map naturally to a probability distribution because there is a finite list to assign probabilities to. Continuous actions break that entirely. There is no finite list of joint angles or forces to enumerate. A probability table has no meaning when the action space is a range of real numbers.

The standard solution is to change what the network outputs. Instead of probabilities over discrete actions, the network outputs the parameters of a continuous distribution—typically a mean and a spread for a Gaussian. The agent then samples an action from that distribution.

Picture a robot arm. The policy does not pick from a menu of moves. It produces a continuous command: a joint angle, a force, a velocity. The network outputs the center of a distribution over possible commands and how spread out that distribution is. The agent samples one command from that distribution and executes it.

The representation shift is the key insight: the network still outputs distribution parameters, not the action itself. The sampling step remains separate. Whether the action space is discrete or continuous, the policy's job is to define a distribution. Only the shape of that distribution changes.

Note: A parameterized policy does not have to be stochastic. A deterministic parameterized policy maps an observation directly to an action—for example, a network that outputs a continuous control signal with no sampling step. This article emphasizes stochastic policies because they are central to policy-gradient exploration, but keep the broader concept in mind: parameterization is about representing the policy with adjustable parameters, and stochasticity is a separate design choice about how the policy explores and learns.

What Changes Compared with a Policy Table

Moving from a table to a parameterized policy is a real trade, not a pure upgrade. Let me be honest about both sides.

What you gain:

• Generalization. The policy produces sensible outputs for unseen observations because similar inputs flow through the same transformation.
• Scale. A compact set of parameters can represent policies for enormous or continuous state spaces that no table could hold.
• A path for gradient updates. Because the policy is a differentiable function of its parameters, you can compute how the parameters should shift to make better actions more likely. This is what policy-gradient methods exploit.

What you lose:

• Inspectability. A table entry is readable: state 42 says take action 3. A network's behavior is distributed across thousands of weights. You cannot look at a weight and understand what it does.
• Localized updates. A table update changes one entry and nothing else. A parameter update changes behavior across many observations at once. That is a strength when the change is good and a risk when it is not—one bad update can degrade behavior everywhere.

One caution about that "path for gradient updates" point: differentiable does not mean smooth in the everyday sense. A neural network can produce sharp changes in action probabilities as parameters shift, and training can be unstable. Gradient methods need usable local sensitivity, not a blanket promise of gentle behavior. This is why you monitor action distributions during training—sudden collapses or spikes tell you the update stepped somewhere unstable.

The practical rule: parameterized policies earn their keep when observations are too numerous or continuous to enumerate. A table is still fine for a tiny toy grid where you can visit every state. Do not reach for a network because it sounds more sophisticated. Reach for it because the state space demands generalization.

Common Misconceptions to Drop

Four mental-model errors show up constantly when learners move from tables to networks. Drop them now and save yourself hours of confusion later.

"The network picks the action." No. The network produces a distribution, or the parameters of a distribution. Sampling picks the action. This distinction matters when you debug why an agent behaves differently across visits to the same state—the network is deterministic; the sampling is not.

"More parameters mean a smarter policy." Parameters are only capacity, not capability. A larger network can represent more complex policies, but only if optimization actually finds good values for those parameters. Representation size is not learning progress.

"A parameterized policy is just a bigger table." This is the original misconception, and it is worth repeating: a table stores per-state answers; a network stores a shared transformation that must generalize. They fail in different ways. A table fails from missing entries. A network fails from a transformation that generalizes badly.

"Stochastic output means the policy is uncertain about the right answer." Sometimes it is, but often the randomness is deliberate. Stochasticity is the exploration mechanism—the way the agent tries actions it does not yet know are good. Treating randomness as hesitation misreads the design.

Before You Touch Policy-Gradient Code

The representation shift is the foundation everything else builds on. Policy-gradient methods adjust parameters to make good actions more likely and bad actions less likely. But none of that makes sense until you can trace what the parameters actually produce.

So before you write a single policy-gradient update, do this: take one observation and push it through a small policy network by hand. Identify the output shape. If the action space is discrete, check that the outputs form a probability distribution over actions. If it is continuous, identify the distribution parameters the network produces. Then ask yourself: which parameters would need to move to make a different action more likely?

That exercise turns the abstraction into something you can verify. Once you can trace an observation through the network and see exactly where the distribution comes from, the parameterized policy stops being a vague idea and becomes a concrete mechanism you control.