Deep Reinforcement Learning Explained: What Neural Networks Add

When you hear "deep reinforcement learning," it's tempting to picture a neural network that somehow learned to think for itself. The network isn't doing the reinforcement learning. It's the engine of representation inside a learning loop that was already well-defined before deep learning existed.

The Confusion at the Heart of "Deep RL"

Here's the mental model most beginners carry: deep RL is a new kind of learning, and the neural network is what makes it work. The network "figures out" the environment. The network "learns" the game. The network is the intelligence.

That model is wrong in a useful way.

Reinforcement learning is the loop: the agent observes a state, takes an action, receives a reward, and updates its understanding. That loop existed for decades before neural networks became practical. Q-learning, temporal difference methods, and policy gradients all define how an agent should learn from interaction. None of them require a neural network to function.

What deep RL actually changes is narrower. It swaps the storage system inside that loop. Instead of a lookup table that holds one value per state-action pair, a neural network estimates those values from input. The learning loop stays the same. The update rules stay the same. Only the representation scales.

So when you hear "deep RL," the useful question is not "what new algorithm is this?" It's "what does the network represent, and which RL update loop is training it?"

From Lookup Table to Learned Function

If you've worked through Q-learning, you know the tabular setting. A Q-table stores one value for every state-action pair. The agent looks up the value, picks the best action, and updates the entry after observing the reward. This works beautifully when states are few and enumerable. A grid world, a simple card game, a small maze.

Real environments break the table. A single game frame is a grid of pixels. A robot's sensor readings are continuous values. The number of possible states is astronomically large, and most of them the agent will never visit twice. A table cannot store what it cannot enumerate, and it cannot generalize to states it has never seen.

The function approximator idea solves this by changing what gets stored. Instead of a table entry for every state, you learn a function that takes a state as input and outputs an estimate. The function is compact. It can produce values for states it has never encountered, because it learns shared features rather than memorizing individual entries.

This is where the neural network enters. In value-based deep RL, the network estimates Q(s, a) or V(s) — the expected future reward from a state or state-action pair. In policy-based deep RL, the network outputs action probabilities or a continuous action directly. Either way, the network is a parameterized function that maps input to output. Training adjusts its weights so the outputs match the same targets tabular RL already used.

Think of a game frame. The network doesn't store a value for "this exact arrangement of pixels." It learns to detect edges, shapes, and patterns — features that predict which actions lead to reward. When a new frame arrives that resembles past frames in useful ways, the network can estimate values for it without ever having seen that exact image before. That generalization is the entire point.

What the Network Actually Learns

Here is the precise mental model: the network is a parameterized function, and training adjusts its weights so its outputs match targets that RL already defined.

For value-based methods, those targets come from the Bellman-style update you already know from Q-learning. The network predicts Q-values. After taking an action and observing a reward and the next state, the agent computes a target using the reward plus the discounted best estimate from the next state. Then it adjusts the network's weights to make its prediction closer to that target. The update rule is the same shape as tabular Q-learning. Only the storage mechanism changed.

For policy-based methods, the network outputs a policy — a distribution over actions. Training nudges the network toward actions that led to higher returns and away from actions that led to lower ones. Again, the learning signal comes from the RL loop, not from the network itself.

The network's real contribution is generalization. It can estimate values or actions for states it has never seen because it learns shared features across states. Two visually different game frames might both contain an enemy approaching from the left. The network learns that the feature "enemy on the left" predicts danger, and it generalizes that knowledge to new frames that share the feature.

Common mistake: Assuming the network "figures out" the environment on its own. It doesn't. The network needs the RL loop to supply targets and experience. Without the loop, the network has nothing to learn from. The network is a representation engine, not a learning rule.

Why Bigger Representation Does Not Fix the Hard Parts

Here is where beginners get disappointed, and where the field's real difficulty lives. Scaling representation solves the state-space problem. It does not solve the learning-loop problems.

Exploration. A network still needs to try actions to discover reward. Deep RL does not tell the agent where to look. If a game rewards a rare sequence of actions, the agent must stumble onto that sequence through trial and error. A bigger network does not make the agent more curious. It just represents what the agent has already learned more flexibly.

Credit assignment. Delayed rewards still need to be traced back to the actions that caused them. If you win a game after fifty moves, which of those fifty moves mattered? The network does not make this attribution automatic. The RL algorithm still has to propagate reward backward through the sequence of decisions, and that problem gets harder, not easier, as the state space grows.

Instability. Neural networks can forget what they learned. During RL training, the data distribution shifts as the agent's policy changes, and the network can oscillate or catastrophically forget earlier lessons. This is why deep RL adds tricks that tabular RL never needed: replay buffers to reuse past experience, target networks to stabilize the learning target, and careful tuning of exploration. These are not optional flourishes. They are the engineering that makes deep RL work at all.

None of these are failures of the concept. They are the real problems you will meet next, and they are worth meeting with the right mental model: the network scales representation, but the RL loop still carries the learning burden.

Deep Learning Versus Reinforcement Learning: Two Jobs, One Loop

The boundary between deep learning and reinforcement learning is worth making crisp, because the two fields are constantly conflated.

Deep learning learns patterns from a fixed dataset. You collect data, label it, and train a network to map inputs to outputs. The data does not change based on the network's behavior. The learning signal is the difference between the network's prediction and the label.

Reinforcement learning learns decisions from interaction. There is no fixed dataset. The agent's actions change what it experiences next. The learning signal is reward, which may arrive long after the action that caused it.

In deep RL, each field contributes one engine. Deep learning supplies the representation and generalization: the ability to take raw pixels or sensor readings and extract useful features. RL supplies the objective, the experience loop, and the update signal: the definition of what "good" means, the mechanism for gathering experience, and the rule for improving decisions.

So no, deep RL is not just deep learning applied to games. The learning loop and the reward signal are what make it RL. If you removed the interaction loop and trained the network on a fixed dataset of game frames with labels, you would have a supervised learning problem, not an RL problem. The network is the engine of representation. RL is the engine of decision.

Where to Go Next in Your RL Path

The durable takeaway is a two-part question you can carry into every deep RL algorithm you meet:

1. What does the network represent? A value function, a policy, or a model of the environment?
2. Which RL update loop is training it? Is it learning from a Q-learning target, a policy gradient, or something else?

Answer those two questions and you can orient yourself inside almost any deep RL paper or library.

The natural next concepts are policy gradients and actor-critic methods. Policy gradients train the network to output actions directly, which handles continuous action spaces that Q-learning struggles with. Actor-critic methods combine both ideas: one network learns a policy, another learns a value function to guide it. After that, you will meet the practical tricks — replay buffers, target networks, and exploration schedules — that make deep RL stable enough to use.

Here is a mental exercise to solidify the model before you move on. Take a tabular Q-learning algorithm you already understand. Now imagine swapping its table for a neural network. What changes? The storage mechanism, the ability to generalize to unseen states, and the need for stability tricks. What stays the same? The exploration problem, the credit-assignment problem, and the Bellman update that drives learning.

That exercise is the whole conceptual foundation of deep RL. The rest is engineering.