This is Part 4 of the Neural Network Architecture series.

What Are Residual Connections in Transformers?

A residual connection is a skip path that adds a layer’s input to its output (x + F(x)), so gradients and signal can flow through depth unchanged. Transformers look like a new paradigm, but they still depend on the same core trick that made deep CNNs practical: skip connections. Residual learning was the breakthrough that made depth trainable in vision (He et al., 2015), and transformers inherit that same stability logic (Vaswani et al., 2017). If the identity-mapping intuition behind x + F(x) is new to you, my earlier post on neural network architecture covers it from first principles — here I assume that foundation and focus on what changes once attention enters the picture.

Without residual paths around attention and feed-forward layers, deep transformer stacks become fragile: gradients degrade, optimization becomes sensitive, and training can collapse.

Residual Connections in a Pre-LN Transformer Block

In a modern Pre-LN transformer block, the core pattern is:

\(x' = x + \text{Attention}(\text{LN}(x))\) \(y = x' + \text{MLP}(\text{LN}(x'))\)

A Pre-LN transformer block does not replace the old representation; it adds a learned update to it. Preserving signal while learning refinement is the entire point of residual design.

Pre-LN Transformer Block (single block)

           +---------------- Attention(LN(x)) ----------------+
           |                                                  v
x ---------+----------------------------------------------> (+) ----> x'

           +------------------ MLP(LN(x')) ------------------+
           |                                                  v
x' --------+----------------------------------------------> (+) ----> y

Why Residual Connections Keep Gradients Alive in Deep Transformers

If a block is $f(x) = x + g(x)$, then:

Even if $\frac{\partial g}{\partial x}$ is poorly conditioned, the identity term $I$ gives gradients a direct path through depth. This is why very deep residual stacks are trainable in practice (He et al., 2015), and why transformers rely on the same mechanism (Vaswani et al., 2017).

Gradient intuition through one residual block

Forward:  x ---> [ g(x) ] ---+
           \                 |
            \----------------+----> y = x + g(x)

Backward: dL/dy splits into:
          1) identity path  --------> direct gradient
          2) residual branch -------> transformed gradient

Pre-LN versus Post-LN, residual scaling, the PyTorch block and failure modes follow below.

References

🔒 Subscribe to keep reading.