This is Part 2 of the Neural Network Architecture series. We will build from foundational concepts to modern innovations like DeepSeek’s mHC architecture.

What is neural network architecture and why do residual connections matter?

Hello, dear friends! Today I want to take you on a journey through one of the most important concepts in modern AI: neural network architecture.

Neural network architecture is the structure of a network, meaning how its layers connect, how information flows and how gradients propagate, and it determines what the network can learn. A residual connection (also called a skip connection) is a shortcut that adds a block’s input to its output, which makes very deep networks far easier to train.

When people talk about AI breakthroughs, they often focus on training techniques, datasets, or compute power. But there is another dimension that is equally crucial: how we structure the network itself. The architecture—how layers connect, how information flows, how gradients propagate—determines what’s possible to build.

This post is the foundation for understanding modern architectural innovations such as Hyper-Connections and mHC. Before we can appreciate recent work like DeepSeek’s mHC architecture or other cutting-edge designs, we need to understand the building blocks they are built upon.

I’ll assume you have basic Python knowledge and have seen neural networks before (if not, you might want to start with my earlier post on artificial neural networks). What we explore today goes deeper: not just “what is a neural network?” but “how do we structure them effectively?”

By the end of this post, you will understand:

• Why network depth matters and what problems it creates
• How residual connections made very deep networks trainable
• Why researchers continue innovating on these foundations
• The mathematical intuition behind these architectural choices

Let us begin.

---

Part 1: Why do deep neural networks suffer from vanishing gradients?

Why do deeper neural networks learn more abstract features?

Neural networks transform data through layers of computation. Each layer learns to recognise increasingly abstract patterns. The list below is a conceptual illustration, not a sequence every network follows, and it mixes examples from image and text models. The layer-by-layer feature visualisations in Zeiler and Fergus (2014) show a similar trend for image models:

• Layer 1: Simple edges and textures (in images) or basic word patterns (in text)
• Layer 2: Combinations of edges forming simple shapes or phrase patterns
• Layer 3: Object parts or semantic relationships
• Layer 4+: Complete objects, concepts, or complex reasoning

Deeper networks can compose simple transformations into increasingly complex representations, which helps them model difficult patterns. Extra layers do not automatically give better accuracy, stronger reasoning or better generalisation, though: architecture, data, optimisation and compute budget all matter.

The vanishing-gradient problem, how ResNet fixed it and practical takeaways follow below.

Part 2: What is a residual connection in ResNet?

🔒 Subscribe to keep reading.

Part 3: How to build a ResNet-18 in PyTorch

🔒 Subscribe to keep reading.

Part 4: What did ResNet achieve on ImageNet and beyond?

🔒 Subscribe to keep reading.

Part 5: What are the limitations of residual connections?

🔒 Subscribe to keep reading.

Part 6: When should you use residual connections in your own networks?

🔒 Subscribe to keep reading.

Final thoughts: residual connections and what comes next

🔒 Subscribe to keep reading.

References

🔒 Subscribe to keep reading.