Neural Network Architecture Series

This series explores how neural network architecture evolved from residual connections to modern experiments with dense links, transformers, and automated design. Each post builds on the previous one while still standing on its own.

Series Progress

All Posts in This Series

Part 0: Artificial Neural Networks 17 Dec, 2021 Artificial neural networks (ANNs) are the cornerstone of Deep Learning algorithms. The name and the architecture are adopted from the human brain's neural network. ANNs are designed to simulate human reasoning based on how neurons communicate. ANNs contain a set of artificial neurons connected.
Part 1: Understanding Neural Network Architecture: From Basics to Residual Connections Coming Soon Before diving into cutting-edge architectural innovations, understand the foundations: how neural networks are structured, why depth creates problems, and how residual connections solved them. This post is currently being written and will be published soon.
Part 2: DeepSeek's mHC: Making Neural Networks Learn Better by Preserving Identity Coming Soon DeepSeek's mHC architecture solves a fundamental tension in neural networks: how to allow rich information flow between layers while maintaining training stability. The result is measurably better performance on reasoning tasks. This post is currently being written and will be published soon.
Part 3: Dense Connections and Information Flow: From DenseNet to Modern Variants Coming Soon Explore dense connectivity patterns, why they improve gradient flow, and when dense connections outperform residual links. This post is currently being written and will be published soon.
Part 4: Attention Mechanisms Meet Residual Connections: How Transformers Use Skip Connections Coming Soon A practical guide to residual connections in transformers: equations, Pre-LN vs Post-LN trade-offs, failure modes, and implementation details for stable training. This post is currently being written and will be published soon.
Part 5: Neural Architecture Search: Automating the Discovery of Better Connections Coming Soon How NAS explores connection patterns and what it reveals about architecture design. This post is currently being written and will be published soon.
Part 6: Bottlenecks, Inverted Residuals, and Mobile Architectures Coming Soon How mobile architectures adapt residual designs to fit edge devices. This post is currently being written and will be published soon.
Part 7: The Future of Connections: From Manifold Constraints to Dynamic Routing Coming Soon A survey of emerging connection patterns and where architecture research may head next. This post is currently being written and will be published soon.
Part 8: Transformers in Practice: From Architecture to a Chatbot Implementation Coming Soon A practical bridge from transformer architecture concepts to implementation: build and evaluate a chatbot workflow with TensorFlow and HuggingFace, with clear trade-offs and deployment guidance. This post is currently being written and will be published soon.

Getting Started

New to this series? Start with Part 0: Artificial Neural Networks.

Each post builds on the previous one, so reading them in order is recommended. Each post also works on its own if you need to jump to a specific topic.