Introduction

It is properly dragging to re-download the same packages after every small bug fix when rebuilding with docker compose build. Change one line in your code, and suddenly pip or npm is fetching the exact same dependencies it fetched five minutes ago. Wasteful, and it adds up to real hours over a week of active development. Luckily, there is a remedy.

BuildKit

Docker includes a powerful engine called BuildKit with quite a robust solution to build cache mounts. This allows creating a persistent cache for package managers, ensuring that dependencies are downloaded only once and reused across subsequent builds.

In short, we can use BuildKit to cache downloaded content, yes, it works! I use it myself, and I love it. Let’s explore the necessary changes to your docker-compose.yml and individual Dockerfiles using few practical examples.

How BuildKit Caching Works

Before BuildKit, Docker’s build cache was primarily layer-based. If a line in your Dockerfile changed, that layer and all subsequent layers would be rebuilt from scratch, including any RUN commands that download packages.

BuildKit introduces a more intelligent caching mechanism. One of its key features is the Use cache mounts in your Dockerfile. This mount type allows you to specify a directory within the build container that should be persisted across builds. When a package manager downloads files, they can be stored in this mounted cache directory. On subsequent builds, the cache is remounted, and the package manager can find the already downloaded files, skipping the lengthy download process.

Enabling BuildKit in Your Docker Compose Workflow

The good news: if you’re on the current docker compose CLI (the docker compose subcommand, not the old standalone docker-compose binary), BuildKit is already the default builder. You don’t need to switch anything on.

That DOCKER_BUILDKIT=1 environment variable you’ll see in older tutorials only matters if you’re still on the legacy docker-compose (v1, hyphenated) tool or calling plain docker build on an old Docker Engine. Docker deprecated v1 back in 2023, so if you’re running it, upgrading to docker compose gets you BuildKit for free:

# legacy docker-compose (v1) only — not needed on the current docker compose CLI
DOCKER_BUILDKIT=1 docker-compose build

On a current install, this is all you need:

docker compose build

A Practical Tutorial: Caching for Common Package Managers

Let’s dive into the practical implementation. We’ll cover how to modify your Dockerfiles to cache dependencies for apt (for Debian/Ubuntu based images), pip (for Python), and npm (for Node.js).

Caching apt Packages

For services built on Debian or Ubuntu, you can cache the apt package lists and the downloaded .deb files.

Before (Traditional Dockerfile):

FROM ubuntu:22.04

RUN apt-get update && apt-get install -y \
    python3-pip \
    nginx

# ... rest of your Dockerfile

In this traditional setup, if you add a new package, the entire RUN command is re-executed, and all packages are downloaded again.

After (with BuildKit Cache Mount):

To leverage BuildKit, you’ll introduce a --mount=type=cache.

# syntax=docker/dockerfile:1

FROM ubuntu:22.04

RUN --mount=type=cache,target=/var/cache/apt,sharing=locked \
    --mount=type=cache,target=/var/lib/apt,sharing=locked \
    apt-get update && apt-get install -y \
    python3-pip \
    nginx

# ... rest of your Dockerfile

Key Changes Explained:

• # syntax=docker/dockerfile:1: This “shebang” is crucial. It tells Docker to use a specific version of the Dockerfile syntax that supports BuildKit features. It must be the very first line of your Dockerfile. You can read more in the Dockerfile syntax documentation.
• --mount=type=cache,target=/var/cache/apt,sharing=locked: This mounts a cache volume at /var/cache/apt, which is where apt stores its package cache. The sharing=locked option ensures that multiple parallel builds using the same cache won’t corrupt it.
• --mount=type=cache,target=/var/lib/apt,sharing=locked: This caches the apt lists and other metadata stored in /var/lib/apt.

Caching pip Packages for Python

Python developers can significantly speed up the installation of pip packages.

Before (Traditional Dockerfile):

FROM python:3.11-slim

WORKDIR /app

COPY requirements.txt .
RUN pip install -r requirements.txt

COPY . .

CMD ["python", "app.py"]

After (with BuildKit Cache Mount):

# syntax=docker/dockerfile:1

FROM python:3.11-slim

WORKDIR /app

COPY requirements.txt .

RUN --mount=type=cache,target=/root/.cache/pip,sharing=locked \
    pip install -r requirements.txt

COPY . .

CMD ["python", "app.py"]

Key Changes Explained:

• --mount=type=cache,target=/root/.cache/pip,sharing=locked: This mounts a cache volume at the default pip cache directory (/root/.cache/pip for the root user, as per pip’s caching documentation). When pip install is executed, it will first check this directory for any existing packages before downloading them.

Caching npm Packages for Node.js

For Node.js applications, caching the npm cache folder is the recommended approach.

Before (Traditional Dockerfile):

FROM node:20

WORKDIR /app

COPY package*.json ./
RUN npm install

COPY . .

CMD ["npm", "start"]

After (with BuildKit Cache Mount):

# syntax=docker/dockerfile:1

FROM node:20

WORKDIR /app

COPY package*.json ./

# The default npm cache directory is /root/.npm on most Linux-based Node images
RUN --mount=type=cache,target=/root/.npm,sharing=locked \
    npm install

COPY . .

CMD ["npm", "start"]

Key Changes Explained:

• --mount=type=cache,target=/root/.npm,sharing=locked: Here, we are mounting a cache volume at npm’s default cache directory (/root/.npm for the root user). Npm uses this folder to store a global cache of packages, which prevents re-downloading them on subsequent npm install runs. You can find your npm cache location by running npm config get cache.

Orchestrating with docker-compose.yml

Now, let’s bring it all together with Docker Compose. While the primary changes are in your Dockerfiles, your docker-compose.yml can be adapted for more advanced caching strategies, such as sharing caches between different services or using a remote cache in a CI/CD environment.

Example docker-compose.yml:

services:
  web:
    build:
      context: ./web
      dockerfile: Dockerfile
      # The 'cache_from' attribute can be used for more advanced caching
      # strategies, like pulling a pre-warmed cache from a registry.
      # For local builds, the Dockerfile changes are the most impactful.
      # See: https://docs.docker.com/compose/compose-file/build/#cache_from
      cache_from:
        - my-registry/my-app-cache:latest

  api:
    build:
      context: ./api
      dockerfile: Dockerfile
    volumes:
      - ./api:/app
    ports:
      - "5000:5000"

Notice there’s no top-level version: key. Compose has ignored it since Compose V2 — if you still have one in your own files, it’s safe to delete; recent Compose versions will warn that it’s obsolete. There’s also no dedicated cache volume here: the caching described above lives entirely in the RUN --mount=type=cache lines inside each Dockerfile, so nothing needs declaring at the Compose level for it to work.

In this docker-compose.yml:

• We’ve defined two services, web and api, each with its own Dockerfile. The caching logic resides within those Dockerfiles.
• The build context for each service points to the directory containing its Dockerfile.
• The cache_from directive (shown for the web service) is a more advanced feature for pulling cache from remote sources like a Docker registry, which is particularly useful in automated build pipelines. You can learn more about it in the Docker Compose build reference. For local development, this is often not necessary.

Verifying the Cache is Working

The first time you run docker compose build after making these changes, you’ll see your packages being downloaded as usual. However, on subsequent runs, you should notice a significant speed improvement during the package installation steps. The output from Docker will indicate that it is using the cache, often showing CACHED next to the relevant build steps.

By embracing BuildKit’s caching capabilities, you can create a much more efficient and enjoyable development experience with Docker Compose. Say goodbye to unnecessary downloads and hello to faster, more streamlined builds.

References

1. What is Docker?

2. Docker Compose overview

3. Discover the possibilities with Meta Llama

4. llama-gpt