Why Docker Compose rebuilds keep re-downloading the same packages

Re-downloading the same packages after every small bug fix is a proper drag when you rebuild with docker compose build. Change one line in your code, and suddenly pip or npm is fetching the exact same dependencies it fetched five minutes ago. Wasteful, and it adds up to real hours over a week of active development. Luckily, there is a remedy.

What BuildKit is and how it caches package downloads

BuildKit is the build engine bundled with modern Docker that adds a cache mount, a persistent directory it re-attaches to a RUN instruction on every build. A cache mount is a build-time storage primitive that survives layer invalidation, so a package manager downloads a given dependency once and reuses it across every subsequent build — even builds where the layer itself has to re-run. That last clause is the whole point, and it is the part Docker’s own BuildKit documentation buries.

I use cache mounts on every containerised project I work on, and they cost three lines of Dockerfile. Below are the changes to your docker-compose.yml and individual Dockerfiles, with worked examples for apt, pip, and npm.

How BuildKit cache mounts work

Before BuildKit, Docker’s build cache was primarily layer-based. If a line in your Dockerfile changed, that layer and all subsequent layers would be rebuilt from scratch, including any RUN commands that download packages.

BuildKit adds a second, finer-grained mechanism on top of layer caching: the cache mount, declared as RUN --mount=type=cache,target=<dir>. The mount names a directory inside the build container that BuildKit stores outside the image and re-attaches on every build. When pip, npm, or apt writes a downloaded archive into that directory, the archive stays on the builder’s disk. On the next build, BuildKit remounts the same directory, the package manager finds its own cache already populated, and it skips the download entirely.

Docker layer caching versus BuildKit cache mounts

Cache mounts and layer caching are complementary mechanisms that fail in different places, and conflating them is the most common reason people think cache mounts “do not work”.

The classic layer cache preserves an entire resulting image layer, so long as none of its inputs changed. Order your Dockerfile so COPY requirements.txt . and RUN pip install -r requirements.txt sit in their own layer, and Docker skips that layer entirely on the next build, provided requirements.txt has not changed. The moment it does change, the whole layer invalidates, and pip runs from a cold start, re-downloading everything.

A cache mount does not preserve the layer at all. A cache mount preserves the downloaded package files sitting on disk inside the mount, independently of whether the layer itself gets rebuilt. So even when requirements.txt changes and that RUN layer invalidates and re-executes, pip still finds most of what it needs already sitting in /root/.cache/pip, and only fetches what genuinely changed. The two mechanisms are not competing; keep your layer ordering sensible, and let the cache mount handle the case layer caching cannot.

How to enable BuildKit in Docker Compose (DOCKER_BUILDKIT=1 is no longer needed)

The good news: if you’re on the current docker compose CLI (the docker compose subcommand, not the old standalone docker-compose binary), BuildKit is already the default builder. You don’t need to switch anything on.

That DOCKER_BUILDKIT=1 environment variable you’ll see in older tutorials only matters if you’re still on the legacy docker-compose (v1, hyphenated) tool or calling plain docker build on an old Docker Engine. Docker ended support for Compose V1 in June 2023 and pulled it from later Docker Desktop releases, so if you’re still running it, upgrading to docker compose gets you BuildKit for free:

# legacy docker-compose (v1) only — not needed on the current docker compose CLI
DOCKER_BUILDKIT=1 docker-compose build

On a current install, this is all you need:

docker compose build

BuildKit cache mount targets for apt, pip, and npm

Each package manager keeps its downloads in a different directory, so each needs its own cache mount target. The three worth knowing are apt (Debian/Ubuntu base images), pip (Python), and npm (Node.js):

How to cache apt packages in a Debian or Ubuntu Dockerfile

On Debian and Ubuntu base images, two directories are worth mounting: /var/cache/apt holds the downloaded .deb archives, and /var/lib/apt holds the package lists apt-get update fetches.

Before (Traditional Dockerfile):

FROM ubuntu:22.04

RUN apt-get update && apt-get install -y \
    python3-pip \
    nginx

# ... rest of your Dockerfile

In this traditional setup, if you add a new package, the entire RUN command is re-executed, and all packages are downloaded again.

After (with BuildKit Cache Mount):

The fix is one extra flag on the RUN line, --mount=type=cache:

# syntax=docker/dockerfile:1

FROM ubuntu:22.04

# Debian/Ubuntu images ship a docker-clean hook (/etc/apt/apt.conf.d/docker-clean)
# that deletes every downloaded .deb the moment apt-get finishes, and apt itself
# discards archives after a successful install. Left as-is, the cache mount below
# is silently emptied on every single build.
RUN rm -f /etc/apt/apt.conf.d/docker-clean && \
    echo 'Binary::apt::APT::Keep-Downloaded-Packages "true";' > /etc/apt/apt.conf.d/keep-cache

RUN --mount=type=cache,target=/var/cache/apt,sharing=locked \
    --mount=type=cache,target=/var/lib/apt,sharing=locked \
    apt-get update && apt-get install -y \
    python3-pip \
    nginx

# ... rest of your Dockerfile

Key Changes Explained:

• rm -f /etc/apt/apt.conf.d/docker-clean: This is the step most BuildKit-and-apt tutorials skip, and it quietly wastes the whole exercise if you skip it too. Debian and Ubuntu base images ship that file specifically to keep images small: it hooks DPkg::Post-Invoke and deletes every downloaded .deb the moment apt-get finishes — so the next build starts from an empty cache again. Removing it is only half the job, though. Modern apt also discards downloaded archives after a successful install on its own, which is why the echo line sets Binary::apt::APT::Keep-Downloaded-Packages "true" — the exact incantation Docker’s own RUN --mount=type=cache reference uses for this. Skip that line and your /var/cache/apt mount stays stubbornly empty, however many times you rebuild.
• # syntax=docker/dockerfile:1: This “shebang” pins the Dockerfile syntax version. On a reasonably current Docker Engine or Docker Desktop, the built-in BuildKit frontend already understands --mount=type=cache without it — but pinning still buys you reproducibility across engine versions, plus access to newer Dockerfile features as they land. When you do include it, it must be the very first line of your Dockerfile. More in the Dockerfile syntax documentation.
• --mount=type=cache,target=/var/cache/apt,sharing=locked: This mounts a cache volume at /var/cache/apt, which is where apt stores its package cache. sharing=locked is not optional here — dpkg takes an exclusive lock on its database, so concurrent access to this cache without locking risks corrupting it.
• --mount=type=cache,target=/var/lib/apt,sharing=locked: This caches the apt lists and other metadata stored in /var/lib/apt, locked for the same reason.

How to cache pip downloads in a Python Dockerfile

Python developers can significantly speed up the installation of pip packages.

Before (Traditional Dockerfile):

FROM python:3.11-slim

WORKDIR /app

COPY requirements.txt .
RUN pip install -r requirements.txt

COPY . .

CMD ["python", "app.py"]

After (with BuildKit Cache Mount):

# syntax=docker/dockerfile:1

FROM python:3.11-slim

WORKDIR /app

COPY requirements.txt .

RUN --mount=type=cache,target=/root/.cache/pip,sharing=locked \
    pip install -r requirements.txt

COPY . .

CMD ["python", "app.py"]

Key Changes Explained:

• --mount=type=cache,target=/root/.cache/pip,sharing=locked: This mounts a cache volume at the default pip cache directory (/root/.cache/pip for the root user, as per pip’s caching documentation). When pip install is executed, it will first check this directory for any existing packages before downloading them.
• On sharing=locked: unlike apt, pip has no exclusive-lock requirement, so the default sharing=shared works fine for a single build. Reach for sharing=locked only if you expect several build stages to hit this same cache id at once.

How to cache npm packages in a Node.js Dockerfile

For Node.js applications, caching the npm cache folder is the recommended approach.

Before (Traditional Dockerfile):

FROM node:20

WORKDIR /app

COPY package*.json ./
RUN npm install

COPY . .

CMD ["npm", "start"]

After (with BuildKit Cache Mount):

# syntax=docker/dockerfile:1

FROM node:20

WORKDIR /app

COPY package*.json ./

# The default npm cache directory is /root/.npm on most Linux-based Node images
RUN --mount=type=cache,target=/root/.npm,sharing=locked \
    npm install

COPY . .

CMD ["npm", "start"]

Key Changes Explained:

• --mount=type=cache,target=/root/.npm,sharing=locked: Here, we are mounting a cache volume at npm’s default cache directory (/root/.npm for the root user). npm uses this folder to store a global cache of packages, which prevents re-downloading them on subsequent npm install runs. You can find your npm cache location by running npm config get cache.
• As with pip above, sharing=locked is a safety margin here rather than a requirement — the default sharing=shared is fine unless you have concurrent writers to the same cache id.

Cache mount permissions when building as a non-root USER (uid and gid)

Production Dockerfiles increasingly drop root inside the build itself, running as USER node or a dedicated appuser for the sake of least privilege. Cache mounts do not automatically follow that switch: BuildKit mounts a cache directory owned by root by default, regardless of which USER is active when the RUN instruction executes. Install as a non-root user without accounting for this, and npm install or pip install fails with a plain permission denied on its own cache directory.

The fix is to tell the mount which user should own it, with uid and gid:

# syntax=docker/dockerfile:1

FROM node:20

WORKDIR /app
USER node

COPY --chown=node:node package*.json ./

RUN --mount=type=cache,target=/home/node/.npm,uid=1000,gid=1000 \
    npm install

COPY --chown=node:node . .

CMD ["npm", "start"]

Key Changes Explained:

• USER node: switches to the image’s non-root node user before installing, following the least-privilege habit you’d want in production anyway.
• uid=1000,gid=1000: matches the official node image’s node user, so BuildKit creates and owns the cache directory as that user rather than root — uid and gid both default to 0, which is exactly the problem. Swap these for whatever id -u/id -g reports for your own non-root user if you’re not on the standard node image.
• target=/home/node/.npm: points at the non-root user’s own cache directory, not /root/.npm — root’s cache is a different, inaccessible path once you’ve dropped privileges.
• COPY --chown=node:node: keeps the copied files owned by the same user, so nothing else in the build trips over a permissions mismatch.

Configuring docker-compose.yml for cached builds

Now, let’s bring it all together with Docker Compose. While the primary changes are in your Dockerfiles, your docker-compose.yml can be adapted for more advanced caching strategies, such as sharing caches between different services or using a remote cache in a CI/CD environment.

Example docker-compose.yml:

services:
  web:
    build:
      context: ./web
      dockerfile: Dockerfile
      # The 'cache_from' attribute can be used for more advanced caching
      # strategies, like pulling a pre-warmed cache from a registry.
      # For local builds, the Dockerfile changes are the most impactful.
      # See: https://docs.docker.com/compose/compose-file/build/#cache_from
      cache_from:
        - my-registry/my-app-cache:latest

  api:
    build:
      context: ./api
      dockerfile: Dockerfile
    volumes:
      - ./api:/app
    ports:
      - "5000:5000"

Notice there’s no top-level version: key. Compose has ignored it since Compose V2, and the Compose file reference now calls it obsolete outright — if you still have one in your own files, it’s safe to delete, and recent Compose versions will warn you about it anyway. There’s also no dedicated cache volume here: the caching described above lives entirely in the RUN --mount=type=cache lines inside each Dockerfile, so nothing needs declaring at the Compose level for it to work.

In this docker-compose.yml:

• We’ve defined two services, web and api, each with its own Dockerfile. The caching logic resides within those Dockerfiles.
• The build context for each service points to the directory containing its Dockerfile.
• The cache_from directive (shown for the web service) is a more advanced feature for pulling cache from remote sources like a Docker registry, which is particularly useful in automated build pipelines. You can learn more about it in the Docker Compose build reference. For local development, this is often not necessary.

Pruning cache mounts and using BuildKit caching in CI/CD

Cache mounts are not free. A cache mount lives on whichever machine or builder ran the build and grows quietly in the background, so learn to see and clear them before your disk does it for you:

docker builder du
docker builder prune --filter type=exec.cachemount

The type=exec.cachemount filter targets only the RUN --mount=type=cache volumes, leaving the rest of your build cache alone.

One distinction matters before you rely on any of this in CI: a BuildKit cache mount is builder-local storage, not a portable artefact. A cache mount is a different mechanism to BuildKit’s exported build cache, the one you configure with --cache-to/--cache-from, including the type=gha backend for GitHub Actions. A GitHub-hosted runner starts from a clean virtual machine on every run, so a RUN --mount=type=cache volume from the previous run simply is not there any more. Docker’s own documentation puts it plainly: BuildKit doesn’t preserve cache mounts in the GitHub Actions cache by default — type=gha caches your image layers, not the contents of your cache mounts. If you genuinely need package-manager caching to survive across ephemeral CI runners, you have three options: a self-hosted, persistent builder; a CI-native dependency-caching action running alongside your Docker build rather than inside it; or the buildkit-cache-dance action, which Docker itself points to — it extracts the cache mount contents into the runner’s cache between builds and injects them back on the next run. The dance works, but it is a workaround, and it looks like one.

How to verify BuildKit cache mounts are working

The first time you run docker compose build after making these changes, you’ll see your packages downloaded as usual. On subsequent runs, watch the package-installation steps: Docker prints CACHED next to steps it skipped entirely, and the RUN steps that do re-execute should finish noticeably faster because the package manager is reading from the mount instead of the network.

If you want to see the cache rather than infer it from the clock, docker builder du grows after the first build and stays put after the second. And if a pip install step still takes as long the second time around, the usual culprit is a cache directory that does not match the user the step runs as — check the uid/gid section above.

When BuildKit cache mounts pay off, and when they do not

🔒 Subscribe to keep reading.

References

🔒 Subscribe to keep reading.