Why Docker Compose rebuilds keep re-downloading the same packages

Re-downloading the same packages after every small bug fix is a proper drag when you rebuild with docker compose build. Change one line in your code, and suddenly pip or npm is fetching the exact same dependencies it fetched five minutes ago. Wasteful, and it adds up to real hours over a week of active development. Luckily, there is a remedy.

What BuildKit is and how it caches package downloads

BuildKit is the build engine bundled with modern Docker that adds a cache mount, a persistent directory it re-attaches to a RUN instruction on every build. A cache mount is a build-time storage primitive that survives layer invalidation, so a package manager downloads a given dependency once and reuses it across every subsequent build — even builds where the layer itself has to re-run. That last clause is the whole point, and it is the part Docker’s own BuildKit documentation buries.

I use cache mounts on every containerised project I work on, and they cost three lines of Dockerfile. Below are the changes to your docker-compose.yml and individual Dockerfiles, with worked examples for apt, pip, and npm.

How BuildKit cache mounts work

Before BuildKit, Docker’s build cache was primarily layer-based. If a line in your Dockerfile changed, that layer and all subsequent layers would be rebuilt from scratch, including any RUN commands that download packages.

BuildKit adds a second, finer-grained mechanism on top of layer caching: the cache mount, declared as RUN --mount=type=cache,target=<dir>. The mount names a directory inside the build container that BuildKit stores outside the image and re-attaches on every build. When pip, npm, or apt writes a downloaded archive into that directory, the archive stays on the builder’s disk. On the next build, BuildKit remounts the same directory, the package manager finds its own cache already populated, and it skips the download entirely.

Docker layer caching versus BuildKit cache mounts

Cache mounts and layer caching are complementary mechanisms that fail in different places, and conflating them is the most common reason people think cache mounts “do not work”.

The classic layer cache preserves an entire resulting image layer, so long as none of its inputs changed. Order your Dockerfile so COPY requirements.txt . and RUN pip install -r requirements.txt sit in their own layer, and Docker skips that layer entirely on the next build, provided requirements.txt has not changed. The moment it does change, the whole layer invalidates, and pip runs from a cold start, re-downloading everything.

A cache mount does not preserve the layer at all. A cache mount preserves the downloaded package files sitting on disk inside the mount, independently of whether the layer itself gets rebuilt. So even when requirements.txt changes and that RUN layer invalidates and re-executes, pip still finds most of what it needs already sitting in /root/.cache/pip, and only fetches what genuinely changed. The two mechanisms are not competing; keep your layer ordering sensible, and let the cache mount handle the case layer caching cannot.

Enabling BuildKit, the apt, pip and npm cache mounts, the Compose configuration and the cleanup steps follow below.

When BuildKit cache mounts pay off, and when they do not

🔒 Subscribe to keep reading.

References

🔒 Subscribe to keep reading.