CI host out of disk: reclaim space from dangling images
Medium

Problem

Your team's CI host is out of disk. The latest pipeline failed mid-build:

failed to register layer: write /var/lib/docker/overlay2/…/diff/app.tar:
no space left on device

The two services on this host (web, cache) are healthy and must stay up. Nobody has cleaned up after months of CI builds, and each build leaves untagged image layers behind. Reclaim the wasted space without touching the running services or the images they use.

Initial setup

  • web (nginx:alpine) and cache (redis:7-alpine) — both Up 3 days.
  • A pile of images, most of them untagged (<none>) leftovers from builds.

Example interaction

$ docker system df
TYPE            TOTAL     ACTIVE    SIZE      RECLAIMABLE
Images          8         2         6.224GB   6.16GB (98%)
...

$ docker images -f dangling=true
REPOSITORY   TAG       IMAGE ID       CREATED   SIZE
<none>       <none>    9f8e7d6c5b4a   …         1.2GB
...

From the RECLAIMABLE column you can see almost all of it is wasted.

Acceptance

You've solved it when:

  • You've used docker system df to confirm the wasted space lives in
images (a large RECLAIMABLE), not containers or volumes, and seen the untagged <none> images with docker images / -f dangling=true.
  • You've removed the dangling images (docker image prune, or
docker system prune) so docker system df reports the Images reclaimable dropped sharply.
  • web and cache are still running and their images
(nginx:alpine, redis:7-alpine) are untouched.

Constraints

  • Tools: docker CLI only. No rm -rf /var/lib/docker/... by hand (that
corrupts the overlay2 store).
  • Don't stop or remove the running services to free space.

Follow-up

  1. After docker image prune, docker system df still shows ~180MB
reclaimable on Images. What is that, and which flag reclaims it?
  1. Why does plain docker system prune (no -a) leave tagged-but-unused
images on disk?
  1. The host filled because builds leak <none> layers. What would stop the
leak at the source (CI cleanup step, --rm, builder cache pruning)?
Live session
Code
SavedNo commands yet