Disk Full: Hunt the Space with du (Un-rotated Backups)
Medium

Problem

db-host is out of disk. Writes are failing with `No space left on device and the orders-api` can't persist state. The Postgres data itself looks fine and RAM is healthy. You're logged in as root.

Nothing in the obvious logs explains it — you'll have to find where the space went. Locate the directory eating the disk, identify the root cause, and name the fix.

Initial setup

  • Host: db-host, Alpine. You are root.
  • Root fs / is at 100%; /home is a separate, healthy disk.
  • A nightly Postgres backup runs from crond.

Acceptance

You've solved it when:

  • You've shown with df -h that / is at 100% (Available ~0) and /home
is fine — so it's the root fs.
  • You've HUNTED the space with du (e.g. du -d1 -h /, then drill into
the big child: du -d1 -h /var, du -d1 -h /var/backups) down to /var/backups/postgres, which holds ~7.5 GiB of nightly dumps — and confirmed with ls -l /var/backups/postgres that they are many dated orders-YYYY-MM-DD.sql files nobody ever deletes.
  • You've named the fix: delete the OLD dumps to reclaim space now (keep
the most recent few), and add retention to the backup job (find /var/backups/postgres -mtime +7 -delete, or a tool retention policy). Root cause: the backup cron (/etc/periodic/daily/pg-backup) writes a dated dump every night with NO cleanup step, so the directory grew unbounded until it filled /.
Live session
Code
SavedNo commands yet