Disk Full: A Runaway Log Filled the Root Filesystem
Easy

Problem

app-host is throwing errors and the orders-api service can't write its files — you're seeing No space left on device in its output. The box has been up for weeks; nobody touched the hardware.

Find which filesystem is full, prove what filled it, and say how you'd recover the space AND stop it from happening again.

Initial setup

  • Host: app-host, Alpine, 1 vCPU, plenty of RAM free.
  • Two filesystems: / (root) and /data.
  • orders-api logs to /var/log/orders-api.log.

Acceptance

You've solved it when:

  • You've shown with df that the root filesystem (/) is at 100%
(Available ≈ 0), and confirmed /data is fine — so it's the root fs, not the data disk, and not a hardware fault.
  • You've identified the runaway file: cat /var/log/orders-api.log
shows the same DB-connection error logged on a tight retry loop with no backoff and no rotation — that's what filled the disk.
  • You've named BOTH halves of the fix: reclaim space now by truncating
the log in place (truncate -s 0 /var/log/orders-api.log or : > /var/log/orders-api.log) — NOT rm (the running process holds the file open, so rm won't free the space until it reopens) — and the durable fix: add logrotate (or copytruncate) and stop the unbounded retry-logging loop.
Live session
Code
SavedNo commands yet