Manoj Yadav

The Breaking Part #2: The Inode Apocalypse

One Tuesday morning, three different Docker containers on my main server (HP T630 #1) started crashing simultaneously.

The container logs were filled with identical error messages:

OSError: [Errno 28] No space left on device
sqlite3.OperationalError: disk I/O error

Naturally, I assumed a log file or a database volume had filled up the SSD. I SSH-ed into the server and ran the standard disk usage command:

df -h

I was greeted by this completely contradictory output:

Filesystem      Size  Used Avail Use% Mounted on
/dev/sda1       120G   24G   90G  21% /

There was over 90 GB of free space. Only 21% of the disk was used.

Yet, when I tried to run a simple touch test.txt, bash hit me with:

touch: cannot touch 'test.txt': No space left on device

How can a Linux filesystem with 90 GB of free space refuse to write a 0-byte text file?


What Actually Happened: Inode Exhaustion

When we think of disk storage, we think of bytes and megabytes. But filesystems like ext4 don’t just store raw data blocks; they also maintain an inode table.

An inode (index node) is a small metadata structure that stores information about a file: permissions, owner, timestamps, and pointers to the physical disk blocks where the content lives.

Crucially: every file, directory, and symlink consumes exactly one inode.

When you format a disk with ext4, the operating system allocates a fixed, predetermined number of inodes based on the size of the drive. If you create millions of tiny files, you can run out of inodes long before you run out of disk space.

Running the inode flag on df confirmed the problem instantly:

df -i
Filesystem       Inodes   IUsed   IFree IUse% Mounted on
/dev/sda1       7864320 7864320       0  100% /

Every single one of the 7.86 million inodes on the disk had been consumed. Not a single free inode remained.


Hunting Down the Culprit

Now I needed to find which directory was hoarding 7.8 million files.

I ran a quick one-liner to count the number of inodes used across top-level directories:

sudo find / -xdev -maxdepth 2 -type d -exec sh -c 'echo -n "{}: "; find "{}" | wc -l' \;

The output quickly singled out a specific Docker volume:

/var/lib/docker/volumes/scraper_cache/_data: 7,420,183

A custom web scraper container I had written had an internal caching layer designed to save scraped HTML snippets to /tmp/cache so it wouldn’t re-download pages unnecessarily.

The bug? I had never implemented a TTL (Time-To-Live) or an eviction policy.

The script was generating thousands of tiny 200-byte cache files every hour, 24 hours a day, for nearly four months. Because each file was tiny, the total disk usage of that cache directory was under 3 gigabytes. But it had swallowed 7.4 million inodes, completely starving the host.


The Recovery (And Why rm -rf Failed)

I navigated to the offending cache directory and tried the obvious command:

rm -rf *

Bash immediately errored out:

-bash: /bin/rm: Argument list too long

When you use the * wildcard, the shell expands all file names into memory before passing them as arguments to rm. With 7.4 million files, the argument string exceeded the kernel’s ARG_MAX buffer limit.

To bypass the shell expansion and delete the files directly as the filesystem iterates over them, I used find with the -delete flag:

find . -type f -delete

It took nearly twenty minutes for the disk controller to purge all 7.4 million inodes.

The moment it finished, df -i dropped back down to 6% usage, and every crashing container immediately recovered and booted cleanly.


The Safeguards Put in Place

To guarantee this never takes down my homelab again, I made three fundamental changes:

1. Inodes Monitored in Zabbix

Most monitoring templates only alert on byte capacity (df -h > 85%). I added an explicit alert trigger in Zabbix for inode consumption:

Trigger: Inode utilization on {HOST.NAME} > 80% for 15m -> Alert to Telegram

2. Ephemeral Storage Belongs in tmpfs

If a container needs a scratchpad or temporary cache directory, it should never write directly to persistent disk storage. I updated the Docker Compose definition to mount temporary caches in volatile RAM via tmpfs:

services:
  scraper:
    image: custom-scraper:latest
    tmpfs:
      - /tmp/cache:size=256M,uid=1000,gid=1000

With tmpfs:

  • The cache writes directly to RAM at gigabytes per second.
  • It can never exceed 256 MB.
  • When the container stops, the inodes are freed instantly without touching SSD endurance.

3. Automated File Lifecycles

Any container writing persistent files to disk must include an automatic pruning job (like tmpwatch or a small Python cron) to evict files older than 48 hours.


Key Takeaway

A disk is full when either its bytes are exhausted or its inodes are exhausted. Check df -i as often as you check df -h.