Manoj Yadav

The Breaking Part #1: The Great DNS Feedback Loop

My blog tagline isn’t just “Building and self-hosting” — it specifically includes “breaking”.

Anyone who claims their homelab runs with 100% uptime is either not self-hosting anything critical, or not tinkering enough. The most valuable engineering lessons come directly from the catastrophic outages you accidentally cause yourself.

This is the first entry in The Breaking Part series: the story of how an innocent attempt to resolve local .home domain names turned into an infinite DNS recursion loop that crippled my home network on a Saturday night.


The Setup

My network runs on a dedicated server VLAN (10.0.20.0/24). On a Raspberry Pi 4B, I run Pi-hole to block network-wide advertisements and trackers.

In my router’s DHCP settings, the Pi-hole’s IP (10.0.20.5) was set as the primary DNS server distributed to every phone, laptop, and smart TV in the house.

Everything worked smoothly for months.


The Innocent Tweak: Local Domain Forwarding

As I started running more internal services across multiple thin clients, I got tired of accessing services via raw IPs. I wanted dashboard.home.example.com and nas.home.example.com to resolve seamlessly.

I decided to enable Conditional Forwarding inside Pi-hole:

  • If a query asked for an internal host, Pi-hole was instructed to forward it upstream to my main gateway router (10.0.10.1), because the router assigned DHCP hostnames.

What I completely forgot was that two weeks prior, during a late-night router configuration session, I had also set the router’s own WAN DNS upstream server to… the Pi-hole.

The trap was officially set.


The Explosion

It took one unrecognized domain query.

A smart TV turned on and sent a reverse lookup (PTR) query for an unknown local address that neither the Pi-hole nor the router had a static record for.

Here is the loop that followed:

[ Client (Smart TV) ] ── (Who is this host?) ──► [ Pi-hole (10.0.20.5) ]
                                                        │
                                                "I don't know it,
                                                asking the Router..."
                                                        ▼
[ Pi-hole ] ◄── "Router forwards upstream to me" ── [ Router (10.0.10.1) ]
     │                                                  ▲
     └────────────── (Loop repeats infinitely) ─────────┘
  1. Pi-hole didn’t recognize the host $\rightarrow$ forwarded it to the router (10.0.10.1).
  2. The router received the query $\rightarrow$ looked at its upstream DNS setting $\rightarrow$ forwarded it back to Pi-hole (10.0.20.5).
  3. Pi-hole received the query again $\rightarrow$ forwarded it back to the router.
  4. Step 2 and 3 repeated at microsecond speeds.

Within 45 seconds, the query log exploded to over 250,000 queries.


The Outage & Diagnosis

First, all websites stopped loading on every phone in the house. Wi-Fi showed full bars, but everything timed out. Within minutes, family members started reporting that “the Wi-Fi is broken.”

I tried opening the Pi-hole web interface (http://10.0.20.5/admin). It wouldn’t load.

I SSH-ed into the Raspberry Pi host directly via IP:

ssh pi@10.0.20.5

Running htop showed the pihole-FTL process pegged at 100% CPU core utilization. The RAM was maxed out, and the process was thrashing.

Checking the query log confirmed the nightmare:

tail -f /var/log/pihole/pihole.log

The log was scrolling so fast it was blurring the terminal: thousands of identical queries bouncing back and forth between 10.0.10.1 and 10.0.20.5 every second.


The Recovery & Safeguards

1. Emergency Kill

To stop the CPU thrashing, I temporarily killed the DNS engine and reset the upstream DNS:

sudo systemctl stop pihole-FTL

I switched the router’s DNS settings back to an external public resolver (Cloudflare 1.1.1.1 and Quad9 9.9.9.9) so the rest of the household regained immediate internet access.

2. Eliminating the Circular Route

I disabled Conditional Forwarding entirely.

Instead of asking the router to resolve local names, I centralized all local DNS inside Pi-hole’s Local DNS Records (/etc/pihole/custom.list). Every internal service is explicitly mapped to its static Traefik reverse proxy IP (10.0.20.2).

3. Adding FTL Query Rate Limiting

To prevent any future rogue container or loop from killing the DNS daemon, I configured strict rate limits in /etc/pihole/pihole-FTL.conf:

# Block any client making more than 1,000 queries within a 60-second window
RATE_LIMIT=1000/60

Lessons Learned

  1. DNS is fragile because it is recursive: Never create a relationship where Server A asks Server B for information while Server B is configured to use Server A as its upstream resolver.
  2. Always separate client DNS from infrastructure DNS: Your router’s internal management interface and WAN uplink should use a hardened public resolver (1.1.1.1 or 9.9.9.9). Only user-facing DHCP clients should point to the internal ad-blocker.
  3. Set query rate limits before you need them: A simple configuration line (RATE_LIMIT) turns a catastrophic network meltdown into a contained, loggable warning.