Building a Cloudflare Failover Watchdog on a 2 GB Linux Laptop

Abstract

A website becomes unreachable from part of the Internet while its origin server remains fully operational. The immediate workaround may be simple: bypass the affected proxy infrastructure and expose the origin directly.

Automating that decision is less simple.

This article documents a small resilience experiment carried out by DDR Lab using an old HP Stream laptop with 2 GB of RAM running antiX Linux. The machine was configured to monitor a production website routed through Cloudflare, independently verify the availability of its IONOS origin server, distinguish confirmed failures from inconclusive DNS conditions, and change the relevant Cloudflare DNS records through the API when a defined failure threshold is reached.

The implementation itself is modest. It uses standard Linux tools, shell scripts and cron. The more interesting problem is not computational but systemic: determining when there is enough evidence to justify an automated intervention in production infrastructure.

1. The incident

The experiment did not begin as a laboratory exercise.

A production website hosted at IONOS and normally routed through Cloudflare became inaccessible from a particular network while the underlying hosting infrastructure remained available.

Once the origin was identified and tested independently, the immediate technical workaround was straightforward: traffic could temporarily bypass the Cloudflare proxy and reach the IONOS server directly.

For an administrator familiar with DNS and web hosting, this is not an extraordinary operation. A few DNS changes can restore an alternative route.

But a manual workaround raises a second question:

Can the decision itself be automated safely?

That question turned out to be considerably more interesting than changing the DNS records.

2. The test platform

The monitoring system was deliberately built on modest hardware.

The machine was an HP Stream laptop with approximately 2 GB of usable RAM running antiX Linux with IceWM. No dedicated server hardware was required.

The relevant software components were equally ordinary:

• curl for HTTP and HTTPS testing;
• dig for DNS queries;
• jq for processing Cloudflare API responses;
• shell scripts for the monitoring and control logic;
• cron for periodic execution.

The watchdog runs every two minutes.

During normal operation the machine spends most of its time effectively idle. The workload consists of short DNS queries and HTTPS requests followed, only when necessary, by API calls.

This matters because the experiment illustrates a useful distinction: resilience mechanisms do not necessarily require powerful hardware. They require reliable decision logic.

3. The first obvious solution was not enough

A naïve watchdog could perform a request such as:

curl https://example.com

and decide that the site is healthy when the request succeeds.

That proved insufficient.

An intermediary or blocking system may return an HTTP response even though the requested application has not actually been reached. HTTP status alone therefore cannot establish that the expected service is available.

The watchdog was consequently modified to look for a stable content marker from the actual website.

Conceptually, the test became:

HTTPS connection succeeds
        +
expected content is present
        =
service considered available

This is still elementary monitoring, but it substantially changes the evidential value of the result.

4. Cloudflare failure is not enough

A second problem immediately appears.

Suppose the Cloudflare route fails.

Bypassing Cloudflare is useful only if the origin server remains available. If both routes are failing, changing DNS may accomplish nothing and may make diagnosis more difficult.

The watchdog therefore performs two logically independent tests:

Cloudflare route
       |
       +-- expected content available?
       |
IONOS origin
       |
       +-- expected content available?

Only one combination potentially justifies intervention:

Cloudflare = FAIL
Origin     = OK

A simultaneous failure produces no bypass.

This implements a simple principle that became central to the experiment:

Do not intervene merely because the current state is undesirable. Intervene only when there is evidence that the proposed alternative state is better.

5. Then DNS became uncertain

Testing revealed another complication.

DNS resolution from the monitoring connection was not perfectly stable. Queries occasionally timed out or temporarily produced results inconsistent with previous and subsequent observations.

The logs therefore contained conditions such as:

NORMAL cloudflare=OK origin=1
DIRECT-DNS origin=1 Cloudflare not authoritative
DNS-UNKNOWN unable to resolve Cloudflare edge IPs; no action
NORMAL cloudflare=OK origin=1

This was useful.

A less conservative implementation could have interpreted DNS uncertainty as evidence of a Cloudflare failure and changed production records unnecessarily.

Instead, the watchdog acquired a third epistemic state:

OK
FAIL
UNKNOWN

UNKNOWN does not mean failure.

It means that the monitoring system does not currently possess enough reliable information to make the decision.

And its response is deliberately uneventful:

no action

6. Consecutive failures

Even a properly observed failure may be transient.

The watchdog therefore does not perform failover after a single unsuccessful Cloudflare test.

It maintains a persistent failure counter.

The intended sequence is:

Cloudflare FAIL / Origin OK -> count 1
Cloudflare FAIL / Origin OK -> count 2
Cloudflare FAIL / Origin OK -> count 3
                              |
                              v
                         request bypass

A successful Cloudflare test resets the counter.

An indeterminate DNS condition does not become evidence merely by being inconvenient.

The result is intentionally biased against intervention.

For infrastructure automation, a false negative may temporarily leave a human-solvable problem unresolved. A false positive can cause the automation itself to alter a healthy production system.

Those risks are not symmetrical.

7. The actuator

Detection and intervention were kept conceptually separate.

A second script acts as the Cloudflare actuator. Using a restricted API token, it changes the proxy state of the relevant web records.

Both IPv4 and IPv6 records must be considered. Disabling the proxy only for an A record while leaving the corresponding AAAA record proxied could leave IPv6 clients following the route the system was attempting to bypass.

The actuator therefore changes four records:

A     example.com
A     www.example.com
AAAA  example.com
AAAA  www.example.com

The API response is checked before the operation is considered successful.

Importantly, the watchdog does not change the domain’s authoritative nameservers.

Cloudflare remains the authoritative DNS provider. The automated intervention changes only whether selected web records are proxied.

Nor does the system touch mail infrastructure. MX, SPF, DKIM, DMARC and other unrelated DNS records remain outside the actuator’s scope.

This is an application of another useful rule:

Automated intervention should have the smallest possible blast radius.

8. Persistent state

The watchdog maintains a small amount of persistent state on disk.

At the simplest level, the monitored service can be in one of two operational states:

NORMAL
BYPASS

Failure and recovery counters are also persistent.

This becomes relevant if the monitoring machine is restarted. The process should not necessarily forget that it previously changed production infrastructure merely because Linux rebooted.

Again, the implementation is technically simple. The important part is deciding what the state means.

9. AI accelerated the implementation

AI assistance was used extensively during the experiment.

That substantially reduced the time required to produce shell code, formulate commands, interpret documentation and iterate through alternative implementations.

It did not make the problem trivial.

The actual workflow remained:

hypothesis
    |
    v
command
    |
    v
observation
    |
    v
interpretation
    |
    v
design change
    |
    v
new test

Several early assumptions had to be modified after observing the real system.

The content check was introduced because HTTP success was not sufficient evidence. DNS uncertainty required a separate non-actionable state. IPv6 had to be included in the actuator. API responses had to be validated rather than merely sent.

The code could be generated quickly.

Determining what the code should safely be allowed to do required considerably more thought.

AI therefore changed the implementation cost without removing the need for systems reasoning.

10. Why an old HP Stream?

There was no technical requirement to use an HP Stream.

A Raspberry Pi, virtual machine, small server or almost any continuously connected Linux system could perform the same task.

The Stream was available, consumed little power and was already running a lightweight Linux distribution.

With graphical applications closed, CPU utilization remained close to idle and memory consumption was comfortably below the machine’s limited capacity. Closing the laptop lid switched off the display without stopping the operating system, and cron continued executing the watchdog.

The machine therefore effectively became a small network appliance.

Its limitations were largely irrelevant because the task itself is small.

That is perhaps the least surprising result of the experiment, but still a useful one:

A system should be sized for the problem it actually has to solve.

11. What has been demonstrated

At the time of writing, the experiment has demonstrated the following:

• independent testing of the Cloudflare and origin routes;
• validation based on expected application content rather than HTTP status alone;
• conservative handling of DNS uncertainty;
• persistent failure counting;
• automatic execution through cron;
• successful API control of the relevant A and AAAA records;
• successful manual testing of both proxy bypass and restoration;
• continued watchdog operation on very limited hardware.

One part deserves further testing before the system can be described as complete: automatic recovery from BYPASS to NORMAL.

Detecting that Cloudflare has become reachable again while the principal hostname is deliberately operating in DNS-only mode requires careful design. A recovery mechanism that depends on the very proxy configuration that has been disabled risks creating a circular dependency.

That problem will be tested separately rather than assumed solved.

12. A modular design

Although the first implementation monitors a single production domain, nothing fundamental in the architecture requires the system to remain single-domain.

The monitoring logic, decision model and Cloudflare actuator can be separated from domain-specific configuration. Each monitored website needs only its own parameters: domain name, origin address, expected content marker, Cloudflare zone and record identifiers, together with independent state and failure counters.

A future implementation can therefore use a common watchdog engine with multiple configuration files:

watchdog
   |
   +-- domain-A.conf  -- NORMAL
   +-- domain-B.conf  -- NORMAL
   +-- domain-C.conf  -- BYPASS
   +-- domain-D.conf  -- NORMAL

A failure affecting one domain would not require changes to the others. The same small Linux machine could monitor several websites and perform selective intervention only where the predefined conditions are satisfied.

The computational cost of doing so is modest. Monitoring additional domains primarily adds DNS queries and short HTTPS requests. These checks can also be staggered to avoid unnecessary simultaneous activity.

More importantly, a multidomain implementation would provide additional observational value. If several domains using different Cloudflare addresses are monitored from the same network, simultaneous and divergent failures may help distinguish a local DNS anomaly, an origin failure, a domain-specific problem and a broader network event.

DDR Lab will document this next stage separately. The objective is to evolve the current single-domain watchdog into a modular multidomain monitoring and failover system capable of protecting a wider web ecosystem without coupling the operational state of one site to another.

13. Status at the time of writing

This experiment originated in a real availability incident. It is therefore worth recording the state of the affected service at the point at which this article was closed.

During the football fixtures taking place on 30 August 2026, the previously affected portal remained accessible from the monitored connection at the time of writing.

The watchdog continued to record intermittent DNS uncertainty and isolated connectivity anomalies, but these observations did not by themselves establish a sustained failure requiring intervention. In accordance with the conservative decision model, inconclusive conditions resulted in no automated change to production infrastructure.

This observation should not be interpreted as evidence that the underlying blocking phenomenon had ceased to exist, nor as evidence that football-related blocking was responsible for every anomaly recorded during the test. It records only what the monitoring system could establish at that particular time.

If a future production event satisfies the defined failover conditions, its sequence will be documented from the original watchdog logs rather than reconstructed retrospectively.

14. From workaround to system

The manual fix took minutes.

The interesting work began afterwards.

Turning a workaround into an autonomous system required defining what constituted evidence, distinguishing failure from uncertainty, identifying the conditions under which intervention could improve the situation, limiting the scope of that intervention and preserving enough state to make the process reversible.

None of those problems required sophisticated computing resources.

They required a model of the system.

That distinction is easily lost when automation is described primarily in terms of code.

In this experiment, the shell scripts were the easy part.

The real problem was deciding when the machine should be allowed to act.

The next step is equally clear: move from one monitored website to many, while preserving the same principles of independent state, minimum intervention, uncertainty management and reversibility.