This article continues the experiment documented in Building a Cloudflare Failover Watchdog on a 2 GB Linux Laptop. The original article describes the design, implementation and initial testing of the watchdog. This follow-up begins with what happened when that design encountered production conditions.
Abstract
The first version of our Cloudflare failover watchdog was designed around a relatively simple principle: if a website became unavailable through its normal delivery path while the origin server remained healthy, the system should be able to bypass the intermediary layer automatically.
That part worked.
Production testing, however, exposed a more subtle problem. After successfully entering bypass mode, the watchdog could misinterpret subsequent DNS observations and return its internal state to NORMAL even though Cloudflare proxying had not actually been restored.
The result was a divergence between logical state and infrastructure state.
A revised version changed the architecture around a stricter principle: uncertainty must not be interpreted as recovery, and a system in bypass mode must remain there until recovery has been positively observed, restoration has been explicitly executed and the resulting infrastructure state has been independently verified.
Only hours after deployment, the revised watchdog encountered another production event. It entered bypass automatically, remained there for approximately two hours and forty-eight minutes, detected five consecutive successful recovery observations, restored Cloudflare and verified the resulting configuration before returning to NORMAL.
This article documents that sequence and the architecture that emerged from it: IFA Trabant — Independent Failover Architecture.
1. The Question Left Open
The previous experiment ended with an intentionally unresolved question.
Detecting failure was relatively straightforward once several safeguards had been introduced. The watchdog did not rely merely on an HTTP status code. It validated expected content, checked the origin independently and required consecutive failures before taking action.
But failover is only half of the problem.
Once an automated system has deliberately changed the network path, how does it determine that the original path is safe to restore?
More importantly, how does it prevent observations made after its own intervention from being interpreted according to assumptions that were valid only before that intervention?
The first production event answered those questions sooner than expected.
The original problem had been deciding when the machine should be allowed to act.
The next problem was deciding when the machine should be allowed to believe that its intervention was no longer necessary.
2. The First Production Event
On 30 August 2026, the watchdog recorded three consecutive observations in which the public Cloudflare path failed while the origin remained available:
16:48:07 WARNING cloudflare=FAIL origin=OK count=1
16:50:05 WARNING cloudflare=FAIL origin=OK count=2
16:52:10 WARNING cloudflare=FAIL origin=OK count=3The configured threshold had therefore been reached.
The watchdog acted:
16:52:11 ACTION Cloudflare bypass requested
16:52:13 STATE BYPASSThe four web records — A and AAAA for both the root domain and www — were changed from proxied to DNS-only operation.
The website could therefore be served directly from the origin.
From the perspective of the original objective, this was a success.
The watchdog had detected a sustained differential failure, confirmed that the origin remained operational and executed the intended mitigation without human intervention.
But the more interesting result came afterwards.
3. When Logical State and Infrastructure State Diverged
The original watchdog continued making observations after entering BYPASS.
Eventually it encountered a DNS condition in which Cloudflare did not appear to be authoritative. A branch originally intended to handle that observation reset the watchdog’s internal state to:
NORMALThere was one problem.
Cloudflare had not been restored.
An API inspection showed that the four relevant records remained:
A @ proxied=false
A www proxied=false
AAAA @ proxied=false
AAAA www proxied=falseThe watchdog believed that the system was in NORMAL.
The infrastructure was still in BYPASS.
In other words:
logical state != infrastructure stateThis was not merely a logging error.
Once the internal model had diverged from reality, subsequent observations could themselves become misleading.
A public content check might succeed because traffic was reaching the origin directly. If the watchdog interpreted that success under the assumption that Cloudflare was active, it could report:
NORMAL cloudflare=OKThe observation — successful delivery of the expected content — was real.
Its interpretation was not.
4. Correct Observation, Wrong Context
This exposed a more general systems problem.
Monitoring data does not necessarily carry its own meaning.
The meaning of an observation can depend on the state in which it was obtained.
A successful request while Cloudflare proxying is enabled and a successful request after Cloudflare has been bypassed are not equivalent observations, even if the returned HTML is identical.
Adding more requests would not solve this.
Running the watchdog every thirty seconds instead of every two minutes would not solve it.
Adding another DNS resolver would not solve it.
The defect was not primarily observational.
It was architectural.
The watchdog needed a stronger state model.
5. Designing Version 4
Version 4 was built around one central rule:
No uncertain observation may implicitly convert BYPASS into NORMAL.
The order of operations was changed accordingly.
When the persistent state is BYPASS, recovery logic is evaluated before normal public DNS interpretation.
An unexpected DNS result can be logged, but it cannot silently change the state.
The revised model can be simplified as follows:
NORMAL
|
| Cloudflare FAIL
| origin OK
| three consecutive observations
v
BYPASS
|
| recovery observed
| five consecutive observations
v
RESTORE
|
| explicit Cloudflare API operation
v
VERIFY
|
| all relevant records confirmed proxied=true
v
NORMALThis introduced another important distinction.
Restoration and recovery are not the same thing.
Recovery is an observation suggesting that the normal delivery layer is available again.
Restoration is an action.
And successful restoration is not assumed merely because the action was requested.
The resulting configuration must be verified.
6. Making NORMAL an Asserted State
In the revised architecture, NORMAL became more than a convenient label.
Following a bypass, the watchdog is permitted to return to NORMAL only after three separate conditions have been satisfied:
- recovery has been observed consistently;
- the restore operation has been executed;
- the Cloudflare API confirms that all four web records are again proxied.
This produces an important invariant:
After failover, NORMAL must describe infrastructure reality, not merely watchdog belief.
If restoration fails, verification fails or the result is uncertain, the system remains in BYPASS.
That is deliberately conservative.
Availability systems are often tempted to convert uncertainty into action.
IFA Trabant does the opposite: where evidence is insufficient, it preserves the last known protective state.
7. Controlled Testing
Before replacing the production watchdog, the new state logic was tested independently.
First, the state was deliberately set to BYPASS without usable recovery information.
The result was:
BYPASS Cloudflare edge still unavailableMost importantly, the state remained:
BYPASSThe condition that had caused the previous version to lose state no longer did so.
A second controlled test provided valid Cloudflare edge information and simulated a recovery sequence.
The watchdog recorded:
RECOVERY Cloudflare edge=OK count=1
RECOVERY Cloudflare edge=OK count=2
RECOVERY Cloudflare edge=OK count=3
RECOVERY Cloudflare edge=OK count=4
RECOVERY Cloudflare edge=OK count=5
ACTION restoring CloudflareThe actuator then confirmed the four records:
A independentedition.com proxied=true
A www.independentedition.com proxied=true
AAAA independentedition.com proxied=true
AAAA www.independentedition.com proxied=trueOnly afterwards did the watchdog write:
STATE NORMAL Cloudflare restored and verifiedThe revised version was then deployed.
8. Production Tested the Fix
Only a few hours later, production provided another test.
On 31 August 2026 the watchdog again observed a sequence in which the Cloudflare path failed while the origin remained available:
21:10:10 WARNING cloudflare=FAIL origin=OK count=1
21:12:02 WARNING cloudflare=FAIL origin=OK count=2
21:14:04 WARNING cloudflare=FAIL origin=OK count=3At the third consecutive observation:
21:14:04 ACTION Cloudflare bypass requested
21:14:06 STATE BYPASSThis time the revised state machine was in control.
For the following period, the watchdog repeatedly recorded:
BYPASS Cloudflare edge still unavailableIt did not infer recovery.
It did not convert DNS uncertainty into NORMAL.
It did not restore Cloudflare merely because the website remained accessible through the origin.
It waited.
9. Recovery
At 23:54 the situation changed.
The watchdog obtained its first successful recovery observation:
23:54:04 RECOVERY Cloudflare edge=OK count=1Two minutes later:
23:56:02 RECOVERY Cloudflare edge=OK count=2Then:
23:58:07 RECOVERY Cloudflare edge=OK count=3
00:00:02 RECOVERY Cloudflare edge=OK count=4
00:02:06 RECOVERY Cloudflare edge=OK count=5Five consecutive successful observations had now been obtained.
The watchdog executed:
00:02:06 ACTION restoring CloudflareFour seconds later:
00:02:10 STATE NORMAL Cloudflare restored and verifiedNo human intervention had been required.
The complete production sequence was therefore:
NORMAL
|
| sustained differential failure
v
BYPASS
|
| approximately 2 h 48 min
| five consecutive recovery observations
v
RESTORE
|
| Cloudflare API verification
v
NORMALThis was the first complete autonomous production cycle of the revised architecture.
10. What the Result Does — and Does Not — Demonstrate
The observed events occurred in the broader context that originally motivated this experiment: intermittent accessibility problems affecting Cloudflare-hosted websites during periods in which dynamic IP blocking measures were of particular interest.
The timing of the production events is therefore noteworthy.
It is not proof of causation.
The experiment does not establish that either failover event was caused by LaLiga, a telecommunications operator, a judicial blocking mechanism or any particular external actor.
The watchdog measures availability from a particular network perspective.
It establishes a narrower set of facts:
- the normal Cloudflare delivery path failed the configured content validation;
- the origin was independently available;
- the condition persisted for the required number of observations;
- bypass restored or preserved the direct delivery path;
- the Cloudflare edge later became consistently reachable again;
- restoration was executed and verified.
Anything beyond those observations requires additional evidence.
This distinction matters.
A resilience system does not need to identify who caused a network failure before mitigating it.
It needs to determine, conservatively and reproducibly, what is failing and what safe action is available.
11. The Hardware Was Never the Difficult Part
The reference implementation continued running on the same modest hardware used in the original experiment:
HP Stream laptop
antiX Linux
approximately 2 GB RAM
During the production failover, observed resource consumption remained low. CPU utilisation was typically only a few percent, and memory consumption remained well below the physical RAM available.
The watchdog spends most of its existence doing almost nothing.
Every two minutes it wakes, gathers a small number of observations, evaluates them and returns to waiting.
The computational requirements are trivial.
The reasoning requirements are not.
A much more powerful computer would not have prevented the state-management defect in the original version.
The correction required a better model, not more processing power.
12. From a Watchdog to IFA Trabant
The original experiment began as a watchdog for a single website.
The production events demonstrated that the more interesting component was not the individual shell script but the decision architecture underneath it.
We therefore use the name:
IFA Trabant
Independent Failover Architecture
IFA Trabant is a state-aware failover architecture designed to detect intermediary-layer availability failures, preserve origin availability and restore the normal delivery path only after verified recovery.
Its current implementation remains deliberately small.
It combines ordinary components:
- Linux;
- DNS observations;
- content validation;
- persistent local state;
- Cloudflare’s API;
- conservative thresholds;
- an independently reachable origin;
- explicit restoration;
- post-action verification.
None of those components is novel in isolation.
The relevant engineering question is how they are combined into a system that is permitted to act autonomously without converting ambiguous observations into unjustified certainty.
13. Initial Production Validation
One successful production cycle does not establish universal reliability.
There remain conditions that have not been observed, failure combinations that have not been tested and network behaviours that may require further changes.
The appropriate conclusion at this stage is therefore deliberately limited:
Initial production validation of IFA Trabant: successful.
Further observation is more useful now than further modification.
The next experimental stage should therefore leave the current implementation substantially unchanged and observe its behaviour across longer network windows.
A future multidomain implementation is also possible, but that is a different problem. Generalising the architecture should follow validation of the single-domain model, not replace it.
14. Conclusion
The first watchdog knew how to switch.
Production demonstrated that this was not enough.
A resilient system must know which observations are meaningful in each state. It must preserve uncertainty rather than translating it into false certainty. And after changing infrastructure, it must verify that external reality agrees with its internal model before declaring recovery.
The first production event exposed the weakness.
The revised architecture was tested, deployed and encountered a second production event only hours later.
This time it entered bypass, remained there while recovery could not be established, waited for five consecutive successful observations, restored Cloudflare, verified the resulting configuration and returned itself to normal operation.
The code remained small.
The architecture became considerably more careful.
That distinction is the real result of the experiment.