IFA Trabant: Extended Production Test and Development Roadmap

This article continues the IFA Trabant production experiment documented in

IFA Trabant: What Production Taught Us About Automated Failover

---

IFA Trabant was developed to answer a narrow operational question: if a legitimate website remains healthy at its origin but becomes unavailable through an intermediary delivery path, can a small independent Linux system detect the condition, change the route when necessary, and later restore the normal configuration without human intervention?

The first production experiments demonstrated that the concept could work.

The next stage was more demanding.

We left IFA Trabant running unattended for almost four days on a 2 GB HP Stream, using antiX Linux and an ordinary domestic 4G connection. During that period the watchdog had to distinguish transient anomalies from qualifying failures, decide when intervention was justified, maintain persistent state, recover autonomously and continue operating with minimal resource consumption.

The documented monitoring window covered approximately 87 hours and 26 minutes.

Three qualifying failover events occurred.

All three were handled autonomously.

All three subsequently returned to the normal Cloudflare delivery path through the configured recovery and verification procedure.

1. Origin of the Experiment

The experiment did not begin in a laboratory.

It began when routine maintenance work on Independent Edition became impossible from a Spanish Internet connection.

Instead of the website, the connection returned a blocking page referring to LaLiga and a judicial measure issued by the Commercial Court of Barcelona.

The origin infrastructure itself remained available.

That observation created the engineering question behind IFA Trabant:

If the origin remains healthy but the normal public delivery path becomes unavailable, can an independent system identify the condition and temporarily provide another route without human intervention?

The initial hypothesis therefore preceded the extended monitoring experiment.

IFA Trabant was not created after discovering a convenient pattern in later telemetry. It was developed in response to an availability problem that had already been directly observed.

2. Experimental Objective

The purpose of the extended test was not simply to prove that a shell script could remain running for several days.

The relevant objectives were more specific:

• verify unattended operation over an extended period;
• observe behaviour under imperfect network conditions;
• confirm that isolated or ambiguous events did not automatically modify infrastructure;
• verify autonomous failover when the configured persistence threshold was reached;
• verify autonomous recovery and restoration;
• observe the resource requirements of the complete monitoring node;
• collect sufficient telemetry to analyse the behaviour retrospectively.

The test was therefore both an endurance test and a state-management test.

3. Test Environment

The test environment was intentionally modest.

No server-class hardware was used.

No enterprise router was used.

No dedicated monitoring infrastructure was used.

This was deliberate.

The purpose was not to demonstrate that sophisticated infrastructure can monitor a website. That proposition requires little experimental confirmation.

The more useful question was how little infrastructure the control function actually required.

4. IFA Trabant Decision Model

IFA Trabant does not respond to every anomalous observation by modifying DNS configuration.

Its operational logic can be summarised in five verbs:

Observe. Warn. Confirm. Act. Verify.

A single failed observation produces evidence, not a conclusion.

Repeated qualifying observations increment the failure counter.

Only when the configured failure threshold is reached while the origin remains available does the watchdog activate the alternate path.

Recovery is deliberately asymmetric.

IFA requires five successful recovery observations before restoring the normal Cloudflare proxy configuration.

The resulting configuration is then verified before the persistent state returns to NORMAL.

A good watchdog is not one that reacts to everything. It is one that knows when observation has become sufficient reason to act.

5. Extended Production Test

The monitoring node was started on Friday 4 September and subsequently left operating through the weekend and into the early hours of Tuesday 8 September.

During the unattended period, IFA continued its two-minute monitoring cycle automatically.

The final telemetry record in the test window showed the system again in normal operation.

The test therefore included normal operation, transient uncertainty, rejected failure sequences, three autonomous failover events and three subsequent autonomous recoveries.

6. Telemetry and Observed Events

The watchdog generated timestamped operational telemetry throughout the test.

These entries describe more than connectivity.

They describe the decision state of the watchdog.

In particular, no state change and no action are operational results.

IFA repeatedly encountered imperfect information without automatically interpreting it as a condition requiring intervention.

7. Autonomous Failover and Recovery

Three complete autonomous failover cycles occurred during the extended test.

In every case the transition to BYPASS occurred only after the configured failure threshold had been satisfied.

The four relevant A and AAAA records were changed to direct-origin operation.

The watchdog then remained in BYPASS while the normal delivery path continued to be considered unavailable.

Recovery required five consecutive positive observations.

Only then did IFA restore the normal Cloudflare proxy state.

The restoration was subsequently verified before the state returned to NORMAL.

8. Rejected and Ambiguous Events

The three successful failover cycles do not by themselves demonstrate conservative automation.

The rejected events are equally important.

During the test there were several sequences in which the failure counter reached one or two but was subsequently interrupted.

There were also observations in which both Cloudflare and the origin failed simultaneously.

This distinction is fundamental.

If the origin itself cannot be confirmed as available, switching away from Cloudflare may provide no benefit.

IFA therefore does not treat every failure as evidence supporting the same action.

It responds to a specific differential condition.

9. Correlation with Football Broadcast Schedules

The test deliberately covered a full weekend of Spanish league football.

This was relevant because the original incident that motivated IFA Trabant occurred in the context of dynamic blocking associated with football broadcasting.

The three qualifying failover events occurred within or close to football broadcast windows.

The strongest temporal correspondence occurred on Saturday and Sunday.

On Saturday, the decisive IFA failure sequence appeared at 16:50, 16:52 and 16:54, with BYPASS activated at 16:54.

On Sunday, the decisive failure sequence appeared at 16:02, 16:04 and 16:06, with BYPASS activated at 16:06, shortly before the 16:15 match window.

On Monday, however, scheduled matches took place without any qualifying BYPASS event.

This negative observation is also relevant.

The working hypothesis is therefore not that football necessarily produces a failover event.

The relevant observation is that several differential-failure events appeared in temporal proximity to known broadcast windows, while other broadcast windows produced no qualifying event.

10. Causal Inference and Alternative Explanations

IFA Trabant does not identify the legal, administrative or network mechanism responsible for an availability failure.

It observes technical effects.

The telemetry establishes that, during the three qualifying events, the monitored public path repeatedly failed while the origin remained available.

The published football schedules provide independent temporal context.

The known dynamic-blocking phenomenon provides the pre-existing working hypothesis that originally motivated development of the watchdog.

Taken together, these elements make collateral IP blocking during football broadcast windows the most plausible explanation for the observed pattern.

This remains an inference rather than a uniquely demonstrated cause.

Alternative explanations — including CDN behaviour, DNS anomalies, routing conditions or access-network effects — cannot be completely excluded from the available telemetry.

This causal uncertainty does not affect the operational logic of IFA.

The watchdog does not need to know why the normal path has failed.

It needs sufficient evidence that the path has failed, that the origin remains available and that activating the alternate route is the lower-risk action.

A watchdog does not need to identify the intruder. It needs to recognise that the protected boundary has been crossed.

11. Resource Consumption and System Stability

The physical monitoring node remained stable throughout the extended test.

During dedicated unattended operation, total system memory consumption generally remained in the approximate 300–340 MB range.

Swap remained unused.

CPU utilisation was normally low.

Disk consumption remained effectively stable.

These figures represent the complete antiX environment, not the isolated IFA process.

At the end of the experiment, ChatGPT, a terminal and an editor were opened interactively on the same Stream while IFA continued operating.

Memory consumption increased substantially and swap became active.

This accidental comparison illustrates the distinction between the resource requirements of the dedicated watchdog and those of a contemporary interactive desktop workload.

The IFA monitoring function itself requires very little computing capacity.

12. Why Linux

Linux is a deliberate architectural choice for IFA Trabant.

A dedicated watchdog benefits from a predictable environment, direct access to standard networking tools, conventional process scheduling, simple text-based configuration and the ability to minimise unrelated background activity.

Network observability is particularly important.

A system designed to interpret network behaviour should ideally generate as little unrelated network activity as possible.

Linux also fits naturally within the technical ecosystem being monitored.

A substantial part of contemporary hosting, web-server, cloud and network infrastructure operates on Linux or Unix-derived systems and conventions.

IFA uses the same basic operational vocabulary: shell scripts, HTTP requests, DNS utilities, process scheduling, logs and filesystem-based state.

Linux was therefore selected not because the same concept would be computationally impossible elsewhere.

It was selected because it allows the monitoring node to remain small, observable and operationally simple.

13. Why antiX Was Used

Linux is an architectural choice.

antiX is an experimental choice.

antiX was selected because the reference HP Stream has only 2 GB of RAM and the test sought to minimise operating-system overhead.

Its lightweight Debian-based environment provided the networking and shell capabilities required by IFA without consuming a significant proportion of the available hardware resources.

IFA Trabant is not intended to depend on antiX.

A future portable implementation should rely on a clearly defined set of standard Linux dependencies rather than distribution-specific features.

antiX is therefore the reference test platform, not a mandatory component of the architecture.

14. Limitations of the Experiment

The experiment should not be interpreted beyond what its telemetry can support.

It used one monitoring node.

It used one domestic 4G access connection.

It monitored one production domain during this test phase.

No simultaneous measurements were collected from multiple independent Spanish access providers.

The watchdog can observe availability conditions but cannot directly identify the mechanism responsible for a failure.

The correlation with football schedules therefore supports a working causal hypothesis but does not uniquely establish it.

Approximately four days of operation also does not constitute long-term reliability certification.

The purpose of the experiment was narrower: to determine whether the architecture behaved coherently enough to justify the next development stage.

15. Next Development Stage

The next experiment will introduce a second independently monitored domain.

Both watchdogs will operate simultaneously from the same physical monitoring node.

The objective is to determine whether the architecture preserves isolation when multiple domains share the same monitoring hardware and Internet connection.

Local failure, local response.

A failure affecting one domain should not automatically modify the operational state of another.

The second-domain experiment will again be conducted during an extended observation period, including a weekend football schedule, so that the resulting telemetry can be compared using the same methodology.

16. The Longer-Term Objective: IFA Trabant Portable

The final objective is not to create a large monitoring platform.

IFA Trabant is intended to remain a small tool for a specific problem.

The potential user may maintain a personal website, a professional portal or a small WooCommerce installation.

That user does not necessarily need enterprise high availability.

The requirement may be considerably simpler: if an intermediary delivery path becomes unexpectedly unavailable while the origin remains healthy, the website should have a practical route of contingency.

A future implementation should therefore be portable.

Turn an unused laptop into an independent website failover node.

The software philosophy is deliberately conservative.

IFA does not need to be the most feature-rich monitoring platform.

It needs to perform one defined function reliably, transparently and with few dependencies.

Small utilities often remain useful precisely because they resist unnecessary expansion.

If the architecture continues to perform satisfactorily through multi-domain and portability testing, publication as open-source software would be a natural destination for the project.

17. Conclusion

IFA Trabant completed approximately 87 hours and 26 minutes of extended production monitoring on a 2 GB HP Stream running antiX Linux through an ordinary domestic 4G connection.

During the test the system encountered numerous imperfect observations.

Most did not produce a state transition.

Three sequences did satisfy the configured differential-failure threshold.

IFA entered BYPASS autonomously on all three occasions.

It subsequently remained in the alternate state until the configured recovery conditions were satisfied.

All three events ended with autonomous restoration and verification of the normal Cloudflare configuration.

The monitoring node remained stable throughout the test with very modest resource consumption.

The experiment therefore demonstrated two complementary properties:

Intervention and restraint.

The watchdog acted when the configured evidence threshold was reached and remained inactive when it was not.

The next development stage is replication rather than redesign.

A second independent domain will be added and monitored under the same methodology.

If that stage succeeds, the project can progress toward a genuinely portable implementation.

No data centre is required.

No artificial complexity is required.

The operational requirement remains simple:

Detect the right failure, make the right change, and otherwise leave the system alone.

Appendix A — IFA Trabant Telemetry

The following cards contain the principal state-changing events recorded during the extended production test.

Failover Event 1

Failover Event 2

Failover Event 3

Final Observed State

The complete telemetry also contains numerous rejected and ambiguous conditions that did not generate failover. These are relevant because they demonstrate that the three BYPASS transitions were not responses to every irregular network observation.

Appendix B — Football Schedule Reference

The following cards reproduce the relevant LaLiga match schedule used as external temporal reference data.

Friday · 4 September 2026

Saturday · 5 September 2026

Sunday · 6 September 2026

Monday · 7 September 2026

The football schedule is used only as external temporal reference data. The IFA telemetry records availability conditions; it does not itself identify the cause of those conditions.

↑ Back to contents