Global Egress Orchestration

I initially explored deploying the edge client as a background application on consumer mobile devices — iOS and Android — to leverage their residential IP addresses without dedicated hardware. Apple's iOS is fundamentally hostile to this use case. The OS aggressively suspends WebSocket threads to preserve battery life, and App Store guidelines strictly prohibit background egress routing.

Strategic abandonment. Rather than engineering fragile, battery-draining workarounds or fighting the OS scheduler, I dropped iOS entirely. Engineering is about knowing what not to build as much as what to build. The deployment strategy pivoted exclusively to headless Alpine Linux containers and repurposed plugged-in ARM hardware — guaranteeing 100% socket uptime without fighting a consumer OS designed to do exactly the opposite.

Residential ISPs frequently reassign IP addresses and deploy Carrier-Grade NAT (CGNAT), making inbound connections to edge nodes impossible. You cannot open a port on a CGNAT address — the NAT gateway doesn't belong to you.

The architecture inverts the connection direction entirely. Edge nodes initiate an outbound WebSocket connection to the Oracle orchestrator on port 443. Because the connection originates from inside the residential network, NAT passes it through without any port forwarding configuration. The orchestrator never connects to the nodes — it waits for nodes to connect to it, then multiplexes task payloads over the persistent socket.

Each node registers itself with a unique ID on connection and is immediately available for task assignment. The Systemd unit ensures the daemon reconnects with a 5-second backoff if the socket drops:

[Unit]
Description=Egress WebSocket Daemon
After=network-online.target
Wants=network-online.target

[Service]
Type=simple
User=egress
ExecStart=/opt/egress-node/client --orchestrator wss://[orchestrator-host]
Restart=always
RestartSec=5
ReadOnlyPaths=/
ReadWritePaths=/tmp

[Install]
WantedBy=multi-user.target

Consumer ISPs drop packets silently. If a residential node loses its connection, the TCP socket on the Oracle server remains "half-open" — the OS doesn't know the peer is gone until it tries to write and the kernel's TCP keepalive eventually fires. This takes minutes. During that window, the orchestrator assumes the node is alive, assigns it HTTP payloads, and blocks waiting for a response that will never arrive. Tasks pile up in the queue; the system grinds to a halt.

Application-layer heartbeat at 15-second intervals — independent of TCP keepalive. Each node sends a PING frame every 15 seconds. A Python SRE service polls Redis continuously. Any node that has not updated its heartbeat timestamp within the timeout window is declared dead, immediately purged from the active pool, and all its locked tasks are requeued to healthy nodes.

{
  "event":           "HEARTBEAT_TIMEOUT",
  "node_id":         "egress-node-region-b-04",
  "last_ping_ms":    15042,
  "status":          "DEAD",
  "action":          "PURGE_AND_REQUEUE",
  "recovered_tasks": [
    { "task_id": "8f9a-4b2c", "requeued_to": "egress-node-region-a-02" },
    { "task_id": "3c1d-9e7f", "requeued_to": "egress-node-region-b-01" }
  ],
  "timestamp":       "2026-05-01T14:22:10Z"
}

The requeue operation is idempotent — tasks carry a unique ID and the orchestrator checks for duplicates before processing, so a node that reconnects after a false-positive timeout cannot cause double-execution.

The edge nodes use repurposed ARM hardware running in residential environments — no server room, no active cooling, ambient room temperature with no airflow management. Running continuous asynchronous HTTP requests at full CPU clock speeds causes thermal runaway: core temperatures climb into throttling territory within minutes, triggering kernel panics and, over weeks, causing permanent hardware degradation.

Rather than relying on the kernel's default thermal governor — which reacts to high temperatures after the fact — I implemented a proactive approach: unconditionally underclock the CPU to a ceiling that keeps core temperatures stable regardless of load. A cron-executed shell script overrides the kernel's frequency governor and hard-caps the maximum clock speed. The tradeoff is approximately 20% of burst throughput, which is fully acceptable: these nodes are I/O-bound (waiting on network responses), not CPU-bound.

#!/bin/sh
# Proactive thermal management: underclock before heat builds, not after
echo "powersave" > /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor

# Hard-limit max frequency to prevent thermal runaway on fanless hardware
MAX_FREQ=1200000
for cpu in /sys/devices/system/cpu/cpu[0-7]*; do
    if [ -f "$cpu/cpufreq/scaling_max_freq" ]; then
        echo $MAX_FREQ > "$cpu/cpufreq/scaling_max_freq"
    fi
done
echo "[INFO] Thermal ceiling applied. Node stable for continuous operation."

The edge nodes are physically inaccessible — deployed in private residential environments across multiple geographies. SSH over a standard public IP is not possible (CGNAT blocks it). A VPN would require a persistent tunnel with its own reliability concerns and exposure. Any management approach requiring physical access is operationally unacceptable.

The management plane runs entirely through Cloudflare Zero Trust. The cloudflared daemon on each node establishes an outbound tunnel to Cloudflare's network — the same reverse-tunnel pattern used for the data plane. SSH access to any node is granted through the Zero Trust dashboard: the engineer authenticates via identity provider, and Cloudflare proxies the SSH session inbound through the existing outbound tunnel. No exposed SSH port. No VPN. No static IP dependency.