Introduction

Connect your servers, sites, and devices into one private, encrypted network — shipped as a single tiny binary that runs anywhere, from a cloud VM to a MikroTik router.

This documentation site is bilingual. Use the 中文 / EN toggle in the top bar to switch languages, or read the 简体中文文档.

What is Subnetra?

Subnetra stitches machines in different places — offices, a data center, roaming laptops, home labs, containers, routers — into a single flat private subnet.

It uses a hub-and-spoke design: a reachable hub relays traffic between spokes, so any node can reach any other by a stable overlay IP even when most of them sit behind NAT. Every packet travels fully encrypted over an ordinary UDP tunnel, and the whole thing is one self-contained binary — nothing else to install, no kernel modules, no daemon zoo.

What you can do with it

• 🏢 Link branch offices to a data center — route whole subnets site-to-site over one private overlay.
• 💻 Give roaming laptops a stable private IP that follows them across Wi-Fi, LTE, and home networks.
• 🧪 Reach home-lab / IoT / container services as if they were on the same LAN.
• 🛰 Publish a LAN from behind NAT — e.g. a MikroTik router exposing 192.168.88.0/24 to the mesh.
• 📦 Run where heavy VPN stacks won’t fit — constrained containers, BusyBox, small ARM boxes, edge routers.

Highlights

• Runs anywhere, installs as one file — a single static binary (under 512 KB on Linux) with zero external dependencies. Drops onto cloud VMs, containers, BusyBox, Raspberry Pi, and MikroTik RouterOS. Builds for amd64 / arm64 / armv7 / armv5, plus a native macOS spoke.
• Encrypted by default, quiet on the wire — every packet is ChaCha20-Poly1305 encrypted with per-link keys and replay protection. There are no magic bytes, and unauthenticated packets are dropped silently — to a port scanner the tunnel looks like nothing is listening. Header obfuscation is on by default (obfuscate, mesh-wide): the 20-byte framing header is XOR-masked per packet so the whole datagram looks random and the NAT keepalive cadence is de-periodized — no protocol fingerprint for a passive observer. Set "obfuscate": false for a readable cleartext header (e.g. packet-capture debugging). Obfuscation hides the protocol fingerprint, not packet length or timing. See Wire Protocol → Header obfuscation.
• A flat private subnet with policy routing — give every node an overlay IP, route whole subnets site-to-site, and let the hub relay spoke-to-spoke so nodes behind NAT still reach each other.
• Just works behind NAT — spokes keep their own NAT pinhole open with a built-in keepalive, and the hub automatically relearns a spoke that roams to a new address — no external pinger, no manual reconnect.
• Change routes live — inject or update forwarding rules at runtime with zero downtime: no restart, no dropped packets.
• Built to be operated — human-readable and JSON status, per-reason drop counters that tell you why traffic isn’t flowing, per-peer health/online flags, and a Prometheus exporter for alerting.
• Simple, declarative config — describe a node as a hub or a spoke and the forwarding table is derived for you. Name your peers so status reads bj-office-gw, not id=2.

Quick start

The fastest path is the container image (the hub is typically a public cloud host):

# 1. Create a config — one hub, one or more spokes.
cp config.example.json config.json
#    Set a UNIQUE 64-hex key on every peer link:  openssl rand -hex 32

# 2. Run it (the tunnel needs the TUN device + NET_ADMIN).
docker run -d --name subnetra \
    --cap-add=NET_ADMIN --device=/dev/net/tun \
    -v "$PWD/config.json":/etc/subnetra/config.json:ro \
    ghcr.io/jamiesun/subnetra:latest

# 3. Check it.
docker exec subnetra subnetra status

Prefer a bare binary? Grab the static build for your architecture from the latest release, then follow the Installation and Quick Start guides. A full hub + two-spoke production walkthrough lives in Production Deployment.

Operate & observe

subnetra status turns the silent, by-design packet drops into countable signals so you can tell why traffic is or isn’t flowing:

subnetra v0.6.0 [running]
mode=raw_direct local_id=1 udp_port=18020 tun=snr0 peers=2
peers:
  id=2 name=bj-office-gw endpoint=203.0.113.2:18020 allowed_src=10.0.0.2/32
  id=3 name=alice-laptop endpoint=203.0.113.3:18020 allowed_src=10.0.0.3/32
traffic: tun_rx / udp_tx / udp_rx / tun_tx / relay / keepalive …
drops:   unknown_peer / auth_or_invalid / spoof / no_route …

subnetra status --json emits the same data as a stable, versioned JSON object — with a per-peer last_seen_age_seconds and online flag — and a drop-in Prometheus exporter turns it into scrapeable metrics. See Observability & Troubleshooting for the full status schema, the drop taxonomy, and alerting examples.

How to read these docs

• New here? Start with Installation and the Quick Start.
• Want to understand the design? Read Architecture and the Security Model.
• Setting up a config? See the Configuration Reference and Roles.
• Going to production? Follow Production Deployment, Containers, RouterOS Spoke, or the macOS Spoke runbook.
• Building another implementation? The Wire Protocol is the normative contract.

Project status

The framework, the pure-algorithm layer, and the syscall data path (TUN, the readiness reactor, AF_UNIX control plane, daemon main loop) are implemented and exercised end-to-end in the dev container, with a native macOS utun/poll(2) spoke behind the comptime src/os/ backend. v1 (raw_direct + PSK + anti-replay

• RCU policy) is the deliverable; v2 reliability modes (kcp_arq, fec_xor) are reserved interface points only — see the Roadmap.

License

MIT © 2026 jettwang.

Installation

Subnetra ships as a single static binary. There is nothing to install system-wide and no shared libraries to manage — pick whichever delivery method fits your environment.

Quick install (interactive)

On Linux or macOS, the fastest path is the install script. It detects your OS and architecture, resolves the latest release, verifies the download against the release SHA256SUMS.txt, and installs both subnetra and subnetrad — pausing for you to confirm before it writes anything:

curl -fsSL https://raw.githubusercontent.com/jamiesun/subnetra/main/install.sh | sh

The script is interactive and only installs the two binaries — it never touches your network, firewall, or services (Subnetra always leaves the host plan to you). If Subnetra is already present in the target directory, it shows the installed version and asks before overwriting. For an unattended run, accept the defaults with --yes:

curl -fsSL https://raw.githubusercontent.com/jamiesun/subnetra/main/install.sh | sh -s -- --yes

To run Subnetra as a managed service, add --service: it installs the hardened systemd (Linux) or launchd (macOS) unit disabled — it never starts it and never touches your network — then prints the steps to finish and enable it. See Deployment for the full service setup.

Prefer to install by hand, or on a platform the script does not cover? Use the release tarballs below, or browse every asset on the Releases page.

The daemon needs two things at runtime regardless of method: the NET_ADMIN capability (to create the TUN device) and access to /dev/net/tun.

Container image

Tagged releases publish a multi-arch image to GHCR. Docker automatically selects the right architecture:

docker pull ghcr.io/jamiesun/subnetra:latest

# The daemon needs NET_ADMIN + the TUN device, and a config.json mounted into
# its working directory (/etc/subnetra).
docker run -d --name subnetra \
    --cap-add=NET_ADMIN --device=/dev/net/tun \
    -v "$PWD/config.json":/etc/subnetra/config.json:ro \
    ghcr.io/jamiesun/subnetra:latest

The image ships a Docker HEALTHCHECK (subnetra status), so docker ps / Compose / Kubernetes report the daemon healthy once it is serving its control socket and unhealthy if it stops responding.

The amd64, arm64 and arm/v7 images are built FROM busybox:musl (they carry the two static binaries, config.example.json, and a tiny BusyBox shell for in-container debugging). The arm/v5 image is built FROM scratch (still static musl, no debug shell) and stitched into the same :latest / :version manifest. See Containers for Compose and Kubernetes details.

Release binaries

Browse and download any release from the Releases page. Each release (vX.Y.Z) attaches static binary tarballs for amd64, arm64, armv7, armv5, mipsel, and mips. The Linux binaries are fully static against musl-libc — ldd reports not a dynamic executable.

MIPS / OpenWrt: mipsel is little-endian (ramips/mt7621/mt7628, most modern OpenWrt devices) and mips is big-endian (ath79/Atheros). See the OpenWrt Spoke guide for picking the right one and the procd service.

Asset names carry the version, so resolve it once, then download, verify, and install:

ARCH=amd64   # one of: amd64 | arm64 | armv7 | armv5 | mipsel | mips
VER=$(curl -fsSLI -o /dev/null -w '%{url_effective}' \
        https://github.com/jamiesun/subnetra/releases/latest | sed 's#.*/tag/##')

curl -fsSLO "https://github.com/jamiesun/subnetra/releases/download/$VER/subnetra-$VER-linux-$ARCH.tar.gz"
curl -fsSLO "https://github.com/jamiesun/subnetra/releases/download/$VER/SHA256SUMS.txt"
sha256sum --ignore-missing -c SHA256SUMS.txt        # verify before installing

tar -xzf "subnetra-$VER-linux-$ARCH.tar.gz"
cd "subnetra-$VER-linux-$ARCH"
sudo install -m 0755 subnetrad subnetra /usr/local/bin/
subnetrad --version

The releases/latest/download/<asset> path always points at the current release, but asset names embed the version — so resolve VER as above, or just use the install script.

Offline / air-gapped install

Devices that cannot reach a container registry can use the per-arch docker load-able image tarballs attached to each release (subnetra-image-<version>-<arch>.tar.gz):

docker load < subnetra-image-<version>-arm64.tar.gz   # -> ghcr.io/jamiesun/subnetra:<version>
docker run -d --name subnetra \
    --cap-add=NET_ADMIN --device=/dev/net/tun \
    -v "$PWD/config.json":/etc/subnetra/config.json:ro \
    ghcr.io/jamiesun/subnetra:<version>

macOS spoke binary

Each release also attaches native macOS binaries for running Subnetra as a spoke — subnetra-<version>-macos-arm64.tar.gz (Apple Silicon) and -amd64.tar.gz (Intel). They are Mach-O binaries that link only libSystem (zero third-party deps).

The install script works on macOS too and clears the Gatekeeper quarantine attribute for you.

tar -xzf subnetra-<version>-macos-arm64.tar.gz
cd subnetra-<version>-macos-arm64
# Gatekeeper quarantines downloaded binaries — clear it (or build from source):
xattr -d com.apple.quarantine subnetrad subnetra 2>/dev/null || true

./subnetra --print-network-plan --config config.json   # preview the host plan
sudo ./subnetrad --config config.json                  # utun creation needs root

Creating the utun interface and applying the ifconfig / route plan both require root. macOS is supported as a spoke only (the hub stays Linux/RouterOS); see the macOS Spoke guide.

Build from source

Requires Zig 0.16.0 or later.

# Native build (defaults to ReleaseSmall; pass -Doptimize=Debug for dev builds)
zig build

# Static cross-compile
zig build -Dtarget=x86_64-linux-musl     # amd64
zig build -Dtarget=aarch64-linux-musl    # arm64
zig build -Dtarget=arm-linux-musleabihf  # armv7 (hard float)
zig build -Dtarget=arm-linux-musleabi    # armv5 (soft float)
zig build -Dtarget=mipsel-linux-musl     # mipsel (LE: ramips/mt7621 — OpenWrt)
zig build -Dtarget=mips-linux-musl       # mips (BE: ath79/Atheros — OpenWrt)

# Run tests
zig build test

# Run the daemon
zig build run

Artifacts are placed in zig-out/bin/: subnetrad (daemon) and subnetra (control tool).

Tuning the peer cap

The peer registry and parsed config are fixed-capacity, zero-allocation arrays, so the maximum number of mesh peers a node can hold is a compile-time build option (-Dmax-peers) — not a runtime config field. It defaults to 32 and is capped at 128:

zig build -Dmax-peers=64                       # raise the local peer cap to 64
zig build -Dmax-peers=128 -Dtarget=aarch64-linux-musl   # combine with a target

A hub manages at most this many spokes. This is a per-node sizing knob — it is never negotiated on the wire, so a spoke that only talks to one hub does not need the hub’s larger cap. The control-plane policy-table size (MAX_POLICY_ENTRIES) is derived from this value, so raising the cap grows the policy capacity automatically (the default 32 yields a 272-entry table).

Raising it much higher trades memory and latency for capacity: the reactor is a single-threaded, per-packet O(N) scan over peers (and, with obfuscate on, a trial de-obfuscation per inbound datagram), so very large meshes are better served by splitting into several hubs than by one hub with hundreds of spokes.

ARMv5 note: ARMv5 has no hardware atomics, so the standard library’s threaded I/O scaffolding references legacy __sync_* intrinsics that musl does not provide. Because Subnetra is strictly single-threaded, src/atomic_shim.zig supplies a provably-correct plain (non-atomic) implementation, gated at comptime and compiled in only for pre-ARMv6 targets — every other architecture is byte-for-byte unaffected.

Verify the install

# Linux: confirm the binary is fully static
ldd ./subnetrad          # -> "not a dynamic executable"
ls -lh ./subnetrad       # -> < 512 KB

# Any platform: print the version banner
./subnetrad --version

Next: head to the Quick Start to bring up a hub and a spoke.

Quick Start

This walkthrough brings up the smallest useful mesh: one hub and two spokes, building a virtual 10.0.0.0/24 overlay. Two spokes is what makes the hub earn its name — it relays traffic between the spokes, which never talk to each other directly. It assumes you have the subnetrad daemon and subnetra control tool installed (see Installation).

Throughout: the hub is at the public address 203.0.113.1:18020, spoke A (overlay 10.0.0.2) at 203.0.113.2, and spoke B (overlay 10.0.0.3) at 203.0.113.3.

1. Generate a per-link key

Every link needs its own 32-byte pre-shared key (64 hex chars). Never reuse one key across peers — so this two-link mesh needs two keys:

openssl rand -hex 32   # → KEY_A, for the hub ↔ spoke-A link
openssl rand -hex 32   # → KEY_B, for the hub ↔ spoke-B link

2. Write the configs

The simplest way is to set a role and let the daemon derive the forwarding policy at boot.

Hub (config.json on 203.0.113.1) — lists both spokes:

{
  "role": "hub",
  "virtual_subnet": "10.0.0.0/24",
  "local_id": 1,
  "listen_ports": [18020, 18023, 18026],
  "peers": [
    { "id": 2, "endpoint": "203.0.113.2:18020", "allowed_src": "10.0.0.2/32", "psk": "…KEY_A…" },
    { "id": 3, "endpoint": "203.0.113.3:18020", "allowed_src": "10.0.0.3/32", "psk": "…KEY_B…" }
  ]
}

Spoke A (config.json on 203.0.113.2):

{
  "role": "spoke",
  "virtual_subnet": "10.0.0.0/24",
  "local_id": 2,
  "local_tun_ip": "10.0.0.2/24",
  "local_routes": ["10.0.0.2/32"],
  "peers": [
    { "id": 1, "endpoint": "203.0.113.1:18020", "allowed_src": "10.0.0.0/24", "psk": "…KEY_A…" }
  ]
}

Spoke B (config.json on 203.0.113.3) — same shape, its own id and address:

{
  "role": "spoke",
  "virtual_subnet": "10.0.0.0/24",
  "local_id": 3,
  "local_tun_ip": "10.0.0.3/24",
  "local_routes": ["10.0.0.3/32"],
  "peers": [
    { "id": 1, "endpoint": "203.0.113.1:18020", "allowed_src": "10.0.0.0/24", "psk": "…KEY_B…" }
  ]
}

Each link carries its own PSK: KEY_A is shared by the hub’s peer 2 and spoke A; KEY_B by the hub’s peer 3 and spoke B — and the two keys differ. The spokes need no listen_ports (a spoke binds the single default port). See the Configuration Reference for every field.

3. Validate before running

--check parses the config, runs every sanity rule, and exits without touching the network:

subnetrad --check --config config.json
# spoke A: subnetra v… (mtu=1452, udp_ports={ 18020 }, mode=raw_direct, local_id=2, peers=1) [config ok]
# hub:     subnetra v… (mtu=1452, udp_ports={ 18020, 18023, 18026 }, mode=raw_direct, local_id=1, peers=2) [config ok]

4. Print and apply the host network plan

The daemon creates the TUN device but does not configure host addressing, routes, or MTU (that would break the zero-dependency guarantee). Ask it to print the exact commands instead:

subnetrad --print-network-plan --config config.json

Review the emitted ip link / ip addr / ip route commands and run them (on macOS the plan emits ifconfig / route). See Host Network Plan for details, including how the safe MTU is computed.

Run this on each spoke. The hub has no local_tun_ip, so its plan only creates the bare TUN device — it is a pure relay with no overlay address.

5. Start the daemons

# On the hub and both spokes (TUN creation needs NET_ADMIN / root):
sudo subnetrad --config config.json

For a real deployment, run it under systemd or launchd instead — see Production Deployment.

6. Verify connectivity

From spoke A, ping spoke B — the packet goes A → hub → B and back, exercising the hub relay:

ping 10.0.0.3

Then check the live counters on any node:

subnetra status

On the spokes you should see udp_tx / udp_rx climbing and the peer online; on the hub the relay_* counters increment as it forwards between the spokes. If traffic is not flowing, the drop counters tell you why — read Observability & Troubleshooting.

The hub here is a pure relay with no overlay address, so there is nothing at 10.0.0.1 to ping. To make the hub itself reachable, see Roles → Reaching the hub itself.

7. Add Site-to-Site routes (optional)

To reach a LAN behind spoke B (e.g. 192.168.3.0/24), the hub needs a relay rule for that prefix. Inject it at runtime — it hot-updates over the control socket with no restart:

# On the hub:
subnetra policy add --src 0.0.0.0/0 --dst 192.168.3.0/24 --action forward --target 3
subnetra policy show
subnetra save              # persist the active policy back to config.json

Spoke B must also deliver that prefix locally (add it to local_routes) and be allowed to source it (widen peer 3’s allowed_src on the hub to cover it). The full Site-to-Site walkthrough is in Production Deployment.

Where to go next

• Roles — auto-derive policy from config.
• Architecture — how the data path works.
• Security Model — keys, epochs, anti-replay.
• Production Deployment — services, secrets, firewall/NAT, upgrades.

Architecture

Subnetra is a Layer-3 overlay: it moves raw IPv4 packets between nodes over an encrypted UDP underlay, arranged as a single-hub hub-and-spoke star. This page explains how a packet travels through the system and how the daemon is structured internally.

Topology

A central hub (typically an overseas relay or colocation node) anchors the mesh. Each spoke (a branch office, a RouterOS container, a Mac) connects to the hub over a private UDP tunnel. Spokes do not talk to each other directly — the hub relays between them by policy. Together they form a virtual subnet (e.g. 10.0.0.0/24) on top of whatever physical leased line connects them, and can route between LANs behind each node (Site-to-Site).

flowchart LR
    subgraph SiteA["Site A · LAN 192.168.1.0/24"]
        S2["Spoke 2<br/>10.0.0.2"]
    end
    subgraph HubBox["Hub · relay"]
        HUB["Hub<br/>10.0.0.1 / id 1<br/>policy + relay"]
    end
    subgraph SiteB["Site B · LAN 192.168.2.0/24"]
        S3["Spoke 3<br/>10.0.0.3"]
    end
    S2 <-->|"encrypted UDP underlay"| HUB
    HUB <-->|"encrypted UDP underlay"| S3

The data path

Every packet crosses the same five stages:

flowchart TD
    TUN["1 · TUN ingress<br/>read raw IPv4"] --> POL{"2 · Policy lookup<br/>longest-prefix CIDR"}
    POL -->|no match| DROP["DROP · counted"]
    POL -->|FORWARD| SEAL["3 · Header + seal<br/>20-byte header · ChaCha20-Poly1305 + 16-byte tag"]
    SEAL --> EGR["4 · Egress dispatch<br/>raw_direct over UDP"]
    EGR -. encrypted UDP underlay .-> RECV["5 · Receive &amp; verify<br/>key_id · epoch · nonce · anti-replay · inner-src"]
    RECV -->|local| DEL["Deliver to local TUN"]
    RECV -->|hub relay| RLY["Relay to another spoke<br/>never back to source"]

1. TUN ingress. The kernel routes a LAN/overlay packet to the virtual L3 device; the reactor reads the raw IPv4 packet non-blocking.
2. Policy lookup. The reactor atomically loads the active policy tree and does a longest-prefix CIDR match on the destination to decide FORWARD (and to which peer) or DROP.
3. Header + seal. It assembles the 20-byte private wire header (version, flags, key_id, session epoch, sequence number) and encrypts the inner packet with ChaCha20-Poly1305, appending the 16-byte tag.
4. Egress dispatch. The sealed datagram is sent over the UDP socket to the peer’s endpoint. v1 uses the raw_direct egress; kcp_arq / fec_xor are reserved for v2.
5. Receive & deliver. The peer verifies key_id, epoch, nonce/anti-replay and the inner source, decrypts, and either writes the inner packet to its own TUN (LOCAL) or relays it to another spoke (hub only, never back to the source).

Internal structure

The daemon is a single-threaded, event-driven reactor. One thread multiplexes three file descriptors and never blocks:

The readiness primitive is selected at comptime by the OS backend:

• Linux — epoll edge-triggered (EPOLLET), reading until EWOULDBLOCK.
• macOS — poll(2) (a spoke-only backend; kqueue is a later milestone).

Source modules:

Two memory tiers

Subnetra splits memory by responsibility instead of applying one blanket rule:

• Data plane (reactor, crypto): strictly zero allocation. All packet buffers are locked into resident memory at startup via a fixed allocator; the hot path never calls alloc/free. Under a saturating large-packet load the RSS line is flat — 0 bytes of jitter.
• Control plane / reliability (uds, policy rebuild, future ARQ/FEC): isolated arenas. These paths may allocate in arenas with independent lifetimes, but must never pollute the data-plane memory line.

Lock-free RCU policy updates

Because the whole daemon is single-threaded, there are no locks anywhere. The data plane reads the policy tree through a single *const PolicyTree pointer with an atomic load. When the control plane injects a rule, it builds a brand-new tree in an arena, then swaps it in with one atomic pointer store (an RCU pattern). The old tree is reclaimed once the event loop is idle. Hot updates are therefore zero-copy and jitter-free — an in-flight TCP throughput test sees no measurable latency spike during a policy swap.

Endpoint learning (NAT / roaming)

Spokes commonly sit behind NAT with changing public addresses. Subnetra carries the sender’s mesh id (key_id) in every datagram. When an authenticated packet arrives from a new underlay address, the hub relearns that peer’s endpoint and replies there — no handshake, no restart. This makes NAT remap and roaming self-healing. A built-in spoke→hub NAT keepalive keeps idle pinholes open (see Roles and Security Model).

For the exact byte layout and receiver rules, see the normative Wire Protocol.

Security Model

Subnetra’s transport security is mandatory in v1 and cannot be deferred. This page describes the threat model and every mechanism that enforces it. The byte-exact rules live in the normative Wire Protocol; this page is the conceptual companion.

Threat model

Subnetra assumes a hostile underlay: an attacker can observe, drop, modify, inject, and replay UDP datagrams, and can actively probe the listening port. The design goals are:

• Confidentiality & integrity of every inner packet.
• Stealth — an active prober cannot distinguish a Subnetra endpoint from a host that simply drops traffic.
• No replay of captured ciphertext into the protected network.
• Compartmentalization — compromising one spoke’s key must not forge any other link.

Per-link pre-shared keys

Each peers[] entry carries its own 32-byte PSK (64 hex chars). There is no mesh-wide shared secret; an old config that still carries a top-level psk is rejected (InvalidPsk), and reusing one PSK across peers is rejected (DuplicatePsk).

From each PSK, a directional link key is derived per ordered pair:

link_key(psk, from_id, to_id) =
    BLAKE2b-256(key = psk, msg = "subnetra-v1-link" || u32_be(from_id) || u32_be(to_id))

So a node’s transmit key to a peer equals that peer’s receive key for traffic from the node, and the two directions use distinct keys.

Why per-link keys are mandatory: under a shared PSK, two independent per-peer monotonic counters could emit the same (key, nonce) pair for different plaintexts, which catastrophically breaks ChaCha20-Poly1305. Giving every directional link its own key makes each link’s nonce space disjoint.

Session epoch — stateless, handshake-free sessions

Subnetra establishes no session on the wire. Instead, each daemon lifetime samples a boot epoch once at startup (wall-clock nanoseconds, u64) and derives a fresh per-session key:

session_key(link_key, epoch) =
    BLAKE2b-256(key = link_key, msg = "subnetra-v1-session" || u64_be(epoch))

The epoch travels in every datagram (8 bytes). The receiver derives the matching key statelessly from the epoch it sees and applies a forward-only rule:

• A larger (later) epoch, once authenticated, supersedes the old session and resets the anti-replay window.
• A smaller (older) epoch is dropped (it would be a cross-epoch replay of a retired session).

Because every restart yields a new key, sequence numbers can safely restart from 1 without ever reproducing a historical (key, nonce) pair — no disk persistence required.

Fail-closed clock: the epoch must be ≥ 2024-01-01T00:00:00Z in nanoseconds and non-zero. A node whose clock cannot satisfy this refuses to start, rather than emit a low/colliding epoch.

Accepted residual limitation: if a node’s wall clock runs backward across a restart (no RTC, not yet NTP-synced), its new epoch may be smaller than the one a peer remembers, and the peer will reject the new session until its clock advances past the old value. This is mitigated operationally (NTP/RTC), never by an in-protocol epoch exchange — there is no handshake by design.

Nonce & anti-replay

The 96-bit AEAD nonce is derived from a 64-bit monotonic counter that each endpoint increments per datagram — it is never fixed or reused. The receiver maintains a sliding-window (bitmap) anti-replay check per session: a sequence number outside the window, or one already seen, is dropped; an in-window out-of-order number is accepted. Without this, historical ciphertext could be replayed into the protected LAN.

Stealth: silent drop, no magic bytes

Subnetra is stateless obfuscation: the ciphertext contains no fixed magic number, and on any authentication or validation failure the packet is dropped silently — never answered with a TCP Reset, an ICMP error, or anything observable. To an active prober sending garbage (or replayed ciphertext) at the UDP port, the endpoint is indistinguishable from a black hole, and its CPU shows no unusual spike.

This defeats active probing. The 20-byte framing header is outside the AEAD, so if it travels in cleartext a passive on-path observer can fingerprint the protocol by its constant version, repeated epoch, and low monotonic seq. The deployment-wide obfuscate setting (on by default) (Wire Protocol → Header obfuscation) XOR-masks the header with a per-packet pad so the whole datagram looks random to such an observer — zero byte overhead, mesh-wide identical — and also de-periodizes the spoke’s NAT keepalive so its cadence is not a fingerprint. It hides the protocol fingerprint only, not packet length or general timing. Set obfuscate: false to opt out (readable cleartext header for packet-capture debugging).

Inner-source binding (anti-spoofing)

Each peer declares an allowed_src CIDR. After decryption, the receiver checks the inner IPv4 source address against that peer’s allowed_src; a packet whose inner source falls outside the allowed range is dropped (counted as spoof). This prevents an authenticated peer from injecting traffic that impersonates another node’s address space.

No-reflect relay guard

When the hub relays between spokes, it never sends a packet back to the peer it came from. Combined with longest-prefix policy routing, this prevents reflection loops.

NAT keepalive (one-way, never acknowledged)

A role=spoke enables a built-in NAT keepalive by default (keepalive_secs = 20): it sends one tiny authenticated datagram to its hub each interval so the spoke’s NAT pinhole stays open and the hub keeps a fresh route back. It is a one-way, never-acknowledged datagram gated purely by static config — it is not a handshake and does not weaken the stateless model. Set keepalive_secs to tune it, or 0 to disable (hub/manual default to 0).

What secrets are never exposed

subnetra status (and --json) deliberately never serialize PSKs or any derived key. Counters, endpoints, and health are observable; secrets are not.

Cryptographic primitives

See the Wire Protocol for the exact key schedule, nonce construction, header serialization, and the full receiver decision sequence — all pinned by known-answer test vectors.

Design Principles

Subnetra is built around a small set of non-negotiable constraints. They are not aspirational — they are binding invariants that shape every module. If a change conflicts with one of these, the change is wrong. (Internally these are the project’s “iron laws”.)

1. Zero third-party dependencies

No WireGuard, no ikcp.c, no network frameworks, no external crypto libraries. Only Zig’s standard library and raw syscalls via std.posix. Even the future v2 reliability layer must be in-house (an arena-based ARQ), never a vendored C library. The payoff is a single auditable artifact with no supply chain.

2. Layered zero dynamic allocation

Memory is constrained by responsibility, not with one blanket rule:

• Data plane (reactor, crypto): strictly allocation-free. All packet buffers live in resident memory fixed at startup; the hot path never allocates.
• Control plane / reliability: independent arenas. Policy rebuilds and the UDS path may allocate in isolated arenas with their own lifetimes, but must never pollute the data-plane memory line.

The acceptance bar — “0 bytes of RSS jitter under load” — applies to the raw_direct data plane; control-plane hot updates may briefly use reclaimable arena memory.

3. Single-threaded, lock-free reactor

One thread; a lock-free, allocation-free readiness loop. No threads, no locks, ever. Because there is no concurrency between the data and control planes, no mutex is needed; policy hot-updates happen via an atomic pointer swap (RCU). The specific readiness primitive is selected at comptime — Linux epoll edge-triggered, macOS poll(2) — but the single-thread / no-lock / no-per-packet- allocation invariant is the law.

4. Stateless obfuscation / stealth

ChaCha20-Poly1305 full encryption with no magic numbers in the ciphertext. On authentication failure, drop silently — never reply with a TCP Reset, ICMP, or anything observable. The endpoint is physically invisible to probing.

That covers active probing; a cleartext framing header would still let a passive observer fingerprint the protocol, which the mesh-wide obfuscate setting (on by default, Wire Protocol → Header obfuscation) masks at zero byte overhead — also de-periodizing the keepalive cadence (fingerprint only — not packet length or timing).

5. Mandatory transport security in v1

A private per-link PSK, a per-endpoint 64-bit monotonic nonce that is never reused, a per-restart session epoch, and a sliding-window anti-replay check. These cannot be deferred to a later version. See the Security Model.

6. A single minimal binary

Default -O ReleaseSmall, zero third-party deps on every platform.

• Linux: fully static against musl-libc; ldd → not a dynamic executable; target ≤ 512 KB.
• macOS: minimal-dynamic — links only libSystem (Apple ships no static libc), with its own recorded size baseline. The ldd-static check is a Linux-only gate.

7. Test-driven

Pure logic ships with tests; zig build test must stay green before any commit. The wire protocol is pinned by machine-checkable known-answer vectors (see Wire Protocol).

8. Stateless, handshake-free transport

Every datagram is self-describing and independently decodable — the per-packet epoch is the entire session-establishment mechanism. Subnetra performs no connection-establishment round-trip, no challenge/response, and no in-band session negotiation — not in v1, and not in v2. Any future transport mode is chosen by static per-link config (the reserved negotiation_version / flags fields), never negotiated on the wire.

Two consequences are accepted by design, not deferred:

1. An on-path attacker may replay a captured datagram of a not-yet-observed epoch to transiently relocate a peer’s endpoint — it self-heals on the peer’s next genuine packet, and an off-path attacker cannot forge it.
2. A node whose wall clock runs backward across a restart is rejected by peers until their clocks advance — mitigated operationally by NTP/RTC, never by an in-protocol epoch exchange.

The KEEPALIVE flag (bit 0) is a one-way, never-acknowledged spoke→hub NAT-pinhole datagram gated by static config. It is not a handshake and does not weaken this law; bits 1–7 stay reserved for static mode selection.

Scope discipline: v1 vs v2

• v1 (delivered): raw_direct data plane + PSK encryption + anti-replay + RCU hot-update policy engine.
• v2 (roadmap, interface only): kcp_arq and fec_xor — in-house reliability modes, selected by static per-link config, never an on-wire handshake. v1 only reserves the egress branch and the header negotiation_version / flags fields; v2 branches return error.NotImplemented.

There is no handshake on the roadmap. See the Roadmap.

Why these constraints?

The target is a steel pipe for a leased line into the most constrained environments imaginable (a RouterOS / BusyBox container). Determinism, a tiny auditable footprint, and stealth matter more than features. The constraints are what make the result deployable where heavier tools cannot go.

Configuration Reference

The daemon reads a single config.json from its working directory (override with --config <path>). If the file is missing it falls back to a compiled-in default. The parser is strict: unknown shapes, invalid CIDRs, or values out of range cause a fail-closed startup. Validate any change with subnetrad --check before deploying.

A minimal example (config.example.json):

{
  "negotiation_version": 1,
  "local_tun_mtu": 1452,
  "listen_ports": [18020, 18023, 18026],
  "virtual_subnet": "10.0.0.0/24",
  "local_id": 1,
  "obfuscate": true,
  "peers": [
    { "id": 2, "endpoint": "203.0.113.2:18020", "allowed_src": "10.0.0.2/32", "name": "bj-office-gw", "psk": "…64 hex…" },
    { "id": 3, "endpoint": "203.0.113.3:18020", "allowed_src": "10.0.0.3/32", "name": "colo-hub",     "psk": "…64 hex…" }
  ]
}

Top-level fields

Peer fields

Each entry of peers[]:

There is no mesh-wide psk. Every link carries its own key. A config that still has a top-level psk is rejected with InvalidPsk; reusing one PSK across peers is rejected with DuplicatePsk.

Peer cap. peers[] has a fixed, zero-allocation capacity chosen at build time by -Dmax-peers (default 32, max 128); a hub manages at most this many spokes, and the policy-table size scales with it. See Tuning the peer cap.

Sanity checks

config.zig enforces these at load (and under --check); any failure aborts startup:

• MTU range: local_tun_mtu must be 68–1500.
• Subnet overlap: the virtual subnet must not collide with the host’s physical subnet in a way that would blackhole traffic.
• Mesh ids: local_id is non-zero and distinct from every peer id.
• Unique PSKs: no PSK is shared across peers (DuplicatePsk); no top-level psk (InvalidPsk).
• Role rules (see Roles):
  • hub rejects a peer whose allowed_src is missing/0.0.0.0/0 or overlaps another peer’s allowed_src.
  • spoke requires exactly one hub peer, at least one local target (local_routes or local_tun_ip), and no 0.0.0.0/0 local route.

MTU and wire overhead

The fixed per-packet overhead is 64 bytes: a 20-byte private header + a 16-byte AEAD tag + 28 bytes of outer IPv4/UDP. The safe tunnel MTU is therefore path_mtu − 64. The default local_tun_mtu = 1452 assumes a 1500-byte underlay. On a smaller path (PPPoE, a VPN underlay), lower it — the host network plan computes and warns about this for you.

Where config meets the rest of the docs

• Bootstrap policy from role: Roles.
• Turning config into host commands: Host Network Plan.
• What the keys/epoch/allowed_src defend against: Security Model.
• Runtime policy injection that overlays this config: CLI Reference.

Roles

Instead of hand-injecting subnetra policy add rules, set a role and let the daemon derive the forwarding table at boot. Three roles are available; role defaults to "manual".

You can always layer extra subnetra policy rules on top of a derived table at runtime.

manual (default)

manual is the original, explicit mode and the default. The daemon derives no policy at boot — the forwarding table starts empty and you install every rule yourself over the control socket. Configs that predate roles keep working unchanged.

What manual changes vs. a derived role:

• No derived policy. You build the table with subnetra policy add.
• No role-specific --check. subnetrad --check still runs the universal sanity checks (MTU range, 16-bit ids, host-subnet overlap), but it does not apply the hub/spoke structural rules (per-peer allowed_src, exactly-one-hub, a local target, no 0.0.0.0/0 local route). A malformed forwarding intent is yours to catch.
• Keepalive defaults to 0. If a manual node sits behind NAT, set keepalive_secs yourself (a spoke does this for you).

What manual does not change — security is identical. Role only chooses the bootstrap policy; it never touches the data plane. Per-link encryption, session-epoch ordering, anti-replay, and — crucially — the per-peer allowed_src inner-source check all run exactly the same. Policy match is destination-only (longest-prefix); each peer’s allowed_src independently binds which inner source addresses that peer may assert. A hand-built manual table therefore cannot be tricked into accepting a spoofed inner source — you give up the derived convenience table and the role-specific guardrails, not the cryptographic guarantees.

When to use manual

• Topologies the hub/spoke shapes can’t express in a single node — e.g. a node that is a spoke upstream and a relay downstream at the same time (the hub/spoke roles each validate one posture; manual lets one node hold both). This is outside the single-tier model the derived roles validate, so the table — and the upstream hub’s allowed_src aggregation — is on you.
• Reproducing a hand-tuned policy table verbatim, or backward compatibility with a pre-role config.

Building the table by hand

Rules are destination-matched longest-prefix; src is permissive (0.0.0.0/0). --target 0 delivers to the local TUN, any other target relays to that peer id:

# On Linux the CLI default already matches the daemon, so no SUBNETRA_SOCK needed.
# Deliver this node's own overlay address locally.
sudo subnetra policy add --src 0.0.0.0/0 --dst 10.0.0.9/32  --action forward --target 0
# Relay a downstream prefix to peer 5; send everything else up to the hub (peer 1).
sudo subnetra policy add --src 0.0.0.0/0 --dst 10.0.0.32/27 --action forward --target 5
sudo subnetra policy add --src 0.0.0.0/0 --dst 10.0.0.0/24  --action forward --target 1
sudo subnetra policy show      # verify ordering
sudo subnetra save             # persist across restarts

Each peer must still carry the right allowed_src for the inner sources it is allowed to assert — that binding is enforced regardless of these rules.

spoke

A home/office spoke that exposes its own overlay IP and routes everything else through the relay needs only:

{
  "role": "spoke",
  "virtual_subnet": "10.0.0.0/24",
  "local_id": 2,
  "local_tun_ip": "10.0.0.2/24",
  "local_routes": ["10.0.0.2/32"],
  "peers": [
    { "id": 1, "endpoint": "203.0.113.1:18020", "allowed_src": "10.0.0.0/24", "psk": "…64 hex…" }
  ]
}

This derives, automatically:

• 10.0.0.2/32 → LOCAL (deliver to this node’s own TUN)
• 10.0.0.0/24 → hub(id 1) (everything else goes through the relay)

To publish a LAN behind the spoke (Site-to-Site), add it to local_routes (e.g. ["10.0.0.2/32", "192.168.2.0/24"]) so the derived table delivers that prefix locally.

Built-in NAT keepalive

A spoke turns on the NAT keepalive by default (keepalive_secs = 20). It sends one tiny authenticated datagram to its hub every interval so an idle spoke’s NAT pinhole stays open and the hub keeps a fresh route back — no external pinger, no cron job. Set keepalive_secs explicitly to tune it, or 0 to disable.

Validation rules for spoke

subnetrad --check enforces:

• exactly one hub peer,
• at least one local target (local_routes or local_tun_ip),
• no 0.0.0.0/0 local route (which would tie the host default route to the tunnel and blackhole it).

hub

The matching hub just lists its spokes; each peer’s allowed_src becomes a forward rule to that peer:

{
  "role": "hub",
  "virtual_subnet": "10.0.0.0/24",
  "local_id": 1,
  "peers": [
    { "id": 2, "endpoint": "203.0.113.2:18020", "allowed_src": "10.0.0.2/32", "psk": "…64 hex…" },
    { "id": 3, "endpoint": "203.0.113.3:18020", "allowed_src": "10.0.0.3/32", "psk": "…64 hex…" }
  ]
}

This derives 10.0.0.2/32 → peer 2 and 10.0.0.3/32 → peer 3. The hub relays between spokes by longest-prefix match, and never reflects a packet back to its source.

Reaching the hub itself

The derived hub table forwards only to spokes — it never delivers to the hub’s own TUN. So by default a hub has no overlay address and is not reachable on the overlay: it is a pure relay. This is usually exactly what you want — the relay exposes nothing addressable on the mesh.

To make the hub itself reachable over the tunnel (to SSH into it, or to host a service on the overlay), do two things:

1. give it an address with local_tun_ip so the network plan configures its TUN, and

2. add a local-delivery rule for that address — either run the node as manual with an explicit table, or layer one rule on top of the derived hub table:

   # deliver the hub's own overlay address to its local TUN
   sudo -E subnetra policy add --src 0.0.0.0/0 --dst 10.0.0.1/32 --action forward --target 0
   

Leave local_tun_ip unset (and add no such rule) to keep the hub relay-only, with nothing on the overlay able to address it.

Validation rules for hub

subnetrad --check rejects:

• a peer with a missing allowed_src (or the permissive 0.0.0.0/0), since the hub could not tell which spoke a packet belongs to;
• two peers whose allowed_src prefixes overlap, which would make forwarding ambiguous.

A hub’s keepalive defaults to 0 (it does not initiate keepalives to spokes).

Ready-to-edit examples

The repository’s deploy/ directory ships editable hub.json, spoke-a.json, and spoke-b.json, plus the service units. The full hub + two-spoke walkthrough is in Production Deployment.

Host Network Plan

subnetrad creates the TUN device but deliberately does not configure host addressing, routes, or MTU. Auto-applying host networking would mean shelling out or linking extra libraries, which breaks the zero-dependency single-binary guarantee. Instead, the daemon prints the exact commands for the loaded config so you can review and run them.

Print the plan

# Print the host networking plan for this node (defaults to a 1500-byte underlay).
subnetrad --print-network-plan --config config.json

# Override the underlay path MTU (e.g. behind a PPPoE / VPN underlay):
subnetrad --print-network-plan --path-mtu 1420 --config config.json

Output is deterministic and print-only — nothing on the host is modified. The backend is selected at comptime, so the plan matches your platform:

• Linux emits ip link / ip addr / ip route commands.
• macOS emits ifconfig / route commands.

What it emits

For the loaded config the Linux plan emits:

• ip link set <tun> mtu <local_tun_mtu> up
• ip addr add <local_tun_ip> dev <tun> — set the optional local_tun_ip config field (e.g. "local_tun_ip": "10.0.0.2/24"); otherwise a placeholder is shown.
• ip route add <subnet> dev <tun> for each peer’s allowed_src — a permissive 0.0.0.0/0 is skipped so you never blackhole the default route.
• an optional TCP MSS clamp hint (nftables / iptables) to avoid PMTU blackholes.

The MTU calculation

The plan computes the safe tunnel MTU from the real wire overhead:

header 20  +  AEAD tag 16  +  outer IPv4/UDP 28  =  64 bytes
max tunnel MTU = path_mtu − 64

If the configured local_tun_mtu exceeds that, the plan prints a warning — this is the classic cause of “small packets work, large transfers stall” (PMTU blackholing). On a standard 1500-byte path, 1436 is the safe default; on a 1420-byte underlay you would lower local_tun_mtu to 1356.

Measure the real path MTU

--print-network-plan assumes a 1500-byte underlay (override with --path-mtu). But the real path is often smaller — PPPoE is 1492, and CGNAT/mobile/tunneled uplinks vary — and the usual way to discover it, kernel Path MTU Discovery, relies on routers returning ICMP “fragmentation needed”. That ICMP is frequently filtered (a PMTU black hole), which is exactly the kind of network an obfuscated overlay runs across.

The mtu-probe tool measures the path actively, end to end, over plain UDP without trusting ICMP, then prints the local_tun_mtu to configure. Run the responder on one node and the prober on the other:

zig build tool:mtu-probe

# on the far node (e.g. the hub, at its public endpoint):
zig-out/tools/mtu-probe --listen 18020

# on the near node — binary-searches the largest datagram that round-trips
# with the Don't-Fragment bit set, then recommends a safe local_tun_mtu:
zig-out/tools/mtu-probe --probe 203.0.113.9:18020
#   underlay path MTU : 1492 bytes   (largest UDP payload that round-tripped: 1464 + 28 IPv4/UDP)
#   subnetra overhead : 64 bytes
#   recommended       : local_tun_mtu = 1428

Set the printed value as local_tun_mtu (and re-run --print-network-plan to confirm the warning is gone). On jumbo-frame paths pass --ceil 9000. See tools/README.md for details.

Apply it

Review the emitted commands, then run them (most require root):

subnetrad --print-network-plan --config config.json | sudo sh   # after reviewing!

Prefer to inspect the output first and paste the commands deliberately, rather than piping straight to a shell — the plan is meant to be auditable.

Why this design

Keeping host networking out of the daemon means:

• the binary stays dependency-free and tiny,
• the same daemon behaves identically across Linux, containers, and macOS,
• operators retain full control and an auditable record of every change to the host’s addressing and routing.

See the Configuration Reference for the fields that feed the plan (local_tun_ip, local_tun_mtu, each peer’s allowed_src) and Production Deployment for wiring it into a service.

Production Deployment

This page condenses the full hub + two-spoke production walkthrough. For the exhaustive version — including traffic shaping, NIC tuning, and benchmarking — see docs/deployment.md in the repository. Ready-to-edit artifacts live in deploy/ (subnetrad.service, net.subnetra.subnetrad.plist, hub.json, spoke-a.json, spoke-b.json).

0. Components

A deployment has one hub (stable public UDP endpoint) and one or more spokes (outbound-only, often behind NAT). Each runs the same subnetrad daemon and subnetra control tool; the difference is configuration (Roles).

1. Install the binary

Use a release tarball or container image. On a bare host:

sudo install -m 0755 subnetrad subnetra /usr/local/bin/

2. Provision config and secrets

Place config.json where the service expects it (the units use /etc/subnetra/config.json), owned by root and mode 0600 because it contains PSKs:

sudo install -d -m 0750 /etc/subnetra
sudo install -m 0600 hub.json /etc/subnetra/config.json
subnetrad --check --config /etc/subnetra/config.json
# subnetra v… (mtu=…, mode=raw_direct, local_id=…, peers=…) [config ok]

Generate each link’s PSK with openssl rand -hex 32 and use a unique value per link. See the Security Model.

3. Host networking

The daemon prints — but never applies — the host plan. Review and run it (see Host Network Plan):

subnetrad --print-network-plan --config /etc/subnetra/config.json

4. Run as a service

Linux — systemd

sudo install -m 0644 deploy/subnetrad.service /etc/systemd/system/subnetrad.service
sudo systemctl daemon-reload
sudo systemctl enable --now subnetrad
journalctl -u subnetrad -f

The unit requests only CAP_NET_ADMIN, grants /dev/net/tun, runs subnetrad --check as ExecStartPre, restarts on failure, and is otherwise sandboxed (ProtectSystem=strict, NoNewPrivileges, restricted address families). Edit the commented ExecStartPost lines to match your --print-network-plan output.

macOS — launchd

A macOS spoke runs as a system daemon (creating a utun needs root):

sudo install -m 0644 deploy/net.subnetra.subnetrad.plist \
    /Library/LaunchDaemons/net.subnetra.subnetrad.plist
sudo launchctl bootstrap system /Library/LaunchDaemons/net.subnetra.subnetrad.plist
sudo launchctl enable system/net.subnetra.subnetrad
sudo tail -f /var/log/subnetrad.log
# subnetra v… (… mode=raw_direct …) tun=utun4 sock=/var/run/subnetra.sock [ready]

The utunN name is kernel-assigned — read it from the [ready] banner and apply the plan after the daemon is up. See the macOS Spoke guide.

5. Install the relay policy (hub)

Shortcut: if your config sets "role": "hub" / "spoke", the daemon derives this whole policy at boot and you can skip this section. See Roles.

For role=manual, install the relay/delivery rules at runtime over the control socket (hot-swapped, no restart). On Linux the CLI default already matches the daemon, so no SUBNETRA_SOCK is needed (set it only for a custom path):

# Hub: relay overlay traffic to the right spoke
sudo -E subnetra policy add --src 0.0.0.0/0 --dst 10.0.0.2/32 --action forward --target 2
sudo -E subnetra policy add --src 0.0.0.0/0 --dst 10.0.0.3/32 --action forward --target 3
sudo -E subnetra policy show
sudo -E subnetra save        # persist a replayable snapshot

# Spoke: deliver tunnelled traffic for the local overlay address to the local TUN (target 0)
sudo -E subnetra policy add --src 0.0.0.0/0 --dst 10.0.0.2/32 --action forward --target 0

6. Operate

subnetra status shows peers, traffic, and per-reason drops; --json is the stable schema for monitoring. See Observability & Troubleshooting.

7. Firewall / NAT

• The hub must accept inbound UDP on all configured listen_ports from the internet. The default is the explicit set 18020, 18023, 18026 (not a range), avoiding WireGuard’s well-known port fingerprint and keeping the node reachable if one port is blocked or throttled.
• Each spoke needs only outbound UDP reachability to the hub — no inbound port-forwarding (the spoke initiates). A spoke therefore only needs one listen port (the role-aware default binds just [18020]); multiple listen_ports are a hub-side feature, because the hub is the reachable endpoint that spokes target.
• If a spoke’s NAT mapping changes, the hub re-learns its new endpoint from the next authenticated datagram. Keep the hub endpoint stable; spokes always initiate.

NAT keepalive (built-in)

An idle spoke’s NAT mapping times out (often ~30 s for UDP), after which inbound relays would blackhole. A role=spoke runs a built-in keepalive by default (keepalive_secs = 20): one tiny authenticated datagram per interval holds the pinhole open and keeps the hub’s learned endpoint fresh. It is allocation-free and adds no thread or external process. Confirm it with the keepalive tx / keepalive rx counters. Set keepalive_secs = 0 to disable (e.g. a spoke not behind NAT).

Hub on a dynamic IP (DDNS)

Endpoints are numeric IP:port and endpoint learning is one-way — a spoke cannot discover a hub that moved. Prefer a stable public IP for the hub. If you must run it behind a dynamic address, solve it operationally on each spoke with a small DDNS watcher that rewrites the endpoint and restarts the (stateless) daemon — no daemon changes.

Hub behind NAT (static port-forward)

A hub does not need a public-IP box — it needs a stable, inbound-reachable UDP endpoint. A host behind NAT qualifies if the edge router has static port-forwards (DNAT) from fixed public UDP ports to the hub’s internal IP:listen_ports:

• Spokes dial the external address. Each spoke’s peer endpoint is one public IP:port from the forward (typically the primary :18020), not the hub’s private address.
• listen_ports are the internal targets. Forward each configured UDP port (for the default, public 18020/18023/18026 → internal 18020/18023/18026). The singular listen_port remains a back-compat alias for a one-port deployment and is ignored when listen_ports is present.
• The mapping must be static. A fixed port-forward, not dynamic PAT that rewrites the source port per flow. If the public IP itself also changes, combine this with the DDNS approach above.
• Same-LAN spokes need the internal endpoint (hairpin). A spoke inside the same NAT often cannot reach the hub through the public IP unless the router does NAT hairpin/loopback — give those spokes the hub’s internal primary IP:port instead.
• CGNAT cannot host a hub. If the “public” address is itself carrier-grade NAT with no inbound port control, you cannot forward to it; that host can only be a spoke.

This is still an ordinary single hub — only its reachability is via DNAT — so it stays inside the validated single-tier model. Endpoint learning remains one-way: keep the external mapping stable; spokes always initiate.

8. High availability

v1 is single-hub by design. The data plane is single-path, stateless, and handshake-free, and the daemon never probes peer health or auto-switches paths (an explicit non-goal). Multi-hub and failover are therefore built around the daemon with ordinary config + OS tooling, driven by the observe-only health in subnetra status --json (online, last_seen_age_seconds, a flat auth_or_invalid). Two patterns are sanctioned.

Pattern A — active/standby hub VIP (recommended)

Two hub boxes sit behind one VRRP/keepalived VIP (or an anycast prefix). They share identical config — same local_id, same per-spoke PSKs, same derived policy — so every spoke sees exactly one peer (the VIP) and needs no special config: a normal role=spoke pointing at the VIP as its single hub.

flowchart LR
    subgraph Spokes
        S2["Spoke · 10.0.0.2"]
        S3["Spoke · 10.0.0.3"]
    end
    VIP(["Hub VIP · 203.0.113.10:18020"])
    subgraph HubPair["Hub pair · shared local_id + PSKs"]
        HA["hub-a · ACTIVE"]
        HB["hub-b · standby"]
    end
    S2 <-->|"encrypted UDP"| VIP
    S3 <-->|"encrypted UDP"| VIP
    VIP --- HA
    VIP -. takeover .- HB

Minimal keepalived on each hub box, with a notify hook that restarts the daemon on takeover:

vrrp_instance subnetra {
    state BACKUP            # BACKUP + nopreempt on both avoids needless flaps
    interface eth0
    virtual_router_id 51
    priority 150            # 150 on hub-a, 100 on hub-b
    nopreempt
    advert_int 1
    virtual_ipaddress { 203.0.113.10/32 }
    notify_master "/usr/local/sbin/subnetra-takeover.sh"
}

# /usr/local/sbin/subnetra-takeover.sh  — runs when this box wins the VIP.
#!/bin/sh
# Restart so the daemon samples a FRESH boot epoch above the old active's (see below).
systemctl restart subnetrad

Two caveats are load-bearing:

• Epoch ordering (the #1 gotcha). Every datagram carries the sender’s boot epoch (wall-clock ns at start) and receivers are forward-only: a session whose epoch is lower than the one a spoke already accepted is dropped before crypto until wall-clock passes it. A long-idle standby that booted before the active presents a lower epoch and is silently blackholed. Mitigation: keep both hubs on NTP and restart the daemon at takeover (the notify_master hook) so it stamps a fresh, higher epoch. This is exactly why active/standby beats active/active here.
• Endpoint re-learn window. Endpoint learning is one-way, so the new active starts with no learned spoke endpoints: hub→spoke relay blackholes until each spoke’s next keepalive re-teaches it (spoke→hub works immediately — the spoke initiates). Recovery is bounded by keepalive_secs (spoke default 20); lower it on the spokes for faster failover.

Pattern B — static dual-hub (independent identities)

Two fully independent hubs — distinct local_id, distinct PSKs, no shared secrets and no epoch coupling — both relaying the same overlay. Every spoke is a peer of both and keeps a primary; you switch by editing the spoke’s policy. This also enables locality: point regional prefixes at the in-region hub so only cross-region destinations traverse a long-haul link (each regional hub must carry the spokes you want locally reachable).

Because role=spoke validates exactly one hub peer, a two-hub spoke runs role=manual and installs its own longest-prefix policy over the control socket:

# Primary path: the whole overlay via hub-1 (id 1); local delivery for self.
sudo -E subnetra policy add --src 0.0.0.0/0 --dst 10.0.0.0/24 --action forward --target 1
sudo -E subnetra policy add --src 0.0.0.0/0 --dst 10.0.0.5/32 --action forward --target 0
sudo -E subnetra policy show
sudo -E subnetra save

# Fail the overlay over to hub-2 (id 2): two /25s out-specify the /24 (longest
# prefix wins), diverting all overlay traffic without touching the /24 rule.
sudo -E subnetra policy add --src 0.0.0.0/0 --dst 10.0.0.0/25   --action forward --target 2
sudo -E subnetra policy add --src 0.0.0.0/0 --dst 10.0.0.128/25 --action forward --target 2

Keep the two hubs’ prefixes non-overlapping when load-splitting — the same discipline role=hub enforces on allowed_src — to avoid split-brain (two relays claiming one destination).

No live policy replace. The control socket is append-only (policy add / policy show / save — there is no replace, del, or clear), and longest-prefix only lets a more specific rule win. To move a prefix you therefore either (a) restart the spoke daemon with an updated config/snapshot — it is stateless, so a restart costs only the keepalive re-learn window — or (b) push a more-specific override as above (the table grows; a clean revert still needs a restart). Design the split to be mostly static; do not treat Pattern B as sub-second failover.

Choosing a pattern

Key-material note (Pattern A). Sharing local_id + PSKs puts identical secrets on two boxes — secure both to the same standard; a compromise of either is a compromise of every spoke link. The daemon itself makes no failover decision in either pattern.

9. Traffic shaping & tuning

On long cross-ISP links the dominant cause of jitter/loss is the underlay, not detection. All shaping is done at the OS layer with tc — no daemon or protocol changes. Cap your egress to ~60–80% of the link’s stable throughput, smooth bursts, and (optionally) tune socket buffers and IRQ/CPU affinity. The kernel sees the real inner five-tuples on the cleartext snr0 device. See docs/deployment.md §9–§10 for the full recipes and the live-overlay benchmark.

Containers

Subnetra is a natural fit for containers: the daemon is a single static binary with no shared-library dependencies. The published image is multi-arch (amd64 / arm64 / armv7 / armv5) and Docker selects the right one automatically.

Requirements

A container running subnetrad needs:

• the NET_ADMIN capability (to create the TUN device),
• access to /dev/net/tun,
• a config.json mounted into its working directory (/etc/subnetra).

Docker

docker pull ghcr.io/jamiesun/subnetra:latest

docker run -d --name subnetra \
    --cap-add=NET_ADMIN --device=/dev/net/tun \
    -v "$PWD/config.json":/etc/subnetra/config.json:ro \
    ghcr.io/jamiesun/subnetra:latest

The image ships a Docker HEALTHCHECK that runs subnetra status, so docker ps reports the container healthy once the daemon is serving its control socket and unhealthy if it stops responding.

Docker Compose

services:
  subnetra:
    image: ghcr.io/jamiesun/subnetra:latest
    container_name: subnetra
    restart: unless-stopped
    cap_add: [NET_ADMIN]
    devices:
      - /dev/net/tun
    volumes:
      - ./config.json:/etc/subnetra/config.json:ro

Apply the host network plan for the overlay on the host or inside the container’s network namespace as appropriate for your topology.

Kubernetes

Run it as a DaemonSet (or Deployment) with the capability and device. A minimal container spec:

securityContext:
  capabilities:
    add: ["NET_ADMIN"]
volumeMounts:
  - name: tun
    mountPath: /dev/net/tun
  - name: config
    mountPath: /etc/subnetra/config.json
    subPath: config.json
volumes:
  - name: tun
    hostPath: { path: /dev/net/tun, type: CharDevice }
  - name: config
    secret: { secretName: subnetra-config }   # config.json contains PSKs — use a Secret

The HEALTHCHECK maps cleanly to a liveness/readiness probe: exec: { command: ["subnetra", "status"] }.

Because config.json carries per-link PSKs, store it in a Kubernetes Secret (or a Compose/host file with 0600 permissions), never in a plain ConfigMap or image layer.

Image internals

The amd64, arm64, and arm/v7 images are built FROM busybox:musl: they carry the two static binaries, config.example.json, and a tiny BusyBox shell plus core utilities for in-container debugging. The daemon itself is fully static and needs nothing from the base.

Because no musl BusyBox publishes linux/arm/v5, the arm/v5 image is built independently FROM scratch (still static musl, but without a debug shell) and stitched into the same :latest / :version manifest. The build cross-compiles with Zig pinned to $BUILDPLATFORM, so no QEMU emulation is needed. See the Dockerfile.

Offline / air-gapped

Devices that cannot reach a registry can docker load the per-arch image tarballs attached to each release — see Installation → Offline install.

RouterOS containers

MikroTik/RouterOS Container is a special case (it manages the container’s Ethernet side through a veth, and image import may need a legacy archive layout). It has its own guide: RouterOS Spoke.

RouterOS Spoke

This guide covers running Subnetra as a spoke inside a RouterOS Container on a MikroTik device, dialing a public Linux hub. The full reference, including scripted (.rsc) bring-up/teardown, is docs/routeros-container.md; scripts live in deploy/routeros/.

Why RouterOS is different

RouterOS Container is not a normal Linux host:

• RouterOS manages the container’s Ethernet side through a veth.
• Subnetra creates its own Linux snr0 TUN device inside the container.
• RouterOS cannot manage that snr0 directly — it routes to the container through the veth gateway.
• RouterOS image import may require a legacy Docker archive layout.

Recommended topology

Put the hub on a public Linux server with a stable UDP endpoint; put NATed RouterOS devices behind it as spokes.

Public Linux Hub
  underlay: 203.0.113.10:18020
  overlay:  10.66.0.1/24

RouterOS office Spoke
  container veth:           172.30.66.2/30
  RouterOS veth side:       172.30.66.1/30
  subnetra TUN in container: 10.66.0.3/24
  published LAN:            192.168.88.0/24

Do not make a NATed RouterOS device the public hub unless its UDP endpoint is stable and reachable from every spoke. LAN addresses here are examples — replace 192.168.88.0/24 with your real LAN.

Prerequisites

• RouterOS v7 with the container package installed.
• Device mode allows containers (/system/device-mode/print).
• A writable storage path for container root dirs and image archives.
• /dev/net/tun visible inside RouterOS containers.
• Outbound UDP from the RouterOS device to the public hub.

/system/package/print
/system/device-mode/print
/container/print

Outline of the bring-up

1. Provision the veth pair and addressing (container side 172.30.66.2/30, RouterOS side 172.30.66.1/30).
2. Import the image (a docker load-able tarball from a release, or pull from GHCR if the device can reach it) and create the container with the mounted config.json, NET_ADMIN, and /dev/net/tun.
3. Set the spoke config (role: spoke, the hub as the single peer, the published LAN in local_routes) — see Roles.
4. Apply routing on RouterOS so LAN traffic for the overlay/remote prefixes is routed to the container’s veth gateway, and the published LAN is reachable across the tunnel.
5. Verify with subnetra status inside the container and a ping across the overlay.

The scripted .rsc files in deploy/routeros/ automate steps 1–4 (and a matching teardown). Follow the full guide for the exact commands, including LAN publishing and the NAT/endpoint notes for NATed spokes.

Endpoint roaming note

Protocol-level endpoint learning means the hub re-learns a NATed spoke’s endpoint from its next authenticated datagram, and the built-in NAT keepalive (role=spoke default) keeps the pinhole open — so a RouterOS spoke behind a changing NAT mapping stays reachable without manual endpoint correction.

OpenWrt Spoke

This guide covers running Subnetra as a spoke on an OpenWrt router (MIPS or ARM home/SOHO CPE) behind NAT, dialing a public Linux hub. OpenWrt is a natural fit: it already ships musl, so the static binary just runs, and a NATed router is exactly what the spoke role is built for.

The ready-to-use procd service is deploy/openwrt/subnetrad.init.

Why OpenWrt is different

• procd, not systemd. Use the provided /etc/init.d/subnetrad init script instead of the systemd unit.
• TUN is a module. Install kmod-tun so the daemon can create its snr0 device via /dev/net/tun.
• BusyBox userland. The doctor.sh preflight is BusyBox-friendly and runs as-is.
• Small flash. Each binary is well under 512 KB; both fit comfortably in the overlay (/usr/sbin, /usr/bin).

Pick the right binary

Subnetra publishes static musl tarballs per architecture. Map your router to one with opkg print-architecture (or uname -m):

Endianness matters for MIPS. mipsel is little-endian (ramips and most modern devices); mips is big-endian (ath79/Atheros). Installing the wrong one fails to exec. When in doubt, check opkg print-architecture.

Install

# 1. TUN module (one-time)
opkg update && opkg install kmod-tun

# 2. Resolve the latest release + your arch, download, verify, install
ARCH=mipsel   # one of: mipsel | mips | armv7 | arm64  (see the table above)
VER=$(uclient-fetch -qO - \
        https://api.github.com/repos/jamiesun/subnetra/releases/latest \
        | sed -n 's/.*"tag_name": *"\([^"]*\)".*/\1/p')
base="https://github.com/jamiesun/subnetra/releases/download/$VER"
cd /tmp
uclient-fetch -q "$base/subnetra-$VER-linux-$ARCH.tar.gz"
uclient-fetch -q "$base/SHA256SUMS.txt"
sha256sum -c SHA256SUMS.txt 2>/dev/null | grep "subnetra-$VER-linux-$ARCH.tar.gz: OK"
tar -xzf "subnetra-$VER-linux-$ARCH.tar.gz"

# 3. Place the binaries (daemon in sbin, client in bin)
install -m 0755 "subnetra-$VER-linux-$ARCH/subnetrad" /usr/sbin/subnetrad
install -m 0755 "subnetra-$VER-linux-$ARCH/subnetra"  /usr/bin/subnetra
subnetrad --version

Configure the spoke

Put this node’s config (with its PSKs) at /etc/subnetra/config.json, root-owned and mode 0600. A minimal spoke that publishes the router’s LAN across the tunnel:

{
  "role": "spoke",
  "virtual_subnet": "10.0.0.0/24",
  "local_id": 7,
  "local_tun_ip": "10.0.0.7/24",
  "local_routes": ["10.0.0.7/32", "192.168.1.0/24"],
  "peers": [
    { "id": 1, "endpoint": "203.0.113.1:18020", "allowed_src": "10.0.0.0/24", "psk": "…64 hex…" }
  ]
}

See Roles for what the spoke role derives, and config-gen / keygen to scaffold a matching hub + spoke set with fresh per-link PSKs.

mkdir -p /etc/subnetra
# (copy your config in, then lock it down)
chmod 0600 /etc/subnetra/config.json
subnetrad --check --config /etc/subnetra/config.json   # validate offline

Install the procd service

# from a checkout, or download the raw file:
uclient-fetch -qO /etc/init.d/subnetrad \
  https://raw.githubusercontent.com/jamiesun/subnetra/main/deploy/openwrt/subnetrad.init
chmod 0755 /etc/init.d/subnetrad
/etc/init.d/subnetrad enable
/etc/init.d/subnetrad start
logread -e subnetrad        # check it came up (and didn't fail --check)

The init script runs subnetrad --check before starting (so a bad config fails fast instead of respawn-looping), keeps the daemon respawned, and bounces it when the network config reloads.

Apply the host network plan

Subnetra never edits routes itself — it only prints the plan. Preview it and apply the result after the service is up (the snr0 device exists only while the daemon runs):

subnetra --print-network-plan --config /etc/subnetra/config.json
# then apply the printed ip/route commands, e.g.:
ip link set snr0 mtu 1400 up
ip addr add 10.0.0.7/24 dev snr0

To make it survive reboots, add the equivalent to /etc/config/network (an interface with proto none bound to snr0, plus static routes) or a small hotplug script — but keep route application on the OpenWrt side, never in the daemon.

Verify

subnetra status                 # peers, traffic, per-reason drops
sh doctor.sh                    # TUN / capabilities / clock preflight (BusyBox-ok)
ping -c3 10.0.0.1               # across the overlay to the hub

Topology notes

• Behind NAT = ideal spoke. The built-in NAT keepalive (role=spoke default, keepalive_secs = 20) holds the pinhole open and keeps the hub’s learned endpoint fresh, so a roaming/CGNAT-changing mapping stays reachable with no manual endpoint correction.
• A router with static port-forwards can be a hub. If this OpenWrt box has stable public UDP ports DNAT’d to its listen_ports, it can run role=hub instead — see Hub behind NAT.
• Time sync. The session key uses a CLOCK_REALTIME boot epoch ordered forward-only, so run sysntpd and let the clock settle before/at start (the init starts late, START=95). See the time-sync note in Production Deployment.

macOS Spoke

macOS is supported as a spoke that dials an existing Linux/RouterOS hub. The data path runs natively on utun + poll(2), selected at comptime behind src/os/. A macOS hub, kqueue, and automatic route mutation are explicitly out of scope.

Because macOS has no network namespaces and hosted-mac CI runners cannot create a utun without elevated privileges, the macOS spoke is runbook-certified rather than CI-gated. The authoritative procedure is docs/macos-spoke-acceptance.md.

Prerequisites

• A real Mac (Apple Silicon or Intel) with sudo access — utun creation needs root.
• A release macOS binary, or Zig 0.16.0+ to build from source.
• A reachable, already-working Linux/RouterOS hub with a stable underlay endpoint and a per-peer PSK issued for this Mac.
• At least one remote overlay target to ping across the tunnel.

The macOS binary is minimal-dynamic — it links only libSystem (still zero third-party deps), so it is not a static executable. Do not run the Linux-only ldd → not a dynamic executable check against it.

Install

From a release tarball (subnetra-<version>-macos-arm64.tar.gz or -amd64):

tar -xzf subnetra-<version>-macos-arm64.tar.gz
cd subnetra-<version>-macos-arm64
# Gatekeeper quarantines downloaded binaries — clear it (or build from source):
xattr -d com.apple.quarantine subnetrad subnetra 2>/dev/null || true

Or build locally with zig build (see Installation).

Configure

Write a spoke config.json (see Roles) with the hub as the single peer and this Mac’s overlay address:

{
  "role": "spoke",
  "virtual_subnet": "10.0.0.0/24",
  "local_id": 4,
  "local_tun_ip": "10.0.0.4/24",
  "local_routes": ["10.0.0.4/32"],
  "peers": [
    { "id": 1, "endpoint": "203.0.113.1:18020", "allowed_src": "10.0.0.0/24", "psk": "…64 hex…" }
  ]
}

Preview the host plan

On macOS the plan emits ifconfig / route commands (the daemon never applies them):

./subnetra --print-network-plan --config config.json

Run

sudo ./subnetrad --config config.json
# subnetra v… (… mode=raw_direct …) tun=utun4 sock=/var/run/subnetra.sock [ready]

The utunN interface name is kernel-assigned — read it from the [ready] banner, then apply the plan with the real name:

sudo ifconfig utun4 inet 10.0.0.4 10.0.0.4 mtu 1400 up
sudo route add -net 10.0.0.0/24 -interface utun4

Run under launchd

For a persistent spoke, install the system daemon plist (it runs as root, restarts on abnormal exit, and logs to /var/log/subnetrad.log):

sudo install -m 0644 deploy/net.subnetra.subnetrad.plist \
    /Library/LaunchDaemons/net.subnetra.subnetrad.plist
sudo launchctl bootstrap system /Library/LaunchDaemons/net.subnetra.subnetrad.plist
sudo launchctl enable system/net.subnetra.subnetrad

Manage it with sudo launchctl kickstart -k system/net.subnetra.subnetrad (restart) and sudo launchctl bootout system/net.subnetra.subnetrad (stop).

KeepAlive may restart the daemon onto a different utunN; re-read the banner and re-apply the plan. Subnetra deliberately leaves routing to you (no automatic route mutation).

Verify

sudo ./subnetra status      # peer online, counters climbing
ping 10.0.0.3               # a remote overlay target across the tunnel

Exit Node & Outbound

Subnetra is a Layer-3 encrypted channel, not a proxy or a rule engine. It does not parse domains, resolve DNS, or match GeoIP — by design those belong in tools built for them. What it does well is carry traffic, fully encrypted and NAT-traversed, to a spoke that has the network access you want. That spoke is the exit, and a small proxy on it turns the mesh into a clean outbound.

The recipe is deliberately simple: run a SOCKS5 proxy on the exit spoke, bound to its overlay IP, and point a rule-capable client (Shadowrocket, mihomo, sing-box) at it. Subnetra is only the secure channel that reaches the proxy; the client decides which destinations take the exit.

Why not build the rules into Subnetra? In-daemon DNS, an L7 router, and a path manager are explicit non-goals. Choosing destinations by domain is a DNS + L7 job; keep that in a mature client and let Subnetra be the transport underneath.

Topology

rule-engine client ─► spoke A ─► hub ─► exit spoke B ─► internet
 (e.g. Shadowrocket)  (10.0.0.2) (relays) (10.0.0.3)      (B's uplink)

The hub relays A ↔ B (spokes never relay each other directly). Only B needs the target access; the client reaches it through B. The device running the client reaches B’s overlay IP (10.0.0.3) through its local spoke A — A itself, or a host on A’s LAN with the overlay routed to it.

1. Run a SOCKS5 proxy on the exit spoke

Run a small SOCKS5 server on B, bound to B’s overlay IP so it is reachable only through the mesh, never on B’s public NIC. zsocks is a tiny zero-dependency SOCKS5 server (also Zig, same philosophy as Subnetra — a single static binary, bounded memory, TCP CONNECT + UDP ASSOCIATE, optional auth):

# Bind to the overlay IP only and require username/password auth.
zsocks --listen 10.0.0.3 --port 1080 --user alice --pass <secret>

Useful flags (see zsocks --help):

UDP ASSOCIATE is supported, so QUIC / HTTP-3 apps work; the advertised UDP relay address defaults to the listen host (10.0.0.3), which is directly reachable over the overlay, so --udp-advertise is not needed here.

Because the proxy binds to the overlay address, all traffic to it stays inside the encrypted mesh and every overlay packet keeps proper overlay source/destination IPs — so the hub’s per-peer allowed_src anti-spoofing stays intact. There is no ip_forward and no masquerade: the proxy re-originates connections from B, and the return path is just its own sockets.

2. Point a rule engine at it

On the client, send only the destinations you choose through the exit. Below is Shadowrocket / Surge config format; mihomo and sing-box are equivalent. The example routes a well-known international music streaming service (Spotify) through the exit and leaves everything else direct:

[Proxy]
via-exit = socks5, 10.0.0.3, 1080, alice, <secret>

[Rule]
DOMAIN-SUFFIX,spotify.com,via-exit
DOMAIN-SUFFIX,scdn.co,via-exit
FINAL,DIRECT

mihomo equivalent:

proxies:
  - name: via-exit
    type: socks5
    server: 10.0.0.3        # exit spoke B's overlay IP — only reachable via Subnetra
    port: 1080
    username: alice
    password: <secret>
rules:
  - DOMAIN-SUFFIX,spotify.com,via-exit
  - DOMAIN-SUFFIX,scdn.co,via-exit
  - MATCH,DIRECT

Everything not matched stays direct (split-tunnel), so only the chosen destinations take the A → hub → B path.

DNS

Domain rules still need names to resolve from the right vantage point: if the client resolves locally it may get endpoints local to A’s region. Let the client resolve DNS through the exit (Shadowrocket’s proxied DNS, or mihomo fake-ip with the resolver reached via the proxy) so matched names resolve from B’s location.

Cautions

• B is the exit. Its IP carries the client’s traffic — real responsibility for whoever runs B, which can see destination metadata (SNI, DNS) even though the TLS payload stays end-to-end encrypted. Always enable proxy auth and bind to the overlay IP only.
• Double hop. Traffic goes client → A → hub → B → target and back: added latency, and the hub now carries this bandwidth — shape it (see Production Deployment → Traffic shaping).
• MTU stacks. You are tunnelling inside a tunnel; size the inner MTU with the Host Network Plan guidance.

Verify

# Through the exit proxy — the IP returned should be B's public IP:
curl -s --socks5-hostname alice:<secret>@10.0.0.3:1080 https://api.ipify.org ; echo

# On the hub — relay counters climb as the traffic flows through:
subnetra status --json | grep relay_

Observability & Troubleshooting

The data plane drops malformed, unauthenticated, replayed, spoofed, unrouted, or oversized packets silently by design (stealth). subnetra status makes those silent drops countable, so you can tell why traffic is not flowing without weakening the stealth property.

subnetra status

subnetrad v0.6.0 [running]
mode=raw_direct local_id=1 udp_port=18020 tun=snr0 peers=2
peers:
  id=2 endpoint=203.0.113.2:18020 allowed_src=10.0.0.2/32
  id=3 endpoint=203.0.113.3:18020 allowed_src=10.0.0.3/32
traffic:
  tun_rx packets=... bytes=...
  udp_tx packets=... bytes=...
  udp_rx packets=... bytes=...
  tun_tx packets=... bytes=...
  relay  packets=... bytes=...
  endpoint_learned=..
  keepalive rx=.. tx=..
drops:
  tun: not_ipv4=.. no_route=.. drop_rule=.. local_loop=.. unknown_target=.. oversized=.. egress_err=.. send_err=..
  udp: unknown_peer=.. auth_or_invalid=.. not_ipv4=.. spoof=.. no_route=.. drop_rule=.. unknown_target=.. no_reflect=.. oversized=.. send_err=..

subnetra status exits non-zero when the daemon is not running, so scripts can detect it. PSKs and derived keys are never printed.

Reading the drop taxonomy

Benign signals: a rising endpoint_learned just counts authenticated peers seen at a new UDP endpoint (roaming / NAT remap). The keepalive rx / tx line counts the built-in spoke→hub NAT keepalive: tx on the emitting spoke, rx on the receiving hub.

Machine-readable status (--json)

For monitoring and automation, subnetra status --json emits the same data as a stable, versioned JSON object — so health can be scraped without parsing free-form text (and still never serializes secrets):

subnetra status --json | jq .

{
  "schema_version": 1,                 // bumped only on a breaking schema change
  "version": "0.5.1",
  "mode": "raw_direct",
  "local_id": 1,
  "listen_port": 18020,
  "tun": "snr0",
  "peers": [
    {
      "id": 2,
      "endpoint": "203.0.113.7:51822",
      "name": "bj-office-gw",            // optional operator label ("" when unset)
      "allowed_src": "10.66.0.2/32",
      "last_seen_age_seconds": 5,        // null if the peer has never authenticated
      "online": true                     // last_seen within the freshness window (~90s)
    }
  ],
  "counters": { "tun_rx_packets": 3, "udp_tx_packets": 0 /* …every data-plane counter… */ }
}

• online / last_seen_age_seconds give a per-peer heartbeat (the freshness window is ~90 s — long enough to tolerate a few missed keepalives without flapping).
• counters carries every counter from the human view, so a scrape never misses a field.
• Pin schema_version in your monitor; it increments only on a breaking change.

Prometheus textfile exporter

There is deliberately no HTTP server in the daemon (extra attack surface, against the single-binary ethos). Instead, deploy/subnetra-textfile-exporter.sh turns subnetra status --json into node_exporter textfile collector metrics (the only prerequisite is jq):

sudo install -m 0755 deploy/subnetra-textfile-exporter.sh /usr/local/bin/
sudo install -m 0644 deploy/subnetra-textfile-exporter.service /etc/systemd/system/
sudo install -m 0644 deploy/subnetra-textfile-exporter.timer   /etc/systemd/system/
sudo systemctl enable --now subnetra-textfile-exporter.timer

It emits (atomically):

Useful alert expressions: subnetra_up == 0, subnetra_peer_online == 0, subnetra_peer_last_seen_age_seconds > 120, and a climbing rate(subnetra_drop_udp_auth_or_invalid_total[5m]) > 0 (PSK/epoch/wire skew) or rate(subnetra_drop_udp_spoof_total[5m]) > 0.

A troubleshooting checklist

1. Daemon up? subnetra status (non-zero exit ⇒ down). Check journalctl -u subnetrad.
2. Config valid? subnetrad --check.
3. Peer online? Look at online / last_seen_age_seconds.
4. Large transfers stall but pings work? MTU/PMTU — recheck the host network plan and add an MSS clamp.
5. auth_or_invalid climbing? PSK mismatch, key-rotation skew, a backward clock/epoch, or a wire-breaking version gap during a partial upgrade (see Upgrade & Release).
6. spoof climbing? A peer’s inner source is outside its allowed_src.
7. no_route? A missing policy rule — check Roles or inject one with subnetra policy add.

Upgrade & Release

Subnetra is a single static binary with no persistent on-disk data-plane state, so a node upgrade is mechanically “swap the binary and restart.” The real risk is wire compatibility across a mesh, and the procedure below manages it.

Upgrade & rollback runbook

Because the transport is fail-closed — a datagram that fails AEAD authentication is silently dropped — a mesh that is half-upgraded across a wire-breaking boundary silently partitions: there is no error, only a climbing auth_or_invalid drop counter on both sides.

Safe procedure:

1. Read the release notes for any wire-breaking change. If the wire is unchanged, nodes interoperate across the upgrade and ordering does not matter.
2. Stage a canary. Upgrade one spoke first and watch subnetra status on both it and the hub — online stays true and auth_or_invalid stays flat.
3. Roll forward the rest. The binary swap is atomic; restart is stateless (a fresh session epoch is derived each lifetime).
4. Rollback is the reverse binary swap + restart. No state migration is involved.

# Per node
sudo install -m 0755 subnetrad subnetra /usr/local/bin/
subnetrad --check --config /etc/subnetra/config.json
sudo systemctl restart subnetrad     # stateless restart; a new epoch is derived
subnetra status                      # confirm peers online, auth_or_invalid flat

Keep the previous binary around for an instant rollback, and watch auth_or_invalid as your partition alarm during the rollout.

Key rotation runbook

PSKs are per link. To rotate one link’s key without a flag day, exploit that each direction/epoch is keyed independently:

1. Generate a new PSK (openssl rand -hex 32).
2. Update both ends of the link to the new PSK.
3. Restart both daemons (or restart one and accept a brief auth_or_invalid blip until the other follows). Because there is no shared mesh key, only this one link is affected.

Watch auth_or_invalid during the change: a transient rise as the two ends cross over is expected; a sustained rise means the two ends disagree on the key.

The full step-by-step is in docs/deployment.md §6 “Key rotation runbook”.

Cutting a release (maintainers)

The release version lives in exactly one place: the .version field of build.zig.zon. It is injected into the daemon banner at build time via the build_options module — never hard-code a version string in src/.

To publish vX.Y.Z:

1. Bump .version in build.zig.zon to the new X.Y.Z (semantic versioning).
2. Commit that bump on main via the normal PR flow.
3. Tag the commit vX.Y.Z — the tag must equal v + the build.zig.zon version. A guard job fails the release if they disagree, so a mismatched tag never ships.
4. Pushing the v* tag triggers .github/workflows/release.yml, which builds the four-arch static binaries, the GHCR multi-arch image, the offline docker load-able per-arch image tarballs, and the macOS spoke binaries, and publishes them all to the GitHub Release with a combined SHA256SUMS.txt.

Do not create a v* tag without first bumping build.zig.zon to match. The release process is documented in docs/release.md.

Verifying downloads

Every release ships a SHA256SUMS.txt. Verify any asset before installing or docker load-ing it:

sha256sum -c SHA256SUMS.txt 2>/dev/null | grep subnetra-<version>-linux-amd64.tar.gz

Wire Protocol

This page is a readable summary of the Subnetra v1 wire protocol. The authoritative, normative specification — with RFC 2119 keywords and known-answer test (KAT) vectors — is docs/PROTOCOL.md. If this prose ever disagrees with the spec or its vectors, the spec/vectors win.

Why a normative spec? Any implementation, in any language, that reproduces the behavior below is a conformant Subnetra endpoint and can join a mesh alongside the reference (pure-Zig) implementation. The behavior is pinned by KAT vectors in tests/protocol-vectors.json, checked against the live code under zig build test and regenerated with zig build vectors.

Model

Subnetra forwards raw IPv4 packets over an encrypted UDP underlay in a single-hub hub-and-spoke topology. Each node has a numeric mesh id (0 < id ≤ 65535) that doubles as the on-wire key_id selector. Every directional link (from_id → to_id) has its own keys. wire_version is 1.

Cryptographic primitives

Key schedule

All integers fed into the KDF are big-endian; labels are ASCII with no NUL.

link_key(psk, from_id, to_id) =
    BLAKE2b-256(key = psk, msg = "subnetra-v1-link" || u32_be(from_id) || u32_be(to_id))

session_key(link_key, epoch) =
    BLAKE2b-256(key = link_key, msg = "subnetra-v1-session" || u64_be(epoch))

The sender uses link_key(psk, local_id, peer_id); the receiver derives the matching key with link_key(psk, peer_id, local_id). psk is a per-link 32-byte secret and MUST NOT be reused across peers.

Nonce

nonce(seq) = u64_le(seq) || 0x00 0x00 0x00 0x00      // 8 bytes LE + 4 zero bytes

The AEAD uses empty AAD; the 16-byte tag follows the ciphertext.

Session epoch

Each daemon lifetime samples a boot epoch once (wall-clock ns, u64), which MUST be ≥ 2024-01-01T00:00:00Z in ns and non-zero. A node that cannot satisfy this fails closed (refuses to start). The epoch travels in every datagram, and the receiver derives the matching session key from it statelessly — there is no handshake.

Datagram format

+------------------+---------------------------+----------------+
|  header (20 B)   |  ciphertext (len(inner))  |  tag (16 B)    |
+------------------+---------------------------+----------------+

Header (20 bytes, fixed)

key_id is an unauthenticated selector — it is not covered by the AEAD (AAD is empty). A forged key_id just selects the wrong key, authentication fails, and the datagram is dropped. It lets a roaming/NATed sender be recognized by identity rather than by source endpoint.

Endianness trap: header epoch and seq are little-endian, but the same epoch fed into the session KDF and the from_id/to_id fed into the link KDF are big-endian. The KAT vectors exist to catch exactly this mistake.

Keepalive (flags bit 0)

A datagram with KEEPALIVE = 0x01 set is a one-way spoke→hub NAT keepalive sealed over an empty inner plaintext, so its total length is 20 + 16 = 36 bytes. It rides the same seq + epoch + anti-replay machinery, is never acknowledged, and is not a handshake. A receiver predating this bit simply drops keepalives (strict flags == 0 check) with no effect on data delivery.

Header obfuscation (optional)

The 20-byte header sits outside the AEAD, so if it travels in cleartext a passive observer can fingerprint the protocol by its constant version, repeated epoch, and low monotonic seq even with no magic bytes. The deployment-wide obfuscate setting (on by default) removes this fingerprint at zero byte overhead: the sender XOR-masks only the header with a per-packet pad

pad = BLAKE2b-256(key = link_key, "subnetra-v1-obfs" || tag)[0..20]

derived from the directional link_key and the datagram’s own cleartext 16-byte tag; the body is already pseudorandom and is left untouched. XOR is symmetric, so a receiver reverses it from the public tag. Because the masked key_id can no longer be read directly, ingress trials each peer’s receive link key (recompute the pad, check version + key_id, then de-mask); AEAD authentication remains the real gate. It is not negotiated — every node MUST set obfuscate identically (a mismatch fails closed) — and it hides the protocol fingerprint only, not packet length or timing. An obfuscating spoke additionally randomizes its NAT-keepalive interval within [keepalive_secs/2, keepalive_secs] so the keepalive cadence is not a fixed-period signature. It is on by default; set obfuscate: false to opt out, leaving the wire byte-identical to v1 (and the header readable in a capture). Full normative detail and KAT vectors: docs/PROTOCOL.md §3.4.

Sender (egress)

To send inner IPv4 packet P to peer D:

1. key = session_key(link_key(psk, local_id, D.id), local_epoch).
2. seq = next value of this link’s monotonic counter (starts at 1, strictly increasing, never repeating within a session/epoch).
3. Emit the header with version=1, flags=0, key_id=local_id, epoch=local_epoch, seq.
4. Append ChaCha20-Poly1305-Seal(key, nonce(seq), "", P).
5. Send to D’s UDP endpoint.

On any event that could reset the counter (e.g. a restart), the sender MUST also obtain a fresh epoch (and therefore a fresh session key).

Receiver (ingress)

On a datagram from source endpoint S, in this normative order:

1. Identity selection by key_id (not by endpoint). No matching peer ⇒ drop. (With header obfuscation on, the header is masked, so this step instead trials each peer’s receive link key to de-mask it; steps 2–9 are unchanged.)
2. Header validation — drop if len < 20, version != 1, a reserved flags bit is set, or epoch == 0.
3. Epoch ordering (forward-only). epoch < cur ⇒ drop before any crypto; epoch == cur ⇒ use the cached key; epoch > cur ⇒ derive a candidate key but do not commit yet.
4. Authenticate & decrypt with the link key and nonce(seq). Failure ⇒ drop. (No state mutated yet — a forged higher epoch or wrong key_id cannot poison the session.)
5. Commit a newer epoch only now (set cur = epoch, cache key, reset the anti-replay window).
6. Anti-replay — apply the 64-entry sliding window to seq; replay/too-old ⇒ drop. (Keepalive short-circuits here: it records the endpoint + last-seen, then stops — no inner packet.)
7. Inner source check — the decrypted source address MUST fall within P’s allowed_src; otherwise drop (anti-spoofing).
8. Endpoint learning — only after steps 4–7, optionally record S as P’s current endpoint (roaming/NAT). Runtime state only; never written to config.
9. Route — deliver locally or (hub only) relay to another peer; a hub MUST NOT reflect back to the source peer.

The order of steps 3–5 (authenticate before mutating receive state) and step 8 (learn the endpoint only after 4–7) is security-critical.

Anti-replay window

Each receive session keeps a 64-bit sliding window (highest + a 64-bit bitmap, bit i = “highest − i was seen”): seq > highest shifts the window forward and accepts; an in-window unseen seq is accepted and marked; a replay or older-than-window seq is dropped.

Accepted residual risks (by design, no handshake)

• Pre-observation epoch replay. An on-path attacker who captures an authenticated datagram for epoch E and replays it before the receiver has observed E can transiently relocate the peer’s learned endpoint. It self-heals on the peer’s next genuine packet, and an off-path attacker cannot forge it.
• Backward clock across restart. A node whose wall clock regresses emits a lower epoch that peers reject until their clocks advance past the old value. Mitigated operationally (NTP/RTC), never by an in-protocol exchange.

Both follow directly from the stateless, handshake-free design.

CLI Reference

Subnetra ships two binaries:

• subnetrad — the daemon (the data plane + control socket).
• subnetra — the control tool (talks to a running daemon over the control Unix domain socket).

subnetrad (daemon)

subnetrad [--config <path>] [--check] [--print-network-plan] [--path-mtu <n>]
          [--version | -V] [--help | -h]

With no action flag, subnetrad runs the daemon: it creates the TUN device, binds the UDP underlay and the control socket, and enters the reactor loop. Creating the TUN device requires CAP_NET_ADMIN (Linux) or root (macOS utun).

# Validate, preview the host plan, then run
subnetrad --check --config /etc/subnetra/config.json
subnetrad --print-network-plan --config /etc/subnetra/config.json
sudo subnetrad --config /etc/subnetra/config.json

subnetra (control tool)

subnetra status [--json]
subnetra policy show
subnetra policy add --src <CIDR> --dst <CIDR> --action forward --target <id>
subnetra save
subnetra --version | --help

policy add arguments

Examples:

# Site-to-Site: reach LAN 192.168.2.0/24 behind spoke id 3
subnetra policy add --src 192.168.1.0/24 --dst 192.168.2.0/24 --action forward --target 3

# Hub: relay overlay traffic to the right spoke
subnetra policy add --src 0.0.0.0/0 --dst 10.0.0.2/32 --action forward --target 2

# Spoke: deliver tunnelled traffic for the local overlay address to the local TUN
subnetra policy add --src 0.0.0.0/0 --dst 10.0.0.2/32 --action forward --target 0

subnetra policy show
subnetra save

Rules injected at runtime layer on top of any policy derived from a role. Use subnetra save to persist the active tree so it survives a restart.

Environment

# Usually unnecessary — the default already matches the daemon. Set it only
# when you run the daemon on a custom socket path:
export SUBNETRA_SOCK=/run/subnetra/subnetra.sock
sudo -E subnetra status

Exit codes

• subnetra status returns non-zero when the daemon is down — convenient for health checks and the Docker HEALTHCHECK.
• subnetrad --check returns non-zero on an invalid config, so it can gate a service start.

Out-of-tree tools

These helpers live under tools/ and are never shipped inside the daemon:

Roadmap

Subnetra deliberately ships a small, finished v1 and reserves a narrow set of v2 interface points. This page describes what is delivered, what is reserved, and — just as importantly — what will never be built.

v1 — delivered

The shipping data plane is raw_direct: a stateless, allocation-free, handshake-free tunnel with:

• ChaCha20-Poly1305 full encryption with per-link keys + per-restart session epoch,
• 64-bit monotonic nonce + sliding-window anti-replay,
• a CIDR longest-prefix policy engine with lock-free RCU hot updates,
• a single-threaded reactor (Linux epoll / macOS poll(2)),
• a normative wire protocol pinned by known-answer vectors.

See the development status table for module-by-module progress.

v2 — reserved interface points only

The PRD reserves two in-house egress modes for lossy/long-haul leased lines. They are design-only today — the branches return error.NotImplemented and no code is authorized until the maintainer signs off on the design RFC (docs/v2-reliability-rfc.md):

The reservation points that already exist in the tree:

Crucially, a v2 mode is selected by static per-link config, never by an on-wire handshake. The negotiation_version / flags fields exist for static mode selection only.

Explicit non-goals

These are not “not yet” — they are never, because they would break the design principles:

• No on-wire handshake / challenge-response / capability exchange. The per-packet epoch is session establishment.
• No in-daemon health-probe or auto-switching path manager. The data plane is single-path; failover is an external decision (see Production Deployment → HA).
• No in-tunnel scheduler / adaptive rate controller. Traffic shaping is done at the OS layer with tc.
• No third-party dependencies. Not even for v2 reliability — the ARQ must be in-house.
• No in-daemon DNS resolver. Endpoints are numeric; a dynamic hub is solved operationally (DDNS watcher) on the spoke.

Changing any non-goal is an RFC that amends the iron laws, not a feature PR — and is intentionally not on the backlog.

The keepalive exception (already in v1)

The only addition under wire_version = 1 is the one-way, never-acknowledged spoke→hub NAT keepalive (flags bit 0). It is backward-compatible and is not a handshake — see the Security Model.

Development

Subnetra is written in pure Zig with zero third-party dependencies — only the standard library and raw syscalls via std.posix. This page covers building, testing, and the local integration harness.

Prerequisites

• Zig 0.16.0 or later.
• For the privileged integration harness: a Linux host (or the provided dev container) with /dev/net/tun and --privileged.

Build & test

# Native build (ReleaseSmall by default; -Doptimize=Debug for dev)
zig build

# Static cross-compile to each shipped target
zig build -Dtarget=x86_64-linux-musl     # amd64
zig build -Dtarget=aarch64-linux-musl    # arm64
zig build -Dtarget=arm-linux-musleabihf  # armv7 (hard float)
zig build -Dtarget=arm-linux-musleabi    # armv5 (soft float)

# Unit tests (must stay green before any commit)
zig build test

# Run the daemon
zig build run

Artifacts land in zig-out/bin/: subnetrad and subnetra.

Useful build steps

Project layout

Test-driven workflow

Pure logic ships with tests. The PRD’s acceptance tests include JSON/sanity checks, CIDR overlap/matching, RCU hot-swap safety, crypto invariance (ciphertext grows by exactly the 16-byte tag), and nonce-monotonic / anti-replay behavior. The wire protocol is pinned by known-answer vectors generated from the live code (zig build vectors) with a drift sentinel in zig build test.

Local integration testing (dev container)

The privileged hub-and-spoke harness is Linux-only, so a reproducible Linux container is provided under .devcontainer/. It is a development/test aid only — the shipped artifact stays a single static musl binary.

# Build the Linux toolchain image (Debian-slim + pinned Zig 0.16.0)
docker build -t subnetra-dev -f .devcontainer/Dockerfile .

# Run the integration / preflight harness
docker run --rm --privileged --device=/dev/net/tun \
    -v "$PWD":/workspace subnetra-dev test/integration/run.sh

test/integration/run.sh builds the binary, enforces the static-link and ≤ 512 KB constraints, smoke-runs the daemon, cross-builds the other musl arch, runs the unit tests, then runs a 3-node hub-and-spoke end-to-end test across network namespaces: real delivery spoke-A → Hub(relay) → spoke-B, on-wire encryption (no plaintext leak onto the underlay), a non-stalling RCU policy hot-update under load, honest drop counters, resilience under underlay loss (netem) with full recovery, and endpoint roaming / NAT remap.

For a throughput/PPS baseline, the sibling test/integration/bench.sh stands up the same star, builds -Doptimize=ReleaseFast (measurement only), and reads achieved pps / Gbps / hub-CPU% from each daemon’s own counters.

Contributing principles

Subnetra is governed by a strict operating contract (AGENT.md). In short: make surgical, goal-aligned changes; keep zig build test green; preserve the zero-dependency, single-threaded, allocation-free (data-plane), handshake-free invariants; and verify the binary still links statically and stays under the size budget before declaring a task done. See the Design Principles.

FAQ

What is Subnetra, in one sentence?

A pure-Zig, zero-dependency Layer-3 UDP tunnel that ships as a single static binary under 512 KB and connects sites and devices in a hub-and-spoke overlay with full per-link encryption.

Is it a VPN? How is it different from WireGuard?

It is a Layer-3 encrypted overlay, so it solves a similar problem to WireGuard. The differences are deliberate trade-offs:

• No handshake. Subnetra is stateless and handshake-free; the per-packet epoch plus a static config replaces session negotiation. There is no Noise handshake, no rekey timer, no roaming handshake.
• Single static binary, no kernel module, no third-party libraries — it runs entirely in userspace on a TUN device and cross-compiles to musl targets down to armv5.
• Hub-and-spoke by design, with an in-binary CIDR policy engine for site-to-site routing and hub relay, hot-updated without a restart.

It is not trying to be a drop-in WireGuard replacement; it targets fixed, operator-managed deployments where a tiny auditable binary and static topology are the priority.

How is it different from n2n?

Both build an encrypted overlay, but the designs are near-opposites:

• Star, not P2P. n2n’s signature is supernode-assisted NAT hole-punching so edges talk directly. Subnetra is strict hub-and-spoke and deliberately has no P2P / hole-punching — every spoke-to-spoke packet relays through the hub, for one predictable path (a non-goal).
• Layer 3, not Layer 2. n2n is an Ethernet (TAP) overlay with broadcast, ARP and any protocol. Subnetra routes IPv4 by CIDR only — no broadcast domain.
• Handshake-free, one fixed cipher. n2n has a registration protocol and per-community selectable ciphers. Subnetra has no registration round-trip and one mandatory AEAD (ChaCha20-Poly1305) with a unique key per link.
• Static, not discovered. n2n finds peers dynamically via supernodes. Subnetra uses static numeric endpoints with no in-daemon discovery or DNS.

Pick n2n for plug-and-play P2P and L2 LAN semantics; pick Subnetra for a tiny, auditable binary with a deterministic single-path topology.

Why no handshake? Isn’t that insecure?

No. Every packet is encrypted and authenticated with ChaCha20-Poly1305 under a per-link key. Replay is stopped by a 64-bit monotonic nonce and a sliding-window filter, and a per-restart session epoch is mixed into key derivation so old captures can’t be replayed across a restart. “Handshake-free” means there is no negotiation round-trip — not that packets are unauthenticated. See the Security Model.

Does it hide that it’s a tunnel? Is it “stealth”?

Partly. Obfuscation is stateless and best-effort: a malformed or unauthenticated datagram is silently dropped with no error reply, so the listener does not advertise itself to blind scanners. Subnetra does not claim to defeat a sophisticated DPI adversary, and it is honest about that — see Design Principles → Stateless obfuscation.

The mesh-wide obfuscate setting is on by default: it masks the 20-byte header so the datagram looks random to a passive observer and de-periodizes the keepalive cadence (it hides the fingerprint, not packet length or timing). Set obfuscate: false to send a readable cleartext header (e.g. for packet-capture debugging), which a passive observer can then fingerprint. See Wire Protocol → Header obfuscation.

What platforms are supported?

• Linux is the production target: x86_64, aarch64, armv7 (hard float), and armv5 (soft float), all static musl. The hub typically runs on Linux.
• macOS is a supported spoke / developer platform via the native utun device (minimal-dynamic, linking only libSystem). It is not a production hub target.

How big / fast is it?

The Linux release binary is a single static musl executable under 512 KB. The data plane is single-threaded, lock-free, and strictly allocation-free on the hot path, so memory use is bounded and predictable. For a throughput baseline on your own hardware, run test/integration/bench.sh.

What MTU should I use?

The fixed wire overhead is 64 bytes (20-byte header + 16-byte tag + 28-byte outer IPv4/UDP). The safe tunnel MTU is therefore path_mtu − 64 — e.g. 1452 on a 1500-byte underlay. Set local_tun_mtu accordingly, and use subnetrad --print-network-plan --path-mtu <n> to compute and preview the host plan. See the Network Plan.

Does Subnetra configure the host network for me?

No — by design it only prints the plan; it never mutates host routing or firewall state. subnetrad --print-network-plan emits the exact ip/route commands so you (or your config-management tool) apply them deliberately and auditable. The daemon does create and manage its own TUN device.

How do PSKs work? Can two links share a key?

Each link (each peer pair, per direction) uses its own 64-hex pre-shared key. Keys must be unique per link — never reuse a PSK across peers. Generate them with zig build tool:keygen. Directional keys are derived from the PSK, so the A→B and B→A directions use different keys.

How do I route a real LAN behind a spoke (site-to-site)?

Set the LAN prefix in remote_routes/local_routes and add policy rules that forward the destination prefix to the right mesh id. The hub relays between spokes. See Configuration → Roles and the policy add examples.

How do I run it on RouterOS / MikroTik?

Via the RouterOS container feature (a static-binary container on the device). See Operations → RouterOS.

The hub has a dynamic IP — now what?

Endpoints are numeric on purpose (no in-daemon DNS). Solve it operationally: run a small DDNS watcher on the spoke that rewrites the hub endpoint and reloads. The spoke’s NAT keepalive keeps the path open. See Security Model → NAT keepalive.

Is there a built-in failover / multi-path?

No. The data plane is intentionally single-path; failover is an external decision (VRRP / health-checked DNS / orchestration). This keeps the daemon small and predictable. See Deployment → High Availability and the Roadmap.

When is v2 (kcp_arq / fec_xor) coming?

Those are reserved interface points, design-only, returning error.NotImplemented until the maintainer approves the design RFC. v1 ships raw_direct only. See the Roadmap.

Why Zig, and why zero dependencies?

To get a tiny, statically-linked, auditable binary with predictable memory and no supply chain — the whole data plane is the standard library plus raw syscalls. The Design Principles (“eight iron laws”) explain the reasoning in full.

Where is the authoritative spec?

The normative on-wire contract is docs/PROTOCOL.md; the product requirements are docs/subnetra-develop.md. This site summarizes and operationalizes them; where they disagree with this site, they win.