High Availability¶

Overview¶

MonsterOps can run as a two-node active/standby pair so RADIUS survives losing a node. It is entirely optional and off by default — a standalone install is always "active" and behaves exactly as it always has, with no extra configuration.

Because all RADIUS state (users, accounting, NAS clients) already lives in PostgreSQL, HA is built on two proven primitives plus a thin orchestration layer:

• PostgreSQL streaming replication — each node runs its own Postgres; the primary streams to a hot standby that can be promoted.
• A floating virtual IP (keepalived / VRRP) — your NAS devices point at one VIP that always lives on whichever node is active.

Only the active node accepts writes and runs the background workers (scheduler, probes, backup receivers, TACACS+, automation …); the standby serves the UI read-only and follows the primary. The VIP moves automatically once the role flips.

Setting it up¶

Install MonsterOps normally on both machines, then run the HA provisioner on each — the first node becomes the primary, the second the standby (its database is re-cloned from the primary):

# On the existing node (becomes primary):
sudo HA_ROLE=primary HA_NODE_ID=node-a HA_PEER_HOST=10.0.0.2 \
     HA_VIP=10.0.0.10 HA_VIP_CIDR=24 HA_IFACE=eth0 \
     VRRP_PASS=<shared-secret> REPL_PASSWORD=<repl-password> deploy/provision-ha.sh

# On the fresh second node (becomes standby — its data dir is wiped & re-cloned):
sudo HA_ROLE=standby HA_NODE_ID=node-b HA_PEER_HOST=10.0.0.1 \
     HA_VIP=10.0.0.10 HA_VIP_CIDR=24 HA_IFACE=eth0 \
     VRRP_PASS=<shared-secret> REPL_PASSWORD=<repl-password> deploy/provision-ha.sh

The provisioner configures streaming replication, keepalived, the MONSTEROPS_HA_* env vars, and points generated NAS snippets at the VIP.

Sharing secrets between the nodes¶

A failover only works if both nodes agree on two secrets, so line these up before you provision the standby:

• MONSTEROPS_SECRET_KEY — signs sessions and derives the key that encrypts stored credentials at rest. If the nodes differ, credentials encrypted on one won't decrypt after a failover.
• The database password — the standby is cloned from the primary with pg_basebackup, which byte-copies the primary's radius role, so the standby ends up authenticating with the primary's DB password.

There are three ways to get them onto the standby — pick one:

1. Pin them at install (recommended). Install both nodes with the same MONSTEROPS_SECRET_KEY and RADIUS_DB_PASSWORD (pass them to install.sh), then provision the standby with SKIP_SECRET_SYNC=1 — nothing to sync, no SSH:

sudo HA_ROLE=standby HA_NODE_ID=node-b HA_PEER_HOST=10.0.0.1 \
     HA_VIP=10.0.0.10 HA_VIP_CIDR=24 HA_IFACE=eth0 \
     VRRP_PASS=<shared-secret> REPL_PASSWORD=<repl-password> \
     SKIP_SECRET_SYNC=1 FORCE=1 deploy/provision-ha.sh

2. Pass them directly. Read them off the primary and hand them to the standby provisioner — still no SSH:

# on the primary — read the two values:
sudo grep -E 'SECRET_KEY|DATABASE_URL' /opt/monsterops/.env
# on the standby — pass them in:
sudo HA_ROLE=standby ... SYNC_SECRET_KEY='<key>' SYNC_DATABASE_URL='<url>' FORCE=1 deploy/provision-ha.sh

3. Let the standby fetch them over SSH (the default). With neither of the above, the standby SSHes into the primary and reads the secrets with sudo. The login user defaults to whoever ran sudo — which is root, and most hosts disable root SSH — so set PEER_SSH_USER=<user> to an account that can sudo on the primary. SSH_COPY=0 turns the SSH attempt off entirely.

sudo HA_ROLE=standby ... PEER_SSH_USER=youradmin FORCE=1 deploy/provision-ha.sh

The Cluster page¶

System → Cluster is an auto-refreshing dashboard that pulls health from both nodes: application and Postgres roles, replication state and lag, replication slots, Postgres health and error counters, and a failover-history table — with an alerts feed (peer unreachable, replica not streaming, lag over threshold, inactive slot, split brain, or a secret-key mismatch between nodes).

Readiness distinguishes a planned switchover (primary reachable → wait for the replica to replay all received WAL → zero data loss) from an unplanned promote (primary unreachable → promote what the standby has, accepting a bounded data-loss window; requires an explicit force).

Failover modes¶

Set MONSTEROPS_HA_FAILOVER_MODE the same on both nodes:

• manual (default) — you click Fail over to this node on the standby (or use the CLI). Failing over fences the old primary first, so you can't end up with two primaries.
• assisted — a debounced health watcher raises a prominent one-click "fail over now" banner once it is confident the primary is down (a human still confirms).
• auto-vip — fully automatic: when the primary dies, keepalived moves the VIP to the healthy standby and MonsterOps auto-promotes whichever node holds the VIP (the VIP is the split-brain arbiter — only its holder may promote). A returning old primary comes back as a backup and self-fences its PostgreSQL read-only so it can't diverge.

Honest limit: auto-vip is safe against a node crash and against a partition where one side keeps the VIP, but not a pathological L2 partition where both sides believe they hold it. For most two-node LAN clusters, auto-vip is the right balance.

From the command line¶

Every action also works from the CLI, directly against the local database — so it works even when the API is read-only on a standby:

monsterops ha status      # role, replication lag, readiness + which failover path applies
monsterops ha promote     # fail over to THIS node (add --force for an unplanned failover)
monsterops ha demote      # put THIS node into read-only standby
monsterops ha vip-check   # exit 0 if this node should hold the VIP (used by keepalived)

Related¶

• Configuration — the MONSTEROPS_HA_* environment variables
• Command-Line Interface — the monsterops ha commands