Availability, High Availability and Disaster Recovery

Communications matter most when other systems are under stress. This chapter covers denial-of-service resistance, the resilience of the single-host profile, the multi-host high-availability profile with second-site recovery, behaviour when an identity provider is unavailable, and backup and recovery.

32.1 Denial of Service and Resource Exhaustion

Table 55: Availability controls

A single-host installation cannot absorb volumetric attacks larger than the customer's network capacity. Deployments on hostile networks should front the edge with the customer's own DDoS protection, or run on private networks.

32.2 Resilience of the Single-Host Profile

The Edge profile is a single host by design. Services restart automatically and are health-checked; clients queue outbound messages, attachments and voice notes in an encrypted outbox and reconnect; the server holds ciphertext for offline devices. Redis runs as two instances on the host as well: an evictable cache and a persistent state instance that never evicts, with the memory cap set per installation. There is no in-profile redundancy: a host failure interrupts service until the host is repaired or the installation is restored on a standby host. Customers that need redundancy use the high-availability profile below. Tosh Defence does not publish an availability SLA figure for self-hosted installations, because availability depends on the customer's infrastructure and operations.

32.3 High-Availability Reference Topology

Status: IMPLEMENTED. The high-availability profile runs on multi-host Docker Compose (Kubernetes packaging is on the roadmap). Its reference topology comprises:

• two edge hosts behind a shared address or the customer's load balancer, each with the TLS certificates and the device CA and revocation list, with the client address preserved end to end;
• N+1 API and WebSocket nodes, with real-time delivery fanned out across nodes through Redis publish and subscribe, and exactly one elected runner for each singleton job, so no scheduled job runs twice;
• Redis split into an evictable cache and a persistent state instance that never evicts, with Sentinel failover;
• PostgreSQL streaming replication with fenced failover, so a former primary can never accept writes, and write-ahead-log archiving to the second site, for both the main and the threat-processor databases;
• distributed MinIO in each site with site replication between sites, keeping AES-256-GCM encryption at rest;
• a three-node OpenBao raft cluster with TLS between nodes, a per-node unseal ceremony under the same M-of-N custody, and encrypted snapshots shipped to the second site;
• several media-server nodes routed through the state Redis, each with the same AES-256 SRTP policy;
• the push relay on more than one host;
• a maintenance mode for planned work.

Host roles and every peer address are set in the installation's environment file and role overlays, so the files shipped in each release stay identical for every customer. Installation, upgrade, backup and restore are orchestrated across the hosts with health gates between steps, and every host performs the release signature, expiry, sequence and downgrade checks itself (Chapter 30). Backups are taken from a replica.

32.4 Behaviour When an Identity Provider Is Unavailable

Federated sign-in is optional and only ever uses the customer's own identity provider (Chapter 13). Its outage is contained by design:

• Administrators. At least one local administrator, signing in with a password and a second factor, always remains: the installation refuses any configuration that would leave none, and federation never disables local sign-in. When the provider cannot be reached, the console directs administrators to a local account.
• End users. The directory is consulted only at enrolment, device restore and new-device sign-in. Everyday sign-in on an existing device and all messaging and calls never contact it, so field devices are unaffected by an outage; enrolment and restore wait until the directory is back, and any exception is an audited owner decision.
• Monitoring. The administrator monitors show the result of the latest provider check.

32.5 Backup

Table 56: Released backup capability

32.6 Disaster Recovery Methodology

The high-availability reference topology, with an asynchronous second site, is designed for a recovery point objective (RPO) of 5 minutes and a recovery time objective (RTO) of 1 hour. These are design targets, not measured claims: each deployment's disaster-recovery drill records the figures it actually achieved, and those recorded figures, not the targets, are what an installation can rely on. An RPO of zero with an RTO of 15 minutes would need synchronous replication and a third site or witness; it is available as a separately priced option and is not part of the built profile.

The drill tooling writes markers at the primary site, fences it, promotes the second site, switches the service names and later fails back. The achieved RPO is measured as the gap between the last marker acknowledged at the primary site and the last marker present at the second site; the achieved RTO runs from the declaration of the disaster to the first successful message and call on iOS, Android and desktop.

For single-host installations, and for any installation's backup-based recovery, RPO and RTO are set by the customer's backup frequency, media handling and standby capacity. The recommended methodology:

1. Classify the installation's criticality and set target RPO (maximum tolerable data loss) and RTO (maximum tolerable outage).¹
2. Schedule backups at an interval no longer than the RPO, and replicate encrypted archives to a recovery site or offline media.
3. Prepare a standby host with the same release and pre-staged images.
4. Separate custody of the backup passphrase, OpenBao unseal shares and CA keys.
5. Test restoration and, in the high-availability profile, site failover on a schedule, measuring the achieved recovery point and recovery time, and record each test.
6. Recover keys: installation keys come back with the backup; end-user message keys are not recoverable by design, and users re-establish sessions after restoration.
7. Recover PKI: re-issue the edge certificate from the customer CA if the original key is unavailable.

Customers should validate restoration on their own infrastructure as part of acceptance and at a regular interval thereafter.

---

Footnotes

1. M. Swanson et al., "Contingency Planning Guide for Federal Information Systems", NIST SP 800-34 Rev. 1, May 2010. ↩