Recovery automation

Getting the data back is the easy half.

The rest of a recovery is a sequence: quiesce the application, stand up somewhere for it to land, restore the database, point DNS at the new thing, check that what came back is what left. That sequence usually lives in a wiki page written after the last incident, by somebody who has since changed jobs, and is performed for the first time under load by whoever happened to be on call.

So the platform holds it instead — as runbooks, made of steps, dispatched onto the same agents that take the backups.

Blueprints

A blueprint is one reusable unit of work. Three engines, because recovery is not one kind of job:

Official blueprints ship with the platform and cover the common shape of a recovery: restore a PostgreSQL service, quiesce an application, mount a recovery volume, verify service health, build a recovery network, bring up standby compute, cut a DNS record over, wait for an endpoint to answer, verify restored checksums, and notify a channel. Your own blueprints sit alongside them in the same catalogue.

Official ones cannot be edited — an update to a blueprint the platform ships is refused rather than silently forked — and one still referenced by a runbook cannot be deleted.

Runbooks

A runbook is an ordered list of steps, each binding values to a blueprint's variables.

Steps are dispatched one at a time, and step n+1 is queued only once step n has actually succeeded. That ordering is the product. A failed restore stops the plan rather than being compounded by everything that was supposed to happen after it.

A step is also never retried automatically. A backup that failed on a flaky network is worth retrying; a recovery step that failed halfway is not, because re-running it blind turns one partial restore into two. The operator decides.

Resolve it without running it

Ask for a runbook's plan and the platform resolves the whole thing — what each step would do, the exact command line it would run, and every reason it could not. A plan that is not ready refuses to start rather than beginning a recovery that will stop in the middle.

The command shown is the command dispatched. An agent that disagrees with it fails the step rather than substituting its own, because the console showed the operator that exact line before they pressed Run.

Rehearsals

A recovery plan nobody has run is a document, not a capability.

A rehearsal rewrites every verb to its read-only counterpart before anything is dispatched: Ansible gets --check --diff, Terraform gets plan, and a destroy becomes a plan too — so the rehearsal shows what would be torn down without tearing it down. The agent does not have to interpret the mode; the action it receives is already correct.

Python is the one engine where a rehearsal is a promise rather than a guarantee, because an interpreter has no dry run. Every official Python blueprint honours the check flag, the starter a custom one begins from honours it, and a script that ignores it fails the rehearsal rather than quietly performing it. The console says so where an operator chooses the mode.

That matters more than it sounds for notifications: a message saying a cutover has happened, sent from a rehearsal, is worse than no message at all — somebody acts on it.

Triggers

Two halves of one question: what makes this plan run?

An event trigger listens to what the estate already notices. A failed backup starts the escalation runbook. A recovery point appearing starts the verification one. An asset missing its RPO starts whatever you decided that should start.

Triggers can be narrowed to a single asset, so "run the failover plan when this database's backup fails" is one trigger rather than one trigger per event type with a filter buried inside the runbook.

The events available:

job.succeeded and backup.completed are deliberately two events. The first says a run finished; the second says something can be restored from — and only the second is what a pipeline waiting on a pre-deployment backup actually asked for.

A trigger whose own runbook produced the event it listens for is refused in code rather than left for an operator to discover as a runaway estate. Escalation still works: one plan failing can start another.

An inbound trigger is a URL somebody else's pipeline calls:

curl -X POST https://smrt.example.test/api/automation/triggers/fire \
  -H "X-Smrt-Trigger-Token: $SMRT_TRIGGER_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"reason": "deploy 4821"}'

A backup taken immediately before a deployment is worth more than one taken six hours earlier on a schedule, and a pipeline that can ask for one is the difference between that and a sleep.

The token goes in a header, never in the URL — a URL is logged by every proxy, load balancer and CI system it passes through, and this token starts code running on machines holding your data. It is hashed at rest, shown exactly once when it is minted, and audited on every use. A wrong token, a disabled trigger and a deleted trigger all answer identically, so a caller holding a bad token learns that it is bad and nothing about what exists.

Notifications

The classic way an estate becomes unrecoverable is a backup failing quietly for a week while every screen in the product shows it to nobody.

Three dialects rather than one generic payload plus a translation layer, because otherwise every customer writes the same adapter.

Delivery is a table, not a channel. An in-process queue loses every pending notification when the process restarts — and the incident that kills it is exactly the one somebody needs to be told about. Five attempts backing off from thirty seconds to five minutes; a 4xx is terminal because a body a receiver refuses will be refused identically next time; 429 is the exception, since it is a request to come back later. Two replicas take different work rather than both posting the same message.

The rendered body is stored verbatim, which makes the first question after an outage nobody noticed — did we tell anyone? — answerable.

Every endpoint has a test button, and it is the single most useful control on the page. A webhook is configured once and then silent until something goes wrong, so without it the first proof that a URL was pasted correctly arrives during an incident. Disabled endpoints can be tested too: the sane order is paste, prove, then enable.

For Slack and Teams the URL is the credential — anyone holding it can post into your channel as us — so it is never returned in full by the API and never written to the audit trail.

How a step reaches a machine

There is exactly one path from the control plane onto an agent: the job queue, handed over on the heartbeat. Automation uses it rather than opening a second one, so a step inherits everything that path already does — it survives an agent being offline when the work is queued, it reports progress back the same way, and every run appears on the same jobs page as every backup.

The payload is self-contained. The agent resolves nothing, fetches nothing and needs no catalogue credentials, because an agent that had to reach a control plane to find out what to run would need one at exactly the moment a disaster makes it doubtful.

Variables never go on a command line, for any engine. A password on an argv is visible in the host's process table to every user on the machine, which is not an acceptable place for the credential to a database somebody is in the middle of restoring. They are written to a private file that is removed with the working directory. Secret-typed variables are never returned by the API, and an update that omits one keeps the stored value — so saving from an editor that cannot display a credential does not clear it.

An agent that does not recognise the engine, the action, or the payload version must fail the step and name what it did not recognise. Silently succeeding is the one outcome that must not happen: it reports a recovery step as done when nothing ran.

What is deliberately not built yet

• Digests. One summary of a quiet night rather than forty messages. A real digest needs a window and a scheduler, and is worth doing properly.
• Escalation timers. A trigger can start an escalation runbook today, but nothing yet tracks whether a human acknowledged anything.
• Cloud accounts as runbook subjects. The seams are in — stable service names, a policy target kind, the account and its scope on the step's own payload, and a Terraform engine that already runs against a cloud. What is missing is the binding between a step and an account, and that is a design question worth answering against a real recovery flow rather than guessing at now.

The DR platform · Security model · Deployment options