A ransomware recovery plan for Proxmox shops

The actual threat model

Assume the attacker eventually gets what your hypervisors have: PVE root, and every credential stored on those hosts. The design question is then simple — with everything on the production side compromised, can they destroy your backup history? If the answer is yes anywhere in the chain, that's the link to fix first. Everything below is that one question applied to each layer.

Phase 1: Prevent — backups production can't destroy

• Back up with a token that can't delete. The credential on your PVE hosts gets the DatastoreBackup role: it can write new snapshots and read its own, but not prune, not delete groups. Prune rights live on a management workstation, never the hypervisors.
• Protect monthly anchors. The protected flag makes a snapshot immune to prune and forget — cheap insurance against both hostile actors and fat-fingered retention.
• Keep an offsite copy on separate credentials. A push-synced offsite PBS — with a write-only token and Remove vanished off — means mass deletion on the primary never propagates. The offsite side enforces its own retention.
• The full permission model is in the immutable backups guide; on hosted plans the offsite server, its patching, and its verification are our side of the wall.

One more that costs nothing: client-side encryption doesn't stop deletion, but it means exfiltrated backup data is ciphertext — half of the modern double-extortion play gone.

Phase 2: Detect — make anomalies loud

• Notifications on every failed job, routed to a mailbox humans watch. A backup that silently stops running is how three-month gaps happen.
• Watch the change rate. Ransomware encrypting inside guests shows up as a sudden spike in backup size — dedup stops matching, and one night's delta balloons. A nightly job that's suddenly 10× its normal size is a page, not a curiosity.
• Scheduled verification re-reads chunks on a cycle, so corruption — malicious or not — is found while good copies still exist. See monitor, verify & prune.
• Check for missing guests monthly — the quiet failure mode that hurts most.

Phase 3: Respond — the first 24 hours

• Isolate before you restore. Cut the affected cluster's network first. Restoring into a live compromise just hands the attacker fresh copies.
• Don't touch backup credentials from compromised machines. Work from a clean workstation; assume anything typed on production hosts is captured.
• Find last-known-good. Snapshot timestamps plus your size-spike detection bracket the infection window. Verified snapshots before the window are your restore set — this is why verification history matters.
• Restore in dependency order onto clean or rebuilt hosts: identity and DNS first, then databases, then applications. Write this order down now; deriving it during the incident is how day one is lost. Restore mechanics are in the check & restore guide.
• Rotate every credential the environment held before reconnecting to anything.

Phase 4: Drill — practice while it's boring

Quarterly, run the plan small: pick a guest, pretend the primary site is gone, restore from the offsite copy to a test VMID, boot it, check the app, record the time. Once a year, run it big: restore the dependency chain — identity, database, app — and measure the real recovery time. The drill's product is a number and a runbook stored outside the environment it describes; a plan that lives only on a VM inside the blast radius is part of the blast.