Production deployment checklist

A single page to walk through before pointing proxxx at a real cluster you care about. Each item is a one-line check + a verifiable command. Treat this as the minimum bar — your shop's own runbook may be stricter.

TIP

This page is for the operator deploying proxxx, not for the PVE cluster itself. Cluster hardening (corosync over a private ring, firewall rules, certificate rotation) is upstream Proxmox material; we link to it but don't restate it.

On-call essential: incident freeze is a fleet-wide write kill-switch — during a maintenance window or a suspected compromise, every mutation is refused before it reaches Proxmox until you thaw.

1. Verify the binary

[ ] Download from a tagged release, not main

TARGET=x86_64-unknown-linux-musl   # or aarch64-apple-darwin
VERSION=$(gh release view --repo fabriziosalmi/proxxx --json tagName -q .tagName | sed 's/^v//')  # latest tag
gh release download v${VERSION} \
  --repo fabriziosalmi/proxxx \
  --pattern "*-${TARGET}.tar.gz" \
  --pattern "*-${TARGET}.tar.gz.sha256"

[ ] Check the SHA-256 sidecar

shasum -a 256 -c proxxx-${VERSION}-${TARGET}.tar.gz.sha256
# → proxxx-...tar.gz: OK

[ ] Verify the sigstore keyless cosign signature (release ≥ next-tag-after-v0.1.6)

gh release download v${VERSION} \
  --repo fabriziosalmi/proxxx \
  --pattern "*-${TARGET}.tar.gz.cosign.bundle"

cosign verify-blob \
  --bundle proxxx-${VERSION}-${TARGET}.tar.gz.cosign.bundle \
  --certificate-identity-regexp 'https://github.com/fabriziosalmi/proxxx/.github/workflows/release.yml@.*' \
  --certificate-oidc-issuer 'https://token.actions.githubusercontent.com' \
  proxxx-${VERSION}-${TARGET}.tar.gz
# → Verified OK

The cert-identity-regexp pins the OIDC subject to this exact workflow path in this exact repo — a leaked sigstore cert from any other workflow or any other repo can't validate against these bundles. The transparency-log inclusion proof is embedded in the bundle, so verification is offline.

[ ] Audit the CycloneDX SBOM (optional but recommended)

gh release download v${VERSION} --repo fabriziosalmi/proxxx \
  --pattern "*.cdx.json" --pattern "*.cdx.json.sha256"
shasum -a 256 -c proxxx-${VERSION}.cdx.json.sha256
grype sbom:proxxx-${VERSION}.cdx.json   # or trivy / cyclonedx-cli

2. Configure access

[ ] Use API tokens, not passwords

Tokens are revocable, scopable, and don't carry full account privilege when --privsep=1. Create with:

ssh root@<node>
pveum user token add operator@pve proxxx --privsep=1
# Grants the TOKEN the same role you grant on the user side:
pveum acl modify /vms/100 -tokens 'operator@pve!proxxx' -roles PVEVMAdmin

[ ] Pin verify_tls = true unless you know why not

Self-signed labs flip this to false for convenience. In production:

verify_tls = true

If you're running PVE behind a real cert (Let's Encrypt, internal CA, ACME via PVE itself), this is the only correct setting. Disabling TLS verification exposes the entire API + WebSocket traffic (including serial-console tickets) to any MITM on the path.

[ ] Store the token secret in a 0600 file or the macOS Keychain, not inline

# Option A: macOS Keychain (macOS only; service `proxxx`, account `<profile>/token_secret` or `token_secret`)
security add-generic-password \
  -s proxxx -a "<profile>/token_secret" -w "<token-uuid>"

# Option B: 0600 file referenced from config.toml (Linux and macOS)
mkdir -p ~/.config/proxxx/secrets
printf '%s' '<uuid>' > ~/.config/proxxx/secrets/pve.token
chmod 600 ~/.config/proxxx/secrets/pve.token
# token_secret_file = "~/.config/proxxx/secrets/pve.token" in config.toml

WARNING

proxxx refuses to read token_secret_file if the file is not mode 0600 (Unix). It will print Security Error: token_secret_file '<path>' has unsafe permissions <mode> and exit. This is intentional — don't chmod 644 to "fix" it.

[ ] Validate the connection works as the deploying user

proxxx ls nodes
proxxx ls guests --format json | jq '.[] | {vmid, name, status}'
proxxx perms <user> --node <node>

3. Configure HITL (if any operator runs destructive ops)

[ ] Provision a dedicated Telegram bot

Don't reuse a bot that's also wired to other systems — the HITL daemon polls and acknowledges every callback, and a shared bot's other listeners may double-fire.

# In Telegram: chat with @BotFather
/newbot → name + username → copy the API token
/setprivacy → DISABLE  (so the bot sees group messages)

[ ] Pin [telegram] in config.toml

[telegram]
bot_token = "<bot-api-token>"
chat_id = "<your-numeric-chat-id>"

The bot token resolves with the same hierarchy as the PVE token: PROXXX_TELEGRAM_BOT_TOKEN env, bot_token_file, keychain, inline.

[ ] Set allowed_approvers — without it the daemon refuses every callback

The callback signature proves proxxx minted the approval keyboard. It does not establish who pressed the button: Telegram delivers the buttons to everyone who can see the message. Without an allowlist, any member of chat_id — including someone added to the group later — could approve a destructive operation, which then runs with the daemon's full PVE credentials.

[telegram]
chat_id = "-1001234567890"
allowed_approvers = [123456789, 987654321]   # numeric ids, not usernames

Get each approver's numeric id by having them message @userinfobot. Usernames are deliberately not accepted: a Telegram handle can be released and re-registered by someone else, so an allowlist keyed on handles is forgeable.

An absent or empty list is treated as "refuse everything" rather than "allow anyone" — the same fail-closed posture as a destructive MCP tool with no matching policy.

[ ] Configure [[policies]] rules

[[policies]]
action  = "delete"
target  = "tag:prod"
channel = "telegram"
require = 2            # two approvals (quorum)

[[policies]]
action  = "stop"
target  = "100"
channel = "telegram"
require = 1

Policies match by action, tag, vmid, or wildcard. The deny-on-timeout is hardcoded to 120 s — if the human doesn't approve in that window, the op is rejected (NOT auto-approved).

[ ] Run the HITL daemon under a process supervisor

# systemd unit at /etc/systemd/system/proxxx-hitl.service:
[Unit]
Description=proxxx HITL approval daemon
After=network-online.target

[Service]
Type=simple
User=proxxx-ops
ExecStart=/usr/local/bin/proxxx hitl serve
Restart=on-failure
RestartSec=5
# Log records go to stderr as well as the rotating file, so
# `journalctl -u proxxx-hitl` shows warnings and errors — including
# the freeze lock becoming unreadable and TLS pinning being skipped.
# Raise verbosity per-crate when debugging:
#   Environment=RUST_LOG=proxxx=debug,russh=debug
Environment=RUST_LOG=proxxx=info
# NOTE: pending approvals, quorum votes, TTL expiry (120 s default),
# and replay protection are persisted in `<config_dir>/hitl_pending.db`
# next to `telegram_hmac.key` (0700 dir / 0600 file), so consumed or
# expired approval buttons remain refused across daemon restarts.

[Install]
WantedBy=multi-user.target

[ ] Test the round-trip end-to-end

proxxx hitl test --action delete --vmid 999
# → Telegram → tap Approve in <120s → daemon runs the op
# → daemon answers callback "✅ Done" → message lifecycle done

4. Configure alerting (optional)

[ ] Define [[alerts]] rules in config.toml

[[alerts]]
name = "node_offline"
when = "node_offline"
for_secs = 120
severity = "critical"
route = ["telegram", "ntfy:proxxx-prod"]
dedup_secs = 600

Predicates available: node_offline, storage_above, replication_failing. The dedup_secs window prevents re-fire spam.

[ ] Run the alert daemon under a supervisor

ExecStart=/usr/local/bin/proxxx alerts watch --interval 30

The daemon persists its dedup window to SQLite (cache schema 1 → 2 since v0.1.2), so a routine restart doesn't re-fire every active alert. Persistence is local to the daemon's host — no shared state across replicas.

[ ] Test each route once

proxxx alerts test --route 'telegram'
proxxx alerts test --route 'ntfy:proxxx-prod'
proxxx alerts test --route 'webhook:https://hooks.example/notify'

5. Configure SSH layer (if running proxxx perms or proxxx patch apply)

[ ] Provision a dedicated SSH key for proxxx

Don't reuse your personal ~/.ssh/id_ed25519. proxxx maintains its own known_hosts at $XDG_CONFIG_HOME/proxxx/known_hosts — giving it a dedicated key keeps the audit trail separate.

ssh-keygen -t ed25519 -f ~/.ssh/proxxx_ops -N "" \
  -C "proxxx-ops@$(hostname)"
ssh-copy-id -i ~/.ssh/proxxx_ops.pub root@<each-node>

[ ] Configure [ssh] block

[ssh]
user = "root"
key_path = "~/.ssh/proxxx_ops"
strict_host_key_checking = "tofu"   # default; pinning happens on first connect

[ ] Verify the round-trip

proxxx perms <user> --node <node>   # exercises the same path
# → table of effective ACLs; if you see the table, SSH works.

6. Lock down the operator host itself

[ ] Treat ~/.config/proxxx/config.toml as a credential file

chmod 600 ~/.config/proxxx/config.toml
ls -l ~/.config/proxxx/

[ ] Confirm shells aren't leaking secrets via history

# Bash:
echo $HISTFILE
grep -E 'PROXXX_TOKEN_SECRET|PROXXX_PBS_TOKEN_SECRET' ~/.bash_history
# Zsh: ~/.zsh_history. Fish: ~/.local/share/fish/fish_history.

If you find tokens in history, rotate them on the cluster side before deleting from history.

[ ] Pin a Rust version + build from source for reproducibility (optional)

git clone https://github.com/fabriziosalmi/proxxx.git
cd proxxx
git checkout v${VERSION}
cargo build --release --target x86_64-unknown-linux-musl
# Compare your binary's sha256 against the release sha256.

7. Operational runbook

[ ] Pin the build into your fleet inventory

proxxx version --json
# → { "version": "0.1.6", "git_sha_short": "...", "audit_ignores_count": 1, ... }

Snapshot this output into your inventory (Ansible facts / Salt grain / Puppet fact) so a security advisory triage can answer "which hosts have <vulnerable version>" instantly.

[ ] Subscribe to release notifications

GitHub repo → Watch → Custom → Releases. Or pin the release feed: https://github.com/fabriziosalmi/proxxx/releases.atom.

[ ] Document your local risk-override policy

--allow-risk bypasses the pre-flight gate. If your shop permits this for any class of op (e.g. patch-apply during a planned window), document who can use it and for what in your runbook. The flag is ungated by design — proxxx trusts the operator who typed --yes AND --allow-risk.

[ ] Test recovery: token revocation

ssh root@<node>
pveum user token remove operator@pve proxxx
# → next proxxx call from the operator host should 401
proxxx ls nodes   # expect: HTTP 401 No ticket

If it doesn't 401, the token wasn't actually scoped — re-issue with --privsep=1 and grant the role to the token path.

8. Harden the MCP server (if you expose proxxx over MCP)

[ ] Keep the transport on loopback, or set mcp_token before exposing it

proxxx mcp serve-http refuses to start on a non-loopback bind (e.g. 0.0.0.0) unless mcp_token is set:

# Local-only (default, safe): binds 127.0.0.1
proxxx mcp serve-http

# Network-exposed without a token → refused
proxxx mcp serve-http --bind 0.0.0.0:8080
# → error: refusing to bind a non-loopback address without a token.
#    Set `mcp_token` (or pass --token), bind to 127.0.0.1, or pass
#    --insecure-bind to override.

mcp_token is a profile config field; --token-file <path> or PROXXX_MCP_TOKEN override it without putting the token in the process listing, which --token does. An empty or whitespace token counts as absent — don't set it to "" and think you're covered. When the server is network-exposed and the token is absent, every request is denied (fail-closed), and that denial survives a SIGHUP that clears the token from a live config.

--insecure-bind overrides the refusal. Never pass it on an untrusted network — it exists for consciously-chosen trusted-segment cases only (see AR-3 in ACCEPTED-RISKS.md).

[ ] Gate every destructive MCP tool with a [[policies]] entry

Over MCP, proxxx is fail-closed: a destructive tool with no matching [[policies]] entry is refused (a typed isError envelope — the PVE gateway is never reached). There is no ungated inline path. The only way to run a destructive tool over MCP is a matching policy that routes it to HITL approval.

Destructive (policy-gated) tools: stop_guest, restart_guest, suspend_guest, delete_guest, create_guest, clone_guest, clone_with_cloudinit, migrate_guest, create_snapshot, delete_snapshot. Non-destructive tools stay inline: start_guest, resume_guest, and every get_* / list_* read.

# Without a matching policy, a destructive tool is simply unavailable
# to the MCP client. Confirm your rules cover every destructive action
# you intend a client to perform.
[[policies]]
action  = "delete"
target  = "*"
channel = "telegram"
require = 1

9. Guard unmanned auto-converge (if you run reconcile watch --auto-converge)

[ ] Set a small max_unmanned_changes before enabling converge_prune

converge_prune = true lets unmanned converge delete drifted resources — but it holds all deletes unless max_unmanned_changes (a per-tick change cap) is also set. With prune on and no cap, deletes are held and only create/update operations converge.

[reconcile]
auto_converge = true
converge_prune = true
max_unmanned_changes = 3       # keep this single-digit
allowed_families = ["lxc"]     # whitelist which families converge unmanned

Size the cap small — single digits. A large cap (e.g. 49) re-opens the 10–49 unmanned-delete band, which is exactly the blast radius the cap exists to close (see AR-4). Use allowed_families to whitelist which resource families are allowed to converge without a human.

10. Protect the audit chain

[ ] Keep audit.key at 0600, and consider a separate volume

The audit log is HMAC-signed and fail-closed on every mutation. Before any PVE POST, PUT, or DELETE request or SSH patch apply command leaves the process, proxxx verifies the existing HMAC chain and writes a write-ahead "sent" entry to audit.db, followed by an "ok <status>" or "error ..." outcome entry when the call completes (with secret parameters redacted). If audit.db or audit.key is unwritable, missing on a non-empty database, has loose permissions, or fails HMAC verification, proxxx refuses the mutation with exit code 8 (AuditWriteRefusal) and sends no request.

proxxx creates the audit dir 0700 and refuses to load a group- or world-readable audit.key (Unix), so keep it owner-only:

chmod 600 <audit-dir>/audit.key
ls -l <audit-dir>/audit.key    # -> -rw------- (0600)

To keep the signing key off the same volume as the database it signs, point PROXXX_AUDIT_KEY at a separate volume — a same-user/root attacker who can rewrite the DB then can't silently re-sign it (AR-5). PROXXX_AUDIT_DIR relocates the DB and key together if you'd rather move the whole directory.

[ ] Run audit verify from a separate trust context

proxxx audit verify        # exits non-zero if the chain has been tampered with

Run this from a host or account that can't write the audit DB — a verifier that shares the operator's write access can't prove much. Wire the non-zero exit into your monitoring.

[ ] Rotate the audit key after a compromise — it no longer costs you the history

The HMAC key is 32 bytes beside the database it protects, so anyone who can read that file can forge the chain. If the host is ever compromised, rotate:

proxxx audit rotate-key      # archives the old key, starts signing with a new one
proxxx audit verify          # every entry still verifies, old and new

Until v0.13.4 rotating meant deleting the key and starting a new chain, which destroyed the verifiability of exactly the history an investigation needs — so in practice the key was permanent. Each row now records which key signed it, and retired keys are archived beside the primary as audit.key.<id>.

Back up the retired keys. They are part of the trail: without one, the rows it signed can no longer be verified, and proxxx audit verify counts them as failures rather than skipping them.

11. Upgrade, rollback and backup

[ ] Know what must survive the host

proxxx keeps state in the platform data and config directories; only the audit trail and cryptographic keys must survive if the machine is rebuilt.

Path	Must be preserved?	Why
audit.db	Yes	The only record of who issued which mutation. Not reconstructible from anything else.
audit.key	Yes	32 bytes. Without it no surviving copy of audit.db can be verified — losing the key alone is enough to make the trail worthless.
audit.key.retired-*	Yes	Keys retired by audit rotate-key. Each still proves the rows it signed; losing one turns those rows into verification failures.
telegram_hmac.key (in <config_dir>)	Optional	32-byte HMAC key signing Telegram inline keyboards (0600). Regenerated automatically if absent, which invalidates any in-flight buttons.
hitl_pending.db (in <config_dir>)	No	SQLite store (0600, alongside telegram_hmac.key) holding pending HITL requests, quorum votes, 120 s TTL expiry, and a 7-day terminal replay log. Recreated automatically on start.
freeze.lock, freeze.<profile>.lock	No	Runtime kill-switch state. Absent means thawed, which is the correct default after a rebuild.
known_certs/ (in <config_dir>)	Optional	TLS certificates pinned on first contact (tls_pin_mode). Without them the next connection pins again, so a changed certificate is no longer detected across the rebuild.
known_hosts (in <config_dir>, or [ssh] known_hosts)	Optional	SSH host keys accepted so far. Without it the next SSH connection trusts on first use again.
secrets/ (in <config_dir>)	Yes, if you use it	Token secret files (token_secret_file), e.g. from token bootstrap. Not reconstructible without issuing new tokens.
schedules.toml (in the local data directory)	Yes, if you use schedule	The recurring operations schedule add created.
<profile>_state.db (in the cache directory)	No	Cluster snapshots and the operation queue, one file per profile. Regenerates from the cluster on next start.
proxxx.log*	No	14 daily rotations, forensic convenience only.

Config (config.toml) lives under the config directory alongside telegram_hmac.key and hitl_pending.db and is recreatable with proxxx init, though backing it up saves re-entering the profile.

Point PROXXX_AUDIT_DIR at a volume that is already backed up if you would rather not add a new backup target:

# systemd unit
Environment=PROXXX_AUDIT_DIR=/var/lib/proxxx/audit

[ ] Upgrade

systemctl stop proxxx-hitl          # let in-flight approvals settle
# verify + install the new binary (section 1)
systemctl start proxxx-hitl
proxxx doctor                       # confirms config, auth and audit chain

Stopping first is deliberate: while pending requests, quorum votes, and replay state persist on disk in <config_dir>/hitl_pending.db, a request approved while the service is stopped has nothing polling getUpdates until the new binary starts (and buttons minted before v0.13.x's <decision>:<id>:<hmac16> grammar are refused with a prompt to request again).

What is compatible across an upgrade:

• Config — backwards compatible. New keys default; old keys keep working.
• Audit DB — backwards compatible. v1 rows keep verifying under the v1 formula while new rows are written as v2.
• Declared state TOML — carries meta.schema_version. A document newer than the binary understands is refused rather than reinterpreted.
• Cache DB — not backwards compatible; see rollback.

[ ] Rollback

systemctl stop proxxx-hitl
# reinstall the previous binary
# delete the profile's state cache (schema is not backwards compatible):
#   Linux: ~/.cache/proxxx/<profile>_state.db
#   macOS: ~/Library/Caches/proxxx/<profile>_state.db
#   or $PROXXX_CACHE_DIR/<profile>_state.db when that is set
rm -f ~/.cache/proxxx/<profile>_state.db
systemctl start proxxx-hitl

The cache database records a schema version and refuses to open when that version is newer than the running binary — an older proxxx will report cache DB schema version N is newer than this binary's M rather than silently misreading it. Deleting <profile>_state.db is safe: it is regenerated from the cluster on the next start. The audit DB and its key need no action, and must not be deleted.

See also

• Configuration schema — every TOML block by section.
• Security model — threat model + invariants.
• Pre-commit gate — what every release passes before tagging.
• ACCEPTED-RISKS.md — residual risks (AR-1…AR-6) knowingly accepted for this release, with the guardrails above cross-referenced.
• Troubleshooting — error message → fix index.
• SECURITY.md — coordinated disclosure contact.