VM Autoscale

Appearance

Changelog ​

Tags are not in chronological order

v1.2.0 was published in December 2025 and v0.1.1 in April 2026 — the two release lines were never reconciled. Entries below are chronological, so 1.2.0 appears after 0.1.1. From 1.3.0 onwards the numbering is monotonic again. Go by the releases page rather than by the highest number.

Unreleased ​

1.7.2 - 2026-09-07 ​

Upgrade note. RAM scaling did not work on a guest whose balloon sits at its memory, which is the default and the state the shipped example configuration produces: every scale-up was rejected by the hypervisor. CPU scaling was unaffected. No configuration changes are required.

Fixed ​

  • RAM scaling was rejected by the hypervisor on the ordinary guest configuration. memory is a ceiling and balloon is the allocation; Proxmox refuses any configuration where the second exceeds the first. The service read memory as the current value and always wrote balloon, so a scale-up computed memory + 512 and issued qm set -balloon with it — answered by balloon value too large (must be smaller than assigned memory) on every guest whose balloon sits at its ceiling, which is the default and the state this project's own example configuration produces. vm_manager.py now reads the balloon as the allocation, moves the balloon alone under the ceiling, and raises both in a single command when the target is above it.
  • A guest that could not scale live could not scale down at all. Where ballooning was not available the service issued qm set -memory alone, which Proxmox rejects for the same reason while the balloon sits above the new ceiling — and changes nothing. An explicit balloon target now travels with the ceiling, which also means a guest rebooting into a higher ceiling actually receives the memory instead of staying pinned at its old allocation. An absent balloon line already tracks memory and balloon: 0 is a deliberate choice, so neither is written back.
  • A scale-down no longer lowers the ceiling of a running guest. That unplugs a DIMM, which the guest may refuse: the testbed returned error unplug memory module after Proxmox had written the new value, leaving the configuration and the guest disagreeing. Deflating the balloon returns the memory just as well, and a ceiling nobody reaches costs nothing.
  • vm_autoscale_billing_degraded was exported and never documented; config_schema.py and version.py were missing from the module reference and check_dependencies.py from the architecture page. All four were found by the documentation tests on their first run, which is the point of them.

Added ​

  • tests/e2e_support.py and tests/test_e2e_real_node.py: an end-to-end suite that drives a real Proxmox node and resizes a real VM. Marked e2e and skipped unless VMA_E2E_HOST is set; it is not in CI and never will be, since CI has no hypervisor to talk to. The entry criterion for a test is that a mock could not have told us. It refuses any VM whose name does not contain testbed, restores a fixed baseline around every test and asserts the VM is actually at it, and leaves the guest powered off if it found it that way. See Contributing.
  • TestMemoryIsACeilingNotAnAllocation in tests/test_real_proxmox_shapes.py, built from a configuration copied verbatim off the node, so the defect above is caught in CI rather than only on hardware.
  • tests/test_docs_match_code.py: the documentation can no longer drift away from the code in silence. The build fails on a configuration key the schema accepts but the reference does not mention, a metric the service exports but the operations guide does not list, a module missing from the module reference, or a version claimed anywhere that is not the one in version.py. REVIEW.md §8.3 recorded this risk and nothing had acted on it — thirteen files once needed correcting in a single change, and every stale claim was found after the code had already moved.

Changed ​

  • The warning logged when memory cannot be applied live now names the single thing standing in the way — the guest is stopped, hotplug: memory is off, or NUMA is disabled — instead of describing two of them at once.

1.7.1 - 2026-09-07 ​

Upgrade note. Two of these are serious enough to warrant upgrading promptly:

  • On a node that writes any warning to stderr — a stale API token entry in /etc/pve/user.cfg is enough — the autoscaler did nothing at all, every cycle, while sending an error notification each time.
  • On any VM without an explicit vcpus line, which is the Proxmox default, a scale-up made the guest smaller.

No configuration changes are required.

Fixed ​

Both found by running the service against a real Proxmox VE 9.1.7 node — the first time this project has ever been exercised against real hardware. 290 tests passed while both of these sat in the scaling path, because every fixture encoded the same assumptions the code did.

  • A scale-up could halve a guest's online CPUs. Proxmox omits vcpus from qm config when every core is online, and _get_current_vcpus read that absence as 1. On a 4-core guest with all cores online, a scale-up therefore issued qm set -vcpus 2 — a reduction — and a scale-down computed max(1 - 1, 1) and dropped it straight to 1, past the one-step-at-a-time contract. Most VMs never set vcpus explicitly, so most VMs were affected. Absence now means the core count.
  • A benign warning on stderr disabled the host gate entirely.check_host_resources raised whenever stderr was non-empty, ignoring the exit status it already had. A node carrying a stale API token entry prints user config - ignore invalid acl token '...' on a successful call, so every VM on that node was skipped every cycle, for the lifetime of the service, with an error notification each time. The gate now judges by exit status.

Added ​

  • tests/test_real_proxmox_shapes.py: 13 tests built from payloads copied verbatim off a real PVE 9.1.7 node rather than invented — a qm config with no vcpus, a successful command that writes to stderr, and the real /cluster/resources JSON. 8 of them fail against the previous commit.

Note ​

The claim in REVIEW.md §3.4 that the old pvesh table scrape was fragile to version drift was not borne out: it still parses correctly on PVE 9.1.7. The defects that justified replacing it were real — the substring match on VMID and the 0.0-on-failure fallback — but the format-drift argument was theoretical and is corrected here.

1.7.0 - 2026-09-05 ​

Upgrade note. Three observable changes:

  • An invalid configuration now refuses to start, listing every problem at once. A configuration that "worked" while quietly ignoring half its keys will stop working. That is the fix, but it will look like new breakage — read the errors, they name the path and the reason.
  • A shrink now needs two consecutive low readings (scale_down_after_cycles), so VMs scale down one cycle later than before. Growth is unchanged and still acts on the first reading.
  • Identical notifications are suppressed for 15 minutes (notification_dedup_seconds). If you were alerting on notification volume rather than content, that signal is gone.

install.sh now installs the latest release tag rather than the default branch. Existing installations are unaffected until the next reinstall.

Everything here comes from acting on REVIEW.md, a full critical review of v1.6.0. Ordered by the risk register in that document.

Fixed ​

  • Billing data could be destroyed by a routine systemctl stop. The state file was opened with mode w, which truncates before the first byte is written, and there was no signal handling anywhere — so SIGTERM killed the process wherever it was, including mid-write. On restart the read failed, logged a warning nobody alerts on, and continued with empty state; the next scaling action then wrote that empty state over the damaged file. Writes are now atomic (temp file, fsync, rename). An unreadable file is quarantined as .corrupt-<timestamp>, writes are disabled for the run so nothing can overwrite it, and the service carries on scaling with billing off.
  • SIGTERM and SIGINT are handled. Only KeyboardInterrupt was caught, which is SIGINT alone. The loop now stops between VMs and waits on an event instead of sleeping, so a stop is acted on at once rather than up to check_interval later. The systemd unit gives it 45 seconds to do that.
  • A failed reconnect inside execute_command reported the wrong error.self.client was left as None, so every remaining attempt raised AttributeError and buried the actual transport failure.
  • Billing timestamps are timezone-aware UTC. They were naive, so any period spanning a DST transition was off by an hour with no way to detect it.
  • install.sh installs a release, not a branch. It cloned the default branch, so the command the README puts first shipped unreleased code and the release tags were consumed by nobody following the documented path.

Added ​

  • A configuration contract (config_schema.py). Validation was one check that four top-level keys existed; everything else was read with an inline .get() and an inline default across 56 sites in three modules. A key that was written, documented and never read produced no error anywhere — which is exactly how scaling_limits, per-VM thresholds, ssh_port and per-VM limits each shipped broken. The whole file is now validated for types, ranges, enumerations, cross-field consistency and referential integrity, and every problem is reported at once. A VM naming a host that does not exist is an error rather than a VM silently skipped forever. Unknown keys are warnings, so a config carrying a newer key still boots.
  • Sustained shrink (scale_down_after_cycles, default 2). Growing fails safe; reclaiming memory from a guest that is using it drives it into swap or to the OOM killer, and a vCPU unplug may be refused. The two were treated identically, on a single sample.
  • Notification deduplication (notification_dedup_seconds, default 900). An unreachable node produced one priority-9 notification per VM per cycle — 240 an hour for twenty VMs, with the real alert somewhere underneath.
  • A version constant, logged at startup and carried as a label on vm_autoscale_build_info. The service could not previously state what it was.
  • pyproject.toml: installable, packageable, with an entry point and upper bounds on dependencies.
  • CI gates: ruff, a coverage floor, pip-audit (which SECURITY.md has been recommending to contributors while CI never ran it), and a check that the shipped config.yaml passes its own validator.
  • 89 tests, 290 total.

Changed ​

  • qm config is fetched once per cycle instead of three times per scaling decision, and the running state is cached rather than re-queried seconds after process_vm established it. Measured: 5 SSH commands per scale_cpu down to 2.
  • Test coverage no longer inverts the risk. autoscale.py was at 63% and ssh_utils.py at 63% while the optional metrics endpoint sat at 100%; the main loop, the entrypoint, the constructor and execute_command were entirely untested. They are now 86% and 93%, 87% overall.
  • Tests build a real VMAutoscaler. Every test patched __init__ away and hand-assembled seven attributes, so the constructor was never executed and adding one field broke ten tests with no behaviour change.
  • process_vm went from 95 lines to 42, with each step named and the near-identical CPU and RAM branches collapsed into one.
  • Bare Exception raises are typed as SSHCommandError and HostResourceUnavailable, so callers and tests can be specific.
  • A webhook script writable by group or others is refused. It executes as root at every billing period boundary.

Not addressed ​

Stated plainly rather than left to inference. From REVIEW.md: there is still no hypervisor abstraction (§3.3), the loop is still sequential with no parallelism (§3.5), host_limits is still global per fleet rather than per node, cooldown state is still in memory, the install location still violates FHS, and the strategic criticism in §2 — the reaction envelope, the billing feature's category error, the commercial model — is not something code can fix.

1.6.0 - 2026-09-05 ​

Upgrade note. A scaling action that fails now raises instead of being logged as done. If something in your fleet has been silently failing to scale, CommandFailed will start appearing in the log. That is the fix working, not new breakage — but it will look like new breakage.

Start with dry_run: true if you want to see what the service intends before letting it act.

Added ​

  • Dry-run mode (dry_run: true). Everything is evaluated and nothing is changed: no command that touches a VM is issued, hotplug auto-configuration included. Reads still happen, so the evaluation is real; the cooldown still applies, so the cadence in the log is the cadence you would get. Notifications carry a [DRY RUN] prefix and billing records nothing.
  • Prometheus metrics endpoint (metrics.enabled), off by default and localhost-bound when on. Exports cycle count and duration, per-VM CPU/RAM, running state and errors, scaling actions by resource and direction, scaling failures, per-node utilisation and gate blocks. A metric that could not be read has its series removed rather than reported as zero. Standard library only — no new dependency.
  • Per-VM scaling_limits. A VM entry may override any of the global limits, so a 2-core web server and a 16-core database can share one instance. Resolution: the VM's own block, then the global section, then the legacy flat keys, then the built-in default.
  • tests/test_dryrun_metrics_limits.py: 31 regression tests (201 total).

Fixed ​

  • A failed qm set is no longer reported as a success. execute_command returns the exit status rather than raising, and every caller discarded it, so a command that exited non-zero still produced Increased cores to 5 (hotplug applied) in the log, a success notification, a billing record and a consumed cooldown — which then delayed the retry by a full scale_cooldown. Commands now run through a checked helper that raises CommandFailed with the command, the exit status and stderr.
  • A failed qm config read no longer invents a value. The getters returned 1 core, 1 vCPU or 512 MB when the command failed, and those fabricated numbers went straight into a scaling decision — the service would happily "scale up from 1 core" a guest that actually had eight. They now raise. An absent key still means the documented Proxmox default, because that is what absence legitimately means.
  • A missing VM is no longer retried for 30 seconds. qm status exiting non-zero was treated as a transient fault and retried three times with backoff. It now fails fast and logs stderr.

1.5.0 - 2026-09-05 ​

Upgrade note. Three behaviour changes land together:

  • A thresholds: block in a VM entry now takes effect. If you wrote one expecting it to work, that VM starts using it — re-read the numbers first.
  • Host keys are now verified. The first connection to each node records its key and a later change is refused. Rebuilt a node? Remove its line from ssh_known_hosts.
  • Billing starts writing CSV reports and calling webhooks where it previously did nothing.

Fixed ​

  • Per-VM thresholds are read. The block has been in the example config.yaml since the beginning and nothing consulted it; every VM used the global scaling_thresholds. Both the flat shape (cpu_high, cpu_low, ram_high, ram_low) and the nested one (cpu: { high, low }) now work, and any bound you omit keeps the global value.
  • Billing generates reports. generate_period_report existed but nothing called it, so enabling billing produced a growing state file and no CSV, no webhook and no report. The main loop now emits one per VM once a billing period elapses; the period clock is persisted, so it survives restarts.
  • Billing records VM start/stop. record_vm_state_change was never called, which is why every report showed 100% uptime. Transitions are now recorded — transitions only, not one entry per poll.
  • Downtime is no longer billed as uptime. _calculate_resource_cost accepted the uptime records and ignored them, so a VM powered off for a week was billed for that week at its last known spec. Cost is now charged only for the hours a VM was actually running.
  • A VM that never changed spec is no longer billed zero. Spec changes are events, so a guest that held one size for the whole period had no records inside it and cost nothing. The spec in effect at period_start is now carried in, and so is the running state.
  • ssh_port has a default again. It was indexed directly rather than via .get(), so omitting it raised KeyError and every VM on that host failed. The shipped example omitted it on host2.

Security ​

  • SSH host keys are verified. The client set AutoAddPolicy without ever loading or saving a known_hosts file, so every connection accepted whatever key it was offered and remembered nothing — no protection at all for a service holding root credentials. New ssh_host_key_policy, default accept-new: a node's key is recorded on first contact and a later change is refused. strict requires a pre-populated ssh_known_hosts; auto reproduces the old behaviour and logs a warning. A mismatch is fatal and is not retried.

Added ​

  • tests/test_thresholds_hostkeys_billing.py: 32 regression tests (170 total) covering threshold resolution and precedence, all three host key policies and the mismatch path, uptime-weighted cost, period carry-in, the persisted report clock, and the autoscaler's own billing plumbing.

Changed ​

  • New logo. The bar-chart-and-arrow mark said "analytics" and only showed growth, which misrepresents a service whose point is moving in both directions. The replacement is a solid allocation between two chevrons, and it stays legible at 16px. favicon.svg keeps the orange tile for the browser tab and navigation; logo-mark.svg is a transparent variant used in the documentation hero.

1.4.0 - 2026-09-05 ​

Upgrade note. A VM whose usage cannot be read now stops scaling instead of being treated as idle and walked down to its minimum. If something in your fleet was quietly shrinking, it will now hold its size and log unavailable — that is the fix working, but a VM with a persistent metric problem will no longer scale at all. Alert on the warning.

Fixed ​

  • Guest metrics no longer come from scraping a table. Usage was read with pvesh get /cluster/resources | grep 'qemu/<vmid>' | awk -F '│' …, which depended on box-drawing separators and fixed column positions — both of which move between Proxmox versions — and whose substring match also caught VMID 1010 when looking for 101. The service now requests --output-format json and matches on type == "qemu" and an exact vmid.
  • An unreadable metric is no longer treated as zero. A parse failure used to report 0.0, which sits below every sensible low threshold, so it was indistinguishable from an idle guest and walked the VM down to its minimum one step per cycle. get_resource_usage() now returns None for a metric it could not read, the log says unavailable, and scaling is skipped for that resource. A genuinely idle guest still reports 0.0 and still scales down.
  • SSH keys of any type now load (Ed25519 included). RSAKey.from_private_key_file was called directly, so an Ed25519 key — the type SECURITY.md recommends — failed to load however valid it was. Key loading now goes through paramiko.PKey.from_path, with a per-class fallback for older paramiko. Encrypted keys remain unsupported, and the error now names the file and the reason.

Added ​

  • 30 regression tests (138 total), including real Ed25519, ECDSA and RSA keys generated on disk and loaded through the production path, and JSON payloads covering the VMID-prefix collision, LXC guests sharing a VMID, missing and non-numeric fields, and the distinction between unreadable and idle.

Removed ​

  • _parse_cpu_usage, _parse_ram_usage and _convert_to_gib — the regex parsers for the old table format.

1.3.0 - 2026-09-05 ​

Upgrade note. Limits you set in scaling_limits were previously ignored; they now take effect. Re-read that section before upgrading, particularly if you configured a max_cores above 8 or a min_ram_mb above 512 and have been running against the hardcoded defaults without realising.

This release also drops Python 3.9. See Removed below.

Fixed ​

  • scaling_limits from config.yaml is now actually enforced. The config validator required the section, but VMResourceManager looked the values up as flat top-level keys under names that did not exist (max_ram/min_ram instead of max_ram_mb/min_ram_mb). Every installation silently ran on the hardcoded defaults 1–8 cores and 512–16384 MB, so the documented min_ram_mb: 1024 (needed because NUMA misbehaves below 1 GB) had no effect. Flat top-level keys are still accepted as a fallback for older configs.
  • CPU and RAM now have independent cooldowns (#30): a single shared timestamp meant any CPU threshold breach suppressed RAM scaling for the same cycle. The cooldown is also consumed only when a scaling command is actually issued — previously the mere check consumed it, so a VM already at its limit was rate-limited for doing nothing.
  • scale_cooldown now applies between polling cycles. VMResourceManager was rebuilt on every iteration of the main loop, resetting the cooldown to zero; the effective interval was check_interval, not scale_cooldown. Managers are now cached per VM and rebound to each cycle's SSH connection. As a side effect, hotplug auto-configuration runs once per VM instead of issuing two extra qm config calls per VM per cycle — and is retried on the next cycle if it failed, so a transient SSH error during the first cycle no longer leaves a VM permanently non-live-scalable.
  • can_scale() and _mark_scaled() reject unknown resource names instead of silently treating them as never rate-limited.

Security ​

  • install.sh no longer leaves config.yaml world-readable. A recursive chmod 755 over the install directory applied to the config file too, which stores the Proxmox root SSH password and SMTP credentials in plain text. The config and its backup are now chown root:root + mode 600, and the backup directory is mode 700 — matching what SECURITY.md already prescribed.

Added ​

  • Documentation site at https://fabriziosalmi.github.io/proxmox-vm-autoscale/ — guide, configuration reference, architecture, module API, a full catalogue of known limitations, threat model, hardening guide and privacy statement. Built with VitePress and deployed by GitHub Actions, with sitemap.xml, llms.txt, .well-known/security.txt and JSON-LD structured data.
  • tests/test_scaling_limits_and_cooldown.py: 24 regression tests covering limit resolution (including a test that the limits shipped in config.yaml are the ones enforced), independent per-resource cooldowns, resource-name validation, and manager reuse and hotplug retry across cycles.

Changed ​

  • install.sh now installs the vm_autoscale.service tracked in the repository instead of generating its own copy inline. The two had drifted: the generated unit had no RestartSec, so a crash-looping service restarted as fast as systemd allowed, and the tracked file was never actually used.
  • install.sh no longer aborts when pip cannot write to the system interpreter. On Proxmox VE 8 and Debian 12+ that is PEP 668's externally-managed-environment, and treating it as fatal failed an installation that was otherwise complete — the apt step already provides paramiko, PyYAML and requests.
  • Dependency floors raised to paramiko>=5.0.0, PyYAML>=6.0.3 and requests>=2.34.2.
  • GitHub Actions bumped to current majors. Dependabot now watches the github-actions and npm ecosystems as well as pip.
  • README trimmed from 343 lines to 131, ARCHITECTURE.md from 290 to 91 and SECURITY.md from 122 to 106, with the long-form material moved to the documentation site rather than maintained in two places and drifting apart.

Removed ​

  • Python 3.9 support. requests 2.34.2 requires Python >= 3.10. Python 3.9 reached end of life in October 2025, and the only Proxmox release shipping it is VE 7, end of life since July 2024. The CI matrix is now 3.10–3.12.
  • release-notes-v0.1.1.md from the repository root — it duplicated the GitHub release verbatim.

0.1.1 - 2026-04-27 ​

Fixed ​

  • Host RAM usage calculation now matches Proxmox WebUI (#38): RAM usage is computed from the used field reported by pvesh, which excludes reclaimable buff/cache. Previously, free + cached was used as the available memory estimate, leading to incorrectly high RAM usage percentages (e.g. ~90% instead of ~66%) and suppressed scaling on hosts with heavy caching.

Added ​

  • tests/test_host_resource_checker.py: full test coverage for HostResourceChecker, including the RAM calculation fix, threshold boundary cases, bytes output handling, and error paths.
  • tests/test_autoscale.py: tests for NotificationManager (config validation, message formatting, routing with Gotify/SMTP fallback), VMAutoscaler config loading, _handle_cpu_scaling / _handle_ram_scaling decision logic, and VMResourceManager scaling helpers (scale_cpu, scale_ram, can_scale, _parse_cpu_usage, _parse_ram_usage).

Changed ​

  • Removed unused cached_mem and free_mem variables from HostResourceChecker.check_host_resources.

1.2.0 - 2025-12-09 ​

Documented retroactively. This tag was published before 0.1.1 despite the higher number — the two release lines were never reconciled. Entries in this file are chronological, so 1.2.0 appears below 0.1.1. From 1.3.0 onwards the numbering is monotonic again.

Added ​

  • Hotplug support (#37): auto_configure_hotplug enables hotplug and NUMA on managed VMs; RAM changes use balloon and CPU changes use vcpus so they apply to a running guest, with a fallback when hotplug is unavailable. cores changes and NUMA itself still require a guest reboot.
  • Billing tracking (#33): billing_tracker.py, recording CPU/RAM spec changes with timestamps, plus period cost calculation, CSV export and webhook support.
  • 30 unit tests covering hotplug and billing.

Known issues in this release ​

Recorded here because the original release notes overstate what shipped:

  • Only spec recording is wired into the service. generate_period_report, export_csv, run_webhook, record_vm_state_change and set_vm_name are public API that nothing calls, so no CSV is produced and no uptime is tracked unless you invoke them yourself.
  • The cost calculation accepts uptime records and ignores them, so downtime is billed as uptime.
  • auto_configure_hotplug ran on every polling cycle, issuing two extra qm config calls per VM per cycle. Fixed in 1.3.0.

[0.1.0-docs] - Unreleased documentation pass ​

Added ​

  • Comprehensive documentation improvements across all markdown files
  • requirements.txt for Python dependency management
  • ARCHITECTURE.md with detailed system architecture documentation
  • Troubleshooting section in README with common issues and solutions
  • Table of contents in README for better navigation
  • Enhanced configuration examples with inline comments
  • Development setup instructions in CONTRIBUTING.md
  • Comprehensive security policy in SECURITY.md
  • This CHANGELOG file to track project changes

Changed ​

  • Enhanced README.md with improved structure and clarity
  • Updated CONTRIBUTING.md with detailed contribution guidelines
  • Expanded SECURITY.md with security best practices and reporting process
  • Improved configuration examples with better annotations
  • Updated prerequisites section with specific version requirements

Fixed ​

  • Typo in config.yaml comment: "doeasnt" → "doesn't"
  • Typo in README: "togheter" → "together"
  • Improved formatting and consistency across documentation
  • Clarified Python version requirement (3.6+)

0.1.0 - Initial Release ​

Added ​

  • Initial release of Proxmox VM Autoscale
  • Automatic CPU and RAM scaling for Proxmox VMs
  • Multi-host support via SSH
  • Gotify notification support
  • Email notification support
  • Systemd service integration
  • Configuration via YAML file
  • Comprehensive logging
  • Host resource safety checks
  • Scaling cooldown periods