Operations

Service control

sudo systemctl start   vm_autoscale.service
sudo systemctl stop    vm_autoscale.service
sudo systemctl restart vm_autoscale.service
sudo systemctl status  vm_autoscale.service

sudo systemctl enable  vm_autoscale.service   # start at boot
sudo systemctl disable vm_autoscale.service

The unit sets Restart=always with RestartSec=10, so systemd brings the process back ten seconds after a crash rather than hammering it. It also sets TimeoutStopSec=45: the service handles SIGTERM, finishes the VM it is on, stops the metrics endpoint and leaves the billing state file intact, so it needs room to do that rather than being killed mid-write. Confirm what is actually installed:

systemctl cat vm_autoscale.service | grep -E 'Restart|ExecStart'

If RestartSec is missing you are running a unit generated by an older installer; reinstall it from the repository copy.

A restart clears cooldown state

Cooldown timers live in memory. After a restart the first cycle can scale immediately, regardless of when the last change happened. Restarting repeatedly — during a config edit session, say — removes the rate limiting entirely for that period.

Logs

Two sinks, both active:

tail -f /var/log/vm_autoscale.log        # file, per logging_config.json
journalctl -u vm_autoscale.service -f    # stdout, captured by systemd

Useful filters:

journalctl -u vm_autoscale.service --since "1 hour ago" -p warning
grep -E 'Scaled (up|down)' /var/log/vm_autoscale.log
grep -c 'Scaled up' /var/log/vm_autoscale.log
journalctl -u vm_autoscale.service --since today | grep 'VM 101'

Log levels

logging_config.json takes precedence over the logging.level key in config.yaml — the latter is only consulted when the JSON file is absent. As shipped, the file handler writes at DEBUG and the console at INFO, so /var/log/vm_autoscale.log is considerably more verbose than journalctl.

To quieten the file, edit logging_config.json:

{
  "handlers": {
    "file": { "level": "INFO" }
  }
}

Restart the service afterwards; logging is configured once at startup.

Log rotation

Nothing rotates /var/log/vm_autoscale.log. At DEBUG, with a large fleet, it grows quickly. Add:

# /etc/logrotate.d/vm_autoscale
/var/log/vm_autoscale.log {
    weekly
    rotate 8
    compress
    delaycompress
    missingok
    notifempty
    copytruncate
}

copytruncate matters: the service holds the file open for its whole lifetime and will keep writing to the rotated inode otherwise.

Reading the log

Line	Meaning
Host CPU Usage: 12.40%, Host RAM Usage: 61.20%	Gate 2 passed for this node
Host pve1 resources maxed out. Skipping scaling.	Gate 2 blocked; no VM on this node scales this cycle
VM 101 is not running. Skipping scaling.	Gate 1 blocked
VM 101 current usage - CPU: 3.20%, RAM: 41.50%	The sample the decision is based on
No CPU scaling required.	Threshold crossed but a limit was already reached
Scaled up vCPUs to 3 ... (hotplug applied)	Live change
... (requires reboot for full effect)	Config changed; guest unaffected until reboot
CPU: unavailable	That metric could not be read; scaling is skipped, not guessed
VM 101: no usage metrics available this cycle.	Neither metric could be read
Error processing VM 101 on host pve1: ...	Per-VM failure; loop continues

unavailable is safe, but it means nothing is scaling

No action is taken on a metric that could not be read — the earlier behaviour of substituting 0.0, which scaled VMs down, is gone. But a VM whose metrics never resolve is a VM that never scales. Alert on it. See troubleshooting.

Changing configuration

Config is read once at startup and fully validated before anything runs. There is no reload signal.

A broken configuration is refused with every problem listed at once:

CRITICAL Refusing to start. 2 configuration problem(s) found:
  - virtual_machines[0].proxmox_host: 'pve-typo' does not match any
    proxmox_hosts name (pve1)
  - scaling_limits.min_cores: 16 is above max_cores (8)

Suspicious-but-usable settings are logged as warnings and the service starts anyway — an unknown key, a dead band narrow enough to flap, a scale_cooldown below check_interval, a metrics endpoint bound off localhost.

sudo cp /usr/local/bin/vm_autoscale/config.yaml /root/config.yaml.bak
sudo nano /usr/local/bin/vm_autoscale/config.yaml
python3 -c "import yaml,sys; yaml.safe_load(open('/usr/local/bin/vm_autoscale/config.yaml'))" \
  && echo "YAML OK"
sudo systemctl restart vm_autoscale.service
journalctl -u vm_autoscale.service -n 30

A malformed config prevents startup:

CRITICAL Failed to start VM Autoscaler: Missing required configuration sections: scaling_limits

Always confirm the service came back up after an edit — Restart=always will keep retrying a process that exits immediately on a bad config, and the failure is only visible in the log.

Pausing scaling

Per VM, without removing its entry:

virtual_machines:
  - vm_id: 101
    proxmox_host: pve1
    scaling_enabled: false

Per resource:

    cpu_scaling: true
    ram_scaling: false

Everything at once: systemctl stop vm_autoscale.service.

Dry run

dry_run: true

Everything is evaluated and nothing is changed. No command that touches a VM is issued — including hotplug auto-configuration — the log records what would have happened, notifications carry a [DRY RUN] prefix, and billing records nothing, because nothing changed.

[WARNING] DRY RUN: no command that changes a VM will be issued.
[INFO]    VM 101 current usage - CPU: 91.20%, RAM: 44.10%
[INFO]    [dry-run] VM 101: would run `qm set 101 -vcpus 3`

Reads still happen, so what you see is a real evaluation against real usage, and the cooldown still applies — the cadence in the log is the cadence you would get. This is the right way to start on a fleet you did not build.

Metrics

An optional Prometheus endpoint, off by default and bound to localhost when enabled:

metrics:
  enabled: true
  bind: 127.0.0.1
  port: 9808
  path: /metrics

It has no authentication

The series name your nodes and VMIDs and report their utilisation, and this process holds root credentials for your hypervisors. Leave it on 127.0.0.1 and let your scraper reach it over a tunnel, or put a reverse proxy that authenticates in front of it. A bind failure is logged and the service carries on without it.

curl -s http://127.0.0.1:9808/metrics

Metric	Type	Labels
vm_autoscale_up	gauge
vm_autoscale_build_info	gauge	version, dry_run
vm_autoscale_billing_degraded	gauge	—
vm_autoscale_cycles_total	counter
vm_autoscale_cycle_duration_seconds	gauge
vm_autoscale_last_cycle_timestamp_seconds	gauge
vm_autoscale_cycle_errors_total	counter
vm_autoscale_vm_running	gauge	vm_id
vm_autoscale_vm_cpu_percent	gauge	vm_id
vm_autoscale_vm_ram_percent	gauge	vm_id
vm_autoscale_vm_errors_total	counter	vm_id
vm_autoscale_metric_unavailable_total	counter	vm_id, resource
vm_autoscale_scaling_actions_total	counter	vm_id, resource, direction
vm_autoscale_scaling_failures_total	counter	vm_id, resource
vm_autoscale_host_cpu_percent	gauge	host
vm_autoscale_host_ram_percent	gauge	host
vm_autoscale_host_gate_blocked_total	counter	host

Absent is not zero

vm_autoscale_vm_cpu_percent is removed for a VM whose CPU could not be read, rather than reported as 0. Emitting a zero would put the same lie into your dashboards that it used to put into the scaling decision. Watch vm_autoscale_metric_unavailable_total for that case.

Alerts worth having:

# The loop has stopped making progress
- alert: VMAutoscaleStalled
  expr: time() - vm_autoscale_last_cycle_timestamp_seconds > 900
  for: 5m

# A VM's metrics cannot be read, so it is no longer scaling
- alert: VMAutoscaleMetricUnavailable
  expr: increase(vm_autoscale_metric_unavailable_total[15m]) > 0

# Scaling is being attempted and failing
- alert: VMAutoscaleScalingFailures
  expr: increase(vm_autoscale_scaling_failures_total[15m]) > 0

# Flapping: up and down on the same VM within the hour
- alert: VMAutoscaleFlapping
  expr: |
    increase(vm_autoscale_scaling_actions_total{direction="up"}[1h]) > 2
    and
    increase(vm_autoscale_scaling_actions_total{direction="down"}[1h]) > 2

Monitoring without the endpoint

Is it alive?

systemctl is-active vm_autoscale.service

Is it doing anything? A healthy service writes at least one host-usage line per cycle. A watchdog on log staleness:

#!/bin/bash
LOG=/var/log/vm_autoscale.log
AGE=$(( $(date +%s) - $(stat -c %Y "$LOG") ))
if [ "$AGE" -gt 900 ]; then
    echo "vm_autoscale log stale: ${AGE}s" >&2
    exit 1
fi

Is it flapping? Repeated up/down on the same VM means your dead band is too narrow:

grep -E 'Scaled (up|down)' /var/log/vm_autoscale.log \
  | grep 'VM 101' | tail -20

systemd-native alerting on failure:

# /etc/systemd/system/vm_autoscale.service.d/alert.conf
[Unit]
OnFailure=status-email@%n.service

Backups

Worth keeping:

• /usr/local/bin/vm_autoscale/config.yaml — contains credentials; encrypt it
• /etc/vm_autoscale/config.yaml.backup — same
• /var/log/vm_autoscale/billing/billing_data.json — if you bill from it
• /etc/systemd/system/vm_autoscale.service and any drop-ins

Never commit config.yaml to a repository. .gitignore covers config.local.yaml, not config.yaml — the tracked config.yaml is the example, and it is easy to commit a real one over it by accident.

Running two instances

For different intervals or different host limits per node group:

sudo cp -r /usr/local/bin/vm_autoscale /usr/local/bin/vm_autoscale_slow
sudo cp /etc/systemd/system/vm_autoscale.service \
        /etc/systemd/system/vm_autoscale_slow.service
sudo nano /etc/systemd/system/vm_autoscale_slow.service   # point at the new dir

autoscale.py hardcodes its config path to /usr/local/bin/vm_autoscale/config.yaml in main(), so a second instance needs its own copy of the directory rather than just its own config file. Make sure the two instances never manage the same VM: they do not share cooldown state and will fight.