LWS Architecture

LWS is built with a modular architecture that separates concerns and makes the codebase maintainable and testable.

High-Level Overview

┌─────────────────────────────────────────────────────────┐
│                      User Interface                      │
├───────────────┬─────────────────────┬───────────────────┤
│   CLI (lws)   │   REST API (api.py) │   Web UI (ui.html)│
├───────────────┴─────────────────────┴───────────────────┤
│                    LWS Core Modules                      │
├──────────────────────────────────────────────────────────┤
│  config  │  ssh  │  proxmox  │  logging  │  utils      │
├──────────────────────────────────────────────────────────┤
│                  Command Executors                       │
├──────────────────────────────────────────────────────────┤
│    Proxmox VE Hosts (Multiple Regions & AZs)            │
└──────────────────────────────────────────────────────────┘

Project Structure

lws/
├── lws.py                      # Main CLI entry point (~3700 lines, 61 commands)
├── api.py                      # REST API server (Flask), drives lws.py via subprocess
├── ui.html                     # Web UI interface
├── config.yaml.example         # Committed config template — cp to config.yaml and edit
├── config.yaml                 # Your local config (gitignored, not in the repository)
│
├── lws_core/                   # Core functionality modules
│   ├── __init__.py            # Module initialization
│   ├── config.py              # Configuration management
│   ├── logging_setup.py       # Logging configuration
│   ├── ssh.py                 # SSH execution utilities
│   ├── proxmox.py             # Proxmox command wrappers
│   └── utils.py               # Shared utility functions
│
├── lws_commands/               # Reserved for a future command-module split — see
│   └── __init__.py            # "Future Enhancements" below. Currently just a stub;
│                               # all 61 commands still live directly in lws.py.
│
├── tests/                     # Unit and integration tests (see coverage note below)
│   ├── conftest.py
│   ├── test_config.py
│   ├── test_proxmox.py
│   ├── test_ssh.py
│   ├── test_utils.py
│   ├── test_python_support.py
│   ├── test_docs_examples.py  # Every `lws` command in the docs parses against the CLI
│   ├── test_docs_site.py      # Site version, navigation and page metadata
│   └── test_api.py            # Targeted tests for api.py's validation/routing logic
│
└── docs/                      # Documentation site (GitHub Pages, Jekyll; see docs/README.md)
    ├── index.html
    ├── _config.yml
    ├── _data/navigation.yml
    ├── _layouts/ _includes/
    └── _pages/

lws_core/ is comprehensively covered (96%). lws.py (the CLI, ~3,700 lines) has no automated tests at all. api.py (the REST API, ~1,200 lines) has tests only for its input-validation and routing logic (test_api.py, 22 tests) — its ~60 endpoint handlers themselves have no tests of their own.

Core Modules

1. Configuration Module (lws_core/config.py)

Handles loading and validating configuration from config.yaml.

Key Functions:

• load_config() - Loads YAML configuration
• validate_config() - Validates configuration structure
• mask_sensitive_info() - Masks passwords/keys for display

Configuration Structure:

regions:
  <region-name>:
    availability_zones:
      <az-name>:
        host: <proxmox-host>
        user: <ssh-user>
        ssh_password: <password>

instance_sizes:
  <size-name>:
    memory: <MB>
    cpulimit: <cores>
    storage: <storage>:<size>

2. SSH Module (lws_core/ssh.py)

Manages SSH connections to Proxmox hosts with retry logic and timeout handling.

Features:

• A fresh SSH connection per command (via sshpass + the system ssh binary — no connection reuse/pooling)
• StrictHostKeyChecking=accept-new: trusts a host’s key on first contact (these hosts are rarely pre-seeded into known_hosts), but rejects a later, changed key for a previously-known host — the actual MITM case this setting exists to catch
• The SSH password is passed via the SSHPASS environment variable (sshpass -e), not as a -p argument, so it never appears in ps//proc/<pid>/cmdline to other local users
• Up to 2 retries when the connection itself fails (ssh exit code 255); a command that ran is never retried
• A per-command timeout, ssh_command_timeout in config.yaml (3600 seconds by default, 0 for none)
• Password sanitization in logs

Key Functions:

• run_ssh_command(host, user, password, command, timeout=..., input_text=None) - Execute SSH command; input_text goes to the remote command’s standard input, which is how secrets reach a command without appearing on its command line

3. Proxmox Module (lws_core/proxmox.py)

Wrappers for executing Proxmox commands locally or remotely.

Key Functions:

• execute_command() - Execute command locally or via SSH
• run_proxmox_command() - Run Proxmox-specific commands
• run_argv(argv, use_local_only, host_details) - Run one argument list, locally or over SSH. OpenSSH joins the remote arguments into a single string that the host’s shell parses again, so run_argv quotes each argument (shlex.join) for the remote copy: the host receives exactly the arguments a local run would.

4. Utilities Module (lws_core/utils.py)

Common utility functions used across the application.

Key Functions:

• get_next_vmid() - Generate next available container ID
• is_container_locked() - Check if container is locked
• process_instance_command() - Generic instance command processor
• build_resize_command() - Build resize command arguments

5. Logging Module (lws_core/logging_setup.py)

Configures logging with both standard and JSON formats.

Features:

• Console logging
• File logging (lws.log)
• JSON logging (lws.json.log)
• Fixed at the ERROR level for the CLI; the REST API has its own api.log_level setting

Command Groups

LWS organizes commands into logical groups:

Configuration (conf)

• show - Display current configuration
• validate - Validate configuration
• backup - Backup configuration to file

Proxmox Hosts (px)

• list - List all Proxmox hosts
• status - Check host resource usage
• reboot - Reboot a Proxmox host
• upload - Upload LXC template
• clusters - List clusters
• Security group management
• Template management

LXC Containers (lxc)

• run - Create and start containers
• show - List/describe containers
• stop / start / reboot - Container lifecycle
• terminate - Destroy containers
• scale - Resize container resources
• snapshot-add / snapshot-rm - Snapshot management
• exec - Execute commands in container
• clone - Clone containers
• migrate - Migrate between hosts
• Backup & restore operations
• Resource monitoring

Docker/Apps (app)

• setup - Install Docker in container
• run - Run Docker containers
• deploy - Manage Docker Compose apps
• logs - View Docker logs
• list - List Docker containers
• remove - Uninstall Docker

Security (sec)

• scan - Security audit containers
• discovery - Network discovery

Data Flow

CLI Command Execution

1. User runs command
   ↓
2. Click parses arguments
   ↓
3. Load configuration (lws_core.config)
   ↓
4. Determine execution mode (local vs remote)
   ↓
5. Execute via SSH or locally (lws_core.ssh/proxmox)
   ↓
6. Parse and display results

API Request Flow

1. HTTP request to API
   ↓
2. Flask routes request
   ↓
3. Validate API key (if required)
   ↓
4. Parse request parameters
   ↓
5. Build lws.py subprocess command
   ↓
6. Execute command
   ↓
7. Format and return JSON response

Design Principles

1. Modularity

• Core functionality separated into focused modules
• Easy to test individual components
• Clear separation of concerns

2. Configuration Over Code

• All infrastructure details in config.yaml
• No hardcoded credentials or hosts
• Easy to manage multiple environments

3. Error Handling

• Comprehensive error messages
• Retry logic for transient failures
• Graceful degradation
• Non-zero process exit codes on failure (a failed operation no longer looks like a success to a caller checking the exit code — including api.py, which keys the HTTP status off it)

4. Security

• Credential masking in logs
• API key authentication (timing-safe comparison; refuses to start with an empty or placeholder key)
• SSH password passed via the SSHPASS environment variable, never in a log message, argv, or process list
• SSH host key verification (StrictHostKeyChecking=accept-new): trust-on-first-use, but a changed key for an already-known host is rejected
• Allow-listed input validation on values that end up in remote commands (security group names, container IDs, firewall rule fields, storage names), on top of the quoting in run_argv
• Firewall security groups are changed through the Proxmox API (pvesh), not by editing files under /etc/pve/firewall
• CORS denies all cross-origin browser access by default; origins must be listed explicitly to allow them

5. Extensibility

• Easy to add new commands
• Plugin-friendly architecture
• Clear interfaces between modules

Performance Considerations

SSH Connection Optimization

• Connection timeout: 15 seconds
• Command timeout: ssh_command_timeout, 3600 seconds by default, so that backups, package installs and migrations can finish
• Up to 2 retries when the connection is refused or times out before the command starts; a command that timed out is not run again, since it may already have changed the host
• ServerAliveInterval: 5 seconds, so a dead connection is noticed long before the command timeout

Parallel Execution

• Multiple containers can be managed in parallel
• Progress bars for batch operations
• ThreadPoolExecutor for concurrent operations

Resource Monitoring

• lxc resources samples CPU, memory, disk and process counts at an interval you choose (--interval, --count)
• lxc scale-check suggests changes from thresholds in config.yaml; it runs only when called

Future Enhancements

Planned architectural improvements:

1. Command Module Extraction
   • Move command groups to lws_commands/
   • Reduce lws.py to thin entry point
2. Database Support
   • Track container state
   • Historical metrics
   • Audit logging
3. Caching Layer
   • Cache Proxmox API responses
   • Reduce SSH overhead
   • Faster status checks
4. Plugin System
   • Custom command plugins
   • Third-party integrations
   • Event hooks
5. Web Dashboard
   • Real-time monitoring
   • Visual resource management
   • Interactive container controls