Changelog

All notable changes to this project will be documented in this file.

The format is based on Keep a Changelog, and this project adheres to Semantic Versioning.

[4.0.1] - 2026-09-07

Fixed

• proxy_core.py puts its own repo root on sys.path. It resolved from src... only because cli.py exported PYTHONPATH before exec'ing mitmdump; driving mitmdump directly — which the deployment guide does — bypassed that and the import failed before the addon was constructed.
• aidlp stats reports PII entities detected. dlp_pii_detected_total is emitted per type, so the command's unlabelled pattern could never match it.
• The Vault HTTP call is bounded by the new vault.timeout (default 10s). The circuit breaker counts failures, so a Vault that was slow but not yet erroring could stall a fetch without incrementing the count that opens it.
• A duplicate dlp: key in README's config.yaml example made it invalid YAML.

Added

• dlp.degrade_to_static_on_ml_timeout (default false). When an ML timeout is the only failure, the request can be forwarded with the static-keyword redaction that already completed instead of failing closed. Off by default: reducing detection coverage is a deliberate choice, not a default. While on, each occurrence is recorded in the request stats, counted on dlp_ml_degraded_total, and logged with the request id.
• README's Observability section documents every exposed metric, grouped by purpose, and states that aidlp stats surfaces only a subset.

[4.0.0] - 2026-09-07

Sweep of the remaining audit findings. Two changes alter an existing contract, hence the major.

⚠️ BREAKING

• Unknown keys inside a config section are now rejected. ProxyConfig, DLPConfig, SecretsProviderConfig and VaultConfig set extra="forbid", so a misspelled nested key fails at startup instead of being dropped while the default silently applied. Unknown top-level sections are still ignored (a stray AIDLP_* variable must not stop the proxy) but are now named in a startup warning.
• GET /_health returns JSON, not plain text. The body is {"status", "version", "details"}. A probe matching on the literal OK needs updating; one checking the status code does not.

Fixed

• secrets_provider.type is a Literal["file","vault"], and type: vault without a vault: section is rejected. "Vault", "VAULT" or a typo used to fall through to the file provider silently.
• ml_threshold is constrained to [0, 1]. Above 1.0 every score comparison was false, so ML redaction stopped flagging anything while stats looked normal.
• DLPAddon.done() now shuts the engine down; it previously only logged, so the worker tasks and the poller outlived the addon.
• shutdown() resets workers/poller_task, so a later start_workers() actually respawns. It used to see the cancelled-but-non-empty lists as live and spawn nothing, leaving an engine with no workers at all. aclose() is the awaiting variant; done() cannot be async because mitmproxy triggers DoneHook through invoke_addon_sync, which rejects coroutine hooks.
• Enqueueing for ML analysis is bounded by ml_timeout. With a full queue the put() blocked forever, hanging the request instead of failing closed.
• A stuck ML worker is recycled by a watchdog. asyncio.to_thread cannot be cancelled, so one pathological input used to occupy a worker indefinitely and drain a fixed pool to zero capacity.
• A missing terms file is announced. Seeding three placeholder words looked exactly like a successful load.
• The 413 response uses the documented {"error": {...}} JSON shape.
• Every flow is counted before the inspection gate; requests with neither body nor query string passed through completely untelemetered.
• Redaction substitutes in one pass instead of repeated slice-assignment, so cost no longer grows with spans x text length.
• docker-compose.yml mounts terms.txt and config.yaml, and the CA volume points at /home/appuser/.mitmproxy — the image runs as appuser, so the documented /root/.mitmproxy persisted nothing and the CA was regenerated on every recreate.
• Both Dockerfile stages are pinned by digest, and the build backend is constrained, so the same commit rebuilds to the same artefact.
• CLI failures exit with distinct codes (3 mitmdump missing, 4 empty term, 5 Vault-managed terms) instead of all returning 1.

Added

• --version and a version command; the running version also appears in the /_health body. A test asserts it never drifts from pyproject.toml.
• dlp.ml_workers and dlp.ml_queue_maxsize: the pool size and queue depth were literals in the source.
• Metrics for the quiet failures: dlp_term_reload_failures_total, dlp_terms_last_reload_success_timestamp_seconds, dlp_term_poller_alive, dlp_ml_workers_alive, dlp_ml_worker_restarts_total, dlp_flows_seen_total.
• /_health consults real state: terms loaded and fresh, circuit breaker, worker liveness, poller liveness.
• The request correlation id reaches the engine, so an ML failure names the request that caused it.
• DLPEngine(dlp_config=...) takes its configuration by parameter instead of reading the global singleton, and metrics startup moved out of the addon constructor into build_addon().

Documentation

• architecture.md no longer claims the two extraction passes run in parallel; they are sequential within a request, and it says why that does not cost accuracy.
• The published image bundles only en_core_web_sm; the config reference now says so next to the nlp_model options.
• /_health is documented, and the availability consequences of the single-instance reference deployment are stated.

[3.1.0] - 2026-09-07

Fixed

• terms.txt is now written atomically. aidlp add-term appended in place, so a kill between open() and the buffered write reaching disk could leave anything from no change at all to a truncated trailing line — which the next start loaded as a redaction keyword. The CLI now snapshots the current file to terms.txt.bak, writes the full contents to a temp file in the same directory, fsyncs it, and renames it into place, fsyncing the directory so the rename itself is durable. It also no longer writes a leading blank line.
• The file provider is polled too. start_workers() created the reload task only when the secrets provider was vault, so after aidlp add-term a running proxy kept redacting from its original in-memory keyword set indefinitely — while the CLI told the operator to "wait for hot-reload". Every provider now gets a poller. The file provider is checked by mtime and size, so an untouched file costs a stat rather than a rebuilt keyword set.
• Loaded terms are validated. Every non-blank line became a keyword verbatim, so a corrupted entry surfaced later as wrong redaction behaviour instead of a load-time error. Entries over 512 characters, or containing control characters, are now skipped with a warning naming the position and reason — never the term itself, since these are secrets. A terms.txt that is not valid UTF-8 is treated as a failed fetch, so the engine keeps its last known-good keywords.

Added

• terms.txt.bak, written before every mutating write, with the restore procedure documented in the configuration reference and the deployment guide and covered by a test. There was previously no way, coded or documented, to recover a damaged terms file.
• dlp.reload_interval (default 60.0): how often the poller re-reads the term source. Previously hardcoded to 60 seconds inside the Vault-only poller.

[3.0.0] - 2026-09-07

⚠️ BREAKING: the proxy no longer relays for anonymous callers

DLPAddon.request forwarded every request once DLP redaction had run, with nothing establishing who the caller was. Combined with a default bind of 0.0.0.0, any host that could reach the port could use the proxy to fetch whatever upstream a request named, and could read the unauthenticated Prometheus endpoint. An open relay is a poor thing for a data-loss-prevention appliance to be.

• Added proxy.auth_token. When set, callers must present Proxy-Authorization: Bearer <token> or Basic base64(anyuser:<token>); anything else gets 407 with a Proxy-Authenticate challenge. The comparison is constant-time, and the header is stripped before the request is forwarded so the secret never reaches the upstream.
• CONNECT is authorised in http_connect, before the tunnel exists — checking only in request() would let an unauthenticated caller open it first.
• proxy.host now defaults to 127.0.0.1 instead of 0.0.0.0.
• Binding a routable interface with no auth_token is refused. The refusal is logged at CRITICAL, which mitmproxy treats as fatal during startup, so the process exits rather than listening at all; verified end to end. If the addon is driven some other way, every request is answered 403 instead of relayed. On loopback without a token it runs normally, with a warning.
• Added proxy.metrics_host (default 127.0.0.1). The Prometheus listener had no addr at all, so it bound every interface; it is deliberately a separate knob, so exposing the proxy does not silently expose its metrics.
• /_health remains reachable without credentials, for container health checks.

docker-compose.yml published 8080:8080, 9090:9090, 9091:9090 and 3000:3000 to every host interface, which bypassed proxy.host entirely. All four are now bound to 127.0.0.1. Inside the container the proxy still binds 0.0.0.0 — it has to, both for port publishing and for Prometheus to scrape dlp-proxy:9090 — which is why the loopback publishing is what actually contains it.

The bundled Grafana shipped GF_SECURITY_ADMIN_PASSWORD=admin. That default is gone; both it and AIDLP_PROXY_AUTH_TOKEN are now declared with :? so compose refuses to start rather than fall back to a credential committed in the repo. See the new .env.example.

Migration

• Running on loopback for local development: nothing to do, beyond a warning.
• Exposing the proxy to anything else: set proxy.auth_token (AIDLP_PROXY__AUTH_TOKEN, e.g. openssl rand -hex 32) and have callers send the Proxy-Authorization header. Without it the proxy will refuse to relay.
• Using docker compose: copy .env.example to .env and fill in both secrets. docker compose up now fails fast if either is missing.
• A shared secret is a floor, not per-caller identity. Beyond a single trusted host, put an authenticating reverse proxy in front.

[2.1.0] - 2026-08-13

Changed (behavioural)

• Environment variables now take precedence over config.yaml. They always should have — the README and this reference both said so — but the file was loaded as constructor arguments, and those outrank every other source in pydantic-settings. The file silently won.

  This was quiet and it mattered. An AIDLP_PROXY__UPSTREAM_INSECURE=false set to harden a deployment could be undone by a leftover upstream_insecure: true in a file, re-disabling the certificate verification that 2.0.0 had just made the default. An AIDLP_DLP__ML_ENABLED=true could likewise be overridden into turning ML redaction off entirely, leaving static term matching as the only protection with nothing reporting the downgrade.

  Precedence is now, highest first: environment → config.yaml → defaults. Sources merge key by key, so one variable no longer discards the rest of a section.

Added

• Startup logs a warning naming every config.yaml key that an environment variable overrides, so the conflict is visible instead of silent.

Migration

If you run with both a config.yaml and AIDLP_* variables setting the same keys, the effective configuration changes with this release. Check the startup warning to see exactly which keys are affected, and confirm the values are the ones you intend — particularly proxy.upstream_insecure and dlp.ml_enabled.

[2.0.0] - 2026-08-13

Removed (BREAKING)

• proxy.ssl_bump and --ssl-bump are deprecated and inert. Through 1.x this setting defaulted to true and its only effect was disabling verification of the upstream server's TLS certificate — despite the name, and despite the documentation describing it as "Enables HTTPS interception". Every stock deployment therefore accepted any certificate the upstream presented, so forwarded prompts could be intercepted and altered in transit.
• upstream.default_scheme, which no code ever read.

Added

• proxy.upstream_insecure (default false) and --upstream-insecure: the explicit, and now only, way to skip upstream certificate verification. While enabled, both the CLI and the mitmproxy addon warn on every startup.
• Query-string values are redacted, on every HTTP method.
• The Docker image is built on every pull request, not only on release tags.

Fixed

• aidlp start could not start: it passed --ssl-version-client and --ssl-version-server, removed from mitmproxy years ago.
• A Vault outage silently emptied the static term list, forwarding secrets in the clear; the term provider now keeps the last known good list.
• Content-Type: application/octet-stream, or no header at all, bypassed body inspection entirely.
• A disconnecting client could permanently kill the ML workers, after which every request hung instead of failing closed.
• The Docker image could not build, and CI installed a dependency set that contradicted pyproject.toml.

Migration

If you reach an upstream through a private CA or a self-signed certificate and change nothing, connections will now fail with a certificate error. Either trust the CA on the host, or opt back in explicitly:

proxy:
  upstream_insecure: true   # accepts ANY upstream certificate

HTTPS interception towards clients is unaffected.

[1.0.0] - 2026-04-28

Added

• Enterprise-grade AI DLP proxy architecture.
• Asynchronous ML worker queue for non-blocking HTTP processing.
• Parallel text redaction combining Presidio and FlashText.
• JSON-aware recursive redaction to preserve API payload structures.
• Multi-architecture Docker builds (linux/amd64 and linux/arm64).
• Pinned GitHub Actions SHAs for supply chain security.
• Comprehensive pydantic-settings based configuration system.
• Hot-reloading of Vault/local terms via background tasks.
• Improved CI pipeline with strict flake8 linting and pytest-cov gating.
• Automated dependency management via Dependabot.