backhaul-agent: public containerized DB-backup client

One docker run on a database's private network starts backing it up:
- backhaul-agent.py: stdlib-only sidecar — every-minute check-in, app-owned
  dump schedule, streams pg_dump/mysqldump/mongodump home chunked, self-updates.
- entrypoint.sh: bootstraps the newest script, installs the once-a-minute cron
  (sources persisted env), runs once immediately, hands off to cron; logs to a
  file surfaced via docker logs.
- Dockerfile: debian-slim + pg_dump 17 (PGDG — required for Postgres 17 servers)
  + mysql client. Mongo tools to follow.
- CI builds + pushes the public image and asserts the dump tools are present.

.gitea/workflows/build.yml

@@ -0,0 +1,61 @@
+# Builds the PUBLIC backhaul-agent image and pushes it to the Gitea registry.
+# No deploy step — this image is pulled by `docker run` on each database's host.
+name: Build and Push backhaul-agent
+
+on:
+  push:
+    branches:
+      - main
+  workflow_dispatch:
+
+concurrency:
+  group: ${{ github.workflow }}-${{ github.ref }}
+  cancel-in-progress: true
+
+jobs:
+  build:
+    runs-on: ubuntu-latest
+    steps:
+      - name: Checkout
+        uses: actions/checkout@v4
+        with:
+          fetch-depth: 1
+
+      - name: Login to Gitea registry
+        run: |
+          echo "${REGISTRY_TOKEN}" | docker login git.skui.io -u "${REGISTRY_USER}" --password-stdin
+        env:
+          REGISTRY_USER: ${{ secrets.REGISTRY_USER }}
+          REGISTRY_TOKEN: ${{ secrets.REGISTRY_TOKEN }}
+
+      - name: Set version tag
+        id: version
+        run: |
+          CURRENT_DATE="$(date -u +%Y.%m.%d)"
+          docker pull git.skui.io/steffen/backhaul-agent:latest 2>/dev/null || true
+          LATEST_TAGS=$(docker image inspect git.skui.io/steffen/backhaul-agent:latest 2>/dev/null | \
+            jq -r '.[0].RepoTags[]?' 2>/dev/null || echo "")
+          LATEST_TAG=$(echo "$LATEST_TAGS" | grep -E "${CURRENT_DATE}\.[0-9]+$" | \
+            sed 's/.*://' | sort -V | tail -1 || echo "")
+          if [[ $LATEST_TAG =~ ^${CURRENT_DATE}\.([0-9]+)$ ]]; then
+            COUNTER=$(( BASH_REMATCH[1] + 1 ))
+          else
+            COUNTER=1
+          fi
+          echo "version=${CURRENT_DATE}.${COUNTER}" >> "$GITHUB_OUTPUT"
+
+      - name: Build image
+        run: |
+          docker build --pull -t git.skui.io/steffen/backhaul-agent:${{ steps.version.outputs.version }} .
+          docker tag git.skui.io/steffen/backhaul-agent:${{ steps.version.outputs.version }} git.skui.io/steffen/backhaul-agent:latest
+
+      - name: Verify the dump tools are present
+        run: |
+          IMG=git.skui.io/steffen/backhaul-agent:${{ steps.version.outputs.version }}
+          docker run --rm --entrypoint sh "$IMG" -c \
+            'pg_dump --version && mysqldump --version && python3 --version'
+
+      - name: Push image
+        run: |
+          docker push git.skui.io/steffen/backhaul-agent:${{ steps.version.outputs.version }}
+          docker push git.skui.io/steffen/backhaul-agent:latest

.gitignore

@@ -0,0 +1,2 @@
+__pycache__/
+*.pyc

Dockerfile

@@ -0,0 +1,26 @@
+# backhaul-agent — the public, containerized DB-backup client. Runs as a sidecar on a
+# database's private Docker network: reaches the DB internally, pushes dumps home over HTTPS.
+# No inbound ports. The agent logic is pure stdlib Python; this image just adds the DB dump
+# tools and a once-a-minute cron.
+FROM debian:bookworm-slim
+
+# postgresql-client-17 from PGDG (Debian's default is 15, and pg_dump refuses a newer server
+# major — nc-db and most modern Postgres are 17). default-mysql-client covers MySQL/MariaDB.
+RUN set -eux; \
+    apt-get update; \
+    apt-get install -y --no-install-recommends \
+        ca-certificates curl gnupg python3 cron default-mysql-client; \
+    install -d /usr/share/postgresql-common/pgdg; \
+    curl -fsSL https://www.postgresql.org/media/keys/ACCC4CF8.asc \
+        -o /usr/share/postgresql-common/pgdg/apt.postgresql.org.asc; \
+    echo "deb [signed-by=/usr/share/postgresql-common/pgdg/apt.postgresql.org.asc] http://apt.postgresql.org/pub/repos/apt bookworm-pgdg main" \
+        > /etc/apt/sources.list.d/pgdg.list; \
+    apt-get update; \
+    apt-get install -y --no-install-recommends postgresql-client-17; \
+    rm -rf /var/lib/apt/lists/*
+
+WORKDIR /app
+COPY backhaul-agent.py /app/backhaul-agent.py
+COPY entrypoint.sh /entrypoint.sh
+RUN chmod +x /entrypoint.sh
+ENTRYPOINT ["/entrypoint.sh"]

README.md

@@ -0,0 +1,50 @@
+# backhaul-agent
+
+The public, containerized client for [backhaul](https://backhaul.skui.io) — a database-backup
+dashboard with proven restore. Run this as a **sidecar on your database's private Docker
+network**: it reaches the database internally and pushes dumps home over HTTPS. **No inbound
+ports are ever opened** — the agent only makes outbound calls.
+
+## How it works
+
+- Once a minute (in-container cron) the agent **checks in**: it heartbeats, reports the
+  databases it can see, delivers any finished dump, and asks what to do.
+- The **app owns the schedule**. When a dump is due, the check-in reply says so; the agent
+  runs `pg_dump`/`mysqldump`/`mongodump` and **streams the result home chunked** — never
+  buffering the whole dump in memory or on disk.
+- The agent **self-updates**: the app advertises the current script version and the agent
+  swaps its own script atomically (after a compile check). Disable with
+  `BACKHAUL_AUTOUPDATE=0`.
+
+Config pulls, data pushes.
+
+## Run it
+
+```bash
+docker run -d --name backhaul-agent --restart unless-stopped \
+  --network <DB_DOCKER_NETWORK> \
+  -e BACKHAUL_URL=https://backhaul.skui.io \
+  -e BACKHAUL_TOKEN=<token from the backhaul admin> \
+  -e BACKHAUL_TARGETS=postgres:<DB_HOST>:5432:<DB_NAME>:<USER>:<PASS> \
+  git.skui.io/steffen/backhaul-agent:latest
+```
+
+- `--network` must be the **same private Docker network as the database**, so the agent can
+  reach it by service name.
+- `BACKHAUL_TARGETS` is a comma-separated list, each entry
+  `engine:host:port:db:user:pass`. Engines: `postgres`, `mysql`, `mariadb`, `mongo`.
+
+The agent registers each target with the app on its first check-in; tune the dump cadence,
+retention, and restore-drill schedule from the backhaul dashboard.
+
+## What's in the image
+
+Debian slim + Python 3 (stdlib only — the agent has no Python dependencies) +
+`postgresql-client-17` (pg_dump 17, required to dump Postgres 17 servers) +
+`default-mysql-client`. MongoDB tools are added in a follow-up.
+
+## Logs
+
+```bash
+docker logs -f backhaul-agent          # check-in activity
+```

backhaul-agent.py

@@ -0,0 +1,177 @@
+#!/usr/bin/env python3
+"""backhaul agent — the containerized DB-backup client. Runs as a sidecar on a database's
+private Docker network: it reaches the DB internally and calls home outbound over HTTPS.
+No inbound ports. Stdlib only; the DB client tools (pg_dump/mysqldump/mongodump) come from
+the image.
+
+Every minute (in-container cron) it CHECKS IN: heartbeats, reports the databases it can see,
+delivers any pending dump result, and asks what to do. The app owns the schedule and replies
+with a dump command when one is due; the agent then dumps and streams it home. The app also
+advertises the current agent-script version, and the agent self-updates (atomic swap after a
+compile check) — same mechanism as snapshoot.
+
+Config (env, from the `docker run`):
+  BACKHAUL_URL     e.g. https://backhaul.skui.io
+  BACKHAUL_TOKEN   agent token from the backhaul admin
+  BACKHAUL_TARGETS comma-separated DB targets, each: engine:host:port:db:user:pass
+                   e.g. postgres:db:5432:nextcloud:nextcloud:nextcloud
+"""
+import json
+import os
+import subprocess
+import sys
+import urllib.request
+
+VERSION = "1.0"
+RESULT_FILE = os.path.join(os.path.dirname(os.path.abspath(__file__)), ".dump-result.json")
+
+
+def targets():
+    """Parse BACKHAUL_TARGETS into dicts. Each: engine:host:port:db:user:pass."""
+    out = []
+    for spec in os.environ.get("BACKHAUL_TARGETS", "").split(","):
+        spec = spec.strip()
+        if not spec:
+            continue
+        parts = spec.split(":")
+        if len(parts) < 6:
+            print(f"skipping malformed target: {spec}", file=sys.stderr)
+            continue
+        engine, host, port, db, user, pw = parts[:6]
+        out.append({"engine": engine, "host": host, "port": port,
+                    "db": db, "user": user, "pw": pw,
+                    "name": f"{engine}/{db}"})
+    return out
+
+
+def db_version(t):
+    """Fetch the engine version string so the app can pick a matching restore image."""
+    try:
+        if t["engine"] == "postgres":
+            env = dict(os.environ, PGPASSWORD=t["pw"])
+            out = subprocess.run(("psql", "-h", t["host"], "-p", t["port"], "-U", t["user"],
+                                  "-d", t["db"], "-tAc", "select version()"),
+                                 capture_output=True, text=True, timeout=30, env=env).stdout
+            return out.strip().split(" on ")[0][:60]
+        if t["engine"] in ("mysql", "mariadb"):
+            out = subprocess.run(("mysql", "-h", t["host"], "-P", t["port"], "-u", t["user"],
+                                  f"-p{t['pw']}", "-N", "-e", "select version()"),
+                                 capture_output=True, text=True, timeout=30).stdout
+            return "MySQL/MariaDB " + out.strip()[:40]
+        if t["engine"] == "mongo":
+            return "MongoDB"
+    except Exception:
+        pass
+    return t["engine"]
+
+
+def dump_cmd(t):
+    """The dump command + environment for an engine. Streams compressed to stdout."""
+    if t["engine"] == "postgres":
+        env = dict(os.environ, PGPASSWORD=t["pw"])
+        # custom format is already compressed and is what pg_restore wants
+        return (("pg_dump", "-h", t["host"], "-p", t["port"], "-U", t["user"],
+                 "-d", t["db"], "-Fc"), env)
+    if t["engine"] in ("mysql", "mariadb"):
+        return (("sh", "-c",
+                 f"mysqldump -h {t['host']} -P {t['port']} -u {t['user']} "
+                 f"-p{t['pw']} --single-transaction --routines {t['db']} | gzip"),
+                dict(os.environ))
+    if t["engine"] == "mongo":
+        return (("mongodump", "--host", t["host"], "--port", t["port"],
+                 "--db", t["db"], "--archive", "--gzip"), dict(os.environ))
+    raise ValueError(f"unknown engine {t['engine']}")
+
+
+def run_dump(t, url, token):
+    """Stream a dump of one target straight to the app's ingest endpoint — never buffer the
+    whole dump in memory or a temp file."""
+    cmd, env = dump_cmd(t)
+    meta = json.dumps({"engine": t["engine"], "version": db_version(t)})
+    req = urllib.request.Request(
+        url.rstrip("/") + "/api/dump",
+        headers={"X-Agent-Token": token, "X-DB-Name": t["name"],
+                 "X-DB-Meta": meta, "Content-Type": "application/octet-stream",
+                 "Transfer-Encoding": "chunked", "User-Agent": f"backhaul-agent/{VERSION}"})
+    proc = subprocess.Popen(cmd, stdout=subprocess.PIPE, env=env)
+    # feed the dump process's stdout straight into the HTTP request body
+    req.data = iter(lambda: proc.stdout.read(1 << 20), b"")
+    with urllib.request.urlopen(req, timeout=3600) as resp:
+        proc.wait()
+        return {"name": t["name"], "ok": resp.status == 200 and proc.returncode == 0}
+
+
+def maybe_self_update(answer, url, token):
+    """Swap our own script when the app advertises a newer version — atomic, compile-checked.
+    Disable with BACKHAUL_AUTOUPDATE=0."""
+    latest = answer.get("agent_latest")
+    if (not latest or latest == VERSION
+            or os.environ.get("BACKHAUL_AUTOUPDATE", "1") == "0"):
+        return
+    me = os.path.abspath(__file__)
+    try:
+        req = urllib.request.Request(url.rstrip("/") + "/api/agent-script",
+                                     headers={"X-Agent-Token": token})
+        with urllib.request.urlopen(req, timeout=30) as r:
+            source = r.read()
+        text = source.decode()
+        if "backhaul-agent" not in text or f'VERSION = "{latest}"' not in text:
+            return
+        compile(text, me, "exec")
+        tmp = me + ".new"
+        with open(tmp, "wb") as f:
+            f.write(source)
+        os.chmod(tmp, os.stat(me).st_mode)
+        os.replace(tmp, me)
+        print(f"self-updated {VERSION} -> {latest}")
+    except Exception as e:
+        print("self-update failed:", e, file=sys.stderr)
+
+
+def checkin(url, token, tgts, pending):
+    """POST the heartbeat: which DBs we see + any finished dump result. The reply tells us
+    what to do next and the latest agent version."""
+    body = json.dumps({"agent_version": VERSION,
+                       "databases": [{"name": t["name"], "engine": t["engine"]} for t in tgts],
+                       "results": pending}).encode()
+    req = urllib.request.Request(url.rstrip("/") + "/api/checkin", data=body,
+                                 headers={"X-Agent-Token": token,
+                                          "Content-Type": "application/json",
+                                          "User-Agent": f"backhaul-agent/{VERSION}"})
+    with urllib.request.urlopen(req, timeout=30) as r:
+        return json.loads(r.read() or b"{}")
+
+
+def main():
+    url = os.environ.get("BACKHAUL_URL", "").rstrip("/")
+    token = os.environ.get("BACKHAUL_TOKEN", "")
+    if not (url and token):
+        sys.exit("BACKHAUL_URL and BACKHAUL_TOKEN must be set")
+    tgts = targets()
+
+    pending = []
+    if os.path.exists(RESULT_FILE):
+        try:
+            pending = json.load(open(RESULT_FILE))
+        except ValueError:
+            pending = []
+
+    answer = checkin(url, token, tgts, pending)
+    if pending:
+        os.remove(RESULT_FILE)
+
+    # run any dump the app asked for (matched by name), record result for next check-in
+    results = []
+    for want in answer.get("dump") or []:
+        t = next((t for t in tgts if t["name"] == want), None)
+        if t:
+            print(f"dumping {t['name']}...", flush=True)
+            results.append(run_dump(t, url, token))
+    if results:
+        json.dump(results, open(RESULT_FILE, "w"))
+
+    maybe_self_update(answer, url, token)
+
+
+if __name__ == "__main__":
+    main()

entrypoint.sh

@@ -0,0 +1,45 @@
+#!/bin/sh
+# backhaul-agent container entrypoint. Bootstraps the newest agent script, installs a
+# once-a-minute cron job (heartbeat + poll for work + self-update), runs once immediately,
+# then hands off to cron. All config comes from env passed to `docker run`.
+set -eu
+: "${BACKHAUL_URL:?set BACKHAUL_URL (e.g. https://backhaul.skui.io)}"
+: "${BACKHAUL_TOKEN:?set BACKHAUL_TOKEN (from the backhaul admin)}"
+
+APP=/app/backhaul-agent.py
+LOG=/var/log/backhaul-agent.log
+touch "$LOG"
+
+# Cron jobs run with a bare environment, so persist the vars the agent needs to a file
+# the cron command sources. Single-quoted to survive spaces/specials in values.
+{
+  printf "export BACKHAUL_URL='%s'\n" "$BACKHAUL_URL"
+  printf "export BACKHAUL_TOKEN='%s'\n" "$BACKHAUL_TOKEN"
+  printf "export BACKHAUL_TARGETS='%s'\n" "${BACKHAUL_TARGETS:-}"
+  printf "export BACKHAUL_AUTOUPDATE='%s'\n" "${BACKHAUL_AUTOUPDATE:-1}"
+} > /app/env
+chmod 600 /app/env
+
+# Bootstrap self-update: pull the current script from the app so a fresh container is already
+# up to date. Compile-check and sanity-check before replacing; keep the bundled copy on any
+# failure. Disable with BACKHAUL_AUTOUPDATE=0.
+if [ "${BACKHAUL_AUTOUPDATE:-1}" != "0" ]; then
+  if curl -fsS -H "X-Agent-Token: $BACKHAUL_TOKEN" "$BACKHAUL_URL/api/agent-script" \
+       -o /tmp/agent.new 2>/dev/null \
+     && python3 -c "compile(open('/tmp/agent.new').read(),'a','exec')" 2>/dev/null \
+     && grep -q backhaul-agent /tmp/agent.new; then
+    cp /tmp/agent.new "$APP"
+    echo "bootstrap: fetched latest agent script"
+  fi
+fi
+
+# Every-minute cron job (system crontab format needs the user field).
+echo "* * * * * root . /app/env; /usr/bin/python3 $APP >> $LOG 2>&1" > /etc/cron.d/backhaul
+chmod 0644 /etc/cron.d/backhaul
+
+# Run once now so the first check-in doesn't wait a full minute.
+( . /app/env; /usr/bin/python3 "$APP" >> "$LOG" 2>&1 ) || true
+
+echo "backhaul-agent up — check-in every minute; logs at $LOG"
+cron                    # daemonize the scheduler
+exec tail -f "$LOG"     # keep PID 1 alive and surface activity in `docker logs`