AHDX/docs/plan.md

AHDX build plan

AHDX is a small self-hosted app for pulling your own Apple Health data out of Apple's export and into a SQLite database you control. It runs in Docker, works on Windows through Docker Desktop, and has a web page for loading the export and looking through it. Everything stays inside the container. Nothing is sent anywhere.

AHDX is short for Apple Health Data eXporter.

Why it exists

You're in the EU, so the health data Apple holds about you is yours to take and keep. Apple will export it, but what you get back is one enormous XML file that's awkward to use. This app turns that file into an ordinary SQLite database and gives you a browser page to import and query it. No account, no cloud, no telemetry.

What Apple actually gives you

On the iPhone: Health app, tap your profile picture, "Export All Health Data". You get export.zip. Inside it:

• apple_health_export/export.xml is the main file and the only one we need to start. It holds <Record> entries (heart rate, steps, weight, sleep, etc.), <Workout> entries, and <ActivitySummary> rows (one per day). This file gets big: real exports run from tens of megabytes to a few gigabytes.
• workout-routes/*.gpx are GPS tracks for outdoor workouts.
• electrocardiograms/*.csv are ECG readings.
• export_cda.xml is a clinical-format copy of the same data. We skip it.

The size is the one real problem to solve. We can't load the whole XML into memory, so we stream it.

Stack

Same shape as the other self-hosted apps here, because it works and there's little to break:

• Python + Flask, server-rendered pages, one CSS file. No frontend framework.
• SQLite via the stdlib sqlite3. No ORM.
• Docker + docker-compose. One service, one volume for the databases and uploads.
• XML parsing with xml.etree.ElementTree.iterparse. It reads the file element by element and lets us throw each one away after we've stored it, so memory stays flat no matter how large the export is.

If we add charts, they get drawn in the browser from data the page already has, with the chart library shipped inside the image. No CDN, so it still works with the machine offline.

Data model, first cut

Three tables cover nearly everything in export.xml:

• records: one row per measurement. Columns: type, source_name, source_version, device, unit, value, value_num, start_date, end_date, creation_date. Index on (type, start_date).
• workouts: activity_type, duration, duration_unit, total_distance, distance_unit, total_energy, energy_unit, start_date, end_date, source_name.
• activity_summary: date, active_energy_burned, active_energy_goal, move_time, exercise_time, stand_hours.

value keeps the raw text from the file. value_num holds the same thing as a REAL when it parses as a number, so numeric types (weight, heart rate) chart cleanly while text types (like sleep state) still survive the round trip.

Later, if you want them: workout GPS points, ECG samples, and the metadata key/value pairs Apple attaches to some records.

The "add a database" part

You said standard SQLite, with a way to add databases. Here's what I'd build. Tell me if you meant something different.

The volume has a databases/ folder. Each health database is one .db file in there. A small registry.db tracks the list and which one is active. The UI gives you:

• a picker to switch the active database,
• "New database": name it, the app creates an empty .db and switches to it,
• import always writes into whichever database is active.

That handles the cases I'd expect: one database per person in the house, a fresh one each year, or a scratch database to test an import before you trust it.

Pages

• / import. Drop export.zip or export.xml. It parses into the active database and shows a live count while it runs.
• /dashboard what's in here: record count per type, the date range covered, number of workouts, last import time.
• /browse pick a type and a date range, read the rows, download them as CSV.
• /databases list, switch, create, delete.
• a status endpoint for Docker's health check.

Import, the tricky bit

A big export takes a while to parse, so import can't block a request until it finishes. I'd run the parse in a background thread and write progress (rows seen, current record type) to a status row that the import page polls. That's enough for a single-user local app; a real job queue would be overkill.

Inserts go in batches, committing every few thousand rows, and each XML element gets cleared right after we read it.

Re-importing is a merge, not a reload. Every export holds your whole history, so loading a newer one shouldn't duplicate the old rows. Apple gives records no stable ID, so we make our own key: a UNIQUE index on (type, start_date, end_date, value, source_name) with INSERT OR IGNORE. Load the same export twice and nothing changes; load a fresher one and only the new rows land. Workouts key on (activity_type, start_date, end_date, source_name), activity summaries on the date. This is what keeps "the latest" flowing into the database without a wipe.

Keeping it current (the interval scan)

You asked for a periodic scan that pulls recent data, free, with nothing behind a paid tier. The limit to be honest about: Apple has no free server-side API. A container on your PC can't reach into the phone and pull Health data by itself. The full export.xml is a manual export from the Health app. So the "interval scan" watches for exports rather than fetching them, and there's a push path for the automatable part.

Two free ways to feed it, both landing in the same merge:

1. Watched inbox. AHDX checks data/inbox/ on a timer (default every 30 minutes, set by AHDX_SCAN_INTERVAL). Drop an export.zip or export.xml in there, or point a synced folder at it, and the container ingests it on its own and moves the file to data/inbox/done/. No clicking.
2. Push endpoint for iOS Shortcuts. A free Shortcut automation on the phone can read recent metrics (steps, heart rate, weight, sleep) on a schedule and POST them as JSON to /ingest. Built into iOS, no third-party app, no paid tier. It won't carry full history the way the manual export does, but it keeps recent data arriving hands-free.

What we can't do for free: a fully automatic full-history export. Apple only does that by hand. Automatic recent data is fine through the Shortcut push. The README will spell out both, with the exact Shortcut steps.

Build order

1. Skeleton. Dockerfile, compose, a Flask app that boots, empty pages, status endpoint. Confirm it runs on Windows via Docker Desktop and the page loads at http://localhost:PORT.
2. Parser and import. Streaming XML into the three tables, background parse with progress. This is the core. Get it solid against a real export.
3. Browse and dashboard. Counts, filtering, CSV export.
4. Multiple databases. Registry, switch, create, delete.
5. Release polish. A README with the export steps and screenshots, a license, a small sample export.xml so people can try it, and a plain statement that the app makes no network calls.

Open-source prep (later)

No git repo yet, per your note. When we set one up:

• License: MIT if you want the widest reuse, AGPL if you want anyone who runs a modified hosted copy to publish their changes. Your call.
• README that leads with "your data, your machine" and the export steps.
• No analytics and no outbound requests, stated plainly, since this is health data.

What I need from you

1. Does AHDX stand for Apple Health Data eXport, or something else?
2. Import scope to start: just export.xml, or accept the whole export.zip and pull the XML out ourselves? The zip is friendlier for you.
3. "Add a database": is the registry idea above what you meant, or did you mean attaching an existing external .db file, or room for other engines (Postgres) down the line?
4. A port preference. I'd default to 8080.
5. If you can drop a real export.xml (or a trimmed chunk of one) into the project, I'll build the parser against actual data. Apple's XML has quirks that only show up in a real file.