mirror of
https://github.com/Termix-SSH/Termix.git
synced 2026-08-29 18:31:33 +00:00
Groundwork for Postgres and MySQL backends (#1134)
* groundwork for postgres and mysql backends #1127 made the repository layer dialect-agnostic. This adds the pieces needed to actually target a second engine, as a foundation only — nothing is wired up and sqlite remains the sole runtime path. - DatabaseDialect covers sqlite, postgres and mysql, resolved from DATABASE_DIALECT and defaulting to sqlite so nothing changes for existing deployments or the desktop build - a column kit holding the per-dialect type choices in one file: booleans are integers on sqlite and native elsewhere, autoincrement differs three ways, and MySQL cannot index unbounded TEXT so key columns need varchar - settings and users declared for all three dialects as a proof slice, chosen because between them they use every construct the real schema does - pg and mysql2 added as dependencies The tests build real queries for all three engines without a server, asserting identifier quoting, placeholder style and boolean storage, so the property the repositories depend on is verified rather than assumed. * verify foreign keys and unique constraints port across dialects The first slice only covered plain columns. The real schema also has 92 foreign keys (80 cascade, 12 set null) and 14 unique columns, so the approach is only viable if those survive the port. Adds audit_logs and ssh_folders to the proof slice: one nullable reference with ON DELETE SET NULL, one required reference with ON DELETE CASCADE, a unique column, and an autoincrement surrogate key — which is spelled three different ways underneath (integer primary key autoincrement, serial, int auto_increment). All of it holds. Worth noting for whoever picks this up: getTableConfig is dialect-specific and silently fails on a table from another dialect, so the test uses each engine's own. * generate the postgres and mysql schemas instead of hand-writing them The proof slice showed the constructs port, but left the maintenance question open. Three hand-written copies of 52 tables is the wrong answer: with foreign keys the copies cross-reference each other, so a renamed table has to land in three places consistently or a key silently points at the wrong one. The mapping is mechanical, so a script does it. schema.ts stays the single source of truth and schema.pg.ts / schema.mysql.ts are derived, covering all 52 tables — the column kit and the two-table portable slice are gone, since the generator now holds those decisions. The transforms are the ones the kit enumerated: integer-backed booleans become native, autoincrement keys become serial or int auto_increment, real becomes double precision or double, and any column that is a primary key, is unique, or sits on either end of a foreign key becomes varchar because MySQL cannot index unbounded TEXT. > termix@2.6.0 lint > node scripts/generate-dialect-schema.cjs --check && eslint . /mnt/c/Users/29037/WebstormProjects/Termix/src/backend/database/routes/homepage-favicon-routes.ts 99:12 warning 'err' is defined but never used unused-imports/no-unused-vars /mnt/c/Users/29037/WebstormProjects/Termix/src/backend/database/routes/homepage-ping-routes.ts 123:12 warning 'err' is defined but never used unused-imports/no-unused-vars /mnt/c/Users/29037/WebstormProjects/Termix/src/backend/database/routes/homepage-rss-routes.ts 144:12 warning 'err' is defined but never used unused-imports/no-unused-vars /mnt/c/Users/29037/WebstormProjects/Termix/src/backend/database/routes/session-log-routes.ts 46:16 warning 'canAccessRecording' is defined but never used. Allowed unused vars must match /^_/u unused-imports/no-unused-vars /mnt/c/Users/29037/WebstormProjects/Termix/src/backend/hosts/vault-signer-core.ts 55:12 warning Unexpected any. Specify a different type @typescript-eslint/no-explicit-any 75:13 warning Unexpected any. Specify a different type @typescript-eslint/no-explicit-any /mnt/c/Users/29037/WebstormProjects/Termix/src/backend/tests/hosts/auth-manager.test.ts 18:73 warning Unexpected any. Specify a different type @typescript-eslint/no-explicit-any /mnt/c/Users/29037/WebstormProjects/Termix/src/backend/tests/utils/shared-host-secrets-manager.test.ts 7:6 warning 'SecretRow' is defined but never used. Allowed unused vars must match /^_/u unused-imports/no-unused-vars /mnt/c/Users/29037/WebstormProjects/Termix/src/backend/utils/auth-manager.ts 510:13 warning 'affectedUsers' is assigned a value but never used. Allowed unused vars must match /^_/u unused-imports/no-unused-vars /mnt/c/Users/29037/WebstormProjects/Termix/src/backend/utils/notification-sender.ts 48:12 warning 'firstErr' is defined but never used unused-imports/no-unused-vars /mnt/c/Users/29037/WebstormProjects/Termix/src/ui/api/ssh-file-operations-api.ts 35:10 warning 'buildFileManagerUrl' is defined but never used. Allowed unused vars must match /^_/u unused-imports/no-unused-vars /mnt/c/Users/29037/WebstormProjects/Termix/src/ui/components/folder-style.tsx 61:14 warning Fast refresh only works when a file only exports components. Use a new file to share constants or functions between components react-refresh/only-export-components 116:14 warning Fast refresh only works when a file only exports components. Use a new file to share constants or functions between components react-refresh/only-export-components 121:14 warning Fast refresh only works when a file only exports components. Use a new file to share constants or functions between components react-refresh/only-export-components 149:17 warning Fast refresh only works when a file only exports components. Use a new file to share constants or functions between components react-refresh/only-export-components /mnt/c/Users/29037/WebstormProjects/Termix/src/ui/components/proxmox/ProxmoxDiscoverDialog.tsx 109:19 warning Unexpected any. Specify a different type @typescript-eslint/no-explicit-any 190:19 warning Unexpected any. Specify a different type @typescript-eslint/no-explicit-any /mnt/c/Users/29037/WebstormProjects/Termix/src/ui/features/homepage/HomepageCanvas.tsx 345:15 warning Empty block statement no-empty 388:15 warning Empty block statement no-empty 415:15 warning Empty block statement no-empty /mnt/c/Users/29037/WebstormProjects/Termix/src/ui/features/homepage/dialogs/SingleHostEditForm.tsx 24:6 warning React Hook useEffect has a missing dependency: 'filter'. Either include it or remove the dependency array. If 'setHosts' needs the current value of 'filter', you can also switch to useReducer instead of useState and read 'filter' in the reducer react-hooks/exhaustive-deps /mnt/c/Users/29037/WebstormProjects/Termix/src/ui/features/homepage/widgets/AlertFeedWidget.tsx 93:6 warning React Hook useEffect has a missing dependency: 'fetchData'. Either include it or remove the dependency array react-hooks/exhaustive-deps /mnt/c/Users/29037/WebstormProjects/Termix/src/ui/features/homepage/widgets/CustomApiWidget.tsx 77:6 warning React Hook useEffect has a missing dependency: 'fetchData'. Either include it or remove the dependency array react-hooks/exhaustive-deps /mnt/c/Users/29037/WebstormProjects/Termix/src/ui/features/homepage/widgets/DockerActivityWidget.tsx 50:6 warning React Hook useEffect has a missing dependency: 'fetchData'. Either include it or remove the dependency array react-hooks/exhaustive-deps /mnt/c/Users/29037/WebstormProjects/Termix/src/ui/features/homepage/widgets/DockerWidget.tsx 16:10 warning Fast refresh only works when a file has exports. Move your component(s) to a separate file react-refresh/only-export-components /mnt/c/Users/29037/WebstormProjects/Termix/src/ui/features/homepage/widgets/FileManagerWidget.tsx 16:10 warning Fast refresh only works when a file has exports. Move your component(s) to a separate file react-refresh/only-export-components /mnt/c/Users/29037/WebstormProjects/Termix/src/ui/features/homepage/widgets/HostGridWidget.tsx 61:6 warning React Hook useCallback has a missing dependency: 'hostIds'. Either include it or remove the dependency array react-hooks/exhaustive-deps 61:7 warning React Hook useCallback has a complex expression in the dependency array. Extract it to a separate variable so it can be statically checked react-hooks/exhaustive-deps /mnt/c/Users/29037/WebstormProjects/Termix/src/ui/features/homepage/widgets/MetricsChartWidget.tsx 168:6 warning React Hook useEffect has a missing dependency: 'fetchData'. Either include it or remove the dependency array react-hooks/exhaustive-deps /mnt/c/Users/29037/WebstormProjects/Termix/src/ui/features/homepage/widgets/PingStatusWidget.tsx 79:6 warning React Hook useEffect has a missing dependency: 'fetchAll'. Either include it or remove the dependency array react-hooks/exhaustive-deps 79:7 warning React Hook useEffect has a complex expression in the dependency array. Extract it to a separate variable so it can be statically checked react-hooks/exhaustive-deps /mnt/c/Users/29037/WebstormProjects/Termix/src/ui/features/homepage/widgets/QuickConnectWidget.tsx 64:10 warning Fast refresh only works when a file has exports. Move your component(s) to a separate file react-refresh/only-export-components /mnt/c/Users/29037/WebstormProjects/Termix/src/ui/features/homepage/widgets/RecentActivityWidget.tsx 82:6 warning React Hook useEffect has a missing dependency: 'fetchData'. Either include it or remove the dependency array react-hooks/exhaustive-deps 82:17 warning React Hook useEffect has a complex expression in the dependency array. Extract it to a separate variable so it can be statically checked react-hooks/exhaustive-deps /mnt/c/Users/29037/WebstormProjects/Termix/src/ui/features/homepage/widgets/SshQuickConnectWidget.tsx 67:6 warning React Hook useCallback has a missing dependency: 'hostIds'. Either include it or remove the dependency array react-hooks/exhaustive-deps 67:7 warning React Hook useCallback has a complex expression in the dependency array. Extract it to a separate variable so it can be statically checked react-hooks/exhaustive-deps 99:17 warning 'online' is assigned a value but never used. Allowed unused vars must match /^_/u unused-imports/no-unused-vars /mnt/c/Users/29037/WebstormProjects/Termix/src/ui/features/homepage/widgets/SshTerminalWidget.tsx 17:10 warning Fast refresh only works when a file has exports. Move your component(s) to a separate file react-refresh/only-export-components /mnt/c/Users/29037/WebstormProjects/Termix/src/ui/features/homepage/widgets/SystemOverviewWidget.tsx 72:6 warning React Hook useEffect has a missing dependency: 'fetchData'. Either include it or remove the dependency array react-hooks/exhaustive-deps /mnt/c/Users/29037/WebstormProjects/Termix/src/ui/features/homepage/widgets/TunnelWidget.tsx 15:10 warning Fast refresh only works when a file has exports. Move your component(s) to a separate file react-refresh/only-export-components /mnt/c/Users/29037/WebstormProjects/Termix/src/ui/features/host-metrics/cards/CpuCard.tsx 14:10 warning 'computeChartData' is defined but never used. Allowed unused vars must match /^_/u unused-imports/no-unused-vars /mnt/c/Users/29037/WebstormProjects/Termix/src/ui/sidebar/FolderPathPicker.tsx 15:17 warning Fast refresh only works when a file only exports components. Use a new file to share constants or functions between components react-refresh/only-export-components 22:17 warning Fast refresh only works when a file only exports components. Use a new file to share constants or functions between components react-refresh/only-export-components /mnt/c/Users/29037/WebstormProjects/Termix/src/ui/sidebar/HostsPanel.tsx 601:52 warning Unexpected any. Specify a different type @typescript-eslint/no-explicit-any ✖ 44 problems (0 errors, 44 warnings) now fails if the generated files are out of date, so editing the schema without regenerating cannot reach main. * select durability behaviour per dialect, and document the backends The onWrite hook every repository receives exists to serialise the in-memory SQLite database back to its encrypted file. On a client-server engine a committed write is already durable and there is nothing to flush, so the factory now installs no hook at all rather than one that does nothing. Repositories call it as this.onWrite?.(), so none of the 43 of them change. Also adds docs/database-backends.md, mostly to be explicit about encryption, which is the part most likely to be misread. Field-level encryption is identical on all three engines and covers every credential. Whole-file encryption has no equivalent on Postgres or MySQL, so host names, snippet contents, audit entries and backups are only as protected as the storage underneath them — that is the operator's responsibility and the docs should not imply otherwise. * generate DDL with drizzle-kit, and give settings a synchronous path Two of the three remaining blockers. DDL: db/index.ts hand-writes 67 CREATE TABLE statements and 122 ADD COLUMN migrations, all in SQLite dialect. Rather than port them, drizzle-kit now generates migrations from the schema modules — 817 lines for Postgres, 869 for MySQL, with the type mapping already correct because the schemas it reads are themselves generated. > termix@2.6.0 schema:migrations > drizzle-kit generate --config=drizzle.config.pg.ts && drizzle-kit generate --config=drizzle.config.mysql.ts Reading config file '/mnt/c/Users/29037/WebstormProjects/Termix/drizzle.config.pg.ts' 52 tables alert_firings 11 columns 0 indexes 2 fks alert_rule_channels 3 columns 0 indexes 2 fks alert_rules 11 columns 0 indexes 2 fks api_keys 9 columns 0 indexes 1 fks audit_logs 13 columns 0 indexes 1 fks c2s_tunnel_presets 8 columns 0 indexes 1 fks command_history 5 columns 0 indexes 2 fks dashboard_service_links 8 columns 0 indexes 1 fks dismissed_alerts 4 columns 0 indexes 1 fks file_manager_pinned 6 columns 0 indexes 2 fks file_manager_recent 6 columns 0 indexes 2 fks file_manager_shortcuts 6 columns 0 indexes 2 fks homepage_items 9 columns 0 indexes 1 fks homepage_layouts 4 columns 0 indexes 1 fks host_access 11 columns 0 indexes 5 fks host_health_checks 7 columns 0 indexes 2 fks host_health_history 8 columns 0 indexes 2 fks host_metrics_history 8 columns 0 indexes 1 fks host_metrics_preferences 6 columns 0 indexes 2 fks ssh_data 94 columns 0 indexes 6 fks network_topology 5 columns 0 indexes 1 fks notification_channels 7 columns 0 indexes 1 fks opkssh_tokens 12 columns 0 indexes 2 fks recent_activity 6 columns 0 indexes 2 fks roles 8 columns 0 indexes 0 fks session_recordings 15 columns 0 indexes 3 fks session_share_participants 6 columns 0 indexes 2 fks session_shares 15 columns 0 indexes 3 fks sessions 11 columns 0 indexes 1 fks settings 2 columns 0 indexes 0 fks shared_host_secrets 15 columns 0 indexes 3 fks snippet_access 8 columns 0 indexes 4 fks snippet_folders 8 columns 0 indexes 1 fks snippets 11 columns 0 indexes 1 fks ssh_credential_usage 5 columns 0 indexes 3 fks ssh_credentials 21 columns 0 indexes 1 fks ssh_folders 9 columns 0 indexes 2 fks sso_providers 8 columns 0 indexes 0 fks sync_tombstones 5 columns 0 indexes 1 fks termix_identities 6 columns 0 indexes 1 fks termix_identity_ca 8 columns 0 indexes 2 fks termix_identity_keys 12 columns 0 indexes 3 fks tmux_session_tags 6 columns 0 indexes 2 fks transfer_recent 7 columns 0 indexes 3 fks trusted_devices 8 columns 0 indexes 1 fks user_open_tabs 9 columns 0 indexes 2 fks user_preferences 23 columns 0 indexes 1 fks user_roles 5 columns 0 indexes 3 fks users 20 columns 0 indexes 0 fks vault_profiles 18 columns 0 indexes 1 fks vault_tokens 8 columns 0 indexes 2 fks webauthn_credentials 12 columns 0 indexes 1 fks No schema changes, nothing to migrate 😴 Reading config file '/mnt/c/Users/29037/WebstormProjects/Termix/drizzle.config.mysql.ts' Reading schema files: /mnt/c/Users/29037/WebstormProjects/Termix/src/backend/database/db/schema.mysql.ts 52 tables alert_firings 11 columns 0 indexes 2 fks alert_rule_channels 3 columns 0 indexes 2 fks alert_rules 11 columns 0 indexes 2 fks api_keys 9 columns 0 indexes 1 fks audit_logs 13 columns 0 indexes 1 fks c2s_tunnel_presets 8 columns 0 indexes 1 fks command_history 5 columns 0 indexes 2 fks dashboard_service_links 8 columns 0 indexes 1 fks dismissed_alerts 4 columns 0 indexes 1 fks file_manager_pinned 6 columns 0 indexes 2 fks file_manager_recent 6 columns 0 indexes 2 fks file_manager_shortcuts 6 columns 0 indexes 2 fks homepage_items 9 columns 0 indexes 1 fks homepage_layouts 4 columns 0 indexes 1 fks host_access 11 columns 0 indexes 5 fks host_health_checks 7 columns 0 indexes 2 fks host_health_history 8 columns 0 indexes 2 fks host_metrics_history 8 columns 0 indexes 1 fks host_metrics_preferences 6 columns 0 indexes 2 fks ssh_data 94 columns 0 indexes 6 fks network_topology 5 columns 0 indexes 1 fks notification_channels 7 columns 0 indexes 1 fks opkssh_tokens 12 columns 0 indexes 2 fks recent_activity 6 columns 0 indexes 2 fks roles 8 columns 0 indexes 0 fks session_recordings 15 columns 0 indexes 3 fks session_share_participants 6 columns 0 indexes 2 fks session_shares 15 columns 0 indexes 3 fks sessions 11 columns 0 indexes 1 fks settings 2 columns 0 indexes 0 fks shared_host_secrets 15 columns 0 indexes 3 fks snippet_access 8 columns 0 indexes 4 fks snippet_folders 8 columns 0 indexes 1 fks snippets 11 columns 0 indexes 1 fks ssh_credential_usage 5 columns 0 indexes 3 fks ssh_credentials 21 columns 0 indexes 1 fks ssh_folders 9 columns 0 indexes 2 fks sso_providers 8 columns 0 indexes 0 fks sync_tombstones 5 columns 0 indexes 1 fks termix_identities 6 columns 0 indexes 1 fks termix_identity_ca 8 columns 0 indexes 2 fks termix_identity_keys 12 columns 0 indexes 3 fks tmux_session_tags 6 columns 0 indexes 2 fks transfer_recent 7 columns 0 indexes 3 fks trusted_devices 8 columns 0 indexes 1 fks user_open_tabs 9 columns 0 indexes 2 fks user_preferences 23 columns 0 indexes 1 fks user_roles 5 columns 0 indexes 3 fks users 20 columns 0 indexes 0 fks vault_profiles 18 columns 0 indexes 1 fks vault_tokens 8 columns 0 indexes 2 fks webauthn_credentials 12 columns 0 indexes 1 fks No schema changes, nothing to migrate 😴 regenerates both. Settings: 27 call sites read settings synchronously, during startup and inside request handlers. better-sqlite3 can do that; Postgres and MySQL cannot, and making all 27 async would push await through code that has no reason to be asynchronous. Settings are a handful of rarely-changing rows read constantly, so they are cached in full — primed at startup, kept in step by SettingsRepository on every set/delete/deleteLike. SQLite keeps reading the database directly and stays authoritative; only the other engines use the cache. Opening a connection is still not done. DatabaseContext.drizzle is typed as BetterSQLite3Database and 43 repositories depend on that inference; the three drizzle instance types are not interchangeable, so widening it is a design decision rather than a mechanical change. * exclude drizzle-kit output from prettier The generated migrations and snapshots are tool output; their formatting is drizzle-kit's to decide, and prettier cannot parse the .sql files at all. * absorb the RETURNING gap so mysql stays reachable MySQL has no RETURNING clause and drizzle's mysql-core does not expose the method, while 156 call sites here read the result of a write. That is the real blocker for MySQL, not the connection layer. Classifying those call sites showed the split is favourable: 92 of them only read .length, which every engine reports — as a returned array on sqlite and postgres, as affectedRows on MySQL. rowsAffected() reads both shapes, so those sites need no change in query shape. insertedId() does the same for the autoincrement key, which MySQL reports as insertId. What is left is the ~34 sites that genuinely consume the returned rows. Those cannot be emulated without reading first, which needs a transaction to stay correct under concurrency, so they will be handled individually rather than behind a helper that quietly adds a round trip. supportsReturning() is the seam for that. Identifying the mysql2 result by its own fields rather than by array shape matters: it hands back [ResultSetHeader, fields], which is an array, so shape alone cannot tell it apart from a returning() result. * name the portable database type, and open remote connections Two pieces of the connection layer. drizzle's three Database classes share no base class and their signatures are incompatible, so there is no honest type that covers all three: a union is not callable and a generic would have to be threaded through 43 repositories and every method on them. DatabaseContext.drizzle is now PortableDatabase, still the SQLite type underneath, but named and documented as the deliberate approximation it is. What makes it safe is that the equivalence is asserted in multi-dialect.test.ts rather than assumed, and the one place the surfaces truly differ — RETURNING — is handled explicitly in mutation-result.ts. connect.ts opens Postgres and MySQL from DATABASE_URL, with the schema module and driver imported lazily so neither is loaded on a SQLite deployment. The URL scheme is checked against the configured dialect first: a postgres:// URL with DATABASE_DIALECT=mysql otherwise surfaces as a driver error deep in a stack that never mentions the actual misconfiguration. * open postgres and mysql at startup * count writes without RETURNING * read affected rows without RETURNING on mysql * insert without RETURNING, and split the sync transactions * stop pretending the generated schemas are used at runtime * run the dialect checks in CI * mysql rejects a bare CURRENT_TIMESTAMP default on text * make the read-back mismatch loud, and stop the next bare returning() * run the repository tests on the real schema * skip the byte-level assertions off sqlite * move generated ids past the seeded ones * keep the export order the same on every engine * stop reading better-sqlite3 fields off every write * read counts as numbers, not whatever the driver returns * make the fixture usable against a live server * upsert on the engine that has no ON CONFLICT * run the repository suite on all three engines in CI * mysql cannot index a text column without a length * document how to actually run on postgres or mysql * keep the sqlite-era migrations off the other engines * concat strings in a way mysql agrees with * run every repository test on every engine * bound how long replicas can disagree about settings * generate the sqlite migrations alongside the others
This commit is contained in:
ZacharyZcR
@@ -0,0 +1,227 @@
|
||||
# Database backends
|
||||
|
||||
Termix runs on SQLite by default. Postgres and MySQL are supported for
|
||||
self-hosted deployments; this document records how the three differ, because the
|
||||
differences are not only about SQL.
|
||||
|
||||
## This is multi-backend, not a migration
|
||||
|
||||
SQLite is not going away. The desktop app embeds its own backend and cannot ship
|
||||
a database server, so it will always run on SQLite. Postgres and MySQL exist for
|
||||
self-hosted deployments that need more than one process to reach the data —
|
||||
multiple replicas, an external backup story, or an existing database estate.
|
||||
|
||||
Anything that assumes a single engine is wrong.
|
||||
|
||||
## Where the schema comes from
|
||||
|
||||
`src/backend/database/db/schema.ts` is the single source of truth, written
|
||||
against `drizzle-orm/sqlite-core`.
|
||||
|
||||
`schema.pg.ts` and `schema.mysql.ts` are **generated** from it:
|
||||
|
||||
```bash
|
||||
npm run schema:generate # rewrite the generated modules
|
||||
npm run schema:check # fail if they are out of date (runs as part of lint)
|
||||
```
|
||||
|
||||
Never edit the generated files. `npm run lint` fails if they drift from the
|
||||
source, so a schema change that forgets to regenerate cannot reach main.
|
||||
|
||||
The transforms are mechanical:
|
||||
|
||||
| sqlite | postgres | mysql |
|
||||
| ------------------------------------------------ | ----------------- | ----------------------- |
|
||||
| `integer(…, { mode: "boolean" })` | `boolean` | `boolean` |
|
||||
| `integer(…).primaryKey({ autoIncrement: true })` | `serial` | `int().autoincrement()` |
|
||||
| `integer` | `integer` | `int` |
|
||||
| `real` | `doublePrecision` | `double` |
|
||||
| `text` used as a key | `varchar(255)` | `varchar(255)` |
|
||||
|
||||
A column becomes `varchar` if it is a primary key, is unique, or sits on either
|
||||
end of a foreign key — MySQL cannot index an unbounded `TEXT`, and both sides of
|
||||
a foreign key must agree.
|
||||
|
||||
## Durability
|
||||
|
||||
On SQLite the database is loaded into memory and serialised back to an encrypted
|
||||
file, so every write needs an explicit flush. That is what the `onWrite` hook
|
||||
each repository receives is for.
|
||||
|
||||
On Postgres and MySQL a committed write is already durable. No hook is installed
|
||||
at all — see `needsExplicitPersist` in `db/dialect.ts`.
|
||||
|
||||
## Encryption: what changes, and what does not
|
||||
|
||||
This is the part most likely to be misread, so it is spelled out.
|
||||
|
||||
### Unchanged on every backend
|
||||
|
||||
**Field-level encryption still applies.** Credentials and other sensitive values
|
||||
are encrypted in the application before they reach the database, under a
|
||||
per-user data key:
|
||||
|
||||
- `ssh_data` — passwords, private keys, key passphrases, sudo/RDP/VNC/Telnet
|
||||
secrets
|
||||
- `ssh_credentials` — passwords, private and public keys
|
||||
- `users` — TOTP secret and backup codes
|
||||
- `vault_tokens`, `opkssh_tokens`, `termix_identity_ca` — certificates and keys
|
||||
- `shared_host_secrets` — re-encrypted per recipient
|
||||
|
||||
Installation-level secrets — the OIDC client secret and LDAP bind password —
|
||||
are encrypted under the system key, since they have no owning user and must be
|
||||
readable during login.
|
||||
|
||||
This is the protection that matters most, and it is identical on all three
|
||||
engines.
|
||||
|
||||
### Different on Postgres and MySQL
|
||||
|
||||
**Whole-file encryption does not exist.** On SQLite the database file itself is
|
||||
encrypted at rest. There is no equivalent for a client-server engine: the data
|
||||
lives in the server's storage, not in a file Termix owns.
|
||||
|
||||
Concretely, on Postgres/MySQL the following are readable by anyone with database
|
||||
access, where on SQLite they were covered by the file encryption:
|
||||
|
||||
- host names, addresses, ports and usernames
|
||||
- folder and snippet names, and **snippet contents**
|
||||
- audit log entries
|
||||
- session recording metadata and paths
|
||||
- user names, roles and API key hashes
|
||||
|
||||
None of these are credentials — those stay encrypted — but together they
|
||||
describe your estate.
|
||||
|
||||
**If you run Postgres or MySQL, encryption at rest is your responsibility**:
|
||||
transparent data encryption, an encrypted volume, or an encrypted filesystem.
|
||||
Termix does not provide it and cannot.
|
||||
|
||||
### Threat model, side by side
|
||||
|
||||
| | SQLite | Postgres / MySQL |
|
||||
| ----------------------------------------------- | ------------------------------------------------ | ------------------------------------------------------------------ |
|
||||
| Stolen database file / volume | credentials encrypted, everything else encrypted | credentials encrypted, **rest depends on your storage encryption** |
|
||||
| Database access without app access | credentials unreadable | credentials unreadable |
|
||||
| Application compromise while a user is unlocked | that user's secrets readable | same |
|
||||
| Backups | inherit file encryption | **plain unless you encrypt them** |
|
||||
|
||||
The second row is the point of field-level encryption, and it holds everywhere.
|
||||
The first and last rows are where the backends genuinely differ.
|
||||
|
||||
## Running on Postgres or MySQL
|
||||
|
||||
Two variables. Unset, nothing changes and SQLite is used exactly as before.
|
||||
|
||||
```
|
||||
DATABASE_DIALECT=postgres
|
||||
DATABASE_URL=postgres://user:password@host:5432/termix
|
||||
```
|
||||
|
||||
```
|
||||
DATABASE_DIALECT=mysql
|
||||
DATABASE_URL=mysql://user:password@host:3306/termix
|
||||
```
|
||||
|
||||
`mariadb://` is accepted for MySQL. The scheme is checked against the dialect
|
||||
before a connection is attempted, so a mismatch fails with a readable message
|
||||
rather than a driver error deep in a stack.
|
||||
|
||||
Point it at an **empty** database. Migrations are applied at startup, from
|
||||
`drizzle/postgres` or `drizzle/mysql`, and drizzle records what it has applied —
|
||||
so several instances against one database are safe, and so is restarting.
|
||||
|
||||
There is no migration path from an existing SQLite database. Exporting one and
|
||||
importing it into Postgres is not something this branch does.
|
||||
|
||||
### Docker
|
||||
|
||||
`drizzle/` ships in the image. A compose service needs only the two variables:
|
||||
|
||||
Added to the compose file in the README, that is one service and two variables:
|
||||
|
||||
```yaml
|
||||
services:
|
||||
termix:
|
||||
image: ghcr.io/lukegus/termix:latest
|
||||
environment:
|
||||
PORT: "8080"
|
||||
DATABASE_DIALECT: postgres
|
||||
DATABASE_URL: postgres://termix:termix@db:5432/termix
|
||||
depends_on:
|
||||
- db
|
||||
|
||||
db:
|
||||
image: postgres:16
|
||||
restart: unless-stopped
|
||||
environment:
|
||||
POSTGRES_USER: termix
|
||||
POSTGRES_PASSWORD: termix
|
||||
POSTGRES_DB: termix
|
||||
volumes:
|
||||
- pgdata:/var/lib/postgresql/data
|
||||
|
||||
volumes:
|
||||
pgdata:
|
||||
```
|
||||
|
||||
`DATA_DIR` is still used for uploads and recordings on every backend. Only the
|
||||
database itself moves.
|
||||
|
||||
## What is verified, and how
|
||||
|
||||
`npm run verify:dialect -- <url>` applies the migrations to an empty database and
|
||||
drives the real repository classes against it, asserting values rather than the
|
||||
absence of exceptions.
|
||||
|
||||
The repository test suite also runs against each engine:
|
||||
|
||||
```
|
||||
TEST_DIALECT=postgres TEST_DATABASE_URL=<url> npx vitest run \
|
||||
src/backend/tests/database/repositories --no-file-parallelism
|
||||
```
|
||||
|
||||
CI runs both, against PostgreSQL 16 and MySQL 8 service containers. Eighteen
|
||||
tests assert on bytes stored by the SQLite driver and skip on other engines;
|
||||
they still run in the SQLite pass.
|
||||
|
||||
Tested against PostgreSQL 16 and MySQL 8. **MariaDB is not a substitute for
|
||||
MySQL when testing** — it accepts DDL that MySQL 8 rejects, which has hidden a
|
||||
real defect here more than once.
|
||||
|
||||
## Known limits
|
||||
|
||||
- The desktop app always uses SQLite. It embeds its own backend and cannot ship
|
||||
a database server.
|
||||
- Repositories import the SQLite table definitions on every engine. That is
|
||||
correct — the query builder needs identifiers and value encoders, and those
|
||||
agree — but it means `PortableDatabase` is a named approximation rather than a
|
||||
guarantee. See `repositories/database-context.ts`.
|
||||
- `getCurrentSettingValue` is a synchronous read. On Postgres and MySQL it comes
|
||||
from a cache primed at startup and kept current by `SettingsRepository`,
|
||||
because those drivers have no synchronous query.
|
||||
|
||||
That cache is per-process, so on a **multi-replica** deployment a setting
|
||||
changed on one instance does not reach the others through the write path. Each
|
||||
replica re-reads the settings table every 30 seconds
|
||||
(`SETTINGS_CACHE_REFRESH_SECONDS`, 0 to disable), which does not make settings
|
||||
immediately consistent — it bounds how long they can disagree. Changing a
|
||||
setting takes effect on the replica that made the change at once, and on the
|
||||
others within the interval.
|
||||
|
||||
- **Importing a backup is SQLite-only.** The restore writes tables in an order
|
||||
that is not dependency-safe and relies on `PRAGMA foreign_keys = OFF`, which
|
||||
has no equivalent here: Postgres needs superuser to disable triggers, and
|
||||
MySQL's session-scoped switch is not guaranteed across a pool. It refuses with
|
||||
a message rather than failing partway through and leaving a half-restored
|
||||
database. Restore into Postgres or MySQL with their own tooling.
|
||||
- **`LIKE` is case-insensitive on SQLite and case-sensitive on Postgres.** The
|
||||
four places that use it match folder path prefixes and settings keys, so the
|
||||
practical effect is that renaming a folder `prod` on SQLite also catches
|
||||
`PROD / api` and on Postgres does not. Postgres is arguably the more correct
|
||||
of the two; nothing was changed to make them agree, because that would alter
|
||||
SQLite behaviour for existing deployments.
|
||||
- The SQLite-era data migrations — legacy shared-credential cleanup, the
|
||||
shared-host-secrets rebuild, per-user field-encryption backfill — do not run on
|
||||
the other engines. A database created by the drizzle migrations never had the
|
||||
shapes they repair.
|
||||
Reference in New Issue
Block a user