mirror of
https://github.com/Termix-SSH/Termix.git
synced 2026-08-29 18:31:33 +00:00
Groundwork for Postgres and MySQL backends (#1134)
* groundwork for postgres and mysql backends #1127 made the repository layer dialect-agnostic. This adds the pieces needed to actually target a second engine, as a foundation only — nothing is wired up and sqlite remains the sole runtime path. - DatabaseDialect covers sqlite, postgres and mysql, resolved from DATABASE_DIALECT and defaulting to sqlite so nothing changes for existing deployments or the desktop build - a column kit holding the per-dialect type choices in one file: booleans are integers on sqlite and native elsewhere, autoincrement differs three ways, and MySQL cannot index unbounded TEXT so key columns need varchar - settings and users declared for all three dialects as a proof slice, chosen because between them they use every construct the real schema does - pg and mysql2 added as dependencies The tests build real queries for all three engines without a server, asserting identifier quoting, placeholder style and boolean storage, so the property the repositories depend on is verified rather than assumed. * verify foreign keys and unique constraints port across dialects The first slice only covered plain columns. The real schema also has 92 foreign keys (80 cascade, 12 set null) and 14 unique columns, so the approach is only viable if those survive the port. Adds audit_logs and ssh_folders to the proof slice: one nullable reference with ON DELETE SET NULL, one required reference with ON DELETE CASCADE, a unique column, and an autoincrement surrogate key — which is spelled three different ways underneath (integer primary key autoincrement, serial, int auto_increment). All of it holds. Worth noting for whoever picks this up: getTableConfig is dialect-specific and silently fails on a table from another dialect, so the test uses each engine's own. * generate the postgres and mysql schemas instead of hand-writing them The proof slice showed the constructs port, but left the maintenance question open. Three hand-written copies of 52 tables is the wrong answer: with foreign keys the copies cross-reference each other, so a renamed table has to land in three places consistently or a key silently points at the wrong one. The mapping is mechanical, so a script does it. schema.ts stays the single source of truth and schema.pg.ts / schema.mysql.ts are derived, covering all 52 tables — the column kit and the two-table portable slice are gone, since the generator now holds those decisions. The transforms are the ones the kit enumerated: integer-backed booleans become native, autoincrement keys become serial or int auto_increment, real becomes double precision or double, and any column that is a primary key, is unique, or sits on either end of a foreign key becomes varchar because MySQL cannot index unbounded TEXT. > termix@2.6.0 lint > node scripts/generate-dialect-schema.cjs --check && eslint . /mnt/c/Users/29037/WebstormProjects/Termix/src/backend/database/routes/homepage-favicon-routes.ts 99:12 warning 'err' is defined but never used unused-imports/no-unused-vars /mnt/c/Users/29037/WebstormProjects/Termix/src/backend/database/routes/homepage-ping-routes.ts 123:12 warning 'err' is defined but never used unused-imports/no-unused-vars /mnt/c/Users/29037/WebstormProjects/Termix/src/backend/database/routes/homepage-rss-routes.ts 144:12 warning 'err' is defined but never used unused-imports/no-unused-vars /mnt/c/Users/29037/WebstormProjects/Termix/src/backend/database/routes/session-log-routes.ts 46:16 warning 'canAccessRecording' is defined but never used. Allowed unused vars must match /^_/u unused-imports/no-unused-vars /mnt/c/Users/29037/WebstormProjects/Termix/src/backend/hosts/vault-signer-core.ts 55:12 warning Unexpected any. Specify a different type @typescript-eslint/no-explicit-any 75:13 warning Unexpected any. Specify a different type @typescript-eslint/no-explicit-any /mnt/c/Users/29037/WebstormProjects/Termix/src/backend/tests/hosts/auth-manager.test.ts 18:73 warning Unexpected any. Specify a different type @typescript-eslint/no-explicit-any /mnt/c/Users/29037/WebstormProjects/Termix/src/backend/tests/utils/shared-host-secrets-manager.test.ts 7:6 warning 'SecretRow' is defined but never used. Allowed unused vars must match /^_/u unused-imports/no-unused-vars /mnt/c/Users/29037/WebstormProjects/Termix/src/backend/utils/auth-manager.ts 510:13 warning 'affectedUsers' is assigned a value but never used. Allowed unused vars must match /^_/u unused-imports/no-unused-vars /mnt/c/Users/29037/WebstormProjects/Termix/src/backend/utils/notification-sender.ts 48:12 warning 'firstErr' is defined but never used unused-imports/no-unused-vars /mnt/c/Users/29037/WebstormProjects/Termix/src/ui/api/ssh-file-operations-api.ts 35:10 warning 'buildFileManagerUrl' is defined but never used. Allowed unused vars must match /^_/u unused-imports/no-unused-vars /mnt/c/Users/29037/WebstormProjects/Termix/src/ui/components/folder-style.tsx 61:14 warning Fast refresh only works when a file only exports components. Use a new file to share constants or functions between components react-refresh/only-export-components 116:14 warning Fast refresh only works when a file only exports components. Use a new file to share constants or functions between components react-refresh/only-export-components 121:14 warning Fast refresh only works when a file only exports components. Use a new file to share constants or functions between components react-refresh/only-export-components 149:17 warning Fast refresh only works when a file only exports components. Use a new file to share constants or functions between components react-refresh/only-export-components /mnt/c/Users/29037/WebstormProjects/Termix/src/ui/components/proxmox/ProxmoxDiscoverDialog.tsx 109:19 warning Unexpected any. Specify a different type @typescript-eslint/no-explicit-any 190:19 warning Unexpected any. Specify a different type @typescript-eslint/no-explicit-any /mnt/c/Users/29037/WebstormProjects/Termix/src/ui/features/homepage/HomepageCanvas.tsx 345:15 warning Empty block statement no-empty 388:15 warning Empty block statement no-empty 415:15 warning Empty block statement no-empty /mnt/c/Users/29037/WebstormProjects/Termix/src/ui/features/homepage/dialogs/SingleHostEditForm.tsx 24:6 warning React Hook useEffect has a missing dependency: 'filter'. Either include it or remove the dependency array. If 'setHosts' needs the current value of 'filter', you can also switch to useReducer instead of useState and read 'filter' in the reducer react-hooks/exhaustive-deps /mnt/c/Users/29037/WebstormProjects/Termix/src/ui/features/homepage/widgets/AlertFeedWidget.tsx 93:6 warning React Hook useEffect has a missing dependency: 'fetchData'. Either include it or remove the dependency array react-hooks/exhaustive-deps /mnt/c/Users/29037/WebstormProjects/Termix/src/ui/features/homepage/widgets/CustomApiWidget.tsx 77:6 warning React Hook useEffect has a missing dependency: 'fetchData'. Either include it or remove the dependency array react-hooks/exhaustive-deps /mnt/c/Users/29037/WebstormProjects/Termix/src/ui/features/homepage/widgets/DockerActivityWidget.tsx 50:6 warning React Hook useEffect has a missing dependency: 'fetchData'. Either include it or remove the dependency array react-hooks/exhaustive-deps /mnt/c/Users/29037/WebstormProjects/Termix/src/ui/features/homepage/widgets/DockerWidget.tsx 16:10 warning Fast refresh only works when a file has exports. Move your component(s) to a separate file react-refresh/only-export-components /mnt/c/Users/29037/WebstormProjects/Termix/src/ui/features/homepage/widgets/FileManagerWidget.tsx 16:10 warning Fast refresh only works when a file has exports. Move your component(s) to a separate file react-refresh/only-export-components /mnt/c/Users/29037/WebstormProjects/Termix/src/ui/features/homepage/widgets/HostGridWidget.tsx 61:6 warning React Hook useCallback has a missing dependency: 'hostIds'. Either include it or remove the dependency array react-hooks/exhaustive-deps 61:7 warning React Hook useCallback has a complex expression in the dependency array. Extract it to a separate variable so it can be statically checked react-hooks/exhaustive-deps /mnt/c/Users/29037/WebstormProjects/Termix/src/ui/features/homepage/widgets/MetricsChartWidget.tsx 168:6 warning React Hook useEffect has a missing dependency: 'fetchData'. Either include it or remove the dependency array react-hooks/exhaustive-deps /mnt/c/Users/29037/WebstormProjects/Termix/src/ui/features/homepage/widgets/PingStatusWidget.tsx 79:6 warning React Hook useEffect has a missing dependency: 'fetchAll'. Either include it or remove the dependency array react-hooks/exhaustive-deps 79:7 warning React Hook useEffect has a complex expression in the dependency array. Extract it to a separate variable so it can be statically checked react-hooks/exhaustive-deps /mnt/c/Users/29037/WebstormProjects/Termix/src/ui/features/homepage/widgets/QuickConnectWidget.tsx 64:10 warning Fast refresh only works when a file has exports. Move your component(s) to a separate file react-refresh/only-export-components /mnt/c/Users/29037/WebstormProjects/Termix/src/ui/features/homepage/widgets/RecentActivityWidget.tsx 82:6 warning React Hook useEffect has a missing dependency: 'fetchData'. Either include it or remove the dependency array react-hooks/exhaustive-deps 82:17 warning React Hook useEffect has a complex expression in the dependency array. Extract it to a separate variable so it can be statically checked react-hooks/exhaustive-deps /mnt/c/Users/29037/WebstormProjects/Termix/src/ui/features/homepage/widgets/SshQuickConnectWidget.tsx 67:6 warning React Hook useCallback has a missing dependency: 'hostIds'. Either include it or remove the dependency array react-hooks/exhaustive-deps 67:7 warning React Hook useCallback has a complex expression in the dependency array. Extract it to a separate variable so it can be statically checked react-hooks/exhaustive-deps 99:17 warning 'online' is assigned a value but never used. Allowed unused vars must match /^_/u unused-imports/no-unused-vars /mnt/c/Users/29037/WebstormProjects/Termix/src/ui/features/homepage/widgets/SshTerminalWidget.tsx 17:10 warning Fast refresh only works when a file has exports. Move your component(s) to a separate file react-refresh/only-export-components /mnt/c/Users/29037/WebstormProjects/Termix/src/ui/features/homepage/widgets/SystemOverviewWidget.tsx 72:6 warning React Hook useEffect has a missing dependency: 'fetchData'. Either include it or remove the dependency array react-hooks/exhaustive-deps /mnt/c/Users/29037/WebstormProjects/Termix/src/ui/features/homepage/widgets/TunnelWidget.tsx 15:10 warning Fast refresh only works when a file has exports. Move your component(s) to a separate file react-refresh/only-export-components /mnt/c/Users/29037/WebstormProjects/Termix/src/ui/features/host-metrics/cards/CpuCard.tsx 14:10 warning 'computeChartData' is defined but never used. Allowed unused vars must match /^_/u unused-imports/no-unused-vars /mnt/c/Users/29037/WebstormProjects/Termix/src/ui/sidebar/FolderPathPicker.tsx 15:17 warning Fast refresh only works when a file only exports components. Use a new file to share constants or functions between components react-refresh/only-export-components 22:17 warning Fast refresh only works when a file only exports components. Use a new file to share constants or functions between components react-refresh/only-export-components /mnt/c/Users/29037/WebstormProjects/Termix/src/ui/sidebar/HostsPanel.tsx 601:52 warning Unexpected any. Specify a different type @typescript-eslint/no-explicit-any ✖ 44 problems (0 errors, 44 warnings) now fails if the generated files are out of date, so editing the schema without regenerating cannot reach main. * select durability behaviour per dialect, and document the backends The onWrite hook every repository receives exists to serialise the in-memory SQLite database back to its encrypted file. On a client-server engine a committed write is already durable and there is nothing to flush, so the factory now installs no hook at all rather than one that does nothing. Repositories call it as this.onWrite?.(), so none of the 43 of them change. Also adds docs/database-backends.md, mostly to be explicit about encryption, which is the part most likely to be misread. Field-level encryption is identical on all three engines and covers every credential. Whole-file encryption has no equivalent on Postgres or MySQL, so host names, snippet contents, audit entries and backups are only as protected as the storage underneath them — that is the operator's responsibility and the docs should not imply otherwise. * generate DDL with drizzle-kit, and give settings a synchronous path Two of the three remaining blockers. DDL: db/index.ts hand-writes 67 CREATE TABLE statements and 122 ADD COLUMN migrations, all in SQLite dialect. Rather than port them, drizzle-kit now generates migrations from the schema modules — 817 lines for Postgres, 869 for MySQL, with the type mapping already correct because the schemas it reads are themselves generated. > termix@2.6.0 schema:migrations > drizzle-kit generate --config=drizzle.config.pg.ts && drizzle-kit generate --config=drizzle.config.mysql.ts Reading config file '/mnt/c/Users/29037/WebstormProjects/Termix/drizzle.config.pg.ts' 52 tables alert_firings 11 columns 0 indexes 2 fks alert_rule_channels 3 columns 0 indexes 2 fks alert_rules 11 columns 0 indexes 2 fks api_keys 9 columns 0 indexes 1 fks audit_logs 13 columns 0 indexes 1 fks c2s_tunnel_presets 8 columns 0 indexes 1 fks command_history 5 columns 0 indexes 2 fks dashboard_service_links 8 columns 0 indexes 1 fks dismissed_alerts 4 columns 0 indexes 1 fks file_manager_pinned 6 columns 0 indexes 2 fks file_manager_recent 6 columns 0 indexes 2 fks file_manager_shortcuts 6 columns 0 indexes 2 fks homepage_items 9 columns 0 indexes 1 fks homepage_layouts 4 columns 0 indexes 1 fks host_access 11 columns 0 indexes 5 fks host_health_checks 7 columns 0 indexes 2 fks host_health_history 8 columns 0 indexes 2 fks host_metrics_history 8 columns 0 indexes 1 fks host_metrics_preferences 6 columns 0 indexes 2 fks ssh_data 94 columns 0 indexes 6 fks network_topology 5 columns 0 indexes 1 fks notification_channels 7 columns 0 indexes 1 fks opkssh_tokens 12 columns 0 indexes 2 fks recent_activity 6 columns 0 indexes 2 fks roles 8 columns 0 indexes 0 fks session_recordings 15 columns 0 indexes 3 fks session_share_participants 6 columns 0 indexes 2 fks session_shares 15 columns 0 indexes 3 fks sessions 11 columns 0 indexes 1 fks settings 2 columns 0 indexes 0 fks shared_host_secrets 15 columns 0 indexes 3 fks snippet_access 8 columns 0 indexes 4 fks snippet_folders 8 columns 0 indexes 1 fks snippets 11 columns 0 indexes 1 fks ssh_credential_usage 5 columns 0 indexes 3 fks ssh_credentials 21 columns 0 indexes 1 fks ssh_folders 9 columns 0 indexes 2 fks sso_providers 8 columns 0 indexes 0 fks sync_tombstones 5 columns 0 indexes 1 fks termix_identities 6 columns 0 indexes 1 fks termix_identity_ca 8 columns 0 indexes 2 fks termix_identity_keys 12 columns 0 indexes 3 fks tmux_session_tags 6 columns 0 indexes 2 fks transfer_recent 7 columns 0 indexes 3 fks trusted_devices 8 columns 0 indexes 1 fks user_open_tabs 9 columns 0 indexes 2 fks user_preferences 23 columns 0 indexes 1 fks user_roles 5 columns 0 indexes 3 fks users 20 columns 0 indexes 0 fks vault_profiles 18 columns 0 indexes 1 fks vault_tokens 8 columns 0 indexes 2 fks webauthn_credentials 12 columns 0 indexes 1 fks No schema changes, nothing to migrate 😴 Reading config file '/mnt/c/Users/29037/WebstormProjects/Termix/drizzle.config.mysql.ts' Reading schema files: /mnt/c/Users/29037/WebstormProjects/Termix/src/backend/database/db/schema.mysql.ts 52 tables alert_firings 11 columns 0 indexes 2 fks alert_rule_channels 3 columns 0 indexes 2 fks alert_rules 11 columns 0 indexes 2 fks api_keys 9 columns 0 indexes 1 fks audit_logs 13 columns 0 indexes 1 fks c2s_tunnel_presets 8 columns 0 indexes 1 fks command_history 5 columns 0 indexes 2 fks dashboard_service_links 8 columns 0 indexes 1 fks dismissed_alerts 4 columns 0 indexes 1 fks file_manager_pinned 6 columns 0 indexes 2 fks file_manager_recent 6 columns 0 indexes 2 fks file_manager_shortcuts 6 columns 0 indexes 2 fks homepage_items 9 columns 0 indexes 1 fks homepage_layouts 4 columns 0 indexes 1 fks host_access 11 columns 0 indexes 5 fks host_health_checks 7 columns 0 indexes 2 fks host_health_history 8 columns 0 indexes 2 fks host_metrics_history 8 columns 0 indexes 1 fks host_metrics_preferences 6 columns 0 indexes 2 fks ssh_data 94 columns 0 indexes 6 fks network_topology 5 columns 0 indexes 1 fks notification_channels 7 columns 0 indexes 1 fks opkssh_tokens 12 columns 0 indexes 2 fks recent_activity 6 columns 0 indexes 2 fks roles 8 columns 0 indexes 0 fks session_recordings 15 columns 0 indexes 3 fks session_share_participants 6 columns 0 indexes 2 fks session_shares 15 columns 0 indexes 3 fks sessions 11 columns 0 indexes 1 fks settings 2 columns 0 indexes 0 fks shared_host_secrets 15 columns 0 indexes 3 fks snippet_access 8 columns 0 indexes 4 fks snippet_folders 8 columns 0 indexes 1 fks snippets 11 columns 0 indexes 1 fks ssh_credential_usage 5 columns 0 indexes 3 fks ssh_credentials 21 columns 0 indexes 1 fks ssh_folders 9 columns 0 indexes 2 fks sso_providers 8 columns 0 indexes 0 fks sync_tombstones 5 columns 0 indexes 1 fks termix_identities 6 columns 0 indexes 1 fks termix_identity_ca 8 columns 0 indexes 2 fks termix_identity_keys 12 columns 0 indexes 3 fks tmux_session_tags 6 columns 0 indexes 2 fks transfer_recent 7 columns 0 indexes 3 fks trusted_devices 8 columns 0 indexes 1 fks user_open_tabs 9 columns 0 indexes 2 fks user_preferences 23 columns 0 indexes 1 fks user_roles 5 columns 0 indexes 3 fks users 20 columns 0 indexes 0 fks vault_profiles 18 columns 0 indexes 1 fks vault_tokens 8 columns 0 indexes 2 fks webauthn_credentials 12 columns 0 indexes 1 fks No schema changes, nothing to migrate 😴 regenerates both. Settings: 27 call sites read settings synchronously, during startup and inside request handlers. better-sqlite3 can do that; Postgres and MySQL cannot, and making all 27 async would push await through code that has no reason to be asynchronous. Settings are a handful of rarely-changing rows read constantly, so they are cached in full — primed at startup, kept in step by SettingsRepository on every set/delete/deleteLike. SQLite keeps reading the database directly and stays authoritative; only the other engines use the cache. Opening a connection is still not done. DatabaseContext.drizzle is typed as BetterSQLite3Database and 43 repositories depend on that inference; the three drizzle instance types are not interchangeable, so widening it is a design decision rather than a mechanical change. * exclude drizzle-kit output from prettier The generated migrations and snapshots are tool output; their formatting is drizzle-kit's to decide, and prettier cannot parse the .sql files at all. * absorb the RETURNING gap so mysql stays reachable MySQL has no RETURNING clause and drizzle's mysql-core does not expose the method, while 156 call sites here read the result of a write. That is the real blocker for MySQL, not the connection layer. Classifying those call sites showed the split is favourable: 92 of them only read .length, which every engine reports — as a returned array on sqlite and postgres, as affectedRows on MySQL. rowsAffected() reads both shapes, so those sites need no change in query shape. insertedId() does the same for the autoincrement key, which MySQL reports as insertId. What is left is the ~34 sites that genuinely consume the returned rows. Those cannot be emulated without reading first, which needs a transaction to stay correct under concurrency, so they will be handled individually rather than behind a helper that quietly adds a round trip. supportsReturning() is the seam for that. Identifying the mysql2 result by its own fields rather than by array shape matters: it hands back [ResultSetHeader, fields], which is an array, so shape alone cannot tell it apart from a returning() result. * name the portable database type, and open remote connections Two pieces of the connection layer. drizzle's three Database classes share no base class and their signatures are incompatible, so there is no honest type that covers all three: a union is not callable and a generic would have to be threaded through 43 repositories and every method on them. DatabaseContext.drizzle is now PortableDatabase, still the SQLite type underneath, but named and documented as the deliberate approximation it is. What makes it safe is that the equivalence is asserted in multi-dialect.test.ts rather than assumed, and the one place the surfaces truly differ — RETURNING — is handled explicitly in mutation-result.ts. connect.ts opens Postgres and MySQL from DATABASE_URL, with the schema module and driver imported lazily so neither is loaded on a SQLite deployment. The URL scheme is checked against the configured dialect first: a postgres:// URL with DATABASE_DIALECT=mysql otherwise surfaces as a driver error deep in a stack that never mentions the actual misconfiguration. * open postgres and mysql at startup * count writes without RETURNING * read affected rows without RETURNING on mysql * insert without RETURNING, and split the sync transactions * stop pretending the generated schemas are used at runtime * run the dialect checks in CI * mysql rejects a bare CURRENT_TIMESTAMP default on text * make the read-back mismatch loud, and stop the next bare returning() * run the repository tests on the real schema * skip the byte-level assertions off sqlite * move generated ids past the seeded ones * keep the export order the same on every engine * stop reading better-sqlite3 fields off every write * read counts as numbers, not whatever the driver returns * make the fixture usable against a live server * upsert on the engine that has no ON CONFLICT * run the repository suite on all three engines in CI * mysql cannot index a text column without a length * document how to actually run on postgres or mysql * keep the sqlite-era migrations off the other engines * concat strings in a way mysql agrees with * run every repository test on every engine * bound how long replicas can disagree about settings * generate the sqlite migrations alongside the others
This commit is contained in:
@@ -0,0 +1,228 @@
|
||||
/**
|
||||
* Generates the Postgres and MySQL schema modules from the SQLite one.
|
||||
*
|
||||
* ## These files produce DDL. They are not used at runtime.
|
||||
*
|
||||
* drizzle-kit reads them to emit the migrations in drizzle/postgres and
|
||||
* drizzle/mysql. Nothing imports them to run a query.
|
||||
*
|
||||
* That is not an oversight. The query builder needs two things from a table
|
||||
* object — the identifiers to interpolate, and the encoders that turn JS values
|
||||
* into driver values — and the sqlite definitions supply both correctly for
|
||||
* every engine, which is why all 44 repositories import schema.ts directly:
|
||||
*
|
||||
* - text and integer encode as themselves everywhere
|
||||
* - integer({ mode: "boolean" }) writes 1/0, which Postgres and MySQL both
|
||||
* accept for a boolean column, and reads back through `Number(v) === 1`,
|
||||
* which is true for JS `true` as well as for 1
|
||||
* - real is a plain number on all three
|
||||
*
|
||||
* What genuinely differs between the dialects is DDL — column types, key
|
||||
* lengths, autoincrement syntax — and DDL is exactly what these files exist to
|
||||
* generate. See scripts/verify-dialects.mjs, which asserts the round-trips
|
||||
* above against real servers rather than trusting this comment.
|
||||
*
|
||||
* The schema is declared once, in sqlite-core, and the other two dialects are
|
||||
* derived. Hand-maintaining three copies of 52 tables would mean a renamed
|
||||
* table has to land in three places consistently or a foreign key silently
|
||||
* points at the wrong one — and the schema is regular enough that the mapping
|
||||
* is mechanical.
|
||||
*
|
||||
* What varies between dialects is small and closed:
|
||||
* - booleans are integers on sqlite, native elsewhere
|
||||
* - autoincrement keys are `integer primary key autoincrement`, `serial`,
|
||||
* and `int auto_increment`
|
||||
* - MySQL cannot index unbounded TEXT, so any column that is a primary key,
|
||||
* is unique, or participates in a foreign key must be varchar
|
||||
* - MySQL rejects a bare DEFAULT CURRENT_TIMESTAMP on a text column, so it is
|
||||
* written as a parenthesised expression default
|
||||
*
|
||||
* Usage: node scripts/generate-dialect-schema.cjs [--check]
|
||||
* --check verifies the committed files match what would be generated,
|
||||
* for CI to catch a schema edit that forgot to regenerate.
|
||||
*/
|
||||
|
||||
const fs = require("fs");
|
||||
const path = require("path");
|
||||
|
||||
const ROOT = path.join(__dirname, "..");
|
||||
const SOURCE = path.join(ROOT, "src/backend/database/db/schema.ts");
|
||||
const TARGETS = {
|
||||
postgres: path.join(ROOT, "src/backend/database/db/schema.pg.ts"),
|
||||
mysql: path.join(ROOT, "src/backend/database/db/schema.mysql.ts"),
|
||||
};
|
||||
|
||||
const KEY_LENGTH = 255;
|
||||
|
||||
/**
|
||||
* Columns that must be varchar rather than text on MySQL. A column qualifies if
|
||||
* it is a primary key, is unique, or is either end of a foreign key.
|
||||
*/
|
||||
function collectKeyColumns(source) {
|
||||
const keyed = new Set();
|
||||
|
||||
// `name: text("col")....primaryKey()` / `.unique()` / `.references(...)`
|
||||
const declaration =
|
||||
/(\w+):\s*text\("([a-z0-9_]+)"\)((?:\s*\.\w+\([^)]*\))*)/g;
|
||||
let match;
|
||||
while ((match = declaration.exec(source)) !== null) {
|
||||
const [, prop, column, modifiers] = match;
|
||||
if (/\.(primaryKey|unique|references)\(/.test(modifiers)) {
|
||||
keyed.add(column);
|
||||
}
|
||||
void prop;
|
||||
}
|
||||
|
||||
// Multi-line form: the modifiers land on following lines.
|
||||
const multiline =
|
||||
/(\w+):\s*text\("([a-z0-9_]+)"\)\s*\n(\s*\.\w+\([\s\S]*?\),)/g;
|
||||
while ((match = multiline.exec(source)) !== null) {
|
||||
if (/\.(primaryKey|unique|references)\(/.test(match[3])) {
|
||||
keyed.add(match[2]);
|
||||
}
|
||||
}
|
||||
|
||||
// A referenced column implies the referencing side too; both must match.
|
||||
const reference = /\.references\(\(\)\s*=>\s*\w+\.(\w+)/g;
|
||||
while ((match = reference.exec(source)) !== null) {
|
||||
keyed.add(camelToSnake(match[1]));
|
||||
}
|
||||
|
||||
// Table-level indexes: `(table) => [uniqueIndex("x").on(table.a, table.b)]`.
|
||||
// These were invisible here at first, and MySQL rejected the migration with
|
||||
// "BLOB/TEXT column used in key specification without a key length" — but
|
||||
// only on MySQL 8; MariaDB took it.
|
||||
const tableIndex = /uniqueIndex\("[a-z0-9_]+"\)\.on\(([^)]*)\)/g;
|
||||
while ((match = tableIndex.exec(source)) !== null) {
|
||||
for (const column of match[1].split(",")) {
|
||||
const name = column.trim().replace(/^\w+\./, "");
|
||||
if (name) keyed.add(camelToSnake(name));
|
||||
}
|
||||
}
|
||||
|
||||
return keyed;
|
||||
}
|
||||
|
||||
function camelToSnake(value) {
|
||||
return value.replace(/[A-Z]/g, (c) => `_${c.toLowerCase()}`);
|
||||
}
|
||||
|
||||
function transform(source, dialect) {
|
||||
const keyed = collectKeyColumns(source);
|
||||
const isPg = dialect === "postgres";
|
||||
let out = source;
|
||||
|
||||
// Autoincrement primary keys, before the plain integer rule below.
|
||||
out = out.replace(
|
||||
/integer\("([a-z0-9_]+)"\)\.primaryKey\(\{\s*autoIncrement:\s*true\s*\}\)/g,
|
||||
(_, col) =>
|
||||
isPg
|
||||
? `serial("${col}").primaryKey()`
|
||||
: `int("${col}").autoincrement().primaryKey()`,
|
||||
);
|
||||
|
||||
// Integer-backed booleans become native ones. Prettier wraps the longer
|
||||
// declarations across lines, so this has to span newlines too.
|
||||
out = out.replace(
|
||||
/integer\(\s*"([a-z0-9_]+)",\s*\{\s*mode:\s*"boolean",?\s*\},?\s*\)/g,
|
||||
(_, col) => `boolean("${col}")`,
|
||||
);
|
||||
|
||||
// Remaining integers.
|
||||
if (!isPg) {
|
||||
out = out.replace(
|
||||
/\binteger\("([a-z0-9_]+)"\)/g,
|
||||
(_, col) => `int("${col}")`,
|
||||
);
|
||||
|
||||
// Timestamps are stored as text (see sql-timestamp.ts). MySQL only accepts
|
||||
// DEFAULT CURRENT_TIMESTAMP on a DATETIME or TIMESTAMP column — on a TEXT
|
||||
// one it is ER_INVALID_DEFAULT, "Invalid default value". Since 8.0.13 an
|
||||
// expression default works on any type, and an expression is written
|
||||
// parenthesised. MariaDB accepts the bare form, which is why this only
|
||||
// surfaces against real MySQL.
|
||||
out = out.replace(/sql`CURRENT_TIMESTAMP`/g, "sql`(CURRENT_TIMESTAMP)`");
|
||||
}
|
||||
|
||||
// Floating point.
|
||||
out = out.replace(/\breal\("([a-z0-9_]+)"\)/g, (_, col) =>
|
||||
isPg ? `doublePrecision("${col}")` : `double("${col}")`,
|
||||
);
|
||||
|
||||
// Key-bearing strings must be indexable.
|
||||
out = out.replace(/\btext\("([a-z0-9_]+)"\)/g, (whole, col) =>
|
||||
keyed.has(col) ? `varchar("${col}", { length: ${KEY_LENGTH} })` : whole,
|
||||
);
|
||||
|
||||
// text("x", { length: n }) is sqlite-only sugar; drop the length.
|
||||
out = out.replace(
|
||||
/\btext\("([a-z0-9_]+)",\s*\{\s*length:\s*\d+\s*\}\)/g,
|
||||
(_, col) => `text("${col}")`,
|
||||
);
|
||||
|
||||
out = out.replace(/\bsqliteTable\(/g, isPg ? "pgTable(" : "mysqlTable(");
|
||||
|
||||
const imports = isPg
|
||||
? `import {\n pgTable,\n text,\n varchar,\n integer,\n serial,\n boolean,\n doublePrecision,\n uniqueIndex,\n} from "drizzle-orm/pg-core";`
|
||||
: `import {\n mysqlTable,\n text,\n varchar,\n int,\n boolean,\n double,\n uniqueIndex,\n} from "drizzle-orm/mysql-core";`;
|
||||
|
||||
out = out.replace(
|
||||
/import\s*\{[^}]*\}\s*from\s*"drizzle-orm\/sqlite-core";/,
|
||||
imports,
|
||||
);
|
||||
|
||||
return `${header(dialect)}\n${out}`;
|
||||
}
|
||||
|
||||
function header(dialect) {
|
||||
return `// GENERATED FILE — do not edit.
|
||||
//
|
||||
// Produced from schema.ts by scripts/generate-dialect-schema.cjs.
|
||||
// Edit the sqlite schema and re-run \`node scripts/generate-dialect-schema.cjs\`.
|
||||
// Target dialect: ${dialect}.
|
||||
//
|
||||
// DDL source for drizzle-kit. NOT imported to run queries — repositories use
|
||||
// schema.ts on every dialect. See the generator header for why that is correct.
|
||||
`;
|
||||
}
|
||||
|
||||
function main() {
|
||||
const check = process.argv.includes("--check");
|
||||
const source = fs.readFileSync(SOURCE, "utf8");
|
||||
let drift = false;
|
||||
|
||||
for (const [dialect, target] of Object.entries(TARGETS)) {
|
||||
const generated = transform(source, dialect);
|
||||
|
||||
if (check) {
|
||||
const current = fs.existsSync(target)
|
||||
? fs.readFileSync(target, "utf8")
|
||||
: "";
|
||||
if (current !== generated) {
|
||||
console.error(
|
||||
`[generate-dialect-schema] ${path.relative(ROOT, target)} is out of date`,
|
||||
);
|
||||
drift = true;
|
||||
}
|
||||
continue;
|
||||
}
|
||||
|
||||
fs.writeFileSync(target, generated);
|
||||
console.log(
|
||||
`[generate-dialect-schema] wrote ${path.relative(ROOT, target)}`,
|
||||
);
|
||||
}
|
||||
|
||||
if (drift) {
|
||||
console.error(
|
||||
"[generate-dialect-schema] run `node scripts/generate-dialect-schema.cjs` and commit the result",
|
||||
);
|
||||
process.exit(1);
|
||||
}
|
||||
}
|
||||
|
||||
module.exports = { transform, collectKeyColumns };
|
||||
|
||||
if (require.main === module) {
|
||||
main();
|
||||
}
|
||||
@@ -0,0 +1,140 @@
|
||||
import { describe, expect, it } from "vitest";
|
||||
import { createRequire } from "node:module";
|
||||
|
||||
const require = createRequire(import.meta.url);
|
||||
const { transform, collectKeyColumns } =
|
||||
require("./generate-dialect-schema.cjs") as {
|
||||
transform: (source: string, dialect: "postgres" | "mysql") => string;
|
||||
collectKeyColumns: (source: string) => Set<string>;
|
||||
};
|
||||
|
||||
const SOURCE = `import { sqliteTable, text, integer, real } from "drizzle-orm/sqlite-core";
|
||||
import { sql } from "drizzle-orm";
|
||||
|
||||
export const users = sqliteTable("users", {
|
||||
id: text("id").primaryKey(),
|
||||
username: text("username").notNull(),
|
||||
isAdmin: integer("is_admin", { mode: "boolean" }).notNull().default(false),
|
||||
wrapped: integer("wrapped", {
|
||||
mode: "boolean",
|
||||
})
|
||||
.notNull()
|
||||
.default(true),
|
||||
score: real("score"),
|
||||
ssoProviderId: integer("sso_provider_id"),
|
||||
});
|
||||
|
||||
export const folders = sqliteTable("folders", {
|
||||
id: integer("id").primaryKey({ autoIncrement: true }),
|
||||
userId: text("user_id")
|
||||
.notNull()
|
||||
.references(() => users.id, { onDelete: "cascade" }),
|
||||
name: text("name").notNull(),
|
||||
syncId: text("sync_id").unique(),
|
||||
cert: text("cert", { length: 8192 }),
|
||||
});
|
||||
`;
|
||||
|
||||
describe("collectKeyColumns", () => {
|
||||
it("finds columns that must be indexable", () => {
|
||||
const keyed = collectKeyColumns(SOURCE);
|
||||
|
||||
// primary key, unique, and both ends of the foreign key
|
||||
expect(keyed.has("id")).toBe(true);
|
||||
expect(keyed.has("sync_id")).toBe(true);
|
||||
expect(keyed.has("user_id")).toBe(true);
|
||||
});
|
||||
|
||||
it("leaves ordinary strings alone", () => {
|
||||
const keyed = collectKeyColumns(SOURCE);
|
||||
|
||||
expect(keyed.has("username")).toBe(false);
|
||||
expect(keyed.has("name")).toBe(false);
|
||||
expect(keyed.has("cert")).toBe(false);
|
||||
});
|
||||
});
|
||||
|
||||
describe("postgres output", () => {
|
||||
const out = transform(SOURCE, "postgres");
|
||||
|
||||
it("is marked generated", () => {
|
||||
expect(out.startsWith("// GENERATED FILE")).toBe(true);
|
||||
});
|
||||
|
||||
it("uses pg-core", () => {
|
||||
expect(out).toContain('from "drizzle-orm/pg-core"');
|
||||
expect(out).not.toContain("sqlite-core");
|
||||
expect(out).toContain("pgTable(");
|
||||
expect(out).not.toContain("sqliteTable(");
|
||||
});
|
||||
|
||||
it("maps autoincrement keys to serial", () => {
|
||||
expect(out).toContain('serial("id").primaryKey()');
|
||||
expect(out).not.toContain("autoIncrement");
|
||||
});
|
||||
|
||||
it("maps integer-backed booleans, including the wrapped form", () => {
|
||||
expect(out).toContain('boolean("is_admin")');
|
||||
// Prettier splits longer declarations across lines; both must convert.
|
||||
expect(out).toContain('boolean("wrapped")');
|
||||
expect(out).not.toMatch(/mode:\s*"boolean"/);
|
||||
});
|
||||
|
||||
it("keeps plain integers and maps real", () => {
|
||||
expect(out).toContain('integer("sso_provider_id")');
|
||||
expect(out).toContain('doublePrecision("score")');
|
||||
});
|
||||
|
||||
it("makes key columns varchar and leaves the rest text", () => {
|
||||
expect(out).toContain('varchar("id", { length: 255 })');
|
||||
expect(out).toContain('varchar("user_id", { length: 255 })');
|
||||
expect(out).toContain('varchar("sync_id", { length: 255 })');
|
||||
expect(out).toContain('text("username")');
|
||||
expect(out).toContain('text("name")');
|
||||
});
|
||||
|
||||
it("drops the sqlite-only text length", () => {
|
||||
expect(out).toContain('text("cert")');
|
||||
expect(out).not.toContain("length: 8192");
|
||||
});
|
||||
});
|
||||
|
||||
describe("mysql output", () => {
|
||||
const out = transform(SOURCE, "mysql");
|
||||
|
||||
it("uses mysql-core", () => {
|
||||
expect(out).toContain('from "drizzle-orm/mysql-core"');
|
||||
expect(out).toContain("mysqlTable(");
|
||||
});
|
||||
|
||||
it("maps autoincrement keys to int auto_increment", () => {
|
||||
expect(out).toContain('int("id").autoincrement().primaryKey()');
|
||||
});
|
||||
|
||||
it("renames integer to int", () => {
|
||||
expect(out).toContain('int("sso_provider_id")');
|
||||
expect(out).not.toMatch(/\binteger\(/);
|
||||
});
|
||||
|
||||
it("maps real to double", () => {
|
||||
expect(out).toContain('double("score")');
|
||||
});
|
||||
|
||||
it("makes key columns varchar — MySQL cannot index unbounded TEXT", () => {
|
||||
expect(out).toContain('varchar("user_id", { length: 255 })');
|
||||
expect(out).toContain('text("name")');
|
||||
});
|
||||
});
|
||||
|
||||
describe("determinism", () => {
|
||||
it("produces identical output for identical input", () => {
|
||||
expect(transform(SOURCE, "postgres")).toBe(transform(SOURCE, "postgres"));
|
||||
expect(transform(SOURCE, "mysql")).toBe(transform(SOURCE, "mysql"));
|
||||
});
|
||||
|
||||
it("keeps foreign key behaviour verbatim", () => {
|
||||
for (const dialect of ["postgres", "mysql"] as const) {
|
||||
expect(transform(SOURCE, dialect)).toContain('onDelete: "cascade"');
|
||||
}
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,140 @@
|
||||
/**
|
||||
* Runs the repository layer against a real Postgres or MySQL server.
|
||||
*
|
||||
* The unit tests only ever see SQLite, so the parts of this codebase that
|
||||
* differ per engine — the RETURNING replacements, the read-then-write
|
||||
* transactions, the value encoders — have no coverage there at all. This is
|
||||
* what covers them, and it needs a live server, which is why it is a script
|
||||
* rather than a test.
|
||||
*
|
||||
* Usage:
|
||||
* npm run verify:dialect -- postgres://user:pass@host:5432/db
|
||||
* npm run verify:dialect -- mysql://user:pass@host:3306/db
|
||||
*
|
||||
* Applies the migrations first, through the same runRemoteMigrations() the
|
||||
* application uses at startup — so a broken migration fails here rather than in
|
||||
* production. Writes real rows: point it at a scratch database.
|
||||
*/
|
||||
|
||||
import { randomUUID } from "crypto";
|
||||
|
||||
const url = process.argv[2];
|
||||
if (!url) {
|
||||
console.error("usage: node scripts/verify-dialects.mjs <DATABASE_URL>");
|
||||
process.exit(2);
|
||||
}
|
||||
|
||||
const scheme = url.split("://", 1)[0].toLowerCase();
|
||||
const dialect = scheme.startsWith("postgres")
|
||||
? "postgres"
|
||||
: scheme === "mysql" || scheme === "mariadb"
|
||||
? "mysql"
|
||||
: null;
|
||||
|
||||
if (!dialect) {
|
||||
console.error(`unsupported URL scheme "${scheme}://"`);
|
||||
process.exit(2);
|
||||
}
|
||||
|
||||
const { drizzle } = await import(
|
||||
dialect === "postgres" ? "drizzle-orm/node-postgres" : "drizzle-orm/mysql2"
|
||||
);
|
||||
|
||||
// No schema option on purpose — see connect.ts.
|
||||
const db = drizzle(url);
|
||||
const context = { dialect, drizzle: db };
|
||||
|
||||
const { runRemoteMigrations } =
|
||||
await import("../src/backend/database/db/migrate.js");
|
||||
await runRemoteMigrations(dialect, db);
|
||||
|
||||
const { UserRepository } =
|
||||
await import("../src/backend/database/repositories/user-repository.js");
|
||||
const { HostRepository } =
|
||||
await import("../src/backend/database/repositories/host-repository.js");
|
||||
const { SettingsRepository } =
|
||||
await import("../src/backend/database/repositories/settings-repository.js");
|
||||
|
||||
let failures = 0;
|
||||
const check = (label, got, want) => {
|
||||
const ok = JSON.stringify(got) === JSON.stringify(want);
|
||||
if (!ok) failures++;
|
||||
console.log(
|
||||
` ${ok ? "ok " : "FAIL"} ${label}` +
|
||||
(ok
|
||||
? ""
|
||||
: `\n got ${JSON.stringify(got)}, want ${JSON.stringify(want)}`),
|
||||
);
|
||||
};
|
||||
|
||||
console.log(`\nverifying ${dialect} at ${url.replace(/:[^:@]*@/, ":***@")}\n`);
|
||||
|
||||
const users = new UserRepository(context);
|
||||
const userId = `verify-${randomUUID()}`;
|
||||
|
||||
// insertReturning: on MySQL this is an insert plus a read inside a transaction.
|
||||
const created = await users.create({
|
||||
id: userId,
|
||||
username: "before",
|
||||
passwordHash: "x",
|
||||
isAdmin: true,
|
||||
});
|
||||
check("insert returns the stored row", created?.username, "before");
|
||||
|
||||
// The one non-identity value encoder in the schema. Booleans are integers in
|
||||
// the sqlite definitions the repositories import, so this asserts that 1/0
|
||||
// survives a round trip through a native boolean column.
|
||||
check("boolean true survives the round trip", created?.isAdmin, true);
|
||||
|
||||
// updateReturning must report the state AFTER the write. Reading first would
|
||||
// return the value the update replaced — silently, with no error.
|
||||
const updated = await users.update(userId, { username: "after" });
|
||||
check("update returns the new value", updated?.username, "after");
|
||||
|
||||
const hosts = new HostRepository(context);
|
||||
const host = await hosts.create({
|
||||
userId,
|
||||
name: "verify",
|
||||
ip: "127.0.0.1",
|
||||
port: 22,
|
||||
username: "root",
|
||||
authType: "password",
|
||||
enableTerminal: true,
|
||||
});
|
||||
check(
|
||||
"autoincrement id came back",
|
||||
typeof host?.id === "number" && host.id > 0,
|
||||
true,
|
||||
);
|
||||
check(
|
||||
"database-assigned createdAt came back",
|
||||
typeof host?.createdAt === "string" && host.createdAt.length > 0,
|
||||
true,
|
||||
);
|
||||
|
||||
// deleteReturning must report the state BEFORE the write. Reading afterwards
|
||||
// would find nothing at all.
|
||||
const settings = new SettingsRepository(context);
|
||||
const prefix = `verify-${randomUUID()}`;
|
||||
await settings.set(`${prefix}-a`, "1");
|
||||
await settings.set(`${prefix}-b`, "2");
|
||||
check(
|
||||
"delete reports the rows it removed",
|
||||
await settings.deleteLike(`${prefix}-%`),
|
||||
2,
|
||||
);
|
||||
check(
|
||||
"and they are actually gone",
|
||||
(await settings.listAll()).filter((row) => row.key.startsWith(prefix)).length,
|
||||
0,
|
||||
);
|
||||
|
||||
await hosts.deleteForUser(userId, host.id);
|
||||
check("host really deleted", await hosts.findById(host.id), null);
|
||||
|
||||
console.log(
|
||||
failures === 0
|
||||
? `\n${dialect}: all checks passed\n`
|
||||
: `\n${dialect}: ${failures} FAILED\n`,
|
||||
);
|
||||
process.exit(failures === 0 ? 0 : 1);
|
||||
Reference in New Issue
Block a user