mirror of
https://github.com/Termix-SSH/Termix.git
synced 2026-08-29 18:31:33 +00:00
ZacharyZcR
8a79e6af53
* groundwork for postgres and mysql backends #1127 made the repository layer dialect-agnostic. This adds the pieces needed to actually target a second engine, as a foundation only — nothing is wired up and sqlite remains the sole runtime path. - DatabaseDialect covers sqlite, postgres and mysql, resolved from DATABASE_DIALECT and defaulting to sqlite so nothing changes for existing deployments or the desktop build - a column kit holding the per-dialect type choices in one file: booleans are integers on sqlite and native elsewhere, autoincrement differs three ways, and MySQL cannot index unbounded TEXT so key columns need varchar - settings and users declared for all three dialects as a proof slice, chosen because between them they use every construct the real schema does - pg and mysql2 added as dependencies The tests build real queries for all three engines without a server, asserting identifier quoting, placeholder style and boolean storage, so the property the repositories depend on is verified rather than assumed. * verify foreign keys and unique constraints port across dialects The first slice only covered plain columns. The real schema also has 92 foreign keys (80 cascade, 12 set null) and 14 unique columns, so the approach is only viable if those survive the port. Adds audit_logs and ssh_folders to the proof slice: one nullable reference with ON DELETE SET NULL, one required reference with ON DELETE CASCADE, a unique column, and an autoincrement surrogate key — which is spelled three different ways underneath (integer primary key autoincrement, serial, int auto_increment). All of it holds. Worth noting for whoever picks this up: getTableConfig is dialect-specific and silently fails on a table from another dialect, so the test uses each engine's own. * generate the postgres and mysql schemas instead of hand-writing them The proof slice showed the constructs port, but left the maintenance question open. Three hand-written copies of 52 tables is the wrong answer: with foreign keys the copies cross-reference each other, so a renamed table has to land in three places consistently or a key silently points at the wrong one. The mapping is mechanical, so a script does it. schema.ts stays the single source of truth and schema.pg.ts / schema.mysql.ts are derived, covering all 52 tables — the column kit and the two-table portable slice are gone, since the generator now holds those decisions. The transforms are the ones the kit enumerated: integer-backed booleans become native, autoincrement keys become serial or int auto_increment, real becomes double precision or double, and any column that is a primary key, is unique, or sits on either end of a foreign key becomes varchar because MySQL cannot index unbounded TEXT. > termix@2.6.0 lint > node scripts/generate-dialect-schema.cjs --check && eslint . /mnt/c/Users/29037/WebstormProjects/Termix/src/backend/database/routes/homepage-favicon-routes.ts 99:12 warning 'err' is defined but never used unused-imports/no-unused-vars /mnt/c/Users/29037/WebstormProjects/Termix/src/backend/database/routes/homepage-ping-routes.ts 123:12 warning 'err' is defined but never used unused-imports/no-unused-vars /mnt/c/Users/29037/WebstormProjects/Termix/src/backend/database/routes/homepage-rss-routes.ts 144:12 warning 'err' is defined but never used unused-imports/no-unused-vars /mnt/c/Users/29037/WebstormProjects/Termix/src/backend/database/routes/session-log-routes.ts 46:16 warning 'canAccessRecording' is defined but never used. Allowed unused vars must match /^_/u unused-imports/no-unused-vars /mnt/c/Users/29037/WebstormProjects/Termix/src/backend/hosts/vault-signer-core.ts 55:12 warning Unexpected any. Specify a different type @typescript-eslint/no-explicit-any 75:13 warning Unexpected any. Specify a different type @typescript-eslint/no-explicit-any /mnt/c/Users/29037/WebstormProjects/Termix/src/backend/tests/hosts/auth-manager.test.ts 18:73 warning Unexpected any. Specify a different type @typescript-eslint/no-explicit-any /mnt/c/Users/29037/WebstormProjects/Termix/src/backend/tests/utils/shared-host-secrets-manager.test.ts 7:6 warning 'SecretRow' is defined but never used. Allowed unused vars must match /^_/u unused-imports/no-unused-vars /mnt/c/Users/29037/WebstormProjects/Termix/src/backend/utils/auth-manager.ts 510:13 warning 'affectedUsers' is assigned a value but never used. Allowed unused vars must match /^_/u unused-imports/no-unused-vars /mnt/c/Users/29037/WebstormProjects/Termix/src/backend/utils/notification-sender.ts 48:12 warning 'firstErr' is defined but never used unused-imports/no-unused-vars /mnt/c/Users/29037/WebstormProjects/Termix/src/ui/api/ssh-file-operations-api.ts 35:10 warning 'buildFileManagerUrl' is defined but never used. Allowed unused vars must match /^_/u unused-imports/no-unused-vars /mnt/c/Users/29037/WebstormProjects/Termix/src/ui/components/folder-style.tsx 61:14 warning Fast refresh only works when a file only exports components. Use a new file to share constants or functions between components react-refresh/only-export-components 116:14 warning Fast refresh only works when a file only exports components. Use a new file to share constants or functions between components react-refresh/only-export-components 121:14 warning Fast refresh only works when a file only exports components. Use a new file to share constants or functions between components react-refresh/only-export-components 149:17 warning Fast refresh only works when a file only exports components. Use a new file to share constants or functions between components react-refresh/only-export-components /mnt/c/Users/29037/WebstormProjects/Termix/src/ui/components/proxmox/ProxmoxDiscoverDialog.tsx 109:19 warning Unexpected any. Specify a different type @typescript-eslint/no-explicit-any 190:19 warning Unexpected any. Specify a different type @typescript-eslint/no-explicit-any /mnt/c/Users/29037/WebstormProjects/Termix/src/ui/features/homepage/HomepageCanvas.tsx 345:15 warning Empty block statement no-empty 388:15 warning Empty block statement no-empty 415:15 warning Empty block statement no-empty /mnt/c/Users/29037/WebstormProjects/Termix/src/ui/features/homepage/dialogs/SingleHostEditForm.tsx 24:6 warning React Hook useEffect has a missing dependency: 'filter'. Either include it or remove the dependency array. If 'setHosts' needs the current value of 'filter', you can also switch to useReducer instead of useState and read 'filter' in the reducer react-hooks/exhaustive-deps /mnt/c/Users/29037/WebstormProjects/Termix/src/ui/features/homepage/widgets/AlertFeedWidget.tsx 93:6 warning React Hook useEffect has a missing dependency: 'fetchData'. Either include it or remove the dependency array react-hooks/exhaustive-deps /mnt/c/Users/29037/WebstormProjects/Termix/src/ui/features/homepage/widgets/CustomApiWidget.tsx 77:6 warning React Hook useEffect has a missing dependency: 'fetchData'. Either include it or remove the dependency array react-hooks/exhaustive-deps /mnt/c/Users/29037/WebstormProjects/Termix/src/ui/features/homepage/widgets/DockerActivityWidget.tsx 50:6 warning React Hook useEffect has a missing dependency: 'fetchData'. Either include it or remove the dependency array react-hooks/exhaustive-deps /mnt/c/Users/29037/WebstormProjects/Termix/src/ui/features/homepage/widgets/DockerWidget.tsx 16:10 warning Fast refresh only works when a file has exports. Move your component(s) to a separate file react-refresh/only-export-components /mnt/c/Users/29037/WebstormProjects/Termix/src/ui/features/homepage/widgets/FileManagerWidget.tsx 16:10 warning Fast refresh only works when a file has exports. Move your component(s) to a separate file react-refresh/only-export-components /mnt/c/Users/29037/WebstormProjects/Termix/src/ui/features/homepage/widgets/HostGridWidget.tsx 61:6 warning React Hook useCallback has a missing dependency: 'hostIds'. Either include it or remove the dependency array react-hooks/exhaustive-deps 61:7 warning React Hook useCallback has a complex expression in the dependency array. Extract it to a separate variable so it can be statically checked react-hooks/exhaustive-deps /mnt/c/Users/29037/WebstormProjects/Termix/src/ui/features/homepage/widgets/MetricsChartWidget.tsx 168:6 warning React Hook useEffect has a missing dependency: 'fetchData'. Either include it or remove the dependency array react-hooks/exhaustive-deps /mnt/c/Users/29037/WebstormProjects/Termix/src/ui/features/homepage/widgets/PingStatusWidget.tsx 79:6 warning React Hook useEffect has a missing dependency: 'fetchAll'. Either include it or remove the dependency array react-hooks/exhaustive-deps 79:7 warning React Hook useEffect has a complex expression in the dependency array. Extract it to a separate variable so it can be statically checked react-hooks/exhaustive-deps /mnt/c/Users/29037/WebstormProjects/Termix/src/ui/features/homepage/widgets/QuickConnectWidget.tsx 64:10 warning Fast refresh only works when a file has exports. Move your component(s) to a separate file react-refresh/only-export-components /mnt/c/Users/29037/WebstormProjects/Termix/src/ui/features/homepage/widgets/RecentActivityWidget.tsx 82:6 warning React Hook useEffect has a missing dependency: 'fetchData'. Either include it or remove the dependency array react-hooks/exhaustive-deps 82:17 warning React Hook useEffect has a complex expression in the dependency array. Extract it to a separate variable so it can be statically checked react-hooks/exhaustive-deps /mnt/c/Users/29037/WebstormProjects/Termix/src/ui/features/homepage/widgets/SshQuickConnectWidget.tsx 67:6 warning React Hook useCallback has a missing dependency: 'hostIds'. Either include it or remove the dependency array react-hooks/exhaustive-deps 67:7 warning React Hook useCallback has a complex expression in the dependency array. Extract it to a separate variable so it can be statically checked react-hooks/exhaustive-deps 99:17 warning 'online' is assigned a value but never used. Allowed unused vars must match /^_/u unused-imports/no-unused-vars /mnt/c/Users/29037/WebstormProjects/Termix/src/ui/features/homepage/widgets/SshTerminalWidget.tsx 17:10 warning Fast refresh only works when a file has exports. Move your component(s) to a separate file react-refresh/only-export-components /mnt/c/Users/29037/WebstormProjects/Termix/src/ui/features/homepage/widgets/SystemOverviewWidget.tsx 72:6 warning React Hook useEffect has a missing dependency: 'fetchData'. Either include it or remove the dependency array react-hooks/exhaustive-deps /mnt/c/Users/29037/WebstormProjects/Termix/src/ui/features/homepage/widgets/TunnelWidget.tsx 15:10 warning Fast refresh only works when a file has exports. Move your component(s) to a separate file react-refresh/only-export-components /mnt/c/Users/29037/WebstormProjects/Termix/src/ui/features/host-metrics/cards/CpuCard.tsx 14:10 warning 'computeChartData' is defined but never used. Allowed unused vars must match /^_/u unused-imports/no-unused-vars /mnt/c/Users/29037/WebstormProjects/Termix/src/ui/sidebar/FolderPathPicker.tsx 15:17 warning Fast refresh only works when a file only exports components. Use a new file to share constants or functions between components react-refresh/only-export-components 22:17 warning Fast refresh only works when a file only exports components. Use a new file to share constants or functions between components react-refresh/only-export-components /mnt/c/Users/29037/WebstormProjects/Termix/src/ui/sidebar/HostsPanel.tsx 601:52 warning Unexpected any. Specify a different type @typescript-eslint/no-explicit-any ✖ 44 problems (0 errors, 44 warnings) now fails if the generated files are out of date, so editing the schema without regenerating cannot reach main. * select durability behaviour per dialect, and document the backends The onWrite hook every repository receives exists to serialise the in-memory SQLite database back to its encrypted file. On a client-server engine a committed write is already durable and there is nothing to flush, so the factory now installs no hook at all rather than one that does nothing. Repositories call it as this.onWrite?.(), so none of the 43 of them change. Also adds docs/database-backends.md, mostly to be explicit about encryption, which is the part most likely to be misread. Field-level encryption is identical on all three engines and covers every credential. Whole-file encryption has no equivalent on Postgres or MySQL, so host names, snippet contents, audit entries and backups are only as protected as the storage underneath them — that is the operator's responsibility and the docs should not imply otherwise. * generate DDL with drizzle-kit, and give settings a synchronous path Two of the three remaining blockers. DDL: db/index.ts hand-writes 67 CREATE TABLE statements and 122 ADD COLUMN migrations, all in SQLite dialect. Rather than port them, drizzle-kit now generates migrations from the schema modules — 817 lines for Postgres, 869 for MySQL, with the type mapping already correct because the schemas it reads are themselves generated. > termix@2.6.0 schema:migrations > drizzle-kit generate --config=drizzle.config.pg.ts && drizzle-kit generate --config=drizzle.config.mysql.ts Reading config file '/mnt/c/Users/29037/WebstormProjects/Termix/drizzle.config.pg.ts' 52 tables alert_firings 11 columns 0 indexes 2 fks alert_rule_channels 3 columns 0 indexes 2 fks alert_rules 11 columns 0 indexes 2 fks api_keys 9 columns 0 indexes 1 fks audit_logs 13 columns 0 indexes 1 fks c2s_tunnel_presets 8 columns 0 indexes 1 fks command_history 5 columns 0 indexes 2 fks dashboard_service_links 8 columns 0 indexes 1 fks dismissed_alerts 4 columns 0 indexes 1 fks file_manager_pinned 6 columns 0 indexes 2 fks file_manager_recent 6 columns 0 indexes 2 fks file_manager_shortcuts 6 columns 0 indexes 2 fks homepage_items 9 columns 0 indexes 1 fks homepage_layouts 4 columns 0 indexes 1 fks host_access 11 columns 0 indexes 5 fks host_health_checks 7 columns 0 indexes 2 fks host_health_history 8 columns 0 indexes 2 fks host_metrics_history 8 columns 0 indexes 1 fks host_metrics_preferences 6 columns 0 indexes 2 fks ssh_data 94 columns 0 indexes 6 fks network_topology 5 columns 0 indexes 1 fks notification_channels 7 columns 0 indexes 1 fks opkssh_tokens 12 columns 0 indexes 2 fks recent_activity 6 columns 0 indexes 2 fks roles 8 columns 0 indexes 0 fks session_recordings 15 columns 0 indexes 3 fks session_share_participants 6 columns 0 indexes 2 fks session_shares 15 columns 0 indexes 3 fks sessions 11 columns 0 indexes 1 fks settings 2 columns 0 indexes 0 fks shared_host_secrets 15 columns 0 indexes 3 fks snippet_access 8 columns 0 indexes 4 fks snippet_folders 8 columns 0 indexes 1 fks snippets 11 columns 0 indexes 1 fks ssh_credential_usage 5 columns 0 indexes 3 fks ssh_credentials 21 columns 0 indexes 1 fks ssh_folders 9 columns 0 indexes 2 fks sso_providers 8 columns 0 indexes 0 fks sync_tombstones 5 columns 0 indexes 1 fks termix_identities 6 columns 0 indexes 1 fks termix_identity_ca 8 columns 0 indexes 2 fks termix_identity_keys 12 columns 0 indexes 3 fks tmux_session_tags 6 columns 0 indexes 2 fks transfer_recent 7 columns 0 indexes 3 fks trusted_devices 8 columns 0 indexes 1 fks user_open_tabs 9 columns 0 indexes 2 fks user_preferences 23 columns 0 indexes 1 fks user_roles 5 columns 0 indexes 3 fks users 20 columns 0 indexes 0 fks vault_profiles 18 columns 0 indexes 1 fks vault_tokens 8 columns 0 indexes 2 fks webauthn_credentials 12 columns 0 indexes 1 fks No schema changes, nothing to migrate 😴 Reading config file '/mnt/c/Users/29037/WebstormProjects/Termix/drizzle.config.mysql.ts' Reading schema files: /mnt/c/Users/29037/WebstormProjects/Termix/src/backend/database/db/schema.mysql.ts 52 tables alert_firings 11 columns 0 indexes 2 fks alert_rule_channels 3 columns 0 indexes 2 fks alert_rules 11 columns 0 indexes 2 fks api_keys 9 columns 0 indexes 1 fks audit_logs 13 columns 0 indexes 1 fks c2s_tunnel_presets 8 columns 0 indexes 1 fks command_history 5 columns 0 indexes 2 fks dashboard_service_links 8 columns 0 indexes 1 fks dismissed_alerts 4 columns 0 indexes 1 fks file_manager_pinned 6 columns 0 indexes 2 fks file_manager_recent 6 columns 0 indexes 2 fks file_manager_shortcuts 6 columns 0 indexes 2 fks homepage_items 9 columns 0 indexes 1 fks homepage_layouts 4 columns 0 indexes 1 fks host_access 11 columns 0 indexes 5 fks host_health_checks 7 columns 0 indexes 2 fks host_health_history 8 columns 0 indexes 2 fks host_metrics_history 8 columns 0 indexes 1 fks host_metrics_preferences 6 columns 0 indexes 2 fks ssh_data 94 columns 0 indexes 6 fks network_topology 5 columns 0 indexes 1 fks notification_channels 7 columns 0 indexes 1 fks opkssh_tokens 12 columns 0 indexes 2 fks recent_activity 6 columns 0 indexes 2 fks roles 8 columns 0 indexes 0 fks session_recordings 15 columns 0 indexes 3 fks session_share_participants 6 columns 0 indexes 2 fks session_shares 15 columns 0 indexes 3 fks sessions 11 columns 0 indexes 1 fks settings 2 columns 0 indexes 0 fks shared_host_secrets 15 columns 0 indexes 3 fks snippet_access 8 columns 0 indexes 4 fks snippet_folders 8 columns 0 indexes 1 fks snippets 11 columns 0 indexes 1 fks ssh_credential_usage 5 columns 0 indexes 3 fks ssh_credentials 21 columns 0 indexes 1 fks ssh_folders 9 columns 0 indexes 2 fks sso_providers 8 columns 0 indexes 0 fks sync_tombstones 5 columns 0 indexes 1 fks termix_identities 6 columns 0 indexes 1 fks termix_identity_ca 8 columns 0 indexes 2 fks termix_identity_keys 12 columns 0 indexes 3 fks tmux_session_tags 6 columns 0 indexes 2 fks transfer_recent 7 columns 0 indexes 3 fks trusted_devices 8 columns 0 indexes 1 fks user_open_tabs 9 columns 0 indexes 2 fks user_preferences 23 columns 0 indexes 1 fks user_roles 5 columns 0 indexes 3 fks users 20 columns 0 indexes 0 fks vault_profiles 18 columns 0 indexes 1 fks vault_tokens 8 columns 0 indexes 2 fks webauthn_credentials 12 columns 0 indexes 1 fks No schema changes, nothing to migrate 😴 regenerates both. Settings: 27 call sites read settings synchronously, during startup and inside request handlers. better-sqlite3 can do that; Postgres and MySQL cannot, and making all 27 async would push await through code that has no reason to be asynchronous. Settings are a handful of rarely-changing rows read constantly, so they are cached in full — primed at startup, kept in step by SettingsRepository on every set/delete/deleteLike. SQLite keeps reading the database directly and stays authoritative; only the other engines use the cache. Opening a connection is still not done. DatabaseContext.drizzle is typed as BetterSQLite3Database and 43 repositories depend on that inference; the three drizzle instance types are not interchangeable, so widening it is a design decision rather than a mechanical change. * exclude drizzle-kit output from prettier The generated migrations and snapshots are tool output; their formatting is drizzle-kit's to decide, and prettier cannot parse the .sql files at all. * absorb the RETURNING gap so mysql stays reachable MySQL has no RETURNING clause and drizzle's mysql-core does not expose the method, while 156 call sites here read the result of a write. That is the real blocker for MySQL, not the connection layer. Classifying those call sites showed the split is favourable: 92 of them only read .length, which every engine reports — as a returned array on sqlite and postgres, as affectedRows on MySQL. rowsAffected() reads both shapes, so those sites need no change in query shape. insertedId() does the same for the autoincrement key, which MySQL reports as insertId. What is left is the ~34 sites that genuinely consume the returned rows. Those cannot be emulated without reading first, which needs a transaction to stay correct under concurrency, so they will be handled individually rather than behind a helper that quietly adds a round trip. supportsReturning() is the seam for that. Identifying the mysql2 result by its own fields rather than by array shape matters: it hands back [ResultSetHeader, fields], which is an array, so shape alone cannot tell it apart from a returning() result. * name the portable database type, and open remote connections Two pieces of the connection layer. drizzle's three Database classes share no base class and their signatures are incompatible, so there is no honest type that covers all three: a union is not callable and a generic would have to be threaded through 43 repositories and every method on them. DatabaseContext.drizzle is now PortableDatabase, still the SQLite type underneath, but named and documented as the deliberate approximation it is. What makes it safe is that the equivalence is asserted in multi-dialect.test.ts rather than assumed, and the one place the surfaces truly differ — RETURNING — is handled explicitly in mutation-result.ts. connect.ts opens Postgres and MySQL from DATABASE_URL, with the schema module and driver imported lazily so neither is loaded on a SQLite deployment. The URL scheme is checked against the configured dialect first: a postgres:// URL with DATABASE_DIALECT=mysql otherwise surfaces as a driver error deep in a stack that never mentions the actual misconfiguration. * open postgres and mysql at startup * count writes without RETURNING * read affected rows without RETURNING on mysql * insert without RETURNING, and split the sync transactions * stop pretending the generated schemas are used at runtime * run the dialect checks in CI * mysql rejects a bare CURRENT_TIMESTAMP default on text * make the read-back mismatch loud, and stop the next bare returning() * run the repository tests on the real schema * skip the byte-level assertions off sqlite * move generated ids past the seeded ones * keep the export order the same on every engine * stop reading better-sqlite3 fields off every write * read counts as numbers, not whatever the driver returns * make the fixture usable against a live server * upsert on the engine that has no ON CONFLICT * run the repository suite on all three engines in CI * mysql cannot index a text column without a length * document how to actually run on postgres or mysql * keep the sqlite-era migrations off the other engines * concat strings in a way mysql agrees with * run every repository test on every engine * bound how long replicas can disagree about settings * generate the sqlite migrations alongside the others