Termix/readme/DATABASE-LAYER-PHASE-0-AUDIT.md

# Database Layer Refactor Phase 0 Audit

Status: Draft  
Branch: `feature/database-layer-refactor`  
Purpose: Establish the current database access inventory, domain map, sensitive field map, and first implementation boundaries before changing runtime persistence.

## 1. Scope

Phase 0 does not change runtime behavior.

It produces the evidence needed for the next implementation phases:

- where database access currently happens
- which modules write directly to the database
- which tables belong to which product domains
- which fields are sensitive
- which direct writes are risky under the current in-memory snapshot model
- which domains should move first into repositories

## 2. Current Evidence

Commands used for the initial audit:

```bash
rg -n "getDb\\(|getSqlite\\(|DatabaseSaveTrigger|SimpleDBOps|db\\.\\$client|\\.prepare\\(" src/backend
rg -n "getDb\\(\\)\\.(insert|update|delete)|await db\\.(insert|update|delete)|db\\.(insert|update|delete)|\\.\\$client\\.prepare\\(\\\"(INSERT|UPDATE|DELETE)|\\.prepare\\(\\\"(INSERT|UPDATE|DELETE)|DatabaseSaveTrigger\\.triggerSave|DatabaseSaveTrigger\\.forceSave" src/backend
rg -n "export const .* = sqliteTable\\(" src/backend/database/db/schema.ts
```

High-level findings:

- Database infrastructure is concentrated in `src/backend/database/db/index.ts`.
- Business database access is spread across route modules, SSH modules, utilities, and auth helpers.
- `SimpleDBOps` is not the only write path.
- There are many direct Drizzle writes and raw SQLite writes.
- Some direct writes manually trigger `DatabaseSaveTrigger`; many write paths do not.
- Schema is SQLite-specific through `sqliteTable` and manual `CREATE TABLE IF NOT EXISTS` / `addColumnIfNotExists`.

## 3. Database Access Hotspots

The most database-heavy files by audit hits:

| File                                                  | Approx. hits | Notes                                                           |
| ----------------------------------------------------- | -----------: | --------------------------------------------------------------- |
| `src/backend/database/db/index.ts`                    |           75 | database init, schema creation, ad-hoc migration, snapshot save |
| `src/backend/database/routes/alert-rules-routes.ts`   |           64 | alert rules, channels, firings, raw SQL deletes                 |
| `src/backend/database/routes/users.ts`                |           54 | users, settings, OIDC/GitHub settings, admin flows              |
| `src/backend/database/database.ts`                    |           27 | import/export, legacy SQLite handling                           |
| `src/backend/ssh/host-metrics.ts`                     |           26 | metrics connection and settings reads                           |
| `src/backend/database/routes/user-settings-routes.ts` |           16 | user settings                                                   |
| `src/backend/dashboard.ts`                            |           15 | dashboard aggregation reads                                     |
| `src/backend/ssh/docker.ts`                           |           14 | host/credential reads for Docker SSH                            |
| `src/backend/ssh/managers/health.ts`                  |           13 | host health checks                                              |
| `src/backend/ssh/alert-engine.ts`                     |           13 | alert evaluation and firing state                               |
| `src/backend/utils/user-crypto.ts`                    |           12 | settings writes for crypto metadata                             |
| `src/backend/ssh/host-metrics-settings-routes.ts`     |           12 | metrics settings                                                |
| `src/backend/ssh/tmux-monitor.ts`                     |           11 | tmux session tags                                               |
| `src/backend/guacamole/routes.ts`                     |           11 | Guacamole config/routes                                         |
| `src/backend/database/routes/host.ts`                 |           10 | host operations and related rows                                |

This confirms the refactor must be domain-by-domain. A mechanical replacement of `getDb()` would be noisy and unsafe.

## 4. Direct Write Risk Inventory

Under the current architecture, a direct write is risky when it bypasses `SimpleDBOps` and does not trigger `DatabaseSaveTrigger`.

### 4.1 Direct Writes That Need Repository Ownership

These areas perform direct writes and should move behind repositories/services:

| Area                  | Representative files                                               | Examples                                                        |
| --------------------- | ------------------------------------------------------------------ | --------------------------------------------------------------- |
| users/settings/auth   | `routes/users.ts`, `utils/auth-manager.ts`, `utils/user-crypto.ts` | sessions, trusted devices, OIDC settings, registration settings |
| RBAC/sharing          | `routes/rbac.ts`, `utils/shared-credential-manager.ts`             | roles, host access, snippet access, shared credentials          |
| hosts/credentials     | `routes/host.ts`, `routes/credentials.ts`, `host-resolver.ts`      | host access cleanup, credential usage                           |
| file manager metadata | `host-file-manager-bookmark-routes.ts`                             | recent, pinned, shortcuts                                       |
| alerts                | `alert-rules-routes.ts`, `ssh/alert-engine.ts`                     | channels, rules, firings                                        |
| metrics               | `host-metrics-preferences-routes.ts`, `managers/health.ts`         | preferences, health checks, history                             |
| terminal logs         | `terminal-session-manager.ts`                                      | session recording metadata                                      |
| tmux                  | `tmux-monitor.ts`                                                  | tmux session tags                                               |
| import/export         | `database/database.ts`                                             | SQLite import and forced save                                   |
| open tabs             | `routes/open-tabs.ts`                                              | tab persistence and cleanup                                     |
| API keys              | `user-api-key-routes.ts`, `utils/auth-manager.ts`                  | API key create/delete/last-used                                 |
| SSO and identity      | `sso-provider-routes.ts`, `termix-id.ts`                           | providers, identity keys/CA                                     |

### 4.2 Immediate Compatibility Rule

Until a domain is migrated to repositories:

- Every direct write must either be moved into a repository or explicitly trigger persistence in the old runtime.
- New code should not add direct `getDb()` writes outside infrastructure or repositories.
- The draft branch should add an enforcement check before Phase 8, not immediately, because the current codebase still violates the target rule widely.

## 5. Table Domain Map

### 5.1 Identity and Authentication

| Tables                 | Notes                                        |
| ---------------------- | -------------------------------------------- |
| `users`                | local/OIDC users, TOTP fields, password hash |
| `sessions`             | JWT sessions                                 |
| `trusted_devices`      | remembered devices                           |
| `api_keys`             | API token hashes/prefixes                    |
| `sso_providers`        | configured SSO providers                     |
| `termix_identities`    | Termix identity records                      |
| `termix_identity_keys` | public/private identity key metadata         |
| `termix_identity_ca`   | CA material, private key is sensitive        |

Suggested repository:

- `userRepository`
- `sessionRepository`
- `trustedDeviceRepository`
- `apiKeyRepository`
- `ssoProviderRepository`
- `termixIdentityRepository`

### 5.2 Hosts and Credentials

| Tables                 | Notes                                            |
| ---------------------- | ------------------------------------------------ |
| `ssh_data`             | primary host table, many config JSON/text fields |
| `ssh_credentials`      | reusable credentials                             |
| `ssh_credential_usage` | usage history                                    |
| `ssh_folders`          | folder metadata                                  |
| `host_access`          | sharing/RBAC access rows                         |
| `shared_credentials`   | encrypted shared credential material             |
| `network_topology`     | topology graph/config                            |

Suggested repository:

- `hostRepository`
- `credentialRepository`
- `credentialUsageRepository`
- `hostAccessRepository`
- `sharedCredentialRepository`
- `hostFolderRepository`
- `networkTopologyRepository`

### 5.3 RBAC

| Tables           | Notes                    |
| ---------------- | ------------------------ |
| `roles`          | role definitions         |
| `user_roles`     | user to role assignments |
| `host_access`    | host-level grants        |
| `snippet_access` | snippet-level grants     |

Suggested repository:

- `roleRepository`
- `accessRepository`

### 5.4 File Manager

| Tables                   | Notes                              |
| ------------------------ | ---------------------------------- |
| `file_manager_recent`    | recently opened paths              |
| `file_manager_pinned`    | pinned paths                       |
| `file_manager_shortcuts` | saved shortcuts                    |
| `transfer_recent`        | host-to-host transfer destinations |

Suggested repository:

- `fileManagerRepository`
- `transferRecentRepository`

### 5.5 Snippets

| Tables            | Notes                   |
| ----------------- | ----------------------- |
| `snippets`        | command snippets        |
| `snippet_folders` | snippet folder metadata |
| `snippet_access`  | snippet sharing         |

Suggested repository:

- `snippetRepository`

### 5.6 Runtime Metadata and Audit

| Tables               | Notes                                                  |
| -------------------- | ------------------------------------------------------ |
| `audit_logs`         | append-only audit trail                                |
| `session_recordings` | terminal recording metadata, log content is file-based |
| `recent_activity`    | host activity feed                                     |
| `command_history`    | terminal command history                               |
| `user_open_tabs`     | UI restore state; currently cleared on startup         |
| `user_preferences`   | per-user UI/preferences                                |
| `settings`           | global settings                                        |

Suggested repository:

- `auditRepository`
- `sessionRecordingRepository`
- `activityRepository`
- `commandHistoryRepository`
- `openTabsRepository`
- `userPreferencesRepository`
- `settingsRepository`

### 5.7 Metrics and Alerts

| Tables                     | Notes                      |
| -------------------------- | -------------------------- |
| `host_metrics_preferences` | metrics layout/preferences |
| `host_health_checks`       | health check definitions   |
| `host_health_history`      | health check results       |
| `host_metrics_history`     | metrics history            |
| `alert_rules`              | alert definitions          |
| `notification_channels`    | webhook/ntfy/etc config    |
| `alert_rule_channels`      | rule/channel joins         |
| `alert_firings`            | firing/ack state           |
| `dismissed_alerts`         | dismissed system alerts    |

Suggested repository:

- `metricsRepository`
- `healthCheckRepository`
- `alertRepository`
- `notificationRepository`

### 5.8 Integrations and Feature Config

| Tables                    | Notes                                   |
| ------------------------- | --------------------------------------- |
| `c2s_tunnel_presets`      | tunnel preset config                    |
| `opkssh_tokens`           | OPKSSH cert/private key cache           |
| `vault_profiles`          | Vault profile config, mostly non-secret |
| `vault_tokens`            | Vault cert/private key cache            |
| `dashboard_service_links` | dashboard links                         |
| `homepage_items`          | homepage widgets/items                  |
| `homepage_layouts`        | homepage layouts                        |
| `tmux_session_tags`       | tmux tag metadata                       |

Suggested repository:

- `tunnelPresetRepository`
- `opksshTokenRepository`
- `vaultRepository`
- `dashboardRepository`
- `homepageRepository`
- `tmuxRepository`

## 6. Sensitive Field Map

The current explicit `FieldCrypto` map encrypts:

| Table                | Fields                                                                                                                                                                          |
| -------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `users`              | `passwordHash`, `clientSecret`, `totpSecret`, `totpBackupCodes`, `oidcIdentifier`                                                                                               |
| `ssh_data`           | `password`, `key`, `keyPassword`, `sudoPassword`, `autostartPassword`, `autostartKey`, `autostartKeyPassword`, `socks5Password`, `rdpPassword`, `vncPassword`, `telnetPassword` |
| `ssh_credentials`    | `password`, `privateKey`, `keyPassword`, `key`, `publicKey`                                                                                                                     |
| `opkssh_tokens`      | `sshCert`, `privateKey`                                                                                                                                                         |
| `termix_identity_ca` | `privateKey`                                                                                                                                                                    |
| `vault_tokens`       | `sshCert`, `privateKey`                                                                                                                                                         |

Additional sensitive fields by table semantics:

| Table                   | Fields / reason                                                                                                                                                |
| ----------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `ssh_credentials`       | `systemPassword`, `systemKey`, `systemKeyPassword` are system-key encrypted credential copies                                                                  |
| `shared_credentials`    | `encryptedUsername`, `encryptedAuthType`, `encryptedPassword`, `encryptedKey`, `encryptedKeyPassword`, `encryptedKeyType` are already encrypted payload fields |
| `api_keys`              | `tokenHash` is not plaintext but is authentication material; `tokenPrefix` may remain plaintext for display                                                    |
| `settings`              | some keys may hold provider secrets or reset codes; repository must classify by key                                                                            |
| `notification_channels` | config may include webhook URLs/tokens; treat config as sensitive unless split                                                                                 |
| `alert_rules`           | rule definitions are usually not secret but can include host/resource metadata                                                                                 |
| `homepage_items`        | widget config can include URLs/API config; classify per widget type                                                                                            |
| `c2s_tunnel_presets`    | config can include connection details; review before plaintext external DB storage                                                                             |
| `termix_identity_keys`  | inspect key material fields before migration; public keys are not secret but private material must never be plaintext                                          |
| `vault_profiles`        | current comments say profile fields are non-secret; keep that invariant explicit                                                                               |

Open privacy decision:

- `ssh_data.ip`, domain fields, usernames, folders, and tags are queryable today.
- Encrypting them improves confidentiality but breaks search/filter/sort unless blind indexes are added.
- Recommended initial migration: keep them plaintext and document privacy implications; add optional privacy mode later.

## 7. Repository Migration Order

Use a vertical slice instead of broad replacement.

### 7.1 First Slice

1. `settingsRepository`
2. `userRepository`
3. `sessionRepository`
4. `hostRepository`
5. `credentialRepository`

Reasons:

- Covers the app's boot/login/core host management path.
- Exercises field encryption.
- Exercises per-user data unlock requirements.
- Exercises transaction and migration behavior.
- Builds reusable patterns for the rest of the backend.

### 7.2 Second Slice

1. `hostAccessRepository`
2. `roleRepository`
3. `sharedCredentialRepository`
4. `auditRepository`
5. `userPreferencesRepository`

Reasons:

- Completes permission and sharing boundaries.
- Removes high-risk direct writes in admin/RBAC flows.
- Moves audit to an append-only repository.

### 7.3 Third Slice

1. `snippetRepository`
2. `fileManagerRepository`
3. `metricsRepository`
4. `alertRepository`
5. `homepageRepository`

Reasons:

- These are broad feature domains with many routes.
- They should reuse patterns from the core slices.

### 7.4 Fourth Slice

1. `vaultRepository`
2. `opksshTokenRepository`
3. `termixIdentityRepository`
4. `tunnelPresetRepository`
5. `tmuxRepository`

Reasons:

- Sensitive token/key cache domains need careful encryption tests.
- Some data is transient and should have retention cleanup.

## 8. Adapter Design Notes

The adapter boundary should expose:

```ts
interface DatabaseAdapter {
  dialect: "sqlite" | "postgres" | "mysql";
  connect(): Promise<void>;
  close(): Promise<void>;
  migrate(): Promise<void>;
  transaction<T>(fn: (tx: DatabaseTransaction) => Promise<T>): Promise<T>;
}
```

The repository layer should not depend on `better-sqlite3`.

For Drizzle, likely options:

- keep dialect-specific Drizzle clients internally
- expose repositories instead of exposing the raw Drizzle client
- keep schema definitions close to migrations, not route handlers

Important: do not make route modules import dialect-specific schema objects after migration.

## 9. Compatibility Shims

During the migration, avoid a flag day.

Add a compatibility database module that lets old code continue to run while new repositories are introduced:

```text
legacy getDb() path
new adapter/repository path
```

Rules:

- New modules use repositories only.
- Migrated modules must not fall back to `getDb()`.
- Legacy path is removed only after all domains move.

## 10. Phase 1 Entry Criteria

Before implementing the adapter skeleton:

- This audit document exists.
- The sensitive field map is reviewed.
- The first vertical slice is accepted.
- The draft PR remains draft.
- No runtime behavior has changed.

## 11. Phase 1 Deliverables

Recommended first implementation PR on this branch:

- `src/backend/database/runtime/config.ts`
- `src/backend/database/runtime/adapter.ts`
- `src/backend/database/runtime/sqlite-adapter.ts`
- `src/backend/database/repositories/settings-repository.ts`
- `src/backend/database/repositories/user-repository.ts`
- `src/backend/database/repositories/session-repository.ts`
- `src/backend/database/repositories/host-repository.ts`
- `src/backend/database/repositories/credential-repository.ts`
- `src/backend/database/repositories/field-encryption-boundary.ts`
- `src/backend/database/repositories/current-settings-repository.ts`
- tests for config parsing and SQLite adapter boot

Started:

- runtime config parser
- SQLite adapter skeleton
- migration metadata table bootstrap
- `SettingsRepository` skeleton and tests
- `UserRepository` and `SessionRepository` skeletons and tests
- `HostRepository` and `CredentialRepository` skeletons and tests
- `FieldEncryptionBoundary` skeleton and tests
- first settings route slice wired through `SettingsRepository`
- user settings route direct `settings` table access moved behind
  `SettingsRepository`
- host metrics settings route direct `settings` table access moved behind
  `SettingsRepository`
- ACME SSL settings route direct `settings` table access moved behind
  `SettingsRepository`
- terminal route direct `settings` table access moved behind
  `SettingsRepository`
- tailscale route direct `settings` table access moved behind
  `SettingsRepository`
- Guacamole route and WebSocket server direct `settings` table access moved
  behind the current settings repository boundary
- auth token expiry and terminal session timeout settings reads moved behind the
  current settings repository boundary
- open tabs, TOTP, and LDAP auth route settings reads moved behind the current
  settings repository boundary
- host metrics polling settings reads moved behind the current settings
  repository boundary
- backend startup settings reads moved behind the current settings repository
  boundary
- user deletion cleanup now removes per-user settings through
  `SettingsRepository.deleteLike`
- password reset route reset code and temporary token settings access moved
  behind `SettingsRepository`
- OIDC utility legacy config fallback reads `oidc_config` through
  `SettingsRepository`
- user route registration/password flags and OIDC config administration moved
  behind the current settings repository boundary
- OIDC authorize/callback temporary state and auto-provision reads moved behind
  the current settings repository boundary
- user login settings reads moved behind the current settings repository
  boundary, completing direct `settings` access cleanup in `routes/users.ts`
- user encryption metadata in `utils/user-crypto.ts` moved behind the current
  settings repository boundary
- database startup and schema migration defaults in `database/db/index.ts`
  moved to local raw settings helpers
- database import/export settings handling in `database/database.ts` moved to
  local helper boundaries
- core `auth-manager.ts` session create/read/update/revoke/list paths started
  using the current session repository boundary
- remaining `auth-manager.ts` session cleanup/middleware/logout paths and
  `user-session-routes.ts` single-session lookup moved behind the current
  session repository boundary
- current user repository factory/write-save hook added, and
  `user-admin-routes.ts` list/admin promotion/admin removal/admin-create user
  paths moved behind the current user repository boundary
- low-risk `routes/users.ts` current-user lookup and admin gate checks moved
  behind the current user repository boundary
- user registration, self-delete, password change hash updates, and admin
  delete-user lookup paths in `routes/users.ts` moved behind the current user
  repository boundary, with first-user admin creation kept transactional inside
  `UserRepository`
- traditional login username lookup and `auth-manager.ts` admin user checks
  moved behind the current user repository boundary
- GitHub and standard OIDC callback user lookup/create/rollback/profile/admin
  sync writes moved behind the current user repository boundary, removing direct
  Drizzle `users` table access from `routes/users.ts`, `user-admin-routes.ts`,
  and `auth-manager.ts`
- API key create/list/delete and API key authentication last-used updates moved
  behind the current API key repository boundary
- trusted device check/add/remove and TOTP trusted-device cleanup moved behind
  the current trusted device repository boundary
- user session routes moved user/admin lookups and admin session username
  enrichment behind the current user repository boundary
- vault admin checks, Termix ID audit username lookup, permission manager admin
  checks, and user data export user lookup moved behind the current user
  repository boundary
- SSH credential OIDC username expansion and tmux monitor audit actor username
  lookup moved behind the current user repository boundary
- user settings route admin checks and audit actor username lookups moved behind
  the current user repository boundary
- ACME SSL route admin checks and audit actor username lookups moved behind the
  current user repository boundary
- audit log route admin checks moved behind the current user repository boundary
- OIDC account link/unlink route user lookups and OIDC field updates moved
  behind the current user repository boundary
- password reset route user lookups, password hash updates, and TOTP reset
  fields moved behind the current user repository boundary
- user deletion helper now removes sessions through the current session
  repository and the final user record through the current user repository
- snippet create/update/delete audit actor username lookups moved behind the
  current user repository boundary
- LDAP login existing-user lookup, encryption rollback delete, admin sync, and
  display-name sync moved behind the current user repository boundary
- TOTP setup/enable/disable/backup-code/login verification user updates and
  session revocation moved behind the current user/session repository
  boundaries
- RBAC host sharing, role assignment, and snippet sharing target-user existence
  checks moved behind the current user repository boundary, with existing
  RBAC/snippet owner username joins retained
- RBAC role list/create/update/delete, user-role assignment/removal/listing,
  and shared host/snippet role-id lookups moved behind the current role
  repository boundary, with existing RBAC/share credential joins retained
- permission manager role permission aggregation, role-id lookups for shared
  host access, and admin role checks moved behind the current role repository
  boundary
- RBAC host/snippet access-list read models moved behind the current RBAC access
  repository boundary, and snippet route shared-access role-id lookups moved
  behind the current role repository boundary
- RBAC shared host/shared snippet read models and the main snippet
  shared-snippet read model moved behind the current RBAC access repository
  boundary
- RBAC host/snippet access grant, revoke, and direct host-access credential
  override writes moved behind the current RBAC access repository boundary,
  with shared credential material creation retained in the existing manager
- permission manager host-access expiration cleanup, shared host-access lookup,
  and last-access timestamp updates moved behind the current RBAC access
  repository boundary
- repository rollout guard added through `DATABASE_LAYER_REPOSITORY_ROLLOUT`
  for the migrated settings/users/sessions/API-key/trusted-device/role/RBAC-access
  slice

Keep it small. Do not wire host or credential routes into the new repositories in
the same first implementation commit.

## 12. Current Unknowns

- Exact Drizzle multi-dialect strategy needs a spike.
- The project may need a migration generator or a custom migration runner.
- Some raw SQL in `routes/users.ts`, `alert-rules-routes.ts`, and `db/index.ts` must be rewritten or isolated.
- `settings` contains mixed public and sensitive values; key-level classification is required.
- `homepage_items.config`, `notification_channels.config`, and tunnel configs may contain embedded secrets.
- Legacy encrypted snapshot fixtures need to be created before migration implementation.

## 13. Decision Log

- Default upgrade target should be persistent SQLite, not PostgreSQL/MySQL.
- PostgreSQL/MySQL migration should be explicit and initially manual/experimental.
- Runtime SSH/WebSocket/tunnel state remains memory-only.
- Field-level encryption is mandatory for every database backend.
- Field-level encryption must use a stable record id; temporary encryption
  contexts are forbidden for newly written repository data.
- Repository migration should start with settings/users/sessions/hosts/credentials.