feat: rebuild desktop sync as linked server accounts, fix packaged plugin bundles and UI scaling

This commit is contained in:
LukeGus committed 2026-09-26 00:41:25 -05:00
1 parent 7c02daccf4
commit 6cd9a2846d
242 files changed
+39127 -8558

No files matched your search

+77 -29
View File
@@ -195,7 +195,7 @@ regenerated `schema.pg.ts`/`schema.mysql.ts` plus drizzle migrations.
**What stays in core.** After 2.9.0, `schema.ts`, the SQLite DDL in
`db/index.ts`, `performance-indexes.ts`, the repository factory and the
drizzle migrations hold only these 36 tables. Anything else is a plugin's,
drizzle migrations hold only these 38 tables. Anything else is a plugin's,
under `p_<id>_`.
| Area | Tables |
@@ -206,7 +206,7 @@ under `p_<id>_`.
| Credentials | `ssh_credentials`, `ssh_credential_usage` |
| Sharing and RBAC | `roles`, `user_roles`, `host_access`, `folder_access`, `credential_access`, `shared_host_secrets`, `shared_host_auth_overrides`, `shared_credential_secrets`, `rbac_known_permissions`, `rbac_applied_defaults` |
| Audit | `audit_logs` |
| Sync | `sync_tombstones` |
| Sync | `sync_records`, `sync_conflicts`, `sync_link` |
| Plugin runtime | `plugins`, `plugin_permission_grants`, `plugin_registries`, `plugin_install_counts`, `plugin_storage`, `plugin_settings`, `plugin_migrations` |
A few core tables still carry columns a plugin reads through its own host or
@@ -222,24 +222,72 @@ they are sealed with this server's keys. An import reads each table by its
`p_` name, or by the legacy name it adopted for a file written before 2.9.
**Sync entities are registered, not hardcoded.** `ctx.sync.registerEntity`
adds an entity to remote sync; core registers its own the same way in
`database/routes/sync-entities.ts`. The wire names are unchanged, because
existing tombstones and rows on the far side match on those strings. The
Electron client asks the server for the ordered list and falls back to its
frozen array for a server that predates the endpoint.
adds an entity to sync between a linked desktop and its server; core
registers its own the same way in `src/backend/sync/entities.ts`. The wire
names are part of the protocol: records on both sides match on them, so never
rename one.
`shouldSync(row)` leaves rows out in both directions: they are not pulled, and
a push, update or tombstone aimed at one is refused with 400. It exists for
state that belongs to one install, such as the workspaces plugin's "Last
Session" row, which would otherwise leave a user with one from each side.
**How sync works.** A desktop links to one server account (Sync in the rail)
and keeps an offline copy of that account's data. Nothing records a change as
it happens. On both sides a reconciler hashes every registered row the user
owns and compares it with `sync_records`:
**Deletes need a tombstone.** Registering an entity does not make a delete
propagate on its own: a pull reads `sync_tombstones` for the entity's wire
name, so a plugin route that deletes a synced row calls
`ctx.sync.recordTombstone(userId, entityType, syncId)` right after, the way
core's own delete routes call `SyncTombstoneRepository.record`. **B2** added
this method to the SDK; the snippets plugin's folder and snippet deletes are
its first callers.
- On a server, a new, changed or missing row bumps the record's `revision`
and gives it the next `seq`. `GET /sync/v2/changes?cursor=` pages through
records after a cursor, so a desktop never compares timestamps from two
clocks.
- On a desktop, a row whose hash differs from the one recorded at the last
sync is a local edit to push. `POST /sync/v2/push` carries the revision the
desktop last saw; if the server moved on, the op is a conflict, the
server's version wins and the local one is kept in `sync_conflicts` for the
user to restore.
- Every op gets its own result, so one refused row never holds up the rest.
A refused row is not sent again until it changes.
- `GET /sync/v2/events` is a server-sent event stream that tells a linked
desktop to pull now.
Because deletes are found by reconciliation, a plugin needs nothing beyond
the registration: a registered row that is gone is a delete. There is no
tombstone table any more.
**What a registration says.**
- `references` translate local ids to syncIds on the way out and back. A
`field` is a column (`credentialId` with `syncField: "credentialSyncId"`)
or a path into a JSON column (`jumpHosts[].hostId`,
`layout.panes[].hostId`); `idType: "string"` keeps an id stored as a
string. A syncId with no row on the receiving side keeps the value the
column already has, and a row written before its target is written again
once the target exists.
- `encryptedFields` are decrypted with the owner's data key for the wire and
encrypted again on arrival. A field that cannot be decrypted keeps the row
out of the feed; it is never sent empty.
- `readOnlyFields` stay on their device: never sent, never overwritten, not
part of the hash.
- `shouldSync(row)` leaves rows out in both directions, for state that
belongs to one install, such as the workspaces plugin's "Last Session" row.
- `permissions` (`create`, `update`, `delete`) are checked on the server
before a pushed change is written.
- `serialize` and `deserialize` run after `references`, for anything they
cannot express. `afterWrite` runs after a synced row is written and
`remove` replaces the plain delete, for rows with side effects of their
own.
- `answersTo` names a plugin entity by a name core uses without knowing the
plugin: core's host quick actions point at `commandSnippet`, which the
snippets plugin's `snippets` entity answers to.
The manifest lists the wire names under `contributes.syncEntities`, so a
server can name them while the plugin is off and a desktop can list them in
its sync settings.
**Plugins on a linked desktop.** The manifest's `desktop` field says how a
linked desktop treats a plugin. `mirror` (the default) follows the server: a
plugin installed there is downloaded from `/sync/v2/plugins/<id>/package`,
verified like any `.tmxplug`, installed, and switched on or off with the
server, with the capabilities an admin granted it there. `local` is decided
on each desktop (serial ports), and `server` never runs on a linked desktop
(ACME, sign-in methods, automations). While linked, the admin toggle for a
mirrored plugin answers 409.
### 6. HTTP and WebSockets
@@ -445,7 +493,7 @@ allowlist (`CORE_SETTINGS_ALLOWLIST`) rather than the whole settings table.
secret field.
- Host fields take `enableDefault` (on the host section), `defaultFrom`,
`shareRead` and `ownerOnly`; see the manifest reference. A host settings
save also bumps the host's `updated_at`, so remote sync carries it.
save also changes the host's sync hash, so sync carries it.
### 8. UI
@@ -573,7 +621,7 @@ boundary and Suspense.
| `t`, `hasPermission` | The plugin's strings and a permission check, for code outside a component (a toast from `activate`) |
| `tabs.open`, `getLayout`, `applyLayout`, `onChange` | Tab control, used by workspaces; `tabs.openRailView` (**B12**) opens a rail view |
| `guest`, `info`, `onDispose` | Guest mode flag, plugin info, extra cleanup |
| `desktop` | `available`, `remoteServerUrl()` and `onRemoteServerChange`: whether the frontend runs in the desktop app and which remote server it syncs with (**C6**) |
| `desktop` | `available`, `remoteServerUrl()` and `onRemoteServerChange`: whether the frontend runs in the desktop app and which server it is linked to (**C6**) |
| `registerComponent` | A component other plugins and core render by id with `PluginComponent` |
| `onSettingsChanged` | Called when one of the plugin's settings changes |
@@ -2166,7 +2214,7 @@ authority for signatures.
| `kv.get/set/delete/list` | `kv:own` | Small key/value state |
| `files.dataDir()` | `files:own` | A per-plugin folder, `<DATA_DIR>/plugin-data/<id>` |
| `db.define/client/refs/persist/dialect` | `db:own` (not `dialect`) | The plugin's own tables through Drizzle; `persist` flushes SQLite to disk |
| `sync.registerEntity/recordTombstone` | none | Adds a table to remote sync, and records a delete for it |
| `sync.registerEntity` | none | Adds a table to sync between a linked desktop and its server |
| `registry.provide/consume/revoke` | none | The low-level named registry services sit on |
| `services.provide/get/providers` | per-service RBAC | Typed, versioned cross-plugin services, checked per call |
| `secrets.get/set/delete/seal/unseal` | `secrets:own` | The plugin's own encrypted values |
@@ -3058,7 +3106,7 @@ upgrade on all three engines, and the checks in step 15 are green.
(`TOPICS.userDeleted` in `src/backend/plugins/events.ts`), emitted from
`deleteUserAndRelatedData` right where the old direct repository call
used to sit. `host.deleted` already existed (B7).
- `sync-entities.ts`, if core synced it (step 9).
- `src/backend/sync/entities.ts`, if core synced it (step 9).
- Anything else `grep -rn "<table_name>\|<tableConst>" src` finds, including
core tests and `src/types`.
4. `npm run schema:generate` (the pg and mysql schema variants), then
@@ -3131,13 +3179,13 @@ is not ready to drop.
### 9. Sync entities
If core synced the data, or it is personal configuration that should follow a
user between the desktop app and a server, register it in `activate`:
`ctx.sync.registerEntity({ type, table, order, ... })`. Keep an existing wire
name exactly and remove core's registration from `sync-entities.ts`. Leave
`electron/remote-sync-entities.cjs` alone: it is the fallback for servers
older than the entity-types endpoint, which still sync the core entity. Give
an entity that references others a higher `order` than they have. Use `shouldSync` for rows that belong to one
install (workspaces leaves out its `last_session` row).
user between a linked desktop and its server, register it in `activate`:
`ctx.sync.registerEntity({ type, table, order, ... })` and list the wire name
under `contributes.syncEntities`. Keep an existing wire name exactly and
remove core's registration from `src/backend/sync/entities.ts`. Give an
entity that references others a higher `order` than they have. Use
`shouldSync` for rows that belong to one install (workspaces leaves out its
`last_session` row). Deletes need no extra call.
### 10. Cross-plugin and core callers
+46 -1
View File
@@ -17,6 +17,37 @@ const FRONTEND_ENTRIES = [
"src/frontend/index.js",
];
/**
* A bundled CommonJS package that calls require("react/jsx-runtime") (or any
* other host-provided package) cannot reach it: the package is external, and
* esbuild's ESM output has no require, so the plugin throws "Dynamic require
* is not supported" as soon as it loads. Each such require is pointed at a
* small ES module that imports the external and re-exports it, which esbuild
* can hand to CommonJS code.
*/
export function externalRequireInterop(externals) {
const names = new Set(externals.filter((name) => !name.includes("*")));
return {
name: "termix-external-require",
setup(build) {
build.onResolve({ filter: /.*/ }, (args) => {
if (args.kind !== "require-call" || !names.has(args.path)) return;
return { path: args.path, namespace: "termix-external-require" };
});
build.onLoad(
{ filter: /.*/, namespace: "termix-external-require" },
(args) => {
const id = JSON.stringify(args.path);
return {
contents: `import * as mod from ${id};\nexport * from ${id};\nexport default mod.default ?? mod;\n`,
loader: "js",
};
},
);
},
};
}
export async function build({ cwd }) {
const manifest = readManifest(cwd);
const pluginId = manifest.id ?? path.basename(cwd);
@@ -74,7 +105,21 @@ export async function build({ cwd }) {
sourcemap: true,
logLevel: "warning",
external: FRONTEND_EXTERNALS,
plugins: [staticUrlImports({ outDir })],
// Vite replaces these in the dev server; a plugin bundle has to have
// them too, or code that reads them throws when the plugin loads.
define: {
"process.env.NODE_ENV": '"production"',
"import.meta.env.DEV": "false",
"import.meta.env.PROD": "true",
"import.meta.env.MODE": '"production"',
"import.meta.env.SSR": "false",
"import.meta.env":
'{"DEV":false,"PROD":true,"MODE":"production","SSR":false,"BASE_URL":"/"}',
},
plugins: [
externalRequireInterop(FRONTEND_EXTERNALS),
staticUrlImports({ outDir }),
],
});
}
@@ -279,6 +279,16 @@
"guest": {
"type": "boolean"
},
"syncEntities": {
"type": "array",
"minItems": 1,
"uniqueItems": true,
"description": "Wire names of the sync entities this plugin registers.",
"items": {
"type": "string",
"pattern": "^[a-zA-Z][a-zA-Z0-9]*$"
}
},
"guestViews": {
"type": "array",
"minItems": 1,
@@ -955,6 +965,11 @@
"enum": ["linux", "win32", "darwin"]
}
},
"desktop": {
"type": "string",
"enum": ["mirror", "local", "server"],
"description": "How a desktop linked to a server treats this plugin: mirror follows the server, local is decided on each desktop, server never runs on a linked desktop. Defaults to mirror."
},
"nativeDependencies": {
"type": "array",
"minItems": 1,
+60 -20
View File
@@ -132,64 +132,104 @@ export interface PluginDatabase {
export type SyncRow = Record<string, unknown>;
export interface SyncEntityReference {
/** Stored column holding a local numeric id, e.g. "credentialId". */
/**
* Where a local numeric id lives. A plain column ("credentialId"), or a
* path into a JSON column: "jumpHosts[].hostId" walks every item of the
* jumpHosts array, "layout.panes[].hostId" walks nested objects. A JSON
* column may be stored as a string or as parsed JSON.
*/
field: string;
/** Wire field holding the portable id, e.g. "credentialSyncId". */
syncField: string;
/**
* Wire field holding the portable id, e.g. "credentialSyncId". Only for a
* plain column; a JSON path is rewritten in place.
*/
syncField?: string;
/** The entity the id points at. */
entityType: string;
/** How a JSON path stores the local id. Defaults to "number". */
idType?: "number" | "string";
}
export interface SyncEntityPermissions {
/** Checked before a pushed row that does not exist yet is written. */
create?: string;
/** Checked before a pushed change to an existing row is written. */
update?: string;
/** Checked before a pushed delete runs. */
delete?: string;
}
export interface SyncWriteEvent {
/** The local id of the row that was written. */
id: number | null;
userId: string;
/** The row as it arrived, before references were resolved. */
wire: SyncRow;
created: boolean;
}
export interface SyncEntityRegistration {
/** Stable wire name. Never change it: tombstones and remote rows match on it. */
/** Stable wire name. Never change it: records on both sides match on it. */
type: string;
/** The table, as returned by ctx.db.define. */
table: unknown;
/** Column holding the owning user id. Defaults to "userId". */
userColumn?: string;
/** Fields DataCrypto translates between plaintext wire and encrypted row. */
/** Fields stored encrypted with the owner's data key. Plaintext on the wire. */
encryptedFields?: readonly string[];
/** Local-id to sync-id translations applied on the way out and back. */
references?: readonly SyncEntityReference[];
/** Lower sorts first. Reference targets must sort before their referrers. */
order?: number;
/** Fields that must never be overwritten by an inbound payload. */
/**
* Fields that stay on this side: never sent, never overwritten by an
* inbound row, and ignored when deciding whether a row changed.
*/
readOnlyFields?: readonly string[];
/** One row per user rather than many, keyed on the owner. */
singleton?: boolean;
/**
* Names core uses when its own rows point at this entity without knowing
* which plugin provides it, e.g. "commandSnippet" for a host quick action.
*/
answersTo?: readonly string[];
/**
* Rows this returns false for are left out of sync in both directions, for
* state that belongs to one install (a per-device "last session").
*/
shouldSync?: (row: SyncRow) => boolean;
/** RBAC permissions a pushed change needs on the server. */
permissions?: SyncEntityPermissions;
/**
* Escape hatch for a row whose references are not plain columns, such as ids
* embedded in a JSON blob. Runs instead of `references`, not alongside it.
* Runs after `references` on the way out, for anything they cannot
* express. Gets the row with references already translated.
*/
serialize?: (
row: SyncRow,
resolveSyncId: (entityType: string, id: number) => Promise<string | null>,
) => Promise<SyncRow>;
/** Runs after `references` on the way in. */
deserialize?: (
row: SyncRow,
resolveId: (entityType: string, syncId: string) => Promise<number | null>,
) => Promise<SyncRow>;
/** Runs after a synced row was written on this side. */
afterWrite?: (event: SyncWriteEvent) => Promise<void>;
/**
* Replaces the plain delete of a synced row, for rows with cleanup of their
* own. Gets the stored row.
*/
remove?: (row: SyncRow, userId: string) => Promise<void>;
}
/** Adds an entity to remote sync between a desktop backend and a server. */
/**
* Adds an entity to sync between a linked desktop and its server.
*
* Deletes need nothing extra: core notices a registered row is gone and
* sends the delete itself.
*/
export interface PluginSync {
registerEntity: (entity: SyncEntityRegistration) => void;
/**
* Records that a row was deleted, so a pull on the other side of remote
* sync removes it too. Call it with the row's syncId right after deleting
* it, for a registered entity whose rows can be deleted (a no-op silently
* loses deletes across devices otherwise). A falsy syncId is ignored.
*/
recordTombstone: (
userId: string,
entityType: string,
syncId: string | null | undefined,
) => Promise<void>;
}
export interface PluginRegistry {
+51
View File
@@ -337,8 +337,27 @@ export interface PluginContributions {
"simple" | "balanced" | "advanced",
Record<string, unknown>
>;
/**
* Wire names of the sync entities this plugin registers with
* ctx.sync.registerEntity, so a server can name them while the plugin is
* off and a desktop can list them in its sync settings.
*/
syncEntities?: string[];
}
/**
* How a desktop linked to a server treats this plugin.
* - "mirror": follows the server, installed and switched on or off with it.
* - "local": each desktop decides for itself (a serial port is local).
* - "server": only makes sense on a server, never runs on a linked desktop.
*/
export type PluginDesktopMode = "mirror" | "local" | "server";
export const PLUGIN_DESKTOP_MODES: readonly PluginDesktopMode[] = [
"mirror",
"local",
"server",
];
export interface PluginHttpContribution {
/**
* Old URLs something outside Termix still calls (a webhook a third party
@@ -405,6 +424,8 @@ export interface PluginManifest {
* dependencies".
*/
nativeDependencies?: string[];
/** How a linked desktop treats this plugin. Defaults to "mirror". */
desktop?: PluginDesktopMode;
}
export const DEFAULT_BACKEND_ENTRY = "dist/backend.js";
@@ -439,6 +460,7 @@ const ALLOWED_TOP_LEVEL = new Set([
"locales",
"platforms",
"nativeDependencies",
"desktop",
]);
const ALLOWED_CONTRIBUTES = new Set([
@@ -455,6 +477,7 @@ const ALLOWED_CONTRIBUTES = new Set([
"http",
"uiPresets",
"auth",
"syncEntities",
]);
const ALLOWED_SETTINGS_FIELD = [
@@ -587,6 +610,14 @@ export function validateManifest(manifest: unknown): string[] {
validateCapabilities(m.capabilities, errors);
validatePlatforms(m.platforms, errors);
validateNativeDependencies(m.nativeDependencies, errors);
if (
m.desktop !== undefined &&
!PLUGIN_DESKTOP_MODES.includes(m.desktop as PluginDesktopMode)
) {
errors.push(
`Field "desktop" must be one of: ${PLUGIN_DESKTOP_MODES.join(", ")}`,
);
}
validateDependencyMap(m.dependencies, "dependencies", errors);
validateDependencyMap(m.optionalDependencies, "optionalDependencies", errors);
validateProvides(m.provides, errors);
@@ -761,6 +792,23 @@ function validateProvides(provides: unknown, errors: string[]): void {
const PROVIDER_NAME_PATTERN = /^[a-z][a-z0-9-]*$/;
function validateSyncEntities(value: unknown, errors: string[]) {
const where = "contributes.syncEntities";
if (!Array.isArray(value) || value.length === 0) {
errors.push(`${where} must be a non-empty array`);
return;
}
const seen = new Set<string>();
for (const name of value) {
if (typeof name !== "string" || !/^[a-zA-Z][a-zA-Z0-9]*$/.test(name)) {
errors.push(`${where} entries must be letters and digits`);
continue;
}
if (seen.has(name)) errors.push(`${where} duplicates "${name}"`);
seen.add(name);
}
}
function validateNameList(value: unknown, where: string, errors: string[]) {
if (!Array.isArray(value) || value.length === 0) {
errors.push(`${where} must be a non-empty array`);
@@ -888,6 +936,9 @@ function validateContributes(
errors.push('Field "contributes.guestViews" needs "contributes.guest"');
}
}
if (contributes.syncEntities !== undefined) {
validateSyncEntities(contributes.syncEntities, errors);
}
validatePermissions(contributes.permissions, pluginId, errors);
validateActions(contributes.actions, errors);
validateActionSlots(contributes.actionSlots, errors);
-12
View File
@@ -260,8 +260,6 @@ export interface FakePluginContext {
tables: PluginTableDefinition[];
/** Sync entities registered through ctx.sync.registerEntity, in order. */
syncEntities: SyncEntityRegistration[];
/** Tombstones recorded through ctx.sync.recordTombstone, in order. */
tombstones: Array<{ userId: string; entityType: string; syncId: string }>;
/**
* WebSocket routes registered through ctx.ws, in order, with the handler
* and options, so a test can hand a route a fake socket.
@@ -412,11 +410,6 @@ export function createFakeContext(
const kv = new Map<string, unknown>();
const tables: PluginTableDefinition[] = [];
const syncEntities: SyncEntityRegistration[] = [];
const tombstones: Array<{
userId: string;
entityType: string;
syncId: string;
}> = [];
const wsRoutes: FakeWsRoute[] = [];
const httpRouters: Array<PluginRouterOptions | undefined> = [];
const listeners = new Map<string, Set<(payload: unknown) => void>>();
@@ -626,10 +619,6 @@ export function createFakeContext(
registerEntity: (entity) => {
syncEntities.push(entity);
},
recordTombstone: async (userId, entityType, syncId) => {
if (!syncId) return;
tombstones.push({ userId, entityType, syncId });
},
},
registry: {
@@ -1241,7 +1230,6 @@ export function createFakeContext(
kv,
tables,
syncEntities,
tombstones,
wsRoutes,
httpRouters,
settings,