Termix/readme/HOST-TO-HOST-TRANSFER.md

Host-to-host file transfer

This document describes the host-to-host copy/move feature. It is intended for operators and contributors. Direct host-to-host routing via SSH tunnels is not implemented; that is documented under Future work.

Overview

Host-to-host transfer copies or moves files from one SSH host to another through the Termix server as a relay. The UI lives in File Manager: right-click files or folders and choose Copy to host… or Move to host….

Compared to copying via your laptop (scp -3 or ssh one 'cat …' | ssh two 'cat …'), Termix keeps the job on the server so transfers continue if you close the browser, and you get integrated progress, cancel, retry, and cleanup.

flowchart LR
  Browser[Browser_UI]
  Termix[Termix_server]
  Src[Source_host]
  Dst[Destination_host]

  Browser -->|start_transfer_API| Termix
  Termix -->|dedicated_SSH_xfer_src| Src
  Termix -->|dedicated_SSH_xfer_dst| Dst
  Termix -->|SFTP_read_then_write| Termix

Data for remote-to-remote paths always passes through the Termix process (pipelined SFTP buffers, or a tar archive stream). There is no source→destination SSH tunnel for file bytes today.

Prerequisites

1. Two hosts with File Manager enabled, saved in Host Manager.
2. Both hosts reachable from the Termix server on SSH (each host’s jump hosts and SOCKS5 proxy chain apply to the Termix→host connection, not host→host).
3. File Manager open on the source host (browse session connected). The destination host should show Ready in the transfer dialog; if it shows authentication required, open File Manager on that host once first.
4. For multi-file or folder transfers, pick a destination directory (not a file path).

The dialog shows a reminder: “Both hosts must be reachable from the Termix server. Direct host-to-host routing is not supported.”

Using the UI

Start a transfer

1. Open File Manager on the source host.
2. Select one or more files or folders.
3. Right-click → Copy to host… or Move to host… (sidebar tree supports the same context menu).
4. In the dialog:
   • Choose destination host (other connected file-manager hosts).
   • Set destination path (type a path or use Browse destination folders).
   • Optionally pick a recent destination (collapsed by default).
   • For multi-item transfers, choose transfer method (Auto / Tar archive / Per-file SFTP); a preview explains what Auto will pick.
   • Pin folders with Add to shortcuts (per-destination host; uses normal file-manager shortcuts, not a separate favourites list).
5. Confirm Copy or Move.

During transfer

• A progress toast shows phase (compressing, transferring, extracting), bytes, speed, and Cancel.
• Transfer monitor (desktop, when authenticated) picks up active transfers started in another tab or window.
• Large single files use segmented SFTP (256 MiB segments) with 2 parallel lanes by default (separate dedicated SSH session pairs per lane). The toast can show total speed and lane count when multiple lanes are active.

After completion or failure

Outcome	What happens
Success	Toast success; destination path saved to recent destinations for that source host; file manager can refresh.
Partial	Some paths failed; toast lists failed paths; source may be kept on move.
Error	Toast with Retry when partial data on destination allows resume.
Cancelled	Toast with optional Clean up destination to remove partial files.

Move deletes source files only after a successful full transfer (same as copy, then source delete). Cancelled or partial moves may leave files on both sides.

Metrics

On success, expanded timing details can include: prepare destination, compress (tar), per-hop throughput (source→server, server→dest), extract, source delete, total duration.

Transfer methods

Method	When used	Behavior
Stream (single file)	Exactly one file, copy or move	Pipelined SFTP read on source → write on destination; segmented above 32 MiB; parallel lanes for throughput.
Tar archive	Multi-file/folder when Auto or user selects Tar, and both sides are Unix with tar	tar -czf on source → one archive streamed through Termix → tar -xzf on destination.
Per-file SFTP	Windows involved, tar unavailable, or Auto chooses it	Each file copied sequentially over SFTP through Termix.

Auto heuristics (see src/backend/ssh/transfer-routing.ts) consider file count, total size, largest file, and compressibility (e.g. many small files → tar; large incompressible sets → per-file SFTP).

Method preview is locked for the current source path set until you change Auto/Tar/Per-file preference, so changing only the destination host does not re-scan the source.

Architecture (implementation)

Component	Role
src/backend/ssh/host-transfer.ts	Transfer engine: sessions, SFTP pipeline, tar path, retry, cancel, progress.
src/backend/ssh/file-manager.ts	HTTP routes, openDedicatedTransferSession, jump/SOCKS connect.
src/backend/ssh/transfer-routing.ts	Tar vs per-file SFTP selection.
src/backend/ssh/transfer-paths.ts	Path normalization (Unix/Windows).
src/ui/.../TransferToHostDialog.tsx	Transfer dialog.
src/ui/.../transferProgressMonitor.tsx	Toasts, cancel, retry, cleanup.
src/ui/.../TransferMonitor.tsx	Global active-transfer polling.

Sessions: Browse sessions identify hosts. Each transfer opens dedicated SSH sessions (xfer:{transferId}:src / :dst) so browsing and transfers do not share channels. Parallel lanes add xfer:{transferId}:src:pN / :dst:pN.

Special case: If the destination host is the same machine as Termix (local SSH endpoint), writes use the local filesystem via fastGet instead of dest SFTP; data still originates from the remote source through Termix.

Persistence: Recent destinations are stored in transfer_recent (per user, per source host). Folder shortcuts use file_manager_shortcuts on the destination host.

HTTP API (file manager service)

Endpoint	Purpose
POST /ssh/file_manager/ssh/transferMethodPreview	Scan source; return resolved tar vs item_sftp and reason.
POST /ssh/file_manager/ssh/transferToHost	Start transfer; body includes sourceSessionId, destSessionId, sourcePaths, destPath, move, methodPreference, optional parallelSegmentCount (1–8, default 2).
GET /ssh/file_manager/ssh/transferStatus/:transferId	Poll progress.
GET /ssh/file_manager/ssh/activeTransfers	List running transfers for user.
POST /ssh/file_manager/ssh/transferCancel/:transferId	Request cancel.
POST /ssh/file_manager/ssh/transferCleanup/:transferId	Remove partial destination artifacts after cancel/failure.
POST /ssh/file_manager/ssh/transferRetry/:transferId	Retry with same snapshot (resume when possible).

Database (main API): GET/POST /host/transfer/recent for recent destinations.

Reliability features

• Resume: Destination file size is probed; SFTP write opens with resume when a partial file exists (per segment on large files).
• Retry: Reconnects dedicated sessions; segment-level and full-copy retries with backoff; fresh SSH pairs after repeated failures (lane reset).
• Stall detection: ~45 s without progress on a segment; hung transfer/reconnect probing on status polls.
• Cancel: Aborts in-flight SFTP; user can clean up destination paths that were created or partially written.
• Overlap guard: Refuses transfer when source and destination paths overlap in a destructive way.

Limitations

1. No direct host-to-host data path — Termix must reach both hosts independently (with each host’s jump/proxy settings).
2. Not the same as S2S SSH tunnels — Tunnels in Host Manager forward TCP ports; they do not carry file-manager transfers today.
3. Throughput — Remote-to-remote speed is bounded by Termix CPU/RAM and min(Termix↔source, Termix↔dest) links; very large files on a small Termix box may be slower than scp -3 from a powerful desktop.
4. Parallel lanes — Writes are out of order on disk; fine for copy, not for playing media from a partially written file. Default is 2 lanes; UI may not expose lane count (API default applies).
5. Tar — Requires tar on both Unix hosts; temporary archive under /tmp on source during transfer.
6. Windows — Tar path disabled; per-file SFTP only for Windows endpoints.
7. Jump hosts on S2S tunnels — Server-to-server tunnel connect does not use jump hosts; only transfer/browse SSH does. A host reachable only via jump may work for transfer but not as an S2S tunnel source until tunnel code is aligned.

Troubleshooting

Symptom	Things to check
No destination hosts listed	Open File Manager on another host; ensure host has File Manager enabled.
Destination “Authentication required”	Connect File Manager on that host once in this session.
Transfer fails immediately	SSH from Termix to both hosts (firewall, jump host, SOCKS5, credentials).
Slow speed	Termix link to slower side; try off-peak; for single huge files, parallel lanes help if CPU/network allow.
Stuck progress	Wait for stall/reconnect; cancel and retry; check server logs for host_transfer / transfer_ssh_*.
Partial files after cancel	Use Clean up destination in the toast.
28 GB / 25 GB style progress	Usually parallel progress accounting; status polls use destination size probes.

Comparison to manual scp between remotes

See Unix & Linux: scp from one remote server to another. Naive scp one:file two:file runs from the first host and fails unless that host can SSH to the second. scp -3 relays through your workstation. Termix relay is analogous to scp -3 through the Termix server, with richer lifecycle management, not analogous to direct ssh source 'scp … dest'.

---

Future work: direct routing via tunnels

The following are planned / discussed enhancements, not shipped in the current build. They build on existing S2S SSH tunnels (src/backend/ssh/tunnel.ts), which already connect source → endpoint using forwardOut from the source host to the endpoint’s SSH port.

Why tunnels matter

Many homelabs have a destination that is only reachable from another host (e.g. NAS on LAN behind a Pi), while Termix runs elsewhere. Today that destination cannot receive a dedicated xfer:dst session from Termix even if an S2S tunnel from Pi → NAS is configured and working.

Possible routes (future)

Route	Termix SSH legs	Data path	Benefit vs today
Relay (current)	2	Termix buffers SFTP	Works when both hosts reachable from Termix.
Tunnel-bridged SFTP	1 (+ bridge)	Still through Termix memory	Dest reached via source forwardOut; fixes reachability; reuses most of current engine.
Direct remote (rsync/scp on source)	1 (control)	Source → dest bytes	Best throughput; Termix orchestrates rsync/scp on source when forward + tools allow.

Integration ideas (not implemented)

1. transfer-bridge module — Shared forwardOut probe and connectDestThroughSource (extracted from tunnel code); lookup matching tunnel_connections on the source host record.
2. Route resolver — Auto-select relay vs bridged vs direct; expose route in method preview (“via Termix” vs “direct host-to-host”).
3. Reuse active S2S tunnel — If Host Manager tunnel source→dest is already connected, reuse endpointClient instead of opening a second bridge.
4. Jump hosts on S2S tunnel source — Align tunnel connect with file-manager jump chains so tunnel and transfer eligibility match.
5. Fallback — Always fall back to current relay when probe or remote rsync fails (Windows, missing tools, forwarding denied).

What would be preserved

Cancel, partial cleanup, retry/resume (rsync --partial on direct path; existing SFTP segment resume on relay/bridged), dedicated sessions, transfer monitor, recent destinations, folder shortcuts, and tar/per-file method selection (with direct-path variants for multi-file).

References

• Internal plan: .cursor/plans/host-to-host_direct_transfer_*.plan.md (if present in your checkout).
• Tunnel implementation: src/backend/ssh/tunnel.ts — connectEndpointThroughSource, establishManagedS2STunnel.
• Transfer engine: src/backend/ssh/host-transfer.ts.

---