.. _arch-worker-agent-split: ==================== Worker / Agent Split ==================== Problem ======= Today the server and executor are fused into a single process. The server receives a webhook, enqueues a ``QueuedRun``, and an in-process tokio task pulls it from an ``mpsc`` channel and executes it — all within the same pod, the same cluster. This is a single-site, single-process assumption. It breaks as soon as any of these are true: - A second deployment context exists (a different cluster, a host with Nix, a build machine with GPU access). - Multiple users or teams share one UI but run on different infrastructure. - The server runs in Kubernetes but a build needs tools that are impractical to containerize (Nix with a warm store, hardware access). A central web UI requires all execution contexts to report back to one server. Without a network boundary between server and executor, each deployment is an island with its own UI, its own run history, its own event stream. That contradicts the multi-user product vision. Current State ============= The run queue architecture from sprint 16 already models the concepts: - ``QueuedRun`` is a work item carrying everything needed to execute: run ID, repo URL, ref, sha, default branch. - A worker task consumes from the queue, resolves workflows, and runs them via the ``Executor`` trait. - The ``EventSender`` (mpsc channel) streams events from executor to persistence and broadcast. The coupling is in the transport: the worker is an in-process task pulling from a tokio ``mpsc`` channel. The split replaces that in-process channel with a network protocol. Proposed Model ============== Server responsibilities ----------------------- The server owns the control plane: - **Identity and authorization** — OAuth login, session management. - **Run lifecycle** — receives triggers (webhook, API, UI), creates ``QueuedRun``, transitions state (queued → claimed → running → finished). - **Event persistence** — receives ``RunEvent`` from workers, writes JSONL files, updates ``RunStore``. - **API and UI** — serves the web interface, SSE endpoints, REST API. - **Run routing** — matches queued runs to workers based on labels. The server never executes builds. Worker responsibilities ----------------------- The worker owns the execution plane: - **Claims work** — polls or subscribes for queued runs that match its capabilities. - **Executes** — clones the repo, discovers workflows, runs them via whichever ``Executor`` is configured (host, k8s, future nix-native). - **Streams events** — sends ``RunEvent`` back to the server as they occur. - **Stateless** — if the worker crashes, the server detects it (via heartbeat timeout) and can re-queue or mark the run as failed. No local state needs recovery. The worker is a new binary: ``myci worker --server --executor ``. ``runs-on`` as Routing Key =========================== The ``runs-on`` field in workflow YAML is currently ignored. With the worker split it becomes the routing mechanism. Workers register with labels describing their capabilities:: myci worker --server https://myci.example \ --executor host \ --label nix=true \ --label arch=x86_64 A workflow declares what it needs: .. code-block:: yaml jobs: build: runs-on: [nix] steps: - run: nix build .#my-image The server matches the ``runs-on`` labels against registered workers and routes the run accordingly. If no worker matches, the run stays queued. What Changes ============ - **Work claim API**: ``POST /api/worker/claim`` — worker polls for queued runs matching its labels. Alternative: WebSocket subscription for push-based delivery. - **Event push**: worker sends ``RunEvent`` back to server. Could reuse the existing event types over HTTP (batched POST) or WebSocket. - **Worker binary**: ``myci worker`` — connects to server, claims work, executes, streams events. - **Run routing**: server matches ``runs-on`` labels to worker capabilities. - **Heartbeat / liveness**: server detects dead workers and re-queues or fails their runs. What Stays the Same =================== - ``Executor`` trait, ``StepRunner``, ``RunContext`` — unchanged. The worker instantiates the executor locally, same as today. - ``RunEvent`` / ``EventKind`` — same types, different transport (network instead of in-process channel). - ``myci run`` (local, no server) — unchanged. Still a self-contained path for local development. - Workflow format, expression evaluation, action resolution — all library code, used by both server and worker. Connection to Federated Vision ============================== The worker split is the concrete first step toward a federated platform vision where MyCI instances act as sovereign peers in a network — receiving events via open protocols, executing locally, and reporting back. The worker model makes this possible: each worker is a peer that connects to a server (or eventually to a federated event bus). The server is one coordination point today; it could become one of many in a federated setup. The architecture does not assume a single server — it assumes a protocol between coordinators and executors. Longer-term directions (Matrix as event bus, CRDT-based state sync, overlay networking) build on this foundation but are not prerequisites for the worker split itself. Migration Path ============== The split does not need to happen all at once: 1. **Extract the worker loop** — move the in-process worker task into a function that can be called either in-process or from a standalone binary. Same code, two entry points. 2. **Add the claim/event APIs** — server exposes work items, accepts events. The in-process path remains as a fallback. 3. **Worker binary** — ``myci worker`` as a thin wrapper around the extracted loop plus HTTP client for claim/event. 4. **Deprecate in-process execution** — once the worker protocol is stable, the server drops the built-in executor.