.. _arch-pipeline-first-class: =============================== Pipeline as First-Class Entity =============================== Problem ======= The current domain model collapses "trigger" and "pipeline/workflow" into a single entity (``RunRecord``). A run triggered by a push can execute multiple workflow files (``ci.yml``, ``docs.yml``), but all jobs land in a flat ``Vec`` with no workflow grouping. Consequences: - Two jobs named ``build`` in different workflows collide. - Aggregate status hides which workflow failed. - Forge commit status is per-run, not per-workflow. Forgejo/GitHub post one check per workflow — MyCI posts one check for everything. - No workflow identity in the event stream. - Users cannot re-run a single workflow. - The runs list cannot show a workflow name column. Current model ============= One ``RunRecord`` per trigger event. All workflows merged. :: {data_dir}/runs/{run_id}.json {data_dir}/events/{run_id}.jsonl RunRecord (1 per trigger) +-- id, repo, ref_name, sha +-- status (aggregate, treated as singular) +-- started_at, finished_at +-- repository_url +-- jobs: Vec (flat -- all workflows merged) +-- name, status, steps EventKind: RunStarted { run_id, repo, ref, sha } JobQueued/Started/Finished { job } StepStarted/Finished { job, step } Log { job, step, stream, content } RunFinished { status } Proposed model ============== A trigger event (push webhook, CLI invocation) is the **cause**. Each matched workflow file becomes its own **run** — the entity with a lifecycle, status, commit check, and event stream. :: TriggerEvent (1 per webhook / CLI invocation) +-- id (correlation ID) +-- repo, ref_name, sha +-- event: String ("push", "pull_request") +-- received_at: DateTime RunRecord (1 per matched workflow) +-- id +-- trigger_id: Option (links back to TriggerEvent) +-- workflow: String (relative path) +-- name: Option (YAML name: field, display label) +-- repo, ref_name, sha (denormalized from trigger) +-- status: RunStatus +-- started_at, finished_at +-- repository_url +-- jobs: Vec +-- (unchanged) ``TriggerEvent`` is lightweight — it records "this happened" and links runs that share a cause. It has no lifecycle or status. ``RunRecord`` is the entity users interact with: it has a status, a commit check, a re-run action, and an event stream. Runs from the same trigger share ``trigger_id``. Grouping is a query (``runs.filter(|r| r.trigger_id == id)``), not nesting. The trigger context (repo, ref, sha) is denormalized onto each run so runs are self-contained — no join needed to display a run. Workflow identifier =================== Use the **relative path** (``.forgejo/workflows/docs.yml``), not just the filename. - Guarantees uniqueness even if multiple workflow directories are supported in the future. - Matches GitHub/Forgejo commit status convention. - Unambiguous for debugging. - The YAML ``name:`` field is the human-readable display label. The path is the stable identifier. The relative path is already computed in ``run_workflows`` (``path.strip_prefix(repo_root)``). ``discover_workflows_from_cache`` would need to prepend the directory prefix. The path is stored on each ``RunRecord`` in the ``workflow`` field. Storage layout ============== Group everything by trigger — the trigger is the causal boundary. Run record and event stream for each run are co-located. :: {data_dir}/triggers/{trigger_id}/ trigger.json (TriggerEvent metadata) {run_id}.json (RunRecord for ci.yml) {run_id}.events.jsonl (event stream for ci.yml) {run_id}.json (RunRecord for docs.yml) {run_id}.events.jsonl (event stream for docs.yml) Why trigger-grouped: - **Mirrors the domain**: everything that exists was caused by a trigger. The filesystem reflects that relationship. - **Co-location**: run record and event stream live next to each other instead of in separate directory trees (``runs/`` and ``events/``). - **Natural access patterns**: list all runs from a trigger = ``readdir``. View a single run = read one file. - **Cleanup is atomic**: trimming old data removes one directory. - **Per-run event streams**: each run has its own JSONL file with independent lifecycle, writer, and broadcast channel. Event stream ============ Each run gets its own JSONL event file. The existing ``RunStarted`` and ``RunFinished`` events already bracket the stream — now scoped to one workflow execution. No new event types needed. The old model (one run = multiple workflows) would have required new bracket events (``PipelineStarted`` / ``PipelineFinished``) to demarcate workflow boundaries within a shared stream, and a workflow filter on the SSE endpoint to select which workflow's events to serve. With one run = one workflow, the existing event types and endpoints work unchanged. The broadcast registry key remains ``run_id``. Each SSE connection subscribes to exactly one run's broadcast. What this unlocks ================= **Per-workflow commit status** ``StatusReporter::report`` takes workflow context. One ``POST /statuses/{sha}`` per run. **Per-workflow re-run** Each run has its own identity. Re-run targets a specific workflow execution. **Workflow display name** ``name`` from YAML ``name:`` field. UI shows name if present, falls back to filename. **Parallel execution** Each run has its own event file, writer, broadcast. No shared mutable state. **Workflow-scoped job names** Jobs scoped within a single run. No collisions across workflows. **Workflow name in runs list** UI shows one row per run (= one workflow execution). API === :: GET /api/runs -> Vec GET /api/runs/{id} -> RunRecord GET /api/runs/{id}/events -> SSE Each run is one workflow execution, so the API needs no workflow filter parameter — ``GET /api/runs/{id}/events`` returns exactly the events for that workflow. The old model would have needed ``?workflow=.forgejo/workflows/docs.yml`` to select from a shared stream. The UI runs list shows one row per run: workflow name (or filename), status, repo, branch, SHA. Runs from the same trigger share ``trigger_id`` for grouping. Migration ========= New fields on ``RunRecord`` (``workflow``, ``name``, ``trigger_id``) use ``#[serde(default)]``. Old JSON files deserialize with these fields as ``None`` / empty — displayed as runs without workflow info. No data migration step needed. The storage layout changes from flat directories to trigger-grouped. A v1→v2 migration moves existing files:: {data_dir}/runs/{run_id}.json {data_dir}/events/{run_id}.jsonl into:: {data_dir}/triggers/{run_id}/ trigger.json (synthesized, minimal) {run_id}.json (moved from runs/) {run_id}.events.jsonl (moved from events/) For old records without a trigger, the run ID serves as the trigger ID (one run = one trigger). The migration synthesizes a minimal ``trigger.json`` from the run's metadata. Trade-offs ========== Pros: - Correct domain model reflecting how workflows actually work. - Per-workflow status in UI and forge commit status. - Enables re-run, concurrency, filtering without further model changes. - Co-located storage (run + events in one directory). - Backward compatible via ``#[serde(default)]`` on new fields. - Event stream stays append-only, no new event types. Cons: - Breaking change across the stack (domain, run store, worker, API, UI). - Storage layout migration needed. - Worker creates multiple records per trigger instead of one. GitLab comparison ================= GitLab supports only one workflow file (``.gitlab-ci.yml``) per repo — effectively one pipeline per trigger. The multi-workflow model (GitHub/Forgejo/MyCI) is more flexible: separate concerns into separate files, each with their own triggers and independent status. Making each workflow execution its own run is the natural consequence of supporting multiple workflows.