# March 2026: From Idea to Working CI

## Goal

Build `myci` from scratch — a platform-agnostic, local-first CI
system in Rust. Get it to the point where it can execute
workflows locally and in Kubernetes pods, with the foundation
for future growth.

## Key Results

1. The ci-cache docs pipeline runs via `myci run --executor k8s`
2. Forgejo push and PR triggers a `myci` run and reports status back
3. A pipeline publishes a package to the Forgejo package registry

## Sprint Overview

- Sprint 17: Dev and testing setup (done)
- Sprint 18: Testing practices (done)
- Sprint 19: Streamed build output in UI (done)
- Sprint 20: Expand conditions support (done)
  - Support `on.push.branches` and `tags`
- Sprint 21: Pull request events + deployment
  - `on.pull_request` webhook + trigger matching
  - Commit status links back to build detail page
  - Deploy to cluster, system-wide webhook in Forgejo
- Sprint 21b: UI and server polish (done)
  - Pipeline as first-class entity, ANSI rendering,
    OIDC graceful degradation, code quality cleanup
- Sprint 22: Credentials and package publishing
  - Expose forge token to pipeline steps
  - End-to-end: pipeline publishes package to Forgejo
- Sprint 23 (stretch): Pages via git-pages
  - Deploy git-pages in cluster
  - Pipeline pushes built docs to git-pages
- Sprint NEW: Focus on e2e testing and core improvement
- TBD: Per-user forge tokens
- TBD: Automatic webhook registration
- TBD: Consolidation (CLI refactor, server config, pipeline dedup)
- TBD: Configuration file for MyCI

## Backlog

- [ ] Refactor `StepRunner` to return async streams instead of accepting
  `EventSender` — see notes below for scope analysis. Also reduces
  parameter count on trait methods (currently 8 params on `run_node`).
- [ ] Revisit `#[cfg(test)]` cross-crate visibility: test constructors
  live in `myci-test` because `#[cfg(test)]` items in `myci-domain`
  are invisible to downstream crates. Track Cargo issue #8379 and
  consider a `test-utils` feature flag if the `myci-test` dependency
  becomes awkward (circular dev-dep).
- [ ] Arch: Extract domain logic into a dedicated crate (or expand
  `myci-domain`). `Pipeline`, `Trigger`, `Job`, `Step`, `glob` and
  expression evaluation are pure domain logic currently in the `myci`
  crate. A crate boundary would enforce that domain has no
  infrastructure dependencies. Trigger: when a second consumer of
  domain logic appears or `myci` crate grows too large.
- [ ] Arch: Redesign expression `Context` as a typed environment
  (minijinja-style). Today `github.*`, `env.*`, `steps.*` are all
  `HashMap<String, String>` with dotted-key hacks for nested access.
  A typed provider model would let each context namespace resolve its
  own lookups, handle nesting naturally, and make future additions
  (`secrets.*`, `matrix.*`, `runner.*`) self-contained.
- [ ] Configuration file for `myci` (executor defaults, image
  defaults)
- [ ] Podspec-like configuration for k8s executor (resource
  limits, volumes, service account)
- [ ] Workspace volume size configurable per runner (once runner
  configuration exists). Currently global via ~MYCI_WORKSPACE_SIZE~.
- [ ] Git server binding in the case of ~myci run~ in a ~Pod~
- [ ] Context handling may need a cleanup after the naive alias introduction
- [ ] Need solution for e2e testing: Forge, MyCI server, in cluster, then test
  - Already needed, at least to a degree to test out the handling of events, and
    also the cache usage in the case of remote repos.
- [ ] Should the aliases be set by default? This should probably be
  configurable.
- [ ] Cache handling
  - Concurrency, e.g. action cache
  - Unification
  - Use something existing instead of implementing a solution
- [ ] Executable documentation to be sure it is correct
- [ ] Action cache invalidation
- [ ] Automatic server start for browser E2E tests (subprocess or in-process)
- [ ] Atomic action cache writes (clone to temp dir, then rename)
- [ ] DRY shutdown/cleanup waiting logic (~myci run~ vs ~myci server~)
  - Paths are fundamentally different (single cleanup vs JoinSet drain),
    only revisit if a third entry point appears
- [ ] ~kubectl logs~ only shows stdout, not stderr (tee limitation);
  evaluate if acceptable long-term
- [ ] Use tracing or prefixed output for executor tee so pod/host
  origin is distinguishable
- [ ] Create file command files (`GITHUB_OUTPUT`, `GITHUB_STATE`), pass paths
  via env
- [ ] `uses:` step support (checkout skipped for now, Node.js actions work)
- [ ] Matrix expansion
- [ ] GitLab CI format plugin
- [ ] Artifact storage
- [ ] Caching
- [ ] Pipeline format plugin system (subprocess-based)
- [ ] K8s tee wrapper shell injection: evaluate if string interpolation
  for commands (ADR-0022) is a concern given workflow YAML comes from
  repo owners
- [ ] Keep pod event watcher alive until pod deletion
  - Currently aborted after step execution; late events (OOM kills,
    evictions, cleanup failures) may be missed
  - Investigate if watcher can run until pod is actually deleted
- [ ] Emit infrastructure lifecycle as distinct event types
  - K8s operations silenced today: `mkdir -p` before tar (null sender),
    tar upload stderr (`eprint!`), pod/PVC deletion (3 sites, tracing only),
    chisel logs (only on failure)
  - User sees nothing between "step started" and first log line
  - Small scope (~1-2h) but risks rework if StepRunner stream refactor
    happens — consider doing them together
  - Run-level infrastructure events needed (e.g. git server start) —
    current `InfraLog` is job-scoped, needs a variant without `job` field
- [ ] Resource monitoring for k8s runs (CPU, memory, disk over time)
  - Poll metrics API + kubelet stats during job execution
  - Display sparklines in build detail UI
  - See ~context/ideas/resource-monitoring.md~
- [ ] Worker/agent split: decouple server from executor
  - Protocol design spike (claim API, event push, heartbeat)
  - Extract worker loop from in-process task to standalone binary
  - `runs-on:` as label-based routing key
  - See {ref}`arch-worker-agent-split`
- [ ] Binary cache infrastructure for k8s Nix builds
  - Deploy attic or harmonia in exp cluster
  - No MyCI code changes, pure infrastructure
  - See {ref}`arch-nix-builds`
- [ ] `secrets.*` expression context and secret management
  - Workflow-level `secrets:` definitions, per-user secret storage
  - Secrets registered in `SecretMasker` automatically
  - Service-level forge token exposure in sprint 22
  - See {ref}`arch-secrets-and-credentials`
- [ ] Unified event-based output rendering
  - Remove direct `println!`/`eprintln!` from executors and orchestration
  - Both `myci run` (CLI) and server render output exclusively from the
    event stream, after secret masking
  - Shared rendering layer that formats events per context (terminal vs
    SSE/JSON), with filtering and verbosity support
- [X] Provide nix environment with all needed tools (git, chisel, ...)
- [X] Extract TestApp and test helpers to myci-test for use from myci-e2e
- [X] test-with compile-time env check: cached builds skip gated tests even when var is set
- [X] K8s tar upload broken pipe on action copy — quality image was
  missing tar and nodejs
- [X] Evaluate `myci run` queue/worker pattern
  - Server queue implemented in sprint 16, CLI evaluation in sprint 17 (no benefit, keep direct execution)
- [X] `myci-dev setup-oauth` also registers the forge webhook
- [X] Fix duplicate ADR numbers
- [X] Node actions require Node in the user's container image
  - Resolved by runner container architecture: Node actions
    execute in the runner container, user's step container only
    runs shell steps
- [X] Server CLI config cleanup (~ServerConfig~ structure and wiring)
  - Generic executor leak resolved by run queue in sprint 16
- [X] Web UI in Rust via WASM
- [X] Basic stdout handling: don't break on `::` workflow commands
- [X] Webhook receiver (server mode)
- [X] Expression evaluation (`${{ }}` syntax)
- [X] Forgejo integration (commit status reporting)
- [X] UI: The runs overview should include a job or pipeline id if possible
- [X] Build detail page (steps, duration, log snippets)
- [X] Q: Running the server locally, should local runs also provide state into
      the web ui?
      - Yes, will do in Sprint 17.

## Notes


### StepRunner async stream refactor (scope analysis)

Currently `EventSender` (mpsc channel) is passed *down* through 29
functions: `run()` → `run_workflows()` → `run_pipelines()` →
`run_jobs()` → `Executor::run_job()` → `execute_steps()` →
`StepRunner::run_shell/run_node/run_docker()` → `read_lines()`.
9 functions send events directly, 20 pass it through.

The clean architecture direction: step runners *return* an async
stream of events instead of accepting a sender. The caller collects
the stream and forwards it. The port produces output, it doesn't
know about event infrastructure.

**Why it's a full sprint:**

- All 3 `StepRunner` trait methods + `Executor::run_job()` need new
  signatures
- 3 implementations (host, k8s, dry_run) all need updating
- `read_lines()` does dual duty (accumulate `Vec<LogLine>` + send
  events) — needs splitting
- K8s has concurrent streams (stdout + stderr + pod events) that
  must merge — stream combinators add complexity
- Host executor has separate stdout/stderr pipes — same merging
- `execute_steps()` sends its own events (StepStarted/Finished)
  wrapping the runner's output — careful interleaving needed
- Dry-run ignores the sender — would return empty streams

**Current approach works correctly.** This is about making the port
direction cleaner: producers return data, they don't push into
infrastructure. Worth doing but not urgent.

### Next scribble

### Forge integration architecture

Research in `context/notes/forge-ci-integration.md`.

External CI systems (Woodpecker, Jenkins, etc.) all integrate the
same way: OAuth + webhooks + commit status API. No proprietary
runner protocol needed.

**OAuth/OIDC** serves two purposes:
- User authentication — user logs into MyCI web UI via the forge
  as identity provider (redirect-based OIDC flow)
- API access — MyCI gets a token to act on the forge: register
  webhooks, read repo files, post commit statuses

**Webhooks** are the trigger — the forge POSTs events (push, PR)
to MyCI's server endpoint.

Minimal server components:
- HTTP endpoint to receive webhook payloads
- OAuth2 client to authenticate with the forge
- API client to post commit statuses back
- Trigger `myci run` on incoming events

**Web UI** layers on top: user authenticates via OAuth, selects
repos to enable (auto-registers webhooks), follows build progress.
This is the Woodpecker model — "log in with Forgejo, flip a
switch, done."

The server side is fully decoupled from the executor. It is
"receive event, run pipeline, report status." The existing
`myci run` with host or k8s executor stays unchanged underneath.

### Incremental forge integration plan

OAuth and a web UI are not prerequisites for CI functionality.
Everything can be built incrementally, each step independently
useful.

**Step 1: `myci run` with status reporting (no server)**

`myci run` detects the current repo's remote URL. Token from
environment (`MYCI_FORGE_TOKEN`) or config file. Posts commit
status (pending before run, success/failure after) via the forge's
REST API. User triggers manually, from cron, or from a git hook.

New code: small HTTP client that POSTs to the commit status API.
The status API is nearly identical across forges:
- Forgejo/GitHub: `POST /repos/{owner}/{repo}/statuses/{sha}`
- GitLab: `POST /projects/{id}/statuses/{sha}`

**Step 2: `myci server` receiving webhooks (minimal HTTP server)**

Thin layer on top of step 1. Listens for webhook payloads, triggers
the same pipeline run, posts status back. Webhook registration is
manual initially (user adds the URL in forge settings).

System-wide integration: Forgejo and GitLab support system-level
webhooks — one URL that fires for all repos on the instance. No
per-repo setup needed.

**Step 3: `myci login` with OAuth (smooth auth)**

CLI-based OAuth login, no web UI needed. Two common flows:
- Device flow: CLI shows URL + code, user authorizes in browser,
  CLI polls for token. Works over SSH.
- Localhost redirect: CLI starts temp server on localhost, opens
  browser, captures redirect with token.

Token stored in config file. Replaces manual PAT creation.

**Later: Web UI**

Web UI is a user experience layer, not a functional prerequisite.
User authenticates via OAuth, selects repos (auto-registers
webhooks), follows build progress. The Woodpecker model.