CMLabs · Psyclone AIOS

# Builders & Supervisors

How a running system builds and heals itself: the node-local Builder that bakes recipes into live components, and the Supervisor that orchestrates it — including how to write your own custom supervisor.

## The idea in one paragraph

Ordinary PsySpec startup is a one-shot: the node reads the spec once and creates everything in it. A Builder makes that same machinery available while the system is running. You hand the Builder a recipe (a PsySpec, plus optional runtime steps like shell commands), and it bakes it — creating components, running steps — streaming progress back as it goes. A Supervisor is the thing that decides what to build and when: it mints recipes, sends them to the right node’s Builder, watches the results, and can pause/retry/cancel a bake mid-flight.

Where this fits. Every Psyclone node already has a Builder. Since the strangler defaults were switched on, normal startup itself now runs through a built-in Supervisor (the Startup Supervisor) that turns your PsySpec into per-node bring-up recipes and only publishes `Psyclone.Ready` once every component exists. So you have already been using this machinery — this page shows how to drive it yourself.

## Builder vs Supervisor

| Builder | Supervisor
--- | --- | ---
Scope | One per node | Any number; a component (crank) that lives on a node
Job | Execute a recipe: create components, run steps, heal | Decide what to bake, dispatch recipes, observe outcomes, steer
Direction | Receives recipes & control; emits progress & data | Emits recipes & control; consumes progress & data
Intelligence | Deterministic step executor | Plain code today (the LLM tier is roadmap)

## A recipe is just a PsySpec you send at runtime

A recipe is written in the same XML as a PsySpec, parsed by the same code path (so `<include>` and `%variable%` substitution work identically). The difference is only when it runs: a recipe is submitted to a Builder and baked step-by-step while the node is live. The component tags create components exactly as at startup; `<cli>` runs a command as a step:

    <psyspec name="AddCamera">
      <module name="Camera7" node="Edge3">
        <crank name="CameraCapture" />
      </module>
      <cli name="Warmup" node="Edge3">./warmup.sh Camera7</cli>
    </psyspec>

Node pinning matters in a recipe. A step with no `node=` only runs on a config node. When a Supervisor sends a recipe to a specific node, it stamps each step with that node’s name for you — but if you hand-write a recipe for a remote node, pin every step with `node="…"` or it will be silently skipped.

The full recipe grammar — headers, `<cli>`/`<attach>`/`<code>`/`<command>`/`<if>` steps, `retryable`, control verbs and `<interactive>` sessions — is documented in the PsySpec XML Reference and, for administrators, the protocol wire format is in the System Guide §13.

## The two-stream conversation

Once a Supervisor sends a recipe, the exchange is a small, fixed set of messages, all correlated by one RecipeID string (`<supervisor>-<recipe>-N`). Everything flows over the normal pub/sub bus, so it works within a node or across nodes, and any component can “listen in” by subscribing:

Message | Direction | Carries
--- | --- | ---
`Psyclone.Builder.Recipe` | Supervisor → Builder | the recipe XML + identity (who, what, which node)
`Psyclone.Builder.Bake.Status` (& `.Success`/`.Failed`/`.Paused`/…) | Builder → Supervisor | progress only — percent, a one-line summary, control acks, the terminal outcome
`Psyclone.Builder.Bake.Data` | Builder → Supervisor | the actual step output (stdout/stderr), lossless and strictly in order
`Psyclone.Builder.BakeControl.*` | Supervisor → Builder | one control verb: Pause / Start / Cancel / SkipStep / RetryStep

The split between Status (progress) and Data (payload) is the important design choice: a dashboard can subscribe to just `Bake.Status` and get a clean progress readout without ever touching the potentially huge command output, while a log collector subscribes to `Bake.Data` and reassembles it gap-free. See the System Guide for the exact fields and guarantees.

## Writing a custom Supervisor

A Supervisor is not declared with a `<supervisor>` tag — that tag is still parsed-and-ignored today. Instead you write an ordinary continuous crank component that owns a `Supervisor` object. This is exactly how the built-in Startup Supervisor works, so you are modelling on shipped code, not inventing a mechanism.

The pattern has three parts:

1. Bind a `Supervisor` to the node with `initForCrank(node, name)`.
2. Dispatch one or more recipes with `startup(targetNode, recipeXML)` (local) or let it target a peer node by name once a `Psyclone.SystemStatus` has revealed that node’s id.
3. Loop, pumping every message your component’s `<trigger>`s deliver into `sup.processMessage(msg)`, and inspect `getBakeState()` / `bakeSucceeded()` to decide what to do next (retry, cancel, dispatch the next recipe, publish a ready signal, …).

    // A minimal custom Supervisor crank (C++). Registered like any crank.
    int8 MySupervisor(PsyAPI* api) {
      Node* node = Node::localNode;
      Supervisor sup;
      sup.initForCrank(node, api->getModuleName().c_str());

      // Send a recipe to the local node's Builder and remember its RecipeID.
      std::string rid = sup.startup(node, myRecipeXML);

      while (api->shouldContinue() && node->isContinuing()) {
        // Bake.Status / Bake.Data arrive here via the component's triggers.
        DataMessage* msg = api->waitForNewMessage(100, NULL);
        if (msg) sup.processMessage(msg);

        // React to outcomes.
        if (sup.bakeSucceeded(rid)) { /* dispatch the next recipe, signal ready... */ }
      }
      return -1;
    }

Declare the crank as a normal module so it lands as a live component (this is the ignition switch the bare `<supervisor>` tag does not give you):

    <module name="MySupervisor" node="Main">
      <crank name="MySupervisor" />
      <!-- subscribe to the reverse streams you want to observe -->
      <trigger name="Status" type="Psyclone.Builder.Bake.*" />
      <trigger name="Data"   type="Psyclone.Builder.Bake.Data" />
      <trigger name="Sys"    type="Psyclone.SystemStatus" />
    </module>

Steering a bake. To pause, resume, cancel, skip or retry a running bake, call `sup.sendControl(recipeID, "pause"|"start"|"cancel"|"skip"|"retry")`. The Builder applies the verb at the next step boundary (Cancel/Terminate is immediate, even mid-`<cli>`) and acknowledges it back on the Status stream as `kind="ack"`.
Listening in. Because the reverse streams are ordinary pub/sub, a Supervisor on one node observes a bake on another for free, and a read-only dashboard is just a component that subscribes to `Psyclone.Builder.Bake.*`. A Supervisor only tracks recipes it issued itself — a pure observer reads the fields off the messages directly.
Honest limits (this build). Supervisors are plain code — the LLM-driven “sim-manager” tier is roadmap. Sending recipes to a remote node’s Builder works; getting the reverse streams back across a node boundary relies on subscription sync and is verified single-node but still being hardened multi-node. Cohort messaging and the system-status API are not built yet. Build against what is on this page, not the older design PDF.