CMLabs · Psyclone AIOS

# 8. Distributed Admin

Running one Psyclone system across many machines: satellite nodes, component placement, spaces, load balancing, migration — and keeping high-volume dataflow local.

## The moving parts

Every Psyclone instance starts a local Node which manages communication and bookkeeping: subscriptions, services, interfaces, the local components, and the local process spaces. A distributed system is simply one master node (started with the PsySpec) plus any number of satellite nodes on other machines — possibly running different operating systems — that the master configures remotely. Conceptually there are three layers an administrator manages:

Layer | Boundary | Admin concern
--- | --- | ---
Node | Machine (one engine process per machine) | Where machines are, ports, libraries per node, network links, SSL
Space | OS process within a node | Crash isolation, restart policy, external processes
Component | Module / whiteboard / catalog | Placement (`node=`/`space=`), migration, load

## Satellite mode: starting the workers

To spread a system across machines, start idle nodes on each worker — Psyclone with no spec, optionally with an explicit port:

    # on each worker machine
    ./bin/linux64/Psyclone port=11000

An idle node waits to be claimed. Then, in the master’s PsySpec, declare the nodes and place components on them by name:

    <node name="Node1" address="localhost" port="11000" />
    <node name="Node2" address="otherhost" port="10000" />

    <module name="Ping" node="Main">  ...  </module>
    <module name="Pong" node="Node1"> ...  </module>

The local (master) node is implicit and referred to as Main. Components with no `node=` attribute run on the startup node. Placement works identically for whiteboards and catalogs (including their `key`/`keytype`, FileCatalog `root` and DataCatalog `interval` attributes) — storage components can live wherever the data is.

Code libraries are distributed globally by default; a node can override where a library comes from:

    <node name="Node3" address="192.168.20.20" port="10000">
      <library name="otherlib" library="../path/mylib.dll" />
    </node>

Tip. Because satellites are configured entirely from the master’s spec, upgrading a deployment usually means editing one file. The bundled `Examples/multinode.xml` and `multinodesimple.xml` specs are runnable references, and PsyProbe’s Nodes section shows per-node performance, spaces and activity live.

## Spaces: process isolation per node

Each node auto-creates the internal space Root (all components as threads in one OS process). Declaring more spaces creates separate OS processes:

    <space name="YTTMSpace" />
    <module name="YTTM" space="YTTMSpace"> ... </module>

If a space crashes, the node restarts the space and recreates its components; private (persistent) data survives. In a multi-node system a plainly-declared space is created on all nodes; restrict it with `<space node="Node1" name="YTTMSpace" />` and place the component with both attributes. External processes can also host a space (`<space name="GUISpace" type="external" />` plus a placeholder module) — the external program links CMSDK and connects with the `PsySpace` API; its crash and reconnection are tolerated like internal spaces. See the User Guide distributed chapter for the developer-side detail.

Warning. Isolation costs copies: messages between spaces cross process boundaries (shared memory), and messages between nodes cross the network. Put chatty component groups in the same space, and experimental or crash-prone code in its own space.

## Topology: keep high-volume dataflow local

The single most important distributed design rule: bandwidth-heavy pipelines stay on one node; only distilled results travel. A camera producing raw frames should be processed (detection, feature extraction) on the machine it is attached to, with only compact events (`Human.Self.Detected`, positions, classifications) crossing the node link. CoCoMaps ran exactly this shape: each robot processed its own vision and audio onboard and shared only observations, roles and tasks with the server. The interactive diagram below shows a typical production topology — drag nodes to explore.

Blue = modules, orange = storage catalogs/whiteboards, charcoal = node engines. Thick edges = high-volume local dataflow (never leaves the machine); thin edges = distilled cross-node messages over the node links.

## Load balancing & migration

Placement is the first-order load-balancing tool: watch PsyProbe’s Node Performance tab (input/output rates, queues, CPU, memory over the last 1/10/30 s) and move components between nodes in the spec. Nodes can be added and removed while the system runs. Runtime component migration exists in the engine but is not yet a hands-off balancing feature (see the Roadmap note below) — today, rebalancing means editing placement in the spec and restarting the affected spaces/nodes.

Runtime module migration is mostly built into the engine (external modules are marked migratable, and per-component private data is designed to survive a move), and the spec accepts a per-component opt-in:

    <module name="Analyser" migration="yes"> ... </module>

Roadmap The `migration` attribute is parsed and sets the component’s migrate-allow flag, but automatic runtime migration as a hands-off admin feature is still being finalised — on the current roadmap it becomes a builder-owned action (`SYS_BUILDER_MOVE`, driving the existing migration machinery); see Roadmap Features. For today’s production systems, treat migration as spec-driven re-placement plus space restarts, and design modules to be movable: keep state in private data or catalogs, not in process-local globals.

## Talking between separate systems

Multi-node is one system spread out. Sometimes you instead want several independent Psyclone systems (one per robot, per site…) that cooperate. The mechanism is the remote query: identical to a local `<query>` plus a host and port:

    <!-- local -->
    <query name="Master" source="CCMMaster" />
    <!-- remote: another Psyclone system -->
    <query name="Master" source="CCMMaster" host="other.hostname" port="10010" />

The remote component receives and answers as normal; the reply is routed back transparently. A component can detect that a query came from another system by the extra entries in the query message: `INTERSYSTEM_IDENTIFICATION` (string), `INTERSYSTEM_ADDRESS` (uint32 IP), `INTERSYSTEM_PORT` (uint16) and `INTERSYSTEM_SOURCENAME`. Chapter 9 shows this pattern federating a whole robot fleet through proxy catalogs.

## Production checklist

- Ports & reachability: every declared `<node>` address/port must be reachable from the master at startup; pick fixed ports and open only those.
- Security: node-to-node and remote-query links carry your dataflow — apply the SSL configuration from chapter 5 (verification is on by default; self-signed requires an explicit `allowselfsigned="yes"`).
- Libraries: ensure each node can load every library its components need (global by default, per-node override available); mind OS differences (`.dll` vs `.so` is handled automatically by name).
- Clocks: DataMessage timestamps are synchronised across computers to microsecond resolution by the platform, but sane NTP on all machines keeps logs and external correlation honest.
- Observability: use PsyProbe’s System Node Communication matrix to verify that high-volume types are not crossing node links, and Node Performance to spot overloaded machines before users do.
- Failure drills: kill a space and a satellite node in staging; confirm space restart and system behaviour match expectations before relying on them in production.

PreviousRecordings & Formats NextAdvanced Applications