8. Distributed Admin

The moving parts

Every Psyclone instance starts a local Node which manages communication and bookkeeping: subscriptions, services, interfaces, the local components, and the local process spaces. A distributed system is simply one master node (started with the PsySpec) plus any number of satellite nodes on other machines — possibly running different operating systems — that the master configures remotely. Conceptually there are three layers an administrator manages:

Satellite mode: starting the workers

To spread a system across machines, start idle nodes on each worker — Psyclone with no spec, optionally with an explicit port:

# on each worker machine
./bin/linux64/Psyclone port=11000

An idle node waits to be claimed. Then, in the master’s PsySpec, declare the nodes and place components on them by name:

<node name="Node1" address="localhost" port="11000" />
<node name="Node2" address="otherhost" port="10000" />

<module name="Ping" node="Main">  ...  </module>
<module name="Pong" node="Node1"> ...  </module>

The local (master) node is implicit and referred to as Main. Components with no node= attribute run on the startup node. Placement works identically for whiteboards and catalogs (including their key/keytype, FileCatalog root and DataCatalog interval attributes) — storage components can live wherever the data is.

Code libraries are distributed globally by default; a node can override where a library comes from:

<node name="Node3" address="192.168.20.20" port="10000">
  <library name="otherlib" library="../path/mylib.dll" />
</node>

Spaces: process isolation per node

Each node auto-creates the internal space Root (all components as threads in one OS process). Declaring more spaces creates separate OS processes:

<space name="YTTMSpace" />
<module name="YTTM" space="YTTMSpace"> ... </module>

If a space crashes, the node restarts the space and recreates its components; private (persistent) data survives. In a multi-node system a plainly-declared space is created on all nodes; restrict it with <space node="Node1" name="YTTMSpace" /> and place the component with both attributes. External processes can also host a space (<space name="GUISpace" type="external" /> plus a placeholder module) — the external program links CMSDK and connects with the PsySpace API; its crash and reconnection are tolerated like internal spaces. See the User Guide distributed chapter for the developer-side detail.

Topology: keep high-volume dataflow local

The single most important distributed design rule: bandwidth-heavy pipelines stay on one node; only distilled results travel. A camera producing raw frames should be processed (detection, feature extraction) on the machine it is attached to, with only compact events (Human.Self.Detected, positions, classifications) crossing the node link. CoCoMaps ran exactly this shape: each robot processed its own vision and audio onboard and shared only observations, roles and tasks with the server. The interactive diagram below shows a typical production topology — drag nodes to explore.

Blue = modules, orange = storage catalogs/whiteboards, charcoal = node engines. Thick edges = high-volume local dataflow (never leaves the machine); thin edges = distilled cross-node messages over the node links.

Load balancing & migration

Placement is the first-order load-balancing tool: watch PsyProbe’s Node Performance tab (input/output rates, queues, CPU, memory over the last 1/10/30 s) and move components between nodes in the spec. Nodes can be added and removed while the system runs. Runtime component migration exists in the engine but is not yet a hands-off balancing feature (see the Roadmap note below) — today, rebalancing means editing placement in the spec and restarting the affected spaces/nodes.

Runtime module migration is mostly built into the engine (external modules are marked migratable, and per-component private data is designed to survive a move), and the spec accepts a per-component opt-in:

<module name="Analyser" migration="yes"> ... </module>

Roadmap The migration attribute is parsed and sets the component’s migrate-allow flag, but automatic runtime migration as a hands-off admin feature is still being finalised — on the current roadmap it becomes a builder-owned action (SYS_BUILDER_MOVE, driving the existing migration machinery); see Roadmap Features. For today’s production systems, treat migration as spec-driven re-placement plus space restarts, and design modules to be movable: keep state in private data or catalogs, not in process-local globals.

Talking between separate systems

Multi-node is one system spread out. Sometimes you instead want several independent Psyclone systems (one per robot, per site…) that cooperate. The mechanism is the remote query: identical to a local <query> plus a host and port:

<!-- local -->
<query name="Master" source="CCMMaster" />
<!-- remote: another Psyclone system -->
<query name="Master" source="CCMMaster" host="other.hostname" port="10010" />

The remote component receives and answers as normal; the reply is routed back transparently. A component can detect that a query came from another system by the extra entries in the query message: INTERSYSTEM_IDENTIFICATION (string), INTERSYSTEM_ADDRESS (uint32 IP), INTERSYSTEM_PORT (uint16) and INTERSYSTEM_SOURCENAME. Chapter 9 shows this pattern federating a whole robot fleet through proxy catalogs.

Production checklist

• Ports & reachability: every declared <node> address/port must be reachable from the master at startup; pick fixed ports and open only those.
• Security: node-to-node and remote-query links carry your dataflow — apply the SSL configuration from chapter 5 (verification is on by default; self-signed requires an explicit allowselfsigned="yes").
• Libraries: ensure each node can load every library its components need (global by default, per-node override available); mind OS differences (.dll vs .so is handled automatically by name).
• Clocks: DataMessage timestamps are synchronised across computers to microsecond resolution by the platform, but sane NTP on all machines keeps logs and external correlation honest.
• Observability: use PsyProbe’s System Node Communication matrix to verify that high-volume types are not crossing node links, and Node Performance to spot overloaded machines before users do.
• Failure drills: kill a space and a satellite node in staging; confirm space restart and system behaviour match expectations before relying on them in production.