CMLabs · Psyclone AIOS

# 10. Distributed Systems

Spread one system across many computers with Nodes, isolate crashes with Spaces, and talk to entirely separate Psyclone systems with remote queries.

## Nodes: one system, many computers

Every Psyclone instance starts a local Node that manages communication and bookkeeping: subscriptions, services, interfaces, local components and local process spaces. To distribute a system, start idle satellite nodes on the other computers — simply run Psyclone with no spec (optionally `port=6789`) — then declare them in the master PsySpec:

    <node name="Node1" address="localhost" port="11000" />
    <node name="Node2" address="otherhost" port="10000" />

The local node is implicit and is referred to as Main. Components are placed with the `node=` attribute; anything without one runs on the startup node:

    <module name="Ping" node="Main"> ... </module>
    <module name="Pong" node="Node1"> ... </module>
    <whiteboard name="WB2" node="Node1" key="count" keytype="integer" />
    <catalog name="MyFiles" node="Node2" type="FileCatalog" root="./" />

Nodes can mix operating systems — a single system can span Windows and Linux machines. Crank libraries are global by default; override per node when a machine needs its own build:

    <node name="Node3" address="192.168.20.20" port="10000">
      <library name="otherlib" library="../path/mylib.dll" />
    </node>

## Spaces: process separation

Within a node, each Space is a separate OS process. Every node auto-creates the internal space Root (all its components run as threads in one process). Declaring more spaces buys crash isolation:

    <space name="YTTMSpace" />
    <module name="YTTM" space="YTTMSpace"> ... </module>

If a component in a space crashes, the node restarts that space and recreates its components — the rest of the system keeps running, and the components' private data survives. This is how commercial-grade and experimental code safely coexist. A plainly-declared space is created on all nodes; restrict it with `<space node="Node1" name="YTTMSpace" />`. Whiteboards and catalogs can be placed in spaces too.

Note. Modules with a Python crank automatically get their own space, so a misbehaving interpreter cannot take down the Root process.

### External spaces

An external space lets a third-party program host Psyclone components inside its own process. Declare the space with `type="external"` and give the module a crank name only:

    <space name="GUISpace" type="external" />
    <module name="CoCoMapsGUI" space="GUISpace">
      <crank name="CoCoMapsGUI" />
    </module>

The external program links CMSDK and connects in:

    PsySpace* space = new PsySpace("GUISpace");
    space->connect(sysid);
    space->start();
    PsyAPI* api = space->getCrankAPI("CoCoMapsGUI");  // retry in a loop on failure
    // ...then use api exactly like an internal module crank...

    space = cmsdk.PsySpace("GUISpace")
    space.connect(sysid)
    space.start()
    api = space.getCrankAPI("CoCoMapsGUI")  # retry in a loop on failure
    # ...then use api exactly like an internal module crank...

Monitor with `space->isConnected()` / `space->hasShutdown()`; on failure delete the space (but not the PsyAPI — it is owned by the PsySpace) and retry. External-space crashes are tolerated and reconnectable just like internal ones. A full walk-through is in Tutorials.

## Topology at a glance

The interactive graph below shows a typical two-computer deployment: drag nodes around, zoom, and note the boundaries — hosts contain Psyclone nodes, nodes contain spaces, spaces contain components.

Blue = modules, orange = whiteboards/catalogs, charcoal = nodes; dashed edge = network link between nodes.

## Communication between separate Psyclone systems

Nodes join computers into one system. Sometimes you instead want several independent Psyclone systems — for example one per robot — to cooperate. That is done with remote requests: a `<query>` pointed at a component in another system by host and port:

    <!-- local query -->
    <query name="Master" source="CCMMaster" />
    <!-- remote query: same code, different spec -->
    <query name="Master" source="CCMMaster" host="other.hostname" port="10010" />

The remote component receives and answers the query exactly as if it were local; the reply is routed back transparently. Because queries are referenced by name in code, switching a component from a local to a remote catalog is a pure spec change — the CoCoMaps slave robots ran a proxy catalog under the same name as the master's CCM so no module ever knew the world model lived on another machine.

### Recognising a remote query

A catalog that wants to know where a query came from can check for the extra entries stamped onto remote queries:

Entry | Type | Meaning
--- | --- | ---
`INTERSYSTEM_IDENTIFICATION` | string | Identification of the calling system
`INTERSYSTEM_ADDRESS` | int | Caller IP as uint32
`INTERSYSTEM_PORT` | int | Caller port as uint16
`INTERSYSTEM_SOURCENAME` | string | Name of the querying component

## Migration and load balancing

Components can be assigned to nodes manually as above, and the platform is designed for runtime migration of components between nodes for load balancing — marking a module `migration="yes"` flags it as movable. The operational side (when and how migration happens, monitoring node load, adding/removing nodes on a live system, and securing node links with SSL) is administration territory: see the System Guide chapter on distributed administration.

Warning. Per the current code audit, the `migration` and `priority` attributes are parsed but the runtime migration machinery is not fully acted on Stub — verify against your build before relying on automatic migration.
PreviousData Messages NextServices & Interfaces