CMLabs · Psyclone AIOS

# 9. Advanced Applications: the CoCoMaps Case Study

How a real multi-robot human–robot-interaction system pushed Psyclone to its limits: four custom catalogs, multi-system federation, roles and tasks, ROS, Python cranks, and PsyProbe as the operator UI.

## The system

CoCoMaps (Collaborative Cognitive Maps, EU Echord++) put two TurtleBot robots and humans in a shared space: speech dialogue with turn-taking, face and human detection, ROS navigation, and joint task execution (including robots operating a simulated wall control panel). Each robot ran its own complete Psyclone system — partly onboard, partly on a server over WiFi, on top of ROS — and the systems cooperated by sharing observations, roles and tasks through one master Collaborative Cognitive Map. It is the best available worked example of “advanced Psyclone”, and everything in this chapter is drawn from its real specs and source.

The heart of the design is a set of four custom catalogs. A catalog in Psyclone is just a crank function with state and query handling — an exported `int8 Fn(PsyAPI*)`, declared with `<catalog type="...">`, whose `<setup>` XML arrives as the `componentsetup` parameter string. All four follow the same skeleton: a `waitForNewMessage` loop that answers `CTRL_QUERY` messages from modules and HTTP calls from PsyProbe from the same code path.

Catalog | Holds | PsyProbe view
--- | --- | ---
CCMCatalog | The shared world model: maps, identities, objects, observations, roles, tasks | 2-D live map (jCanvas)
TaskManagerCatalog | A hierarchical task machine defined in JSON | Interactive task tree (vis.js)
TimeSeriesCatalog | Typed, capped time-series extracted from messages | Live scrolling chart (Smoothie)
ControlPanel | A screen/menu console with options and PINs | The panel’s own rendered HTML

## System shape: one federation, many Psyclone systems

Each robot is an independent Psyclone system; the master robot (or server) runs the real CCM, and every other system runs a same-named proxy. Drag the diagram to explore the real dataflow:

Blue = modules/pipelines, orange = catalogs, charcoal = system boundaries. The CCM proxy on each slave forwards queries to the master over a remote `<query host=... port=...>`; modules never know the CCM is remote.

- Perception → CCM: ROS robot status and vision modules post `Self.Position.Data`, `Human.Self.Detected`, `input.speech.detected`…; CCMCollector catalogs map these into typed observations and push them to the CCM.
- Roles drive modes: a RoleNegotiator per robot takes/leaves CCM roles (Searcher/Communicator…); small gating modules translate `Self.Role.*` into `dialog.on/off`, `navigation.auto.on/off`, `cmd.input.audio.on/off` — switching the whole system’s operating mode via messages.
- Tasks close the loop: TaskNegotiator/TaskExecutor modules negotiate CCM tasks; executing one posts `Robot.Request.Navigate.NamedPoint` (resolved against CCM named points) and panel commands: robot ⇆ world model ⇆ actuation.
- Dialogue stack: microphone → Nuance speech reco/TTS, pitch tracking, and a turn-taking model run as a Python crank (`<crank name="YTTM" language="python2" script="%YTTMDir%/yttm.py"/>`), with the GUI in its own external space.
- ROS is just messages: the ROSInterface library wraps navigation/odometry/camera topics as cranks (including a simulator variant), so ROS is another message source/sink inside the psySpec.
- Observability & replay: a `MessagesOfInterest` whiteboard subscribes wildcards (`dialog.*`, `Role.*`, `Task.*`…) and a ReplayCatalog records ~30 message types for the repeatable-demo `*_static_*` spec variants (see chapter 7).
- PsyProbe as the app UI: `<psyprobe location="%PsyDir%/html">` with an `<alias name="robot" .../>` serving a custom robot web UI, plus manual-test `<post>` entries so operators can inject commands from the browser.

Tip. One spec family served master, slave, virtual (simulator) and static-replay variants, kept portable with `<variable>` definitions and `<include file="system.inc"/>` — the same technique any multi-deployment Psyclone project should use.

## CCMCatalog: the Collaborative Cognitive Map

The CCM is the shared world model: maps (floor plans with origin/offset/scale/bitmap), identities (known robots and humans, with portrait and face images), objects (live entities with type, identity link, first/last-seen, presence, media streams), timestamped observations (typed time series per object — e.g. Location points, String utterances), vantage points and named points (navigation targets), plus distributed roles and tasks with full assignment history. One master CCM serves the whole federation.

### Behaviour

Startup reads three parameters — `SystemID` (this system’s ID in the federation), `MaxKeep` (max observations kept per typed store, default 1000) and `StorageDir` (disk directory, default `CCMData`) — then parses its `<setup>` (maps, identities, vantage/named points, roles). The main loop waits on `api->waitForNewMessage(100)` and answers `CTRL_QUERY` messages whose `Query` string entry selects the operation:

Group | Queries | Notes
--- | --- | ---
Data ingest | `CreateObject`, `UpdateObject`, `AddDataToObject`, `AddStreamToObject`, `AddImage`, `UpdateMap`, `AddObservations` | Fed mostly by CCMCollector bundles
Reads | `map`, `identity`, `identities`, `object`, `objects`, `all`, plus generic `query=range`/`value` observation queries | Also served over HTTP to PsyProbe (`/api/query?from=<name>&query=...`); web calls are detected via an `HTTP_OPERATION` entry and answered with a `JSON` string entry
Navigation | `GetNamedPoint`, `GetVantagePoint`, `SetVantagePoint`, `FreeVantagePoint` | Resolve targets for the navigation stack
Federation | `CCMHeartbeat`, `GetLocalSystemID`, `post` | `post` relays a message to the other systems via queued posts
Roles | `takerole`, `leaverole`, `giverole` | On success also posts `RoleAssigned`/`RoleLeft`/`RoleGiven`
Tasks | `createtask`, `canceltask`, `assigntask`, `accepttask`, `updatetask`, `completetask` | Posts `TaskCreated/Assigned/Accepted/Updated/Completed/Cancelled`; a background check posts `TaskTimeout`

Replies go back with `api->queryReply(inMsg->getReference(), status, replyMsg)`. Roles and tasks are `EXCLUSIVE` or `SHARED`, task states run Created → Assigned → Accepted → Status → Success/Failed/Timedout/Cancelled, and every transition is kept in a history log.

### PsySpec declaration (real, trimmed — master)

    <catalog name="CCMMaster" type="CCMCatalog">
      <parameter name="SystemID" type="Integer" value="%SystemID%" />
      <parameter name="StorageDir" type="String" value="%DataDir%/CCMData" />
      <trigger name="ObjectInfo" type="Object.Information" />
      <post name="CommandNavigateTo" type="Robot.Request.Navigate" />
      <post name="RoleAssigned" type="Role.Assigned" />
      <post name="TaskCreated"  type="Task.Created" />
      <post name="TaskTimeout"  type="Task.Timeout" />
      <!-- ...more Robot.Command.* / Report.* / Role.* / Task.* posts... -->
      <setup>
        <map name="main" view="10" invertx="yes" />
        <identity id="1" type="robot" name="RobotOne" nickname="Turtle1"
                  image="%DataDir%/turtlebot.bmp" />
        <identity id="3" type="human" name="Thor List" nickname="Thor"
                  image="%DataDir%/thor.bmp" face="%DataDir%/thor_face720.bmp" />
        <vantagepoints>
          <point id="1" label="%VantagePoint1Name%" data="%VantagePoint1Data%"
                 overlap="%VantagePoint1Overlap%" />
        </vantagepoints>
        <namedpoints>
          <point label="ControlPanel1" data="%PanelPoint1Data%" />
        </namedpoints>
        <roles>
          <role name="Searcher" type="Shared" />
          <role name="Communicator" type="Exclusive" />
        </roles>
      </setup>
    </catalog>

### The proxy: transparent federation

Slave systems run CCMProxyCatalog under the same name as the master CCM (“must be named the same as the master CCM in every other system”). It forwards queries to the real master through a remote query route and re-emits the master’s events locally through the same post list — so slave modules are written exactly as if the CCM were local:

    <catalog name="CCMMaster" type="CCMProxyCatalog">
      <parameter name="SystemID" type="Integer" value="%SystemID%" />
      <query name="Master" source="CCMMaster" host="%MasterAddress%" port="%MasterPort%" />
      <trigger name="ObjectInfo" type="Object.Information" />
      <!-- same post list as the master, so events re-emit locally -->
    </catalog>

### Feeding it: CCMCollector

CCMCollector is a bridge catalog that turns ordinary pub/sub messages into CCM observations. Its `<setup>` holds `<obs>` entries with `<mapping>` children; each trigger extracts fields from the message and sends one `Query=AddObservations` bundle to the CCM via `api->queryCatalog(&resultMsg, "Master", outMsg)`:

    <catalog name="PositionCollector1" type="CCMCollector">
      <query name="Master" source="CCMMaster" />
      <trigger name="RobotStatus"    type="Self.Position.Data" />
      <trigger name="HumanDetection" type="Human.Self.Detected" />
      <trigger name="IncomingSpeech" type="input.speech.detected" />
      <setup>
        <obs name="Location" obstype="Location" trigger="HumanDetection">
          <mapping entry="x" key="PosX" />
          <mapping entry="y" key="PosY" />
        </obs>
        <obs name="Utterance" obstype="String" trigger="IncomingSpeech">
          <mapping entry="val" key="Utterance" />
        </obs>
      </setup>
    </catalog>

### PsyProbe element

The catalog registers its own view with `api->addPsyProbeCustomView("Cognitive Map", "elements/ccmcatalog.html")` — a jCanvas 2-D map that draws the floor-plan bitmap, plots objects at their latest Location observations, overlays identity portraits (fetched via `/api/query?...&query=identity&datatype=image&id=N`), shows role/task badges per identity, and lets an operator click to post commands. It is the canonical example of a catalog serving both DataMessage queries (to modules) and JSON/image HTTP queries (to PsyProbe) from one query loop.

Companion cranks in the same source complete the pipeline: RoleNegotiator (default/primary/secondary role climbing with `UpgradeRole`/`DowngradeRole` triggers), TaskNegotiator (reacts to `PerformTask`/`CancelTask` and CCM task events), TaskExecutor (reacts to `TaskAssigned`/`TaskTimeout`/`TaskCancelled` and drives robot actions), and CCMTester (a periodic self-test).

## TaskManagerCatalog: hierarchical task scripting

A declarative hierarchical task machine. Tasks (with arbitrarily nested subtasks) are defined in a JSON file; each task lists message triggers and message posts per event type (`fail, success, complete, stop, timeout, start, next, previous, restart, startnow`), plus `timeout` and start/between delays. The catalog turns the whole tree into pub/sub behaviour: incoming messages advance tasks, timers fire timeouts, and transitions post new messages.

Its single parameter is `TaskFile` (required). Two implementation details make it a masterclass in advanced catalog technique:

- Runtime subscription registration. After parsing the JSON it collects all trigger/post types and builds a `<subscription>` XML fragment at runtime, registering it with `api->addSubscription(...)` — subscriptions generated from a data file, not the psySpec. Posts that would self-trigger are dropped with a warning.
- Timer queue + trigger matching. The loop computes its wait from the next pending timer, matches incoming types via `api->typeToText(inMsg->getType())`, advances the tree, and emits `api->postOutputMessage(post)` for each resulting post.

    <psySpec>
      <catalog name="TaskManager" type="TaskManagerCatalog">
        <parameter name="TaskFile" type="String" value="tasks.json" />
      </catalog>
    </psySpec>

And the task file it consumes (real, trimmed — the robot searches for, then identifies, a participant):

    [{ "name": "Search Participant",
       "triggers": { "start":   ["task.participants.none"] },
       "posts":    { "success": ["task.conversation.start"] },
       "delays":   { "start": 1000, "between": 1000 },
       "subtasks": [
         { "name": "Search for Participant", "timeout": 120000,
           "triggers": { "success": ["task.participants.tracked"],
                         "fail":    ["task.participants.none"] },
           "posts":    { "start":   ["task.participants.search"],
                         "timeout": ["task.participants.none"] } },
         { "name": "Identify Participant", "timeout": 30000, "...": "..." } ] }]

For PsyProbe it registers a TaskView tab (`elements/hierarchicaldiagram.html`) and publishes the live task tree as private data (`api->setPrivateData("JSON", ..., "application/json")`). The viewer is built on vis.js Network — the same offline bundle this guide uses — fetching the JSON via `/api/getcomponentdata` and rendering an interactive node graph with live status.

## TimeSeriesCatalog: typed time-series & live charting

Records values extracted from messages into named, colored, capped datasets (integer, float, string, time) and serves them to queriers and to a live scrolling PsyProbe chart. CoCoMaps used it to visualise turn-taking dynamics — who has the floor, speech on/off, pitch — in real time. It has no parameters at all: everything is in the `<setup>`, with two tag kinds:

- `<dataset name datatype color shoulders shading maxkeep maxfrequency />` — declares a series (default `maxkeep` 1000; `maxfrequency` f becomes a minimum ms-apart). `shoulders` is `"0"` (drop to zero around each sample) or `"last"` (repeat previous value) and controls the step shape; `shading` is fill opacity.
- `<store dataset trigger key timekey durationkey value datatype />` — binds a message trigger to a dataset: extract `key` from the message (single values, arrays or maps — spread across the duration), or store a fixed `value` (e.g. 50 when a state turns on).

    <catalog name="TimeSeriesCatalog" type="TimeSeriesCatalog">
      <trigger name="SpeechOff"      type="input.speech.off"/>
      <trigger name="IHaveTurn"      type="dialog.on.i-have-turn"/>
      <trigger name="OtherHasTurn"   type="dialog.on.other-has-turn"/>
      <trigger name="IStartSpeaking" type="output.audio.started"/>
      <trigger name="IStopSpeaking"  type="output.audio.ended"/>
      <setup>
        <dataset name="OtherPitch" datatype="integer" color="orange" shading="0"  shoulders="0" />
        <store dataset="OtherPitch" trigger="SpeechOff" key="pitchvalue" datatype="float" />

        <dataset name="ISpeak" datatype="integer" color="purple" shading="50" shoulders="last" />
        <store dataset="ISpeak" trigger="IStartSpeaking" value="50" datatype="integer" />
        <store dataset="ISpeak" trigger="IStopSpeaking"  value="0"  datatype="integer" />

        <dataset name="IHaveTurn" datatype="integer" color="blue" shading="80" shoulders="last" />
        <store dataset="IHaveTurn" trigger="IHaveTurn"    value="20" datatype="integer" />
        <store dataset="IHaveTurn" trigger="OtherHasTurn" value="0"  datatype="integer" />
      </setup>
    </catalog>

Queries: `Query=<datasetname>` (with `sinceid`/`sincetime`/`maxcount`/`maxage`/`minage`) returns the dataset as JSON; `List=all` returns metadata for every dataset. The PsyProbe view (`elements/timeseriescatalog.html`) is a live strip chart built on SmoothieChart: it discovers datasets via `list=all`, then polls `query=<dataset>&sinceid=<lastid>` per series; clicking pauses/resumes the stream. (Smoothie for streaming, vis.js for graphs, Chart.js for static charts — three complementary options for custom views.)

## ControlPanel: an interactive screen/menu catalog

ControlPanel simulates an operator console: a hierarchy of screens with selectable options, optional PIN-protected screens, and flashing selection feedback. Robots (via dialogue and tasks) or humans (via the PsyProbe view) navigate it — CoCoMaps demos had a robot walk to a wall panel and shut down “Generator 1”. It is a catalog rather than a module because it holds queryable state (the current screen) rather than transforming message streams.

Parameters: `inifile` (path to a JSON screen definition — despite the name; required) and `print` (debug). Operations return a `PanelResponse`: `SUCCESS, UNKNOWN, BUSY, NOACCESS, ENTERPIN, NONE, FAILED`. Queries distinguish web calls (an `HTTP_OPERATION` entry present) from module queries: `HTML` (rendered screen for PsyProbe), `OptionSelect`, `BackNavigation`, `Reset`, `EnterPIN`, `AddPIN`, `ReadScreen`, `GetUpdatedView`.

### The bridge pattern: ControlPanelController

A companion crank bridges pub/sub to the panel: for every incoming trigger it copies the message, sets `Query=<triggerName>` and calls `api->queryCatalog(&resultMsg, "ControlPanel", msg, 5000)`; based on the `PanelResponse` it posts `<TriggerName>Success` or `<TriggerName>Failed` with an `Error` string (Option Unknown / Panel Busy / Access Denied / PIN Required / Catalog Query Timeout…). This convention-based response posting is a reusable pattern for bridging query/reply into pub/sub anywhere:

    <module name="PanelController1">
      <query name="ControlPanel" source="Panel1" />
      <trigger name="Reset"        type="Console.Command.Reset" />
      <trigger name="OptionSelect" type="Console.Command.Select" />
      <trigger name="EnterPIN"     type="Console.Command.EnterPIN" />
      <crank name="PanelController1" function="PsySystem::ControlPanelController" />
      <post name="OptionSelectSuccess"  type="Console.Command.Select.Success" />
      <post name="OptionSelectUnknown"  type="Console.Command.Select.Unknown" />
      <post name="OptionSelectPIN"      type="Console.Command.Select.EnterPIN" />
      <post name="EnterPINSuccess"      type="Console.Command.EnterPIN.Success" />
    </module>

    <catalog name="Panel1" type="ControlPanel">
      <parameter type="String" name="inifile" value="panel.json" />
      <parameter type="String" name="print"   value="yes" />
    </catalog>

The same spec scripts a full panel interaction with a `MessageScript` crank — a `<setup><track>` of timed `<post timems="+5000" name="OptionSelect" key="OptionName" value="status_screen"/>` entries — Psyclone’s built-in way to script test interactions. The PsyProbe element (`elements/controlpanel.html`) is a thin shell that polls `query=html` and injects the panel’s own rendered HTML; clicks post back `OptionSelect`/`EnterPIN`/`Reset`. So across the four catalogs you see three PsyProbe integration styles: canvas drawing (CCM), charting (TimeSeries), and catalog-rendered HTML (ControlPanel).

Note. A fourth, built-in relative completes the family: MessageDataCatalog stores whole last-N messages or extracted blobs per named store — CoCoMaps used it to expose robot camera frames (USB/Color/Depth/IR/Registered/Map as `image/bmp` + raw) and robot status (JSON/text/XML) to PsyProbe. Contrast: MessageDataCatalog stores messages/blobs, TimeSeriesCatalog stores extracted scalar series, and the CCM stores a structured world model.

## What to steal for your own systems

- Catalogs are cranks with state. All four are plain exported functions; the `<setup>` XML arrives as the `componentsetup` parameter. Start from any module and add `CTRL_QUERY` handling.
- One query loop, two audiences. Serve module `queryCatalog` calls and PsyProbe HTTP (detect `HTTP_OPERATION`, answer with a `JSON` entry) from the same code path — the pattern behind every custom PsyProbe tab.
- Runtime subscriptions (`api->addSubscription`) let data files, not just the psySpec, define dataflow.
- Same-name proxy catalogs give transparent multi-system federation — slave modules don’t know the CCM is remote.
- Convention-based response posting (`<Trigger>Success`/`<Trigger>Failed`) bridges query/reply into pub/sub.
- Variables + `<include>` keep one spec family portable across master/slave/virtual/replay variants.

PreviousDistributed Admin NextTesting (CMSDK & PsyTest)