CMLabs · Psyclone AIOS

# 8. Catalogs

Catalogs are components you can ask questions: data stores, recorders and bridges to other systems, all answering synchronous queries alongside normal pub/sub traffic.

## What is a catalog?

A catalog is an advanced module. Like any module it can have `<trigger>`, `<crank>` and `<post>` elements and take part in publish/subscribe dataflow — but in addition it answers synchronous queries. A caller sends a query and blocks (up to a timeout) for the reply; the catalog receives the query as a special control message and answers it with `queryReply`. This makes catalogs the natural home for two things:

- Queryable data stores — files, key/value data, recorded message streams, world models — with a simple query language.
- Conduits to other systems — search services, recognition servers, other robots: the catalog translates Psyclone queries into whatever the external system speaks.

In the PsySpec a catalog is declared with the `<catalog>` element; its `type` attribute names the crank function that implements it (bare names get the `PsySystem::` prefix and resolve in the bundled System library):

    <catalog name="MessageDataCatalog" type="MessageDataCatalog">
      ...triggers, parameters, setup...
    </catalog>

Note. A whiteboard is itself a special catalog that stores only messages and adds a rich retrieval query language. This chapter covers the general-purpose catalogs and how to build your own.

## The query/reply interaction

A catalog query is synchronous for the caller: `queryCatalog` blocks until the catalog calls `queryReply` or the timeout expires. Inside the catalog the query arrives as an ordinary message of type `CTRL_QUERY`.

Queries are declared in the PsySpec with `<query>` and referred to by name in code — the same name/type indirection used for triggers and posts, so a query can be repointed at another catalog (or a remote system) without touching code. Result status codes are shared with whiteboard retrieves:

Code | Value | Meaning
--- | --- | ---
`QUERY_FAILED` | 1 | General failure
`QUERY_TIMEOUT` | 2 | No reply within the timeout
`QUERY_NAME_UNKNOWN` | 3 | No query with this name declared
`QUERY_COMPONENT_UNKNOWN` | 4 | Target component not found
`QUERY_QUERYFAILED` | 5 | The catalog answered with a failure
`QUERY_SUCCESS` | 6 | Reply received
`QUERY_NOT_AVAILABLE` | 7 | Target exists but cannot serve queries
`QUERY_NOT_REACHABLE` | 8 | Target unreachable (e.g. remote host down)

## Built-in catalogs

### File Catalog

Exposes a directory tree as a queryable store. Reads and writes files by name below a root directory:

    <catalog name="MyFiles" type="FileCatalog" root="./">
      <parameter name="ReadOnly" type="String" value="no" />
    </catalog>

    <!-- in the querying module -->
    <query name="MyFiles" source="MyFiles" subdir="test" ext="txt" binary="yes" />

In code the string-based query API names the entry and the operation:

    char* result = NULL; uint32 datasize = 0;
    uint8 status = api->queryCatalog(&result, datasize, "MyFiles", "test", "read");
    // reads ./test/test.txt ; write with operation "write" plus data+datasize
    if (status == PsyAPI::QUERY_SUCCESS) { /* use result */ }
    delete [] result;   // caller owns the returned buffer

    result = api.queryCatalog("MyFiles", "test", "read")
    # reads ./test/test.txt ; write with operation "write" plus a data argument
    if result is not None:
        pass  # use result (no manual buffer management in Python)

Setting `ReadOnly` to `yes` blocks write operations. The `subdir`, `ext` and `binary` attributes on the `<query>` declaration scope which files the query addresses.

### Data Catalog

A central key/value datastore backed by a single file. Entries are read and written by name exactly like the File Catalog; the in-memory store is flushed to disk at the configured interval:

    <catalog name="MyData" type="DataCatalog" interval="2000" root="./mydata.dat" />
    <query name="MyData" source="MyData" />

### Replay Catalog

Records live message activity in one system and plays it back in another (or the same one, later) — the foundation of Psyclone's repeatable-testing workflow. Recording is just a catalog with triggers for the types to capture:

    <catalog name="Replay1" type="ReplayCatalog" root="./replay1"
             maxsize="10240000" maxcount="2000">
      <trigger name="Video"  type="robot.sensor.video" />
      <trigger name="Bumper" type="robot.sensor.bumper" />
    </catalog>

It keeps the most recent 2,000 messages and at most ~10 MB, whichever limit is hit first. Playback uses the same catalog with `<post>` entries instead: posted types may differ from the recorded ones, but the trigger/post names must match the recording. Messages replay at their recorded intervals; `interval="1000"` overrides the posting interval, and `rotate="yes"` loops from the beginning when the recording ends.

Tip. The CoCoMaps robots used a ReplayCatalog (`maxsize="1000000000"`, ~30 message types) to record live demos and re-run them offline against the full processing stack. See the System Guide case study.

### Request Store Catalog

Collects messages system-wide, keeps the most recent, and serves them (whole or per field) over HTTP — the easiest way to expose live data to a browser or to custom PsyProbe views:

    <catalog name="RequestStore" type="RequestStore">
      <trigger name="VideoFrame" type="robot.sensor.video" />
      <setup>
        <store name="VideoFrame" trigger="VideoFrame" datatype="raw" mimetype="image/bmp" />
        <store name="Status" trigger="Status" key="content" mimetype="application/json" />
      </setup>
    </catalog>

Fetch with `GET /api/query?from=RequestStore&query=VideoFrame`. Store attributes:

Attribute | Meaning
--- | ---
`name` | Request name used in the URL (required)
`trigger` | Which catalog trigger feeds this store (required)
`key` | A user data entry, or the header fields `time` (µs since year 0), `timetext`, `type`, `size`, `content`; omitted ⇒ the whole message rendered as JSON/XML/HTML/TEXT per mimetype
`keytype` | Reserved — not currently used
`datatype` | `raw` = raw pixels; the message must contain width/height and a valid bitmap is built
`mimetype` | Required; standard (`application/json`, `text/html`, `text/xml`) or short forms `json`/`xml`/`text`
`maxkeep` | History count kept (default 1)
`maxfrequency` | Max messages kept per second

The closely related MessageDataCatalog uses the same store table to expose whole messages and binary blobs; CoCoMaps used it to serve robot camera frames (USB/Color/Depth/IR) and status JSON to its operator UI.

## Creating custom catalogs

A custom catalog is just a crank function with state and query handling: an exported `int8 MyCatalog(PsyAPI* api)` in a library, declared with `<catalog type="MyLib::MyCatalog">`. Two query APIs exist on the caller side:

    // string-based
    uint8 queryCatalog(char** result, uint32 &resultsize, const char* name,
                       const char* query, const char* operation = NULL,
                       const char* data = NULL, uint32 datasize = 0,
                       uint32 timeout = 5000);
    // message-based (full DataMessage in, DataMessage out)
    uint8 queryCatalog(DataMessage** resultMsg, const char* name,
                       DataMessage* msg, uint32 timeout = 5000);

    # string-based: returns the result data (or None)
    result = api.queryCatalog(name, query, operation=None, data=None,
                              timeout=5000)
    # message-based (full DataMessage in, DataMessage out)
    resultMsg = api.queryCatalog(name, msg, timeout=5000)

On the catalog side the crank runs a normal continuous loop and detects queries by message type:

    while (api->shouldContinue()) {
      DataMessage* inMsg = api->waitForNewMessage(100);
      if (!inMsg) continue;
      if (inMsg->getType() == PsyAPI::CTRL_QUERY) {
        uint32 id = (uint32) inMsg->getReference();
        // string-query params arrive as string entries: Subdir, Ext, Binary, Operation
        DataMessage* reply = new DataMessage();
        reply->setString("Answer", "42");
        api->queryReply(id, PsyAPI::QUERY_SUCCESS, reply);
      } else {
        // ordinary trigger message: ingest data, post output, etc.
      }
    }

    while api.shouldContinue():
        inMsg = api.waitForNewMessage(100)
        if inMsg is None: continue
        if inMsg.getType() == cmsdk.PsyAPI.CTRL_QUERY:
            id = inMsg.getReference()
            # string-query params arrive as string entries: Subdir, Ext, Binary, Operation
            reply = cmsdk.DataMessage()
            reply.setString("Answer", "42")
            api.queryReply(id, cmsdk.PsyAPI.QUERY_SUCCESS, reply)
        else:
            pass  # ordinary trigger message: ingest data, post output, etc.

Three `queryReply` forms are available: status only; status plus a raw data buffer (`data, size, count`); and status plus a `DataMessage`. Anything the crank keeps in local variables (or private data) is the catalog's store — there is no framework beyond the query protocol.

## Real-world custom catalogs: CoCoMaps

The CoCoMaps multi-robot project built its whole shared world model as custom catalogs. The CCMCatalog (Collaborative Cognitive Map) holds maps, identities, live objects, timestamped observations, navigation points, and distributed roles and tasks; one master serves several robots, each slave running a same-named CCMProxyCatalog that forwards queries to the master. A CCMCollector turns ordinary pub/sub messages into world-model observations. Trimmed from the real spec:

    <catalog name="CCMMaster" type="CCMCatalog">
      <parameter name="SystemID" type="Integer" value="%SystemID%" />
      <parameter name="StorageDir" type="String" value="%DataDir%/CCMData" />
      <trigger name="ObjectInfo" type="Object.Information" />
      <post name="RoleAssigned" type="Role.Assigned" />
      <post name="TaskCreated"  type="Task.Created" />
      <setup>
        <map name="main" view="10" invertx="yes" />
        <identity id="1" type="robot" name="RobotOne" nickname="Turtle1" />
        <roles>
          <role name="Searcher" type="Shared" />
          <role name="Communicator" type="Exclusive" />
        </roles>
      </setup>
    </catalog>

The TimeSeriesCatalog shows a different flavour: its `<setup>` declares named, colour-coded datasets and `<store>` bindings that extract values from trigger messages into capped time series, served both to querying modules and to a live scrolling PsyProbe chart. One query loop serves module `queryCatalog` calls and PsyProbe HTTP requests (detected via an `HTTP_OPERATION` entry, answered with a JSON string) — the pattern behind every custom PsyProbe tab.

Note. The full CCM architecture — federation, roles/tasks negotiation, collectors, the Cognitive Map web view — is a System Guide case study: see Advanced applications: CoCoMaps.
PreviousWhiteboards NextData Messages