8. Catalogs

What is a catalog?

A catalog is an advanced module. Like any module it can have <trigger>, <crank> and <post> elements and take part in publish/subscribe dataflow — but in addition it answers synchronous queries. A caller sends a query and blocks (up to a timeout) for the reply; the catalog receives the query as a special control message and answers it with queryReply. This makes catalogs the natural home for two things:

• Queryable data stores — files, key/value data, recorded message streams, world models — with a simple query language.
• Conduits to other systems — search services, recognition servers, other robots: the catalog translates Psyclone queries into whatever the external system speaks.

In the PsySpec a catalog is declared with the <catalog> element; its type attribute names the crank function that implements it (bare names get the PsySystem:: prefix and resolve in the bundled System library):

<catalog name="MessageDataCatalog" type="MessageDataCatalog">
  ...triggers, parameters, setup...
</catalog>

The query/reply interaction

Queries are declared in the PsySpec with <query> and referred to by name in code — the same name/type indirection used for triggers and posts, so a query can be repointed at another catalog (or a remote system) without touching code. Result status codes are shared with whiteboard retrieves:

Built-in catalogs

File Catalog

Exposes a directory tree as a queryable store. Reads and writes files by name below a root directory:

<catalog name="MyFiles" type="FileCatalog" root="./">
  <parameter name="ReadOnly" type="String" value="no" />
</catalog>

<!-- in the querying module -->
<query name="MyFiles" source="MyFiles" subdir="test" ext="txt" binary="yes" />

In code the string-based query API names the entry and the operation:

char* result = NULL; uint32 datasize = 0;
uint8 status = api->queryCatalog(&result, datasize, "MyFiles", "test", "read");
// reads ./test/test.txt ; write with operation "write" plus data+datasize
if (status == PsyAPI::QUERY_SUCCESS) { /* use result */ }
delete [] result;   // caller owns the returned buffer

result = api.queryCatalog("MyFiles", "test", "read")
# reads ./test/test.txt ; write with operation "write" plus a data argument
if result is not None:
    pass  # use result (no manual buffer management in Python)

Setting ReadOnly to yes blocks write operations. The subdir, ext and binary attributes on the <query> declaration scope which files the query addresses.

Data Catalog

A central key/value datastore backed by a single file. Entries are read and written by name exactly like the File Catalog; the in-memory store is flushed to disk at the configured interval:

<catalog name="MyData" type="DataCatalog" interval="2000" root="./mydata.dat" />
<query name="MyData" source="MyData" />

Replay Catalog

Records live message activity in one system and plays it back in another (or the same one, later) — the foundation of Psyclone's repeatable-testing workflow. Recording is just a catalog with triggers for the types to capture:

<catalog name="Replay1" type="ReplayCatalog" root="./replay1"
         maxsize="10240000" maxcount="2000">
  <trigger name="Video"  type="robot.sensor.video" />
  <trigger name="Bumper" type="robot.sensor.bumper" />
</catalog>

It keeps the most recent 2,000 messages and at most ~10 MB, whichever limit is hit first. Playback uses the same catalog with <post> entries instead: posted types may differ from the recorded ones, but the trigger/post names must match the recording. Messages replay at their recorded intervals; interval="1000" overrides the posting interval, and rotate="yes" loops from the beginning when the recording ends.

Request Store Catalog

Collects messages system-wide, keeps the most recent, and serves them (whole or per field) over HTTP — the easiest way to expose live data to a browser or to custom PsyProbe views:

<catalog name="RequestStore" type="RequestStore">
  <trigger name="VideoFrame" type="robot.sensor.video" />
  <setup>
    <store name="VideoFrame" trigger="VideoFrame" datatype="raw" mimetype="image/bmp" />
    <store name="Status" trigger="Status" key="content" mimetype="application/json" />
  </setup>
</catalog>

Fetch with GET /api/query?from=RequestStore&query=VideoFrame. Store attributes:

The closely related MessageDataCatalog uses the same store table to expose whole messages and binary blobs; CoCoMaps used it to serve robot camera frames (USB/Color/Depth/IR) and status JSON to its operator UI.

Creating custom catalogs

A custom catalog is just a crank function with state and query handling: an exported int8 MyCatalog(PsyAPI* api) in a library, declared with <catalog type="MyLib::MyCatalog">. Two query APIs exist on the caller side:

// string-based
uint8 queryCatalog(char** result, uint32 &resultsize, const char* name,
                   const char* query, const char* operation = NULL,
                   const char* data = NULL, uint32 datasize = 0,
                   uint32 timeout = 5000);
// message-based (full DataMessage in, DataMessage out)
uint8 queryCatalog(DataMessage** resultMsg, const char* name,
                   DataMessage* msg, uint32 timeout = 5000);

# string-based: returns the result data (or None)
result = api.queryCatalog(name, query, operation=None, data=None,
                          timeout=5000)
# message-based (full DataMessage in, DataMessage out)
resultMsg = api.queryCatalog(name, msg, timeout=5000)

On the catalog side the crank runs a normal continuous loop and detects queries by message type:

while (api->shouldContinue()) {
  DataMessage* inMsg = api->waitForNewMessage(100);
  if (!inMsg) continue;
  if (inMsg->getType() == PsyAPI::CTRL_QUERY) {
    uint32 id = (uint32) inMsg->getReference();
    // string-query params arrive as string entries: Subdir, Ext, Binary, Operation
    DataMessage* reply = new DataMessage();
    reply->setString("Answer", "42");
    api->queryReply(id, PsyAPI::QUERY_SUCCESS, reply);
  } else {
    // ordinary trigger message: ingest data, post output, etc.
  }
}

while api.shouldContinue():
    inMsg = api.waitForNewMessage(100)
    if inMsg is None: continue
    if inMsg.getType() == cmsdk.PsyAPI.CTRL_QUERY:
        id = inMsg.getReference()
        # string-query params arrive as string entries: Subdir, Ext, Binary, Operation
        reply = cmsdk.DataMessage()
        reply.setString("Answer", "42")
        api.queryReply(id, cmsdk.PsyAPI.QUERY_SUCCESS, reply)
    else:
        pass  # ordinary trigger message: ingest data, post output, etc.

Three queryReply forms are available: status only; status plus a raw data buffer (data, size, count); and status plus a DataMessage. Anything the crank keeps in local variables (or private data) is the catalog's store — there is no framework beyond the query protocol.

Real-world custom catalogs: CoCoMaps

The CoCoMaps multi-robot project built its whole shared world model as custom catalogs. The CCMCatalog (Collaborative Cognitive Map) holds maps, identities, live objects, timestamped observations, navigation points, and distributed roles and tasks; one master serves several robots, each slave running a same-named CCMProxyCatalog that forwards queries to the master. A CCMCollector turns ordinary pub/sub messages into world-model observations. Trimmed from the real spec:

<catalog name="CCMMaster" type="CCMCatalog">
  <parameter name="SystemID" type="Integer" value="%SystemID%" />
  <parameter name="StorageDir" type="String" value="%DataDir%/CCMData" />
  <trigger name="ObjectInfo" type="Object.Information" />
  <post name="RoleAssigned" type="Role.Assigned" />
  <post name="TaskCreated"  type="Task.Created" />
  <setup>
    <map name="main" view="10" invertx="yes" />
    <identity id="1" type="robot" name="RobotOne" nickname="Turtle1" />
    <roles>
      <role name="Searcher" type="Shared" />
      <role name="Communicator" type="Exclusive" />
    </roles>
  </setup>
</catalog>

The TimeSeriesCatalog shows a different flavour: its <setup> declares named, colour-coded datasets and <store> bindings that extract values from trigger messages into capped time series, served both to querying modules and to a live scrolling PsyProbe chart. One query loop serves module queryCatalog calls and PsyProbe HTTP requests (detected via an HTTP_OPERATION entry, answered with a JSON string) — the pattern behind every custom PsyProbe tab.