CMLabs · Psyclone AIOS

# 6. Debugging & Traces

Following messages through a live system: log levels, per-module logging, PsyProbe's activity and subscription views, message traces and the classic pitfalls.

## The debugging toolbox

Psyclone gives you four complementary views into a running system:

- Logs — verbose/debug output per topic and per component (console, node log, per-module log files).
- PsyProbe Activity — the live last-10-in/last-10-out message view per component.
- PsyProbe Subscriptions & Dataflow — who is subscribed to what, with manual test firing, and the graphical dataflow map.
- Recordings — capture message traffic with a Replay Catalog and replay it offline (see Recordings & Formats).

## Verbose and debug levels in practice

Levels run 0 (errors only) to 9 (everything); the default is 1/1. The workflow that scales:

    # 1. Production baseline
    ./Psyclone spec=sys.xml verbose=1 debug=0

    # 2. A subscription doesn't fire? Raise just those topics
    ./Psyclone spec=sys.xml verbose-subscriptions=7 verbose-triggers=7

    # 3. Node join problems? Network + sync
    ./Psyclone spec=sys.xml verbose-network=5 debug-sync=5

Topics: System, Network, Memory, Process, Sync, Node, Space, Component, Subscriptions, Triggers, Timing (full table in Administering Psyclone).

## Isolating one module Shipped

Per-module `verbose`/`debug`/`logfile` attributes are now fully wired — the highest-signal way to debug one component in a busy system:

    <module name="FaceFinder" verbose="7" debug="5" logfile="./log/facefinder.log">
      <trigger name="frame" type="robot.sensor.video" />
      <crank name="find" function="Perception::FaceFinder" />
      <post name="faces" type="percept.faces" />
    </module>

Everything else stays at baseline; FaceFinder's narrative goes to its own file. Your own cranks contribute via `api->logPrint(level, ...)` — those lines also appear in the component's Log tab in PsyProbe, and log lines from external processes/spaces are forwarded to node logging automatically (via `CTRL_LOGPRINT`).

## Reading logs

- Startup prints the resolved configuration — verify components, nodes and ports before anything else, and wait for SYSTEM READY.
- PsyProbe's top-line Log view aggregates all components with a free-text filter; each component's Log tab shows just its own output.
- Watch for negative post results in your own logging: `POST_FAILED (-1)` error, `POST_NOSPEC (-2)` posted before any trigger arrived, `POST_OUTOFCONTEXT (-3)` the context changed and the crank is no longer active.

## Inspecting subscriptions and dataflow in PsyProbe

Most “my module never fires” problems are solved in three PsyProbe views:

- System Subscriptions — every subscription, hierarchical by message type. Check the exact type string your trigger registered (typos and wildcard scope are the usual suspects). Every trigger and post has a [test] button that fires it manually with an empty message — the fastest way to prove the downstream path works.
- Component Activity — the last 10 incoming and 10 outgoing messages, live. Hover for full content, click for a static window; incoming messages show their source component. If input arrives but output never appears, the problem is inside the crank (or a filter/context gate).
- Dataflow — the graphical component/flow map, with filters for messages and components on big systems. Great for spotting a broken link in a pipeline at a glance.

Tip. Work backwards: find the last component whose Activity tab shows the message, then check the next hop's trigger in System Subscriptions — wrong type string, an over-strict `<filter>`, a `maxage` that skips lagging messages, or an inactive context are the four usual gates.

## Message traces and timing

Every DataMessage carries a header like an email (time, from, to, type) with microsecond-resolution timestamps synchronised across computers — so end-to-end latency of a pipeline can be read straight off the message headers in Activity windows. The Performance tabs show data volume over the last 1/10/30 seconds per component and per node (input/output/queues/CPU/memory); a growing queue pinpoints the bottleneck component. For repeatable traces, record the flow with a Replay Catalog and replay it against a test spec — identical input, every run (chapter 7).

## Common pitfalls

Symptom | Cause & fix
--- | ---
Crashes/garbage after changing a struct shared between modules | Shared-memory layouts must match across all binaries. After changing any shared struct, run `make clean` and rebuild everything — stale objects with the old layout are a classic source of “impossible” crashes.
Interval trigger never fires | An `interval=` timer only starts after the module has been triggered once by a real message — give it a kick-start trigger (e.g. `Psyclone.Ready`).
Module silently does nothing | Check for `POST_OUTOFCONTEXT`: a context switch deactivated the crank. External modules must check `getCurrentTriggerContext()` themselves.
Debug build loads the wrong library | Debug builds first look for `ExamplesDebug.dll` / `libExamplesDebug.so`, then the release name — a stale debug library shadows your fresh release build.
Messages vanish with no subscriber | Unmatched messages are discarded unless a `ttl` keeps them in shared memory. Verify subscriptions in PsyProbe before assuming loss.
Trigger skips messages under load | A `maxage` filter is doing its job — deliberately skipping stale messages. Confirm whether that is wanted before “fixing” it.

PreviousSecurity & SSL NextRecordings & Formats