6. Debugging & Traces

The debugging toolbox

Psyclone gives you four complementary views into a running system:

• Logs — verbose/debug output per topic and per component (console, node log, per-module log files).
• PsyProbe Activity — the live last-10-in/last-10-out message view per component.
• PsyProbe Subscriptions & Dataflow — who is subscribed to what, with manual test firing, and the graphical dataflow map.
• Recordings — capture message traffic with a Replay Catalog and replay it offline (see Recordings & Formats).

Verbose and debug levels in practice

Levels run 0 (errors only) to 9 (everything); the default is 1/1. The workflow that scales:

# 1. Production baseline
./Psyclone spec=sys.xml verbose=1 debug=0

# 2. A subscription doesn't fire? Raise just those topics
./Psyclone spec=sys.xml verbose-subscriptions=7 verbose-triggers=7

# 3. Node join problems? Network + sync
./Psyclone spec=sys.xml verbose-network=5 debug-sync=5

Topics: System, Network, Memory, Process, Sync, Node, Space, Component, Subscriptions, Triggers, Timing (full table in Administering Psyclone).

Isolating one module Shipped

Per-module verbose/debug/logfile attributes are now fully wired — the highest-signal way to debug one component in a busy system:

<module name="FaceFinder" verbose="7" debug="5" logfile="./log/facefinder.log">
  <trigger name="frame" type="robot.sensor.video" />
  <crank name="find" function="Perception::FaceFinder" />
  <post name="faces" type="percept.faces" />
</module>

Everything else stays at baseline; FaceFinder's narrative goes to its own file. Your own cranks contribute via api->logPrint(level, ...) — those lines also appear in the component's Log tab in PsyProbe, and log lines from external processes/spaces are forwarded to node logging automatically (via CTRL_LOGPRINT).

Reading logs

• Startup prints the resolved configuration — verify components, nodes and ports before anything else, and wait for SYSTEM READY.
• PsyProbe's top-line Log view aggregates all components with a free-text filter; each component's Log tab shows just its own output.
• Watch for negative post results in your own logging: POST_FAILED (-1) error, POST_NOSPEC (-2) posted before any trigger arrived, POST_OUTOFCONTEXT (-3) the context changed and the crank is no longer active.

Inspecting subscriptions and dataflow in PsyProbe

Most “my module never fires” problems are solved in three PsyProbe views:

• System Subscriptions — every subscription, hierarchical by message type. Check the exact type string your trigger registered (typos and wildcard scope are the usual suspects). Every trigger and post has a [test] button that fires it manually with an empty message — the fastest way to prove the downstream path works.
• Component Activity — the last 10 incoming and 10 outgoing messages, live. Hover for full content, click for a static window; incoming messages show their source component. If input arrives but output never appears, the problem is inside the crank (or a filter/context gate).
• Dataflow — the graphical component/flow map, with filters for messages and components on big systems. Great for spotting a broken link in a pipeline at a glance.

Message traces and timing

Every DataMessage carries a header like an email (time, from, to, type) with microsecond-resolution timestamps synchronised across computers — so end-to-end latency of a pipeline can be read straight off the message headers in Activity windows. The Performance tabs show data volume over the last 1/10/30 seconds per component and per node (input/output/queues/CPU/memory); a growing queue pinpoints the bottleneck component. For repeatable traces, record the flow with a Replay Catalog and replay it against a test spec — identical input, every run (chapter 7).

Common pitfalls