CMLabs · Psyclone AIOS

# 10. Testing (CMSDK & PsyTest)

Two complementary suites ship with Psyclone: the CMSDK unit-test framework (fast, isolated, per-object) and PsyTest (an XML-driven whole-system function + performance test).

## Two suites, two jobs

Suite | What it exercises | How you run it | Runtime
--- | --- | --- | ---
CMSDK unit tests | Individual SDK objects and engine behaviours: networking, SSL, timers, message index, config variables, per-module logging… | `Psyclone test=cmsdk` (or a single `test=<name>`) | Seconds to a couple of minutes
PsyTest | A full running Psyclone system: pub/sub, contexts, signals, whiteboards, catalogs — with performance numbers | `Psyclone spec=Examples/psytest.xml` | ~35 s

## The CMSDK unit-test framework

All unit tests are registered into one unified framework and driven from the Psyclone binary itself. The basics:

    # run the full default suite
    ./bin/linux64/Psyclone test=cmsdk

    # list every registered test, grouped by category, with descriptions
    ./bin/linux64/Psyclone test=list

    # run a single test in isolation
    ./bin/linux64/Psyclone test=network_sslverify
    ./bin/linux64/Psyclone test=psy_configvars

(`test=all` and `test=sdk` are accepted aliases for `test=cmsdk`.) A test is any function `bool Func()` registered by name; the suite prints one status line per test and exits 0 only if every test passed — ready for CI.

### Options

Option | Default | Meaning
--- | --- | ---
`fork=0` | fork=1 | Run tests in-process instead of one forked process per test — essential under a debugger. With forking on (the default where supported), a hung test is killed on timeout and the framework reports which progress phase it died in.
`verbose=1` | off | Show the tests’ verbose diagnostics (`unittest::detail(...)` output).
`out=<file>` | auto | Override the path of the performance JSON written after a run.
`outdir=<dir>` | auto | Directory for output files.
`compare=<oldfile>` | — | Compare this run’s recorded metrics against a previous run’s JSON and show deltas — a lightweight performance-regression check.

### Interpreting the output

Each test prints a `[ PASS ]` or `[ FAIL ]` line; failures include the reason set by the test (`unittest::fail("...")`). While a test runs on a terminal, a live progress line shows percent and current action (`unittest::progress(pct, action)`); piped/redirected output stays clean. Tests can also record named performance metrics (`unittest::metric(name, value, unit)`) which appear on the PASS line and land in the performance JSON used by `compare=`.

### Tests worth knowing by name

Test | Verifies
--- | ---
`psy_configvars` | `%variable%` substitution uses the spec’s `<variable>` defaults and that a `name=value` command-line argument overrides them (command line wins).
`psy_modulelogging` | Per-module `verbose`/`debug`/`logfile` attributes take effect.
`network_sslverify` | SSL certificate verification default (verify-peer; self-signed rejected unless allowed).
`network_sslhostca` | SSL hostname verification and custom-CA (`cafile`/`capath`) handling.

Warning. The SSL tests require the SSL-enabled build (`make ssl` on Linux, the “Release SSL” configuration on Windows). On a non-SSL binary they are not available — a skipped/absent SSL test on a plain build is expected, not a regression.
Tip. Debugging one test: `./bin/linux64/Psyclone test=<name> fork=0 verbose=1` under `gdb` keeps everything in a single process with full diagnostics.

## PsyTest: the whole-system function + performance test

PsyTest is a self-checking Psyclone system: an ordinary PsySpec (`Examples/psytest.xml`) whose modules (cranks in `Examples/src/PsyTest.cpp`, library `Examples`) exercise the main single-node features in a chain of stages, each printing one pass/fail line with performance data:

    ./bin/linux64/Psyclone spec=Examples/psytest.xml

The stage chain runs sequentially, each stage activating the next via a Done post to a sibling `Test.*` context:

`Intro → TimeTest → PingTest → PingTestUDP → Signals → Contexts → Catalogs → Done`

Stage | Exercises | Reports
--- | --- | ---
TimeTest | Clock round-trip timing, master + slaves | rounds, replies, avg rtt
PingTest | Reliable (guaranteed) ping/pong message ring | msgs, throughput, avg age + tail latency
PingTestUDP | Best-effort ring (`guaranteed="no"`) | msgs, throughput, avg age
Signals | Signal emit/receive ring | signals, throughput, avg age
Contexts | Dynamic context switching + `interval` timer triggers | switches routed ok, heartbeats
Catalogs | Whiteboard, File and Data catalog retrieval | steps, ops ok, ms
Summary | Aggregates all stages | `PSYTEST RESULT: SUCCESS/FAILED`, then shuts the system down

### Tuning the run

The spec exposes its knobs as `<variable>` definitions (`Nodes`, `PingCount`, `PingCycles`, `SignalCount`, `ContextRounds`…), all overridable from the command line. The ping stages also take a `PayloadSize` parameter (bytes of message payload per ball; 0 = empty), so you can measure throughput and latency at realistic message sizes:

    # heavier ping load with 4 KB payloads
    ./bin/linux64/Psyclone spec=Examples/psytest.xml PingCount=100 PingCycles=100 PayloadSize=4096

### Tail latency

The ping masters collect a per-message latency sample for every ball and report tail-latency percentiles alongside the averages, formatted as:

    p50=42.1us p90=61.3us p95=70.8us p99=112.5us p99.9=413.0us

Averages hide stalls; p99/p99.9 expose scheduling hiccups, queue buildup and GC-like pauses. Track the percentiles across builds if latency matters to your application.

### Reading the output

- Each stage prints exactly one `[ PASS ]`/`[ FAIL ]` line (log level 0) with its performance data; per-message chatter is at verbose level ≥ 2 (`verbose=2` to see it).
- Stall watchdogs: the ping/signal masters detect a stalled ring (no progress for ~3 s) and report `[ FAIL ] … stalled` instead of hanging forever.
- During the context-switching stage, `Msg filtered in getTriggersForMsg, filter 1` lines are expected and benign: they are the inactive responder’s pings being correctly filtered — proof the switch worked.
- A process-global result table aggregates per-stage results for the final verdict (all modules run in one process on a single node).
- PsyProbe runs throughout at `http://localhost:10000` — watching the Activity and Performance tabs during a PsyTest run is an excellent way to learn both.

## Adding your own tests

### A new CMSDK unit test

Write a `bool` function using the `unittest::` helpers and register it (engine-level tests live in `Psyclone/src/PsycloneTests.cpp`; SDK tests register next to the object they test, e.g. the SSL tests in `NetworkManager.cpp`):

    static bool UnitTest_MyThing() {
        unittest::progress(10, "setting up");
        // ... exercise the object ...
        if (bad) { unittest::fail("expected X, got Y"); return false; }
        unittest::metric("ops_per_s", ops, "ops/s");
        return true;
    }
    // at registration time:
    UnitTestRunner::instance().registerTest("my_thing", UnitTest_MyThing,
        "one-line description", "category", true /* in default run */);

It is then runnable as `test=my_thing`, appears in `test=list`, and (with the last argument true) joins the default `test=cmsdk` suite. Use `unittest::detail(...)` for verbose-only diagnostics and call `unittest::progress()` at phase boundaries so a timeout can report where a hang occurred.

### A new PsyTest stage

Add a crank to `Examples/src/PsyTest.cpp` (declared `extern "C"` in `include/PsyTest.h`), declare its module in `Examples/psytest.xml` under a new sibling `Test.*` context, chain it in by having the previous stage’s Done post activate your context (and yours activate the next), report into the shared result table, and print a single `[ PASS ]/[ FAIL ]` line at level 0 with a stall watchdog if your stage forms a ring. The existing stages are short and idiomatic — copy the nearest one.

Note. For scripted application-level tests (timed message injections against your own spec), the built-in `MessageScript` crank used in the CoCoMaps ControlPanel spec is often all you need — a `<setup><track>` of timed `<post>` entries, no C++ required.
PreviousAdvanced Applications NextRoadmap Features