CMLabs · Psyclone AIOS

# 12. Running Psyclone in Production

Service management, health checks, log rotation, network exposure, backup, runbooks, change management, upgrades, capacity and secrets — the operational glue around the runtime, including an honest list of what you must build yourself today.

Psyclone gives you the runtime; production operability — service supervision, health endpoints, log rotation, backup tooling, validated deploys — is largely your responsibility today. This chapter is the checklist: for each concern it gives the concrete recipe where one exists, links the chapter that covers the underlying mechanism, and says plainly where the platform has a gap you must work around. Nothing here duplicates chapters 1–11; it wires them together for an operator on call.

## Service & daemon management

Psyclone has no built-in engine supervisor. A crashed process space is restarted by its node (see chapter 8), but if the engine/node process itself dies, nothing inside Psyclone restarts it — only an external supervisor (systemd on Linux, a Windows service wrapper) brings it back. Run every production engine and every satellite node under one. A minimal systemd unit:

    # /etc/systemd/system/psyclone.service
    [Unit]
    Description=Psyclone AIOS engine
    After=network-online.target
    Wants=network-online.target

    [Service]
    User=psyclone
    WorkingDirectory=/opt/psyclone            # deploy dir containing html/ and the spec
    ExecStart=/opt/psyclone/Psyclone spec=/opt/psyclone/robot.xml port=10000 html=/opt/psyclone/html
    Restart=on-failure
    RestartSec=5

    [Install]
    WantedBy=multi-user.target

    sudo systemctl daemon-reload
    sudo systemctl enable --now psyclone
    journalctl -u psyclone -f          # follow the node log if it goes to stdout

Satellite nodes get the same unit with `ExecStart=/opt/psyclone/Psyclone port=11000` (no spec — satellite mode, chapter 8).

Stopping. The clean shutdown path in Psyclone is message-driven: a component posts `Psyclone.Shutdown`, which drives `CTRL_SYSTEM_SHUTDOWN` to all nodes and processes (chapter 2). What happens on SIGTERM / Ctrl-C is not documented or guaranteed by the current code base — we do not claim that in-flight messages are drained or that catalogs flush on signal.

Verify on your build. Before relying on `systemctl stop` (which sends SIGTERM), test in staging: does the process exit promptly, are in-flight messages lost, does the DataCatalog end up consistent? If you need guaranteed-clean shutdown, trigger the `Psyclone.Shutdown` message path first (e.g. via a small admin component or a PsyProbe test post, chapter 3), then let systemd stop the process.

## Health checks & liveness

There is no dedicated `/health` endpoint today Roadmap — treat that as a feature gap and plan for it. Practical interim signals:

- Startup: watch the node log for the SYSTEM READY line (chapter 6) before declaring a deploy live.
- Human health: PsyProbe’s main page, Activity and Node Performance tabs (chapter 3, chapter 8) are the live dashboard.
- Scripted liveness (poor man’s probe): the PsyProbe HTTP API (chapter 3) answers plain GETs, so a monitoring script can poll it. Declare a RequestStore that a heartbeat module posts into, then:

    # Liveness: does the engine answer HTTP at all, and is the store fresh?
    curl -fsS 'http://127.0.0.1:10000/api/query?from=RequestStore&query=Status' || alert
    # Component-published metrics (module publishes JSON it computes itself):
    curl -fsS 'http://127.0.0.1:10000/api/getcomponentdata?compid=15&name=Health&format=json'

An HTTP timeout or connection refusal means the engine is down or wedged; a stale timestamp in the store means the publishing module (and whatever feeds it) has stalled even though HTTP still answers. Have a module publish queue depths or last-processed timestamps into the store to get a crude backlog gauge. A first-class health/readiness endpoint is a strong candidate for the roadmap; until then this scrape is the supported pattern.

## Logging in production

Two shipped mechanisms (chapter 1): the node log (system-wide `verbose=`/`debug=` and per-topic forms) and the per-module `logfile=` attribute for routing one component’s output to its own file. Run production at `verbose=1 debug=0` and raise levels per topic/module only while investigating.

Warning — logs are unbounded. Psyclone has no built-in log rotation or size cap. A long-running node at elevated verbosity will fill the disk. Rotate externally.

    # /etc/logrotate.d/psyclone
    /opt/psyclone/log/*.txt {
        daily
        rotate 14
        compress
        missingok
        notifempty
        copytruncate   # the long-running process keeps the file handle open;
                       # copytruncate rotates without needing a reopen signal
    }

Verify on your build. The exact default log destination and line format are not pinned down in this guide — confirm on your build where node and per-module log lines land (stdout vs file) and test that `copytruncate` interacts correctly with Psyclone’s file handles (a small write-after-rotate test in staging is enough). If the node logs to stdout under systemd, journald does the rotation for you instead.

## Ports & network exposure

Everything a firewall/security-group review needs in one place. Ports actually used by a deployment:

Port | What rides it | Where set
--- | --- | ---
Main port (default 10000) | Inter-system comms, PsyProbe UI + HTTP API, Telnet console, Message protocol (autodetected on one port) | `port=` cmdline / `<psySpec port=>` (ch. 1)
Dedicated PsyProbe port (optional) | PsyProbe only, if configured; `defaultport="no"` removes it from the main port | `<psyprobe port=>` (ch. 3)
Per-node satellite ports | Master ↔ satellite node traffic, one port per node | `<node port=>` / satellite `port=` (ch. 8)
Custom interface ports | Declared `<interface port=>` HTTP/Telnet/Message/RAW endpoints | ch. 4 (custom services currently stubbed)
Remote-query ports | Cross-system `<query>` with host+port | ch. 8

Warning — no authentication exists. Network ACLs are your only access control. PsyProbe is an unauthenticated HTTP control surface — it can inject context changes and fire test posts, not just observe (ch. 3). The Telnet console is an unauthenticated interactive console on the main port (ch. 4). The interface `<authentication>` tag and encryption attributes are parsed-but-stubbed Stub (ch. 5). Anyone who can reach the main port can control the system. Bind to loopback or a private/management interface, firewall every port in the table to trusted hosts only, disable the Console (`<interface protocol="Telnet" service="none" />`) if unused, and front PsyProbe with an authenticating reverse proxy (nginx + auth + TLS) when humans need remote access.

## Backup, recovery & the state inventory

Know what is persistent before you need the restore. Everything stateful in a stock deployment:

State | What it is | Where configured | Copy while running?
--- | --- | --- | ---
DataCatalog storage | Persistent catalog data on disk | `<catalog>` `root`/parameters (User Guide ch. on Catalogs) | Not guaranteed — verify crash-consistency for your build; prefer stop-or-snapshot
FileCatalog roots | File-backed catalog directories | `<catalog>` `root` | Files being written may be torn — verify
Recording roots | ReplayCatalog recordings | `<catalog type="ReplayCatalog" root=>` (ch. 7) | Live recording appends — copy completed recordings, or pause first
CCM StorageDir | CoCoMaps-style component disk store (default `CCMData`) | `StorageDir` parameter (ch. 9) | Verify — component-managed writes
Per-component private data | Anything your own modules write to disk | Your spec/parameters | You know your write pattern — document it
Specs, launcher scripts, certs | The deployment itself | Deploy dir | Yes (static) — keep in version control

Whiteboards are in-memory only: their contents are lost on restart. If a whiteboard holds state you cannot recompute, a module must persist it (to a catalog or file) itself.

Basic approach. Backup: stop the system (or take a filesystem snapshot) and copy the directories in the table plus the spec. Restore: put the files back in the same paths, redeploy the matching binary + spec, restart, watch for SYSTEM READY. There is no backup tooling in the platform today Roadmap — script it, and test the restore in staging before you need it.

## Failure semantics & runbook

Symptom | What to do
--- | ---
Engine/node process crash | systemd restarts it (above). Collect the node log tail and any core dump before it is overwritten; check which space/component was active at crash time (ch. 6).
Space crash | The owning node restarts the space automatically (ch. 8); repeated restarts of the same space point at the component inside it — raise its per-module `verbose`/`debug` (ch. 1).
Hung system (HTTP answers but nothing flows) | Raise `verbose` per topic, inspect PsyProbe Activity and queues, and capture a native stack of the process (`gdb -p <pid>` / eu-stack) if you can, before restarting.
Satellite node drop | See ch. 8 for node lifecycle. The exact rejoin/failover behaviour after a network partition should be drilled in staging and confirmed for your build — do not assume components migrate or resume automatically.
Message backlog | A growing queue means a downstream bottleneck. There is no documented backpressure or queue limit; messaging is backed by shared memory, so watch shared-memory usage and treat sustained growth as an incident. Fix by speeding up/parallelising the consumer or shedding input.
SSL handshake failures | Raise `verbose-network`, look for OpenSSL error strings in the log, then check `cafile`/`capath`/`allowselfsigned` scope per ch. 5 (expired cert and hostname mismatch are the usual causes).

## Change management: deploying spec changes

There is no dry-run or validate mode today Roadmap — the only way to find out whether a spec change works is to load it. Safe procedure: edit the spec → start it on a staging instance (same binary, different port/machine) and watch for SYSTEM READY and expected components in PsyProbe → then deploy to production.

Warning — silent failure modes. The parser is permissive: unknown or typo’d attributes are silently ignored, and misdeclared `system=` triggers are silently ignored rather than rejected. The multi-alias parse quirk on PsyProbe aliases (ch. 3) means a list can partially apply. And `%variable%` substitution is literal text replacement (ch. 1) — a bad command-line value can produce invalid XML or a semantically wrong spec with no warning at the substitution step. A typo can therefore deploy “successfully” while a trigger simply never fires.

- Keep specs (and launcher scripts) in version control; deploy by tag, roll back by tag.
- Validate XML well-formedness pre-deploy after variable substitution where possible: `xmllint --noout mysystem.xml`.
- Diff the resolved startup configuration printout (ch. 6) against the previous deploy — missing components mean a silently-ignored declaration.

## Upgrade & rollback

The hard rule comes from the shared-memory ABI (ch. 6 pitfall): when a CMSDK shared-memory struct changes between versions, every node and every native library must be rebuilt and deployed together. Mixed-version nodes across such a change are unsafe — keep the master and all satellites on the same build, always.

- Upgrade: stage the new binary + libraries + spec on all machines, stop all nodes, deploy everywhere, restart satellites then master, watch SYSTEM READY.
- Rollback: redeploy the previous binary + spec from version control the same way. Keep the last known-good deploy directory intact on each machine.
- Data across versions: recording and DataCatalog on-disk format compatibility across engine versions is not documented — verify by replaying a recording from the old version on the new build in staging before you depend on it.

## Capacity & tuning

Honest status: hard limits (max nodes, components, message size, queue depth) are not documented — establish your envelope empirically. What is known:

- Keep high-volume dataflow node-local; ship only distilled results across node links (ch. 8). Placement is the first-order tuning knob.
- Spaces are processes: isolation costs a process (and IPC) per space. Group components that can share fate; isolate the crash-prone ones.
- Messaging rides shared memory; sizing and overflow behaviour are not documented — monitor shared-memory usage on loaded nodes and verify overflow behaviour for your build.
- Baseline before go-live: use PsyTest (ch. 10) on production-like hardware to measure throughput and p99 latency for your message sizes and shapes, and re-run it after upgrades to catch regressions.

## Secrets

Warning. Command-line `key=value` %variables% (ch. 1) are visible to every local user via `ps`/`/proc` — never pass credentials, API keys or passwords that way, and never commit them in the spec.

- Put secret-bearing settings in a separate spec fragment pulled in via `<include>`, owned by the service user with mode 600, excluded from version control.
- Or inject at launch from the environment/secret store via a launcher script that writes a transient mode-600 fragment (systemd `LoadCredential=` pairs well with this).
- There is no platform secret store today; the roadmap’s `<llm>` design (ch. 11) introduces `apikeyref`-style logical secret names — not built yet.

## Known operability gaps (roadmap candidates)

Plan around these. As of this build, Psyclone ships with no engine supervisor (use systemd), no health endpoint (scrape PsyProbe HTTP), no log rotation (use logrotate/journald), no authentication on PsyProbe/Telnet/interfaces (use network ACLs + reverse proxy), no spec validate/dry-run mode (use staging + xmllint), and no backup tooling (script stop/snapshot + copy). These are known gaps, not undocumented features — build your operational layer around them, and check the roadmap and release notes as they close.
PreviousRoadmap Features NextOverview