12. Running Psyclone in Production
Service management, health checks, log rotation, network exposure, backup, runbooks, change management, upgrades, capacity and secrets — the operational glue around the runtime, including an honest list of what you must build yourself today.
Psyclone gives you the runtime; production operability — service supervision, health endpoints, log rotation, backup tooling, validated deploys — is largely your responsibility today. This chapter is the checklist: for each concern it gives the concrete recipe where one exists, links the chapter that covers the underlying mechanism, and says plainly where the platform has a gap you must work around. Nothing here duplicates chapters 1–11; it wires them together for an operator on call.
Service & daemon management
Psyclone has no built-in engine supervisor. A crashed process space is restarted by its node (see chapter 8), but if the engine/node process itself dies, nothing inside Psyclone restarts it — only an external supervisor (systemd on Linux, a Windows service wrapper) brings it back. Run every production engine and every satellite node under one. A minimal systemd unit:
# /etc/systemd/system/psyclone.service
[Unit]
Description=Psyclone AIOS engine
After=network-online.target
Wants=network-online.target
[Service]
User=psyclone
WorkingDirectory=/opt/psyclone # deploy dir containing html/ and the spec
ExecStart=/opt/psyclone/Psyclone spec=/opt/psyclone/robot.xml port=10000 html=/opt/psyclone/html
Restart=on-failure
RestartSec=5
[Install]
WantedBy=multi-user.targetsudo systemctl daemon-reload
sudo systemctl enable --now psyclone
journalctl -u psyclone -f # follow the node log if it goes to stdoutSatellite nodes get the same unit with ExecStart=/opt/psyclone/Psyclone port=11000 (no spec — satellite mode, chapter 8).
Stopping. The clean shutdown path in Psyclone is message-driven: a component posts Psyclone.Shutdown, which drives CTRL_SYSTEM_SHUTDOWN to all nodes and processes (chapter 2). What happens on SIGTERM / Ctrl-C is not documented or guaranteed by the current code base — we do not claim that in-flight messages are drained or that catalogs flush on signal.
systemctl stop (which sends SIGTERM), test in staging: does the process exit promptly, are in-flight messages lost, does the DataCatalog end up consistent? If you need guaranteed-clean shutdown, trigger the Psyclone.Shutdown message path first (e.g. via a small admin component or a PsyProbe test post, chapter 3), then let systemd stop the process.Health checks & liveness
There is no dedicated /health endpoint today Roadmap — treat that as a feature gap and plan for it. Practical interim signals:
- Startup: watch the node log for the SYSTEM READY line (chapter 6) before declaring a deploy live.
- Human health: PsyProbe’s main page, Activity and Node Performance tabs (chapter 3, chapter 8) are the live dashboard.
- Scripted liveness (poor man’s probe): the PsyProbe HTTP API (chapter 3) answers plain GETs, so a monitoring script can poll it. Declare a RequestStore that a heartbeat module posts into, then:
# Liveness: does the engine answer HTTP at all, and is the store fresh?
curl -fsS 'http://127.0.0.1:10000/api/query?from=RequestStore&query=Status' || alert
# Component-published metrics (module publishes JSON it computes itself):
curl -fsS 'http://127.0.0.1:10000/api/getcomponentdata?compid=15&name=Health&format=json'An HTTP timeout or connection refusal means the engine is down or wedged; a stale timestamp in the store means the publishing module (and whatever feeds it) has stalled even though HTTP still answers. Have a module publish queue depths or last-processed timestamps into the store to get a crude backlog gauge. A first-class health/readiness endpoint is a strong candidate for the roadmap; until then this scrape is the supported pattern.
Logging in production
Two shipped mechanisms (chapter 1): the node log (system-wide verbose=/debug= and per-topic forms) and the per-module logfile= attribute for routing one component’s output to its own file. Run production at verbose=1 debug=0 and raise levels per topic/module only while investigating.
# /etc/logrotate.d/psyclone
/opt/psyclone/log/*.txt {
daily
rotate 14
compress
missingok
notifempty
copytruncate # the long-running process keeps the file handle open;
# copytruncate rotates without needing a reopen signal
}copytruncate interacts correctly with Psyclone’s file handles (a small write-after-rotate test in staging is enough). If the node logs to stdout under systemd, journald does the rotation for you instead.Ports & network exposure
Everything a firewall/security-group review needs in one place. Ports actually used by a deployment:
| Port | What rides it | Where set |
|---|---|---|
| Main port (default 10000) | Inter-system comms, PsyProbe UI + HTTP API, Telnet console, Message protocol (autodetected on one port) | port= cmdline / <psySpec port=> (ch. 1) |
| Dedicated PsyProbe port (optional) | PsyProbe only, if configured; defaultport="no" removes it from the main port | <psyprobe port=> (ch. 3) |
| Per-node satellite ports | Master ↔ satellite node traffic, one port per node | <node port=> / satellite port= (ch. 8) |
| Custom interface ports | Declared <interface port=> HTTP/Telnet/Message/RAW endpoints | ch. 4 (custom services currently stubbed) |
| Remote-query ports | Cross-system <query> with host+port | ch. 8 |
<authentication> tag and encryption attributes are parsed-but-stubbed Stub (ch. 5). Anyone who can reach the main port can control the system. Bind to loopback or a private/management interface, firewall every port in the table to trusted hosts only, disable the Console (<interface protocol="Telnet" service="none" />) if unused, and front PsyProbe with an authenticating reverse proxy (nginx + auth + TLS) when humans need remote access.Backup, recovery & the state inventory
Know what is persistent before you need the restore. Everything stateful in a stock deployment:
| State | What it is | Where configured | Copy while running? |
|---|---|---|---|
| DataCatalog storage | Persistent catalog data on disk | <catalog> root/parameters (User Guide ch. on Catalogs) | Not guaranteed — verify crash-consistency for your build; prefer stop-or-snapshot |
| FileCatalog roots | File-backed catalog directories | <catalog> root | Files being written may be torn — verify |
| Recording roots | ReplayCatalog recordings | <catalog type="ReplayCatalog" root=> (ch. 7) | Live recording appends — copy completed recordings, or pause first |
| CCM StorageDir | CoCoMaps-style component disk store (default CCMData) | StorageDir parameter (ch. 9) | Verify — component-managed writes |
| Per-component private data | Anything your own modules write to disk | Your spec/parameters | You know your write pattern — document it |
| Specs, launcher scripts, certs | The deployment itself | Deploy dir | Yes (static) — keep in version control |
Whiteboards are in-memory only: their contents are lost on restart. If a whiteboard holds state you cannot recompute, a module must persist it (to a catalog or file) itself.
Basic approach. Backup: stop the system (or take a filesystem snapshot) and copy the directories in the table plus the spec. Restore: put the files back in the same paths, redeploy the matching binary + spec, restart, watch for SYSTEM READY. There is no backup tooling in the platform today Roadmap — script it, and test the restore in staging before you need it.
Failure semantics & runbook
| Symptom | What to do |
|---|---|
| Engine/node process crash | systemd restarts it (above). Collect the node log tail and any core dump before it is overwritten; check which space/component was active at crash time (ch. 6). |
| Space crash | The owning node restarts the space automatically (ch. 8); repeated restarts of the same space point at the component inside it — raise its per-module verbose/debug (ch. 1). |
| Hung system (HTTP answers but nothing flows) | Raise verbose per topic, inspect PsyProbe Activity and queues, and capture a native stack of the process (gdb -p <pid> / eu-stack) if you can, before restarting. |
| Satellite node drop | See ch. 8 for node lifecycle. The exact rejoin/failover behaviour after a network partition should be drilled in staging and confirmed for your build — do not assume components migrate or resume automatically. |
| Message backlog | A growing queue means a downstream bottleneck. There is no documented backpressure or queue limit; messaging is backed by shared memory, so watch shared-memory usage and treat sustained growth as an incident. Fix by speeding up/parallelising the consumer or shedding input. |
| SSL handshake failures | Raise verbose-network, look for OpenSSL error strings in the log, then check cafile/capath/allowselfsigned scope per ch. 5 (expired cert and hostname mismatch are the usual causes). |
Change management: deploying spec changes
There is no dry-run or validate mode today Roadmap — the only way to find out whether a spec change works is to load it. Safe procedure: edit the spec → start it on a staging instance (same binary, different port/machine) and watch for SYSTEM READY and expected components in PsyProbe → then deploy to production.
system= triggers are silently ignored rather than rejected. The multi-alias parse quirk on PsyProbe aliases (ch. 3) means a list can partially apply. And %variable% substitution is literal text replacement (ch. 1) — a bad command-line value can produce invalid XML or a semantically wrong spec with no warning at the substitution step. A typo can therefore deploy “successfully” while a trigger simply never fires.- Keep specs (and launcher scripts) in version control; deploy by tag, roll back by tag.
- Validate XML well-formedness pre-deploy after variable substitution where possible:
xmllint --noout mysystem.xml. - Diff the resolved startup configuration printout (ch. 6) against the previous deploy — missing components mean a silently-ignored declaration.
Upgrade & rollback
The hard rule comes from the shared-memory ABI (ch. 6 pitfall): when a CMSDK shared-memory struct changes between versions, every node and every native library must be rebuilt and deployed together. Mixed-version nodes across such a change are unsafe — keep the master and all satellites on the same build, always.
- Upgrade: stage the new binary + libraries + spec on all machines, stop all nodes, deploy everywhere, restart satellites then master, watch SYSTEM READY.
- Rollback: redeploy the previous binary + spec from version control the same way. Keep the last known-good deploy directory intact on each machine.
- Data across versions: recording and DataCatalog on-disk format compatibility across engine versions is not documented — verify by replaying a recording from the old version on the new build in staging before you depend on it.
Capacity & tuning
Honest status: hard limits (max nodes, components, message size, queue depth) are not documented — establish your envelope empirically. What is known:
- Keep high-volume dataflow node-local; ship only distilled results across node links (ch. 8). Placement is the first-order tuning knob.
- Spaces are processes: isolation costs a process (and IPC) per space. Group components that can share fate; isolate the crash-prone ones.
- Messaging rides shared memory; sizing and overflow behaviour are not documented — monitor shared-memory usage on loaded nodes and verify overflow behaviour for your build.
- Baseline before go-live: use PsyTest (ch. 10) on production-like hardware to measure throughput and p99 latency for your message sizes and shapes, and re-run it after upgrades to catch regressions.
Secrets
key=value %variables% (ch. 1) are visible to every local user via ps//proc — never pass credentials, API keys or passwords that way, and never commit them in the spec.- Put secret-bearing settings in a separate spec fragment pulled in via
<include>, owned by the service user with mode 600, excluded from version control. - Or inject at launch from the environment/secret store via a launcher script that writes a transient mode-600 fragment (systemd
LoadCredential=pairs well with this). - There is no platform secret store today; the roadmap’s
<llm>design (ch. 11) introducesapikeyref-style logical secret names — not built yet.