# T1.4 — External Module Executable Auto-Start & Lifecycle

## Scope

The `<executable>` child of an external `<module>` (parsed in `Node::componentSpecFromXML`)
currently only stores a command line; it is never launched. This feature launches the
declared executable and manages its full lifecycle (crash detection, restart with backoff,
output routing, clean shutdown).

## Ownership decisions (settled)

1. **`<executable>` stays a module child**; `type="external"` stays a space-level
   attribute on `<space>`. No executable parsing on `<space>`.
2. **The external space is owned by the PsyNode.** The executable is launched when the
   external space is created/registered during spec parsing (node startup) — i.e. by the
   *first* module declaring an `<executable>` for that space. A second `<executable>`
   declaration targeting the same already-launched space is **ignored with a warning**.
3. The space (and its ProcessMap entry) may outlive the connected process; the launcher
   restarts the child into the same space id.

## XML

```xml
<space name="MySpace" type="external"/>

<module name="Test" space="MySpace" type="external">
  <executable autorestart="yes" restartdelay="3000" output="logfile"
              logfile="myspace-%spacename%.log" workdir="/opt/mymodule"
              shutdownondelete="yes" startuptimeout="10000"
              restartlimit="5" restartwindow="60000" maxrestartdelay="30000"
              stopsignal="SIGTERM" stoptimeout="3000" enabled="yes">
    <env name="MY_VAR" value="hello-%port%"/>
    ./bin/MyExternalModule %port%
  </executable>
  ...
</module>
```

Element **text** = command line (with substitutions). `<env>` children set child
environment variables (also substituted).

## Attributes

| Attribute | Default | Meaning |
|---|---|---|
| `enabled` / `autostart` | `yes` | Parse but do not auto-launch when `no` (manual start/debug). |
| `autorestart` | **`yes`** (default flipped from old parser's `no`) | Restart the child if it exits/crashes/hangs. |
| `restartdelay` | `3000` ms | Base delay before a restart. |
| `restartlimit` | `0` (= unlimited) | Max restarts within `restartwindow`; when exceeded, give up (state = FAILED) and log. |
| `restartwindow` | `60000` ms | Sliding window for `restartlimit`. |
| `maxrestartdelay` | `30000` ms | Cap for exponential backoff (delay doubles per consecutive fast crash, capped here). |
| `startuptimeout` | `10000` ms | Max wait for the child's PsySpace to reach `PSYPROC_ACTIVE` after launch; on timeout the launch is declared failed (counts as a crash for restart purposes). |
| `shutdownondelete` | `yes` | Kill the child when the owning module is deleted or the node shuts down. |
| `output` | `ignore` | `ignore` \| `console` \| `logfile` \| `psyclonelog` \| `messages` (see below). |
| `logfile` | — | Log file path for `output="logfile"` (substituted). |
| `consoleoutput` | `no` | Legacy compat: `yes` == `output="console"` (child inherits node's stdout/stderr). |
| `workdir` | inherit | Child working directory (substituted). |
| `stopsignal` | `SIGTERM` | POSIX signal name/number sent on graceful stop (`SIGTERM`, `SIGINT`, `SIGKILL`, ...). Windows: delivered as a console `CTRL_BREAK` to the child's process group, then `TerminateProcess` after `stoptimeout`. |
| `stoptimeout` | `3000` ms | Wait after `stopsignal` before escalating to `SIGKILL`. |

## Output routing

- `ignore` — stdout/stderr redirected to `/dev/null`.
- `console` — inherit the node's stdout/stderr (legacy `consoleoutput="yes"`).
- `logfile` — captured line-wise and appended to `logfile` with `[out]`/`[err]` prefixes.
- `psyclonelog` — captured and forwarded through `LogPrint` (LOG_NODE, level 1 for
  stdout, 0 for stderr) so lines land in the node log.
- `messages` — captured lines posted as DataMessages typed
  `<ModuleName>.Stdout` / `<ModuleName>.Stderr` (message content = the line).

## `%…%` substitutions (cmdline, logfile, workdir, env values)

| Token | Value |
|---|---|
| `%nodeid%`, `%port%` | Node network port (the system id an external space connects to). |
| `%spacename%` | External space name. |
| `%spaceid%` | External space (process) id. |
| `%moduleid%` | Owning module's component id. |
| `%modulename%` | Owning module's name. |
| `%specdir%` | Directory of the spec/config file, when known (else empty). |

## Crash / hang detection

A monitor thread (in `ExternalExecManager`) polls each managed child every ~250 ms:

1. **OS exit**: `utils::GetProcessStatus(pid)` → `PROC_TERMINATED` (waitpid reap).
2. **Hung process**: once the space has been seen `>= PSYPROC_ACTIVE`, staleness of
   `ProcessMemory::getProcessStatus(spaceID, lastseen)` beyond 10 s while the OS process
   still runs ⇒ treated as hung: child is stopped (stopsignal → SIGKILL) and restarted.
3. **Startup timeout**: after launch, if the space does not reach `PSYPROC_ACTIVE`
   within `startuptimeout`, the launch is failed and the child killed (restart rules apply).

Restarts obey `restartdelay` with exponential backoff (×2 per consecutive crash, capped
at `maxrestartdelay`) and stop permanently when `restartlimit` restarts occurred within
`restartwindow` ms.

## Shutdown

`Node::terminate()` / `Node::~Node` stop the manager: each child with
`shutdownondelete="yes"` receives `stopsignal`, then `SIGKILL` after `stoptimeout`.
Module deletion (when/if a runtime delete path exists) should call
`ExternalExecManager::removeByModule(compID)`; note: Psyclone currently has no runtime
module-delete path, so node shutdown is the operative case.

## Implementation notes

- New: `Psyclone/include/ExternalExec.h`, `Psyclone/src/ExternalExec.cpp`
  (`cmlabs::ExternalExecManager`, owned by `Node`, friend of `Node` for message posting).
- `ComponentSetup` (shared-memory binary contract) is **not** changed; the extended
  options live in a node-local `ExternalExecSpec`.
- POSIX implemented fully (fork/execvp with pipes, setenv, chdir, kill).
- Windows implemented fully via new Utils helpers `NewProcessEx` (CreateProcess with a
  merged env block, working directory, and inheritable stdout/stderr capture pipes;
  `CREATE_NEW_PROCESS_GROUP` for graceful stop), `ReadProcessOutput` (non-blocking
  `PeekNamedPipe`/`ReadFile` drain), and `SendProcessBreak` (`GenerateConsoleCtrlEvent`
  `CTRL_BREAK`). Kill-on-node-exit uses the existing global Job Object
  (`JOB_OBJECT_LIMIT_KILL_ON_JOB_CLOSE`) that all `NewProcess*` children join.
  Crash detection, restart/backoff/limit, output-mode routing and line-splitting are
  platform-independent and shared by both paths. (Windows code compiles by inspection
  but has not been run here — no Windows host available.)
- Parser default for `autorestart` flipped to `yes`.
- **macOS:** builds and runs the same POSIX path as Linux (fork/execvp, pipes, setenv,
  chdir, waitpid, kill). `PR_SET_PDEATHSIG` is Linux-only, so on macOS `shutdownondelete`
  is enforced by a small detached "nanny" grandchild: the launched child does
  `setpgid(0,0)` to lead its own group, then forks a nanny that `kqueue`-watches the node
  PID for `NOTE_EXIT` and `kill(-childgroup, SIGKILL)`s when the node dies — covering a
  hard node crash too (the monitor thread cannot, since it dies with the node). Compile-
  checked by inspection only (no macOS host here); needs one `make && Psyclone test=cmsdk`
  on a Mac to confirm.
- Node accessor `getExternalExec()` added (returns the manager or NULL) for tests/diagnostics.

## Tests (in the standard unit harness)

The former `test/t14/` bash suite has been ported into the built-in CMSDK unit-test
framework so it builds and runs cross-platform every time (`Psyclone test=cmsdk`, or
individually e.g. `Psyclone test=psy_extexec_basic`). Registered in
`Psyclone/src/PsycloneTests.cpp`:

| Test | Covers |
|---|---|
| `psy_extexec_basic` | auto-launch, space becomes ACTIVE, `workdir`, `<env>` substitution, `output=logfile` stdout+stderr routing, no orphan on shutdown |
| `psy_extexec_restart` | crash → `autorestart` with backoff, `output=psyclonelog` |
| `psy_extexec_limit` | `restartlimit`/`maxrestartdelay` stop the restart storm (FAILED) |
| `psy_extexec_startuptimeout` | non-connecting child + `autorestart=no` → stopped, no restart |
| `psy_extexec_messages` | `output=messages` mode launches/runs |
| `psy_extexec_duplicate` | duplicate `<executable>` on a shared external space ignored (one managed entry) |
| `psy_extexec_disabled` | `enabled=no` registers but does not launch |
| `psy_extexec_hang` | child drops its space while the process stays alive → hang detected → restart |

- Each test boots a real `Node` in-process from an inline spec on a unique port, lets it
  launch a real external child, asserts via `ExternalExecManager::getState()`/`count()`
  (and logfile readback for routing), then `terminate()`s. No shell/`pgrep`/`kill`/signals
  in the test logic — all cross-platform.
- **Test child:** `Examples/src/PsyTestChild.cpp` (modes `run`/`crashafter`/`noconnect`/`hang`).
  The `hang` mode calls `PsySpace::shutdown()` then stays alive (portable stand-in for a
  hung/SIGSTOP process). Wired into `Examples/Makefile` as a second exe target
  (`PsyTestChild`), mirroring `ExternalModules`.
- **Windows build TODO (Thor):** add a `PsyTestChild` VS project by copying
  `Examples/ExternalModules.vs2022.vcxproj` and swapping the source file to
  `src\PsyTestChild.cpp` and the output name to `PsyTestChild` (the child source itself
  is already cross-platform — uses `utils::GetCurrentDir`/`GetLocalProcessID`, no
  `unistd.h`). Not generated here since VS project XML can't be validated headless.
- **`test/t14/`** is now redundant (kept for manual reference; safe to delete once the
  harness tests are trusted on Windows).
