> ## Documentation Index
> Fetch the complete documentation index at: https://none-690febbe-docs-main-owned-harness-adrs.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.

# Running simulator programs

> Submit one RunSpec through the shared local-Kubernetes or GKE execution path.

Simulator programs are ordinary `.mjs` modules loaded by the in-cluster
controller. The repository-local submitters accept one module path and run it
through the same Temporal, Kubernetes, Kueue, Agent Sandbox, controller, and
`Run.execute` path.

## Package entry points

| Import                       | Purpose                                                                         |
| ---------------------------- | ------------------------------------------------------------------------------- |
| `@moltzap/simulator`         | `RunSpec`, `Run.execute`, event catalogs, customer services, and run outcomes   |
| `@moltzap/simulator/agents`  | Container runtime descriptors and exact OpenClaw and NanoClaw gateway contracts |
| `@moltzap/simulator/network` | Router, transport, link, endpoint, and nominal capability contracts             |
| `@moltzap/simulator/ledger`  | Completed-ledger types, validation, and artifact inspection                     |

The experiment module owns its agents, customer events, customer Effect, and
completion policy. The selected profile owns every platform object.

## Write a controller-loadable module

Export exactly one named `runSpec`:

```ts theme={null}
import { RunSpec } from "@moltzap/simulator";
import {
  openClawRuntime,
} from "@moltzap/simulator/agents";
import { Duration, Effect, Schema } from "effect";
import { controllerServicesFromEnvironment } from "/opt/moltzap/dist/cluster/controller/services.js";

class ExperimentTimedOut extends Schema.TaggedError<ExperimentTimedOut>()(
  "ExperimentTimedOut",
  {},
) {}

const alice = openClawRuntime({
  tools: {
    deny: ["*"],
    elevated: { enabled: false },
    exec: { mode: "deny" },
  },
  sandbox: { mode: "off" },
  workspaceFiles: [
    { relativePath: "IDENTITY.md", content: "You are Alice." },
  ],
});

export const runSpec = RunSpec.define({
  id: "acme.echo/v1",
  events: [],
  agents: { alice },
  cluster: controllerServicesFromEnvironment(),
  execute: ({ agents, network }) =>
    Effect.gen(function* () {
      const diagnostic = yield* network.endpoint("diagnostic");
      const conversation = yield* diagnostic.open(
        agents.alice.agent,
      );
      yield* conversation.send("hello");
    }).pipe(
      Effect.timeoutFail({
        duration: Duration.minutes(5),
        onTimeout: () => ExperimentTimedOut.make({}),
      }),
    ),
});
```

The absolute cluster-services import is available inside the repository-built
controller image. It constructs the selected profile's private Layer from the
validated controller environment. Experiment code does not receive raw
Kubernetes, Kueue, Sandbox, or Temporal objects.

The controller requires the exact value returned by `RunSpec.define`. It
loads the mounted module once and invokes `Run.execute(runSpec)` once; there is
no fallback execution entry point or automatic replay.

Every roster runtime must provide a distributed application-container
realization. Its bridge resolves only after the application is usable and
returns that runtime's exact `.gateway` plus `.termination` observation. The
customer Effect starts after all roster entries pass the same readiness gate.

## Run on the local Kubernetes profile

Build the shared controller/support image:

```bash theme={null}
pnpm nx run @moltzap/simulator:local-controller-image
```

The command prints an immutable `pinnedImage`. Use it to create the pinned kind
profile:

```bash theme={null}
pnpm nx run @moltzap/simulator:local-cluster-create -- \
  --artifacts "$PWD/.moltzap/local-artifacts" \
  --image CONTROLLER_IMAGE_AT_SHA256
```

The cluster setup prints its exact kube context, tool paths, queue names,
Temporal address, and artifact roots. It refuses to replace an existing
cluster.

Submit the module through the local profile:

```bash theme={null}
MOLTZAP_CONTROLLER_IMAGE=CONTROLLER_IMAGE_AT_SHA256 \
MOLTZAP_SUPPORT_IMAGE=CONTROLLER_IMAGE_AT_SHA256 \
MOLTZAP_TEMPORAL_ADDRESS=127.0.0.1:7233 \
pnpm nx run @moltzap/simulator:local-run -- path/to/experiment.mjs
```

`MOLTZAP_SUPPORT_IMAGE` defaults to `MOLTZAP_CONTROLLER_IMAGE` for this local
path. Both values must be digest-pinned. The checked-in
`packages/simulator/local/README.md` records the component versions and smoke
modules.

## Run on the GKE profile

Provision the checked-in Terraform profile, install its pinned add-ons, push
the controller/support image, and acquire the explicit kube context as
described in `packages/simulator/gke/README.md`. Then submit the same module:

```bash theme={null}
MOLTZAP_KUBE_CONTEXT=EXPLICIT_KUBE_CONTEXT \
MOLTZAP_GKE_ARTIFACT_BUCKET=PROFILE_ARTIFACT_BUCKET \
MOLTZAP_TEMPORAL_ADDRESS=TEMPORAL_HOST:7233 \
MOLTZAP_CONTROLLER_IMAGE=REGISTRY/CONTROLLER@sha256:DIGEST \
MOLTZAP_SUPPORT_IMAGE=REGISTRY/CONTROLLER@sha256:DIGEST \
pnpm nx run @moltzap/simulator:gke-run -- path/to/experiment.mjs
```

The GKE submitter validates the checked-in profile, uses only the explicit
context and bucket, and calls the same Temporal submission code as the local
profile. The repository does not select production Temporal hosting or high
availability.

Static GKE validation does not contact Google Cloud or a cluster:

```bash theme={null}
pnpm nx run @moltzap/simulator:gke-profile-check
```

Passing that check is not a live qualification claim. The GKE acceptance gate
still requires a caller-authorized project, the small smoke, an OpenClaw
evaluation, readable retained artifacts, and zero run-owned residue.

## Express completion policy in `execute`

The customer Effect returns, fails, or is interrupted according to its own
logic. Use Effect's `Clock`, `Schedule`, race, timeout, `Deferred`, Stream, and
Scope primitives for deadlines, quiescence, supervised work, and explicit
stop conditions.

A runtime exit after readiness is committed as typed ledger evidence. It does
not implicitly end the customer Effect. One program may fail fast on that
evidence while another continues observing the remaining society.

The run returns a `ProgramFinished` or `ClusterLost` outcome after ledger
allocation succeeds. `ProgramFinished.exit` preserves customer success, typed
failure, defect, or interruption. Infrastructure acquisition, append,
controller, teardown, or completion failures stay distinct from behavioral
results.

The submitters print one final JSON result containing the run namespace and
bounded controller result. Applications decide how to map that result into
their own exit codes, retries, operator messages, and report states.

## Sweeps remain application orchestration

A single simulator invocation is one definition-bound society and one ledger.
Schedules, matrices, retries, sharding, naming, resumption, and aggregation
stay in the calling application. For example, `packages/evals` submits every
case-condition cell as its own `RunSpec` through the selected local or GKE
profile, then persists the terminal attempt in its report database.

This keeps suite orchestration failures separate from the evidence produced by
an individual society.

## Inspect completed artifacts

After a run publishes a completed receipt, both profiles retain exported files
under the same relative path:

```text theme={null}
{namespace}/ledger/{ledgerRef}/{manifest.json,records.ndjson,completion.json}
```

Local files are written directly below the artifact root selected during
cluster setup. GKE runs build the active ledger on controller-local POSIX
storage, then export the three completed artifacts to the Terraform-owned
Cloud Storage bucket with `completion.json` last. The active GKE ledger is not
a recovery guarantee for controller or node loss before that export finishes.
Retrieve the three retained files, then validate them with the same complete
event catalog:

```ts theme={null}
import {
  EventCatalog,
  coreEvents,
} from "@moltzap/simulator";
import {
  openLedgerArtifacts,
} from "@moltzap/simulator/ledger";
import {
  runSpec,
  experimentEvents,
} from "./experiment.mjs";

const catalog = EventCatalog.merge(
  coreEvents,
  experimentEvents,
);

const ledger = yield* openLedgerArtifacts(
  catalog,
  receipt.ledger,
  artifacts,
  runSpec.id,
);
```

Opening validates the definition identity, exact catalog, schemas, digests,
run identity, count, event identities, and logical sequence before exposing
reusable typed streams. It does not start a society.
