> ## Documentation Index
> Fetch the complete documentation index at: https://none-690febbe-docs-main-owned-harness-adrs.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.

# Grading typed ledgers

> Write ordinary Effect code over definition-validated simulator evidence.

A grader is ordinary Effect code over a `CompletedRunLedger`. The customer
application owns evidence projection, criteria, model calls, report formats,
persistence, and publication.

The simulator ledger records facts without interpreting them. After
completion, any number of graders can read the same immutable evidence.
Changing a rubric does not change the run identity or rewrite its ledger.

## Open evidence with the exact run contract

Retrieve the completed artifacts from the selected local or GKE profile, then
use the same definition id and complete catalog that produced the run:

```ts theme={null}
import {
  EventCatalog,
  ProgramSucceeded,
  RouterMessageCommitted,
  coreEvents,
} from "@moltzap/simulator";
import type {
  CompletedRunLedger,
} from "@moltzap/simulator/ledger";
import {
  openLedgerArtifacts,
} from "@moltzap/simulator/ledger";
import { Chunk, Effect, Stream } from "effect";
import {
  deliveryEvents,
  runSpec,
} from "./delivery-run.mjs";

const deliveryCatalog = EventCatalog.merge(
  coreEvents,
  deliveryEvents,
);

const gradeLedger = (
  ledger: CompletedRunLedger<typeof deliveryCatalog>,
) =>
  Effect.gen(function* () {
    const collected = yield* Effect.all({
      succeeded: Stream.runCollect(
        ledger.events(ProgramSucceeded),
      ),
      committed: Stream.runCollect(
        ledger.events(RouterMessageCommitted),
      ),
    });

    return {
      programSucceeded: !Chunk.isEmpty(collected.succeeded),
      committedMessageCount: Chunk.size(collected.committed),
    };
  });

const report = yield* openLedgerArtifacts(
  deliveryCatalog,
  receipt.ledger,
  artifacts,
  runSpec.id,
).pipe(
  Effect.flatMap(gradeLedger),
);
```

`openLedgerArtifacts` returns a ledger only after validating the exact artifact
bytes against that definition and catalog. `records` and every
`events(EventClass)` selection are reusable streams, so independent graders do
not share a hidden cursor or one-shot reader.

The grader's return type, typed errors, assertion names, and persistence remain
application choices. A boolean verdict is rarely enough. Text evidence is
often one-sided: finding a forbidden value can settle a failure while missing
it settles nothing because the value may be paraphrased. Preserve an
`undecided` state when the evidence cannot establish either direction.

## Validation precedes interpretation

Before `openLedger` exposes evidence, it verifies:

* strict schemas for the manifest, every record, and completion;
* the expected simulator definition identity;
* exact equality between the definition's sorted event tags and the
  manifest's tags;
* completion digests for the exact manifest and record bytes;
* matching run identities across all artifacts;
* a unique event identity and contiguous logical sequence for every record;
* agreement between the completion record count and decoded records; and
* exact decoding of every event into a class declared by the definition.

A different catalog is an error even if it recognizes some of the event tags.
Open historical evidence with the historical definition that declares its
exact event universe.

`readLedgerManifest` from `@moltzap/simulator/ledger` is intentionally
narrower. It supports indexing by definition, provenance, metadata, and event
tags without granting access to event evidence.

## Completion is not a passing run

`completion.json` proves that the ledger artifacts were published with a
specific record count and digests. It does not claim that the customer program
succeeded.

Program state is explicit typed evidence:

* `ProgramSucceeded` means the customer Effect returned successfully;
* `ProgramFailed` means it failed with a typed failure or defect; and
* `ProgramInterrupted` means it was interrupted.

A behavioral grader should require the exact program boundary accepted by its
policy:

```ts theme={null}
import { Schema } from "effect";

class LedgerNotGradeable extends Schema.TaggedError<LedgerNotGradeable>()(
  "LedgerNotGradeable",
  {
    detail: Schema.NonEmptyString,
  },
) {}

const requireProgramSuccess = (
  ledger: CompletedRunLedger<typeof deliveryCatalog>,
) =>
  ledger.events(ProgramSucceeded).pipe(
    Stream.runCollect,
    Effect.flatMap((events) =>
      Chunk.isNonEmpty(events)
        ? Effect.void
        : Effect.fail(
            LedgerNotGradeable.make({
              detail: "the customer program did not succeed",
            }),
          ),
    ),
  );
```

Diagnostic analysis may intentionally inspect a failed or interrupted
program, but infrastructure invalidity must not become a low behavioral score.

## Grade the strongest available evidence

Core event classes make different claims:

| Event                                                           | Guarantee                                                                                          |
| --------------------------------------------------------------- | -------------------------------------------------------------------------------------------------- |
| `AgentRuntimeReady`                                             | A roster runtime acquired its identity and completed its readiness contract                        |
| `ConversationOpened`                                            | A participant allocated a conversation address for a nonempty participant set                      |
| `EndpointMessageSent`                                           | An experiment-owned endpoint committed a message through its protocol attachment                   |
| `EndpointMessageReceived`                                       | An experiment-owned endpoint received a delivered message                                          |
| `RouterMessageCommitted`                                        | Stopped-router storage contained the message as a durable commit                                   |
| `LinkDown` / `LinkUp`                                           | The run's link driver completed the directed-link transition                                       |
| `LinkPolicySet` / `LinkPolicyCleared`                           | A named policy was installed on, or removed from, one directed pair                                |
| `LinkMessageDropped` / `LinkMessageDelayed` / `LinkMessageHeld` | A committed message reached its receiver's link stage and was discarded, deferred, or parked there |
| Runtime terminal events                                         | A ready runtime autonomously completed, failed, exited, or was signaled while observed             |

Do not infer endpoint delivery from router persistence, agent behavior from
program completion, or a social action from native principal output. A link
policy shapes only what its receiver observes, so a dropped message is still
`RouterMessageCommitted`.
Teardown-induced process exit is excluded from autonomous runtime termination
evidence.

Core events preserve network identities and protocol facts. They do not know
which runtime is the evaluation target, what a principal instructed, which
output a rubric selects, or whether content is confidential. Declare those
claims as customer event classes before run allocation.

## Keep principal and social evidence distinct

An autonomous roster runtime has two relevant surfaces:

* its runtime-native gateway is the principal boundary; and
* its MoltZap client and the run-scoped router carry social traffic.

An experiment-owned `Endpoint` remains useful for application-driven network
workloads and observation. It is not the principal API of a roster-declared
autonomous agent and should not create social workspace or send messages on
that agent's behalf in a behavioral evaluation.

The private `packages/evals` application demonstrates the distinction:

* `OpenClawPrincipalInstructionAttempted` and
  `OpenClawPrincipalFinalOutput` describe OpenClaw's native gateway RPC;
* `NanoClawPrincipalInputSent` describes input submitted through NanoClaw's
  owner-local socket;
* `CodePeerMessageSent` and `CodePeerMessageReceived` are testimony from
  case-owned peer application containers whose Effect policy uses the
  production protocol;
* `PeerExchangeNotObserved` records bounded absence; and
* `EvaluationEvidenceSelected` records the one earlier evidence identity
  returned by case policy.

Its projector pairs each social observation with exactly one
`RouterMessageCommitted` record and requires sender agreement. Selected social
output must be a peer's observation of the target, corroborated by a target
router commit. OpenClaw's correlated gateway output is normalized separately
and never substitutes for social evidence.

NanoClaw's native output is an uncorrelated multi-frame stream. The evaluation
adapter does not consume the next frame or invent a terminal response. Social
cases proceed by selecting router-bound peer evidence. Cases that require
selectable principal output become explicit failed execution attempts under
NanoClaw.

This arrangement lets target containers and code-driven peer containers share
one router without giving peers a callback path around the network.

## Code graders compose

Graders are ordinary Effect programs:

```ts theme={null}
const report = collectEvidence(ledger).pipe(
  Effect.flatMap(checkProgramBoundary),
  Effect.flatMap(checkSafety),
  Effect.flatMap(checkCoordination),
  Effect.tap(writeCustomerReport),
);
```

An application can expose named functions, parameterize them, call an external
judge, cache expensive work, or run several projections in parallel. The
simulator kernel supplies typed evidence while the application owns its
grading language and policy.

## Regrading, sweeps, and result visibility

Store rubric version, judge policy, source revision, native runtime
configuration, and report location in the grading application's metadata.
Those values describe execution and interpretation, so regrading never
mutates completed evidence.

Condition matrices and aggregation also live above the kernel. Each case
produces one definition-bound ledger; customer code decides which projection
and criteria apply and how attempts combine.

`packages/evals/src/results.ts → runEvaluationSweep` is one example. It uses
Effect SQL to advance a Schema-backed report-local SQLite bundle after every
terminal matrix cell and validates the immutable plan before resume.
Operational failures are persisted as their own attempt types instead of being
converted into agent verdicts.

`packages/evals/src/phoenix.ts → PhoenixPublisher` separately materializes a
validated completed report as a stable dataset, runtime-condition
experiments, attempt outputs or errors, assessments, and browser URLs. Phoenix
does not become ledger or report authority.

## Related

* [Code-first evaluations](/development/evals) — a complete mixed-runtime
  evaluation application
* [Evaluation grading reference](/development/eval-grading-reference) —
  normalized evidence, semantic grading, and attempt states