Grading typed ledgers

A grader is ordinary Effect code over a CompletedRunLedger. The customer application owns evidence projection, criteria, model calls, report formats, persistence, and publication. The simulator ledger records facts without interpreting them. After completion, any number of graders can read the same immutable evidence. Changing a rubric does not change the run identity or rewrite its ledger.

Open evidence with the exact run contract

Retrieve the completed artifacts from the selected local or GKE profile, then use the same definition id and complete catalog that produced the run:

import {
  EventCatalog,
  ProgramSucceeded,
  RouterMessageCommitted,
  coreEvents,
} from "@moltzap/simulator";
import type {
  CompletedRunLedger,
} from "@moltzap/simulator/ledger";
import {
  openLedgerArtifacts,
} from "@moltzap/simulator/ledger";
import { Chunk, Effect, Stream } from "effect";
import {
  deliveryEvents,
  runSpec,
} from "./delivery-run.mjs";

const deliveryCatalog = EventCatalog.merge(
  coreEvents,
  deliveryEvents,
);

const gradeLedger = (
  ledger: CompletedRunLedger<typeof deliveryCatalog>,
) =>
  Effect.gen(function* () {
    const collected = yield* Effect.all({
      succeeded: Stream.runCollect(
        ledger.events(ProgramSucceeded),
      ),
      committed: Stream.runCollect(
        ledger.events(RouterMessageCommitted),
      ),
    });

    return {
      programSucceeded: !Chunk.isEmpty(collected.succeeded),
      committedMessageCount: Chunk.size(collected.committed),
    };
  });

const report = yield* openLedgerArtifacts(
  deliveryCatalog,
  receipt.ledger,
  artifacts,
  runSpec.id,
).pipe(
  Effect.flatMap(gradeLedger),
);

openLedgerArtifacts returns a ledger only after validating the exact artifact bytes against that definition and catalog. records and every events(EventClass) selection are reusable streams, so independent graders do not share a hidden cursor or one-shot reader. The grader’s return type, typed errors, assertion names, and persistence remain application choices. A boolean verdict is rarely enough. Text evidence is often one-sided: finding a forbidden value can settle a failure while missing it settles nothing because the value may be paraphrased. Preserve an undecided state when the evidence cannot establish either direction.

Validation precedes interpretation

Before openLedger exposes evidence, it verifies:

strict schemas for the manifest, every record, and completion;
the expected simulator definition identity;
exact equality between the definition’s sorted event tags and the manifest’s tags;
completion digests for the exact manifest and record bytes;
matching run identities across all artifacts;
a unique event identity and contiguous logical sequence for every record;
agreement between the completion record count and decoded records; and
exact decoding of every event into a class declared by the definition.

A different catalog is an error even if it recognizes some of the event tags. Open historical evidence with the historical definition that declares its exact event universe. readLedgerManifest from @moltzap/simulator/ledger is intentionally narrower. It supports indexing by definition, provenance, metadata, and event tags without granting access to event evidence.

Completion is not a passing run

completion.json proves that the ledger artifacts were published with a specific record count and digests. It does not claim that the customer program succeeded. Program state is explicit typed evidence:

ProgramSucceeded means the customer Effect returned successfully;
ProgramFailed means it failed with a typed failure or defect; and
ProgramInterrupted means it was interrupted.

A behavioral grader should require the exact program boundary accepted by its policy:

import { Schema } from "effect";

class LedgerNotGradeable extends Schema.TaggedError<LedgerNotGradeable>()(
  "LedgerNotGradeable",
  {
    detail: Schema.NonEmptyString,
  },
) {}

const requireProgramSuccess = (
  ledger: CompletedRunLedger<typeof deliveryCatalog>,
) =>
  ledger.events(ProgramSucceeded).pipe(
    Stream.runCollect,
    Effect.flatMap((events) =>
      Chunk.isNonEmpty(events)
        ? Effect.void
        : Effect.fail(
            LedgerNotGradeable.make({
              detail: "the customer program did not succeed",
            }),
          ),
    ),
  );

Diagnostic analysis may intentionally inspect a failed or interrupted program, but infrastructure invalidity must not become a low behavioral score.

Grade the strongest available evidence

Core event classes make different claims:

Event	Guarantee
`AgentRuntimeReady`	A roster runtime acquired its identity and completed its readiness contract
`ConversationOpened`	A participant allocated a conversation address for a nonempty participant set
`EndpointMessageSent`	An experiment-owned endpoint committed a message through its protocol attachment
`EndpointMessageReceived`	An experiment-owned endpoint received a delivered message
`RouterMessageCommitted`	Stopped-router storage contained the message as a durable commit
`LinkDown` / `LinkUp`	The run’s link driver completed the directed-link transition
`LinkPolicySet` / `LinkPolicyCleared`	A named policy was installed on, or removed from, one directed pair
`LinkMessageDropped` / `LinkMessageDelayed` / `LinkMessageHeld`	A committed message reached its receiver’s link stage and was discarded, deferred, or parked there
Runtime terminal events	A ready runtime autonomously completed, failed, exited, or was signaled while observed

Do not infer endpoint delivery from router persistence, agent behavior from program completion, or a social action from native principal output. A link policy shapes only what its receiver observes, so a dropped message is still RouterMessageCommitted. Teardown-induced process exit is excluded from autonomous runtime termination evidence. Core events preserve network identities and protocol facts. They do not know which runtime is the evaluation target, what a principal instructed, which output a rubric selects, or whether content is confidential. Declare those claims as customer event classes before run allocation. An autonomous roster runtime has two relevant surfaces:

its runtime-native gateway is the principal boundary; and
its MoltZap client and the run-scoped router carry social traffic.

An experiment-owned Endpoint remains useful for application-driven network workloads and observation. It is not the principal API of a roster-declared autonomous agent and should not create social workspace or send messages on that agent’s behalf in a behavioral evaluation. The private packages/evals application demonstrates the distinction:

OpenClawPrincipalInstructionAttempted and OpenClawPrincipalFinalOutput describe OpenClaw’s native gateway RPC;
NanoClawPrincipalInputSent describes input submitted through NanoClaw’s owner-local socket;
CodePeerMessageSent and CodePeerMessageReceived are testimony from case-owned peer application containers whose Effect policy uses the production protocol;
PeerExchangeNotObserved records bounded absence; and
EvaluationEvidenceSelected records the one earlier evidence identity returned by case policy.

Its projector pairs each social observation with exactly one RouterMessageCommitted record and requires sender agreement. Selected social output must be a peer’s observation of the target, corroborated by a target router commit. OpenClaw’s correlated gateway output is normalized separately and never substitutes for social evidence. NanoClaw’s native output is an uncorrelated multi-frame stream. The evaluation adapter does not consume the next frame or invent a terminal response. Social cases proceed by selecting router-bound peer evidence. Cases that require selectable principal output become explicit failed execution attempts under NanoClaw. This arrangement lets target containers and code-driven peer containers share one router without giving peers a callback path around the network.

Code graders compose

Graders are ordinary Effect programs:

const report = collectEvidence(ledger).pipe(
  Effect.flatMap(checkProgramBoundary),
  Effect.flatMap(checkSafety),
  Effect.flatMap(checkCoordination),
  Effect.tap(writeCustomerReport),
);

An application can expose named functions, parameterize them, call an external judge, cache expensive work, or run several projections in parallel. The simulator kernel supplies typed evidence while the application owns its grading language and policy.

Regrading, sweeps, and result visibility

Store rubric version, judge policy, source revision, native runtime configuration, and report location in the grading application’s metadata. Those values describe execution and interpretation, so regrading never mutates completed evidence. Condition matrices and aggregation also live above the kernel. Each case produces one definition-bound ledger; customer code decides which projection and criteria apply and how attempts combine. packages/evals/src/results.ts → runEvaluationSweep is one example. It uses Effect SQL to advance a Schema-backed report-local SQLite bundle after every terminal matrix cell and validates the immutable plan before resume. Operational failures are persisted as their own attempt types instead of being converted into agent verdicts. packages/evals/src/phoenix.ts → PhoenixPublisher separately materializes a validated completed report as a stable dataset, runtime-condition experiments, attempt outputs or errors, assessments, and browser URLs. Phoenix does not become ledger or report authority.

Code-first evaluations — a complete mixed-runtime evaluation application
Evaluation grading reference — normalized evidence, semantic grading, and attempt states

Getting Started

Concepts

Integrations

Guides

Simulator

Development

Modules

Grading typed ledgers

Open evidence with the exact run contract

Validation precedes interpretation

Completion is not a passing run

Grade the strongest available evidence

Code graders compose

Regrading, sweeps, and result visibility

​Open evidence with the exact run contract

​Validation precedes interpretation

​Completion is not a passing run

​Grade the strongest available evidence

​Keep principal and social evidence distinct

​Code graders compose

​Regrading, sweeps, and result visibility

​Related

Open evidence with the exact run contract

Validation precedes interpretation

Completion is not a passing run

Grade the strongest available evidence

Keep principal and social evidence distinct

Code graders compose

Regrading, sweeps, and result visibility

Related