> ## Documentation Index
> Fetch the complete documentation index at: https://docs.output.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Evals

> Workflow evaluation using datasets, typed evaluators, and deterministic or LLM-based assertions

The `@outputai/core/evals` module lets you test workflow quality across datasets - without modifying your workflow code. You define evaluators with `verify()`, write datasets in YAML, and run them with the CLI. Each dataset case feeds a saved workflow input/output pair through your evaluators, and the framework reports pass/partial/fail verdicts per case.

This is the complement to [evaluator steps](https://docs.output.ai/evaluators/evaluator-step.md) that run inside workflows. Those evaluators power generate-evaluate-retry loops in production. Evaluation workflows answer a different question: "across a set of known inputs, does my workflow still produce acceptable output?" For the full guide, see [Evaluation Workflow](https://docs.output.ai/evaluators/workflow-evaluators.md).

## What's in the Module

```typescript theme={null}
import {
  // Eval workflow
  evalWorkflow,

  // Evaluator creation
  verify,

  // Deterministic + LLM assertion helpers
  Verdict,

  // Verdict and criticality values
  VERDICT,
  CRITICALITY,

  // Dataset and eval output schemas
  DatasetSchema,
  EvalOutputSchema
} from '@outputai/core/evals';

// LLM judge functions
import { judgeVerdict, judgeScore, judgeBoolean, judgeLabel } from '@outputai/core/llm';
```

| Export | Description |
| - | - |
| `evalWorkflow` | Define an eval workflow that tests datasets against evaluators |
| `verify` | Create a typed evaluator with Zod schemas for input/output |
| `Verdict` | Deterministic assertion helpers (equals, gte, contains, etc.) and LLM result wrappers |
| `VERDICT`, `CRITICALITY` | The `pass`/`partial`/`fail` and `required`/`informational` values |
| `DatasetSchema`, `EvalOutputSchema` | Zod schemas for dataset cases and eval workflow output |
| `judgeVerdict` | LLM judge that returns pass/partial/fail (from `@outputai/core/llm`) |
| `judgeScore` | LLM judge that returns a numeric score (from `@outputai/core/llm`) |
| `judgeBoolean` | LLM judge that returns true/false (from `@outputai/core/llm`) |
| `judgeLabel` | LLM judge that returns a string label (from `@outputai/core/llm`) |

The judges live in `@outputai/core/llm` because eval workflows run in Temporal's workflow sandbox, which cannot load the AI SDK. Import them only in evaluator files.

## Creating Evaluators with verify()

`verify()` creates a typed evaluator that receives the workflow's input, output, and optional ground truth from the dataset. It wraps `evaluator()` from `@outputai/core` so it integrates with both the eval workflow and the Temporal worker.

```typescript theme={null}
import { verify } from '@outputai/core/evals';
import { z } from '@outputai/core';

const myEvaluator = verify(
  {
    name: 'my_evaluator',
    input: z.object({ /* workflow input schema */ }),
    output: z.object({ /* workflow output schema */ })
  },
  ({ input, output, context }) => {
    // input: typed workflow input
    // output: typed workflow output
    // context.ground_truth: Record<string, unknown> from dataset YAML
    return Verdict.equals(output.result, context.ground_truth.expected);
  }
);
```

The `input` and `output` schemas are optional — they default to `z.any()` if omitted. The check function receives a `CheckContext`:

| Field | Type | Description |
| - | - | - |
| `input` | `TInput` | The workflow input from the dataset |
| `output` | `TOutput` | The workflow output (cached or freshly executed) |
| `context.ground_truth` | `Record<string, unknown>` | Ground truth values from the dataset YAML |

### Basic Example

A deterministic evaluator that checks a sum calculation:

```typescript tests/evals/evaluators.ts theme={null}
import { verify, Verdict } from '@outputai/core/evals';
import { z } from '@outputai/core';

export const evaluateSum = verify(
  {
    name: 'evaluate_sum',
    input: z.object({ values: z.array(z.number()) }),
    output: z.object({ result: z.number() })
  },
  ({ input, output }) =>
    Verdict.equals(output.result, input.values.reduce((a, b) => a + b, 0))
);
```

### Ground Truth Example

Evaluators can read per-evaluator ground truth from the dataset. The framework merges global ground truth with evaluator-specific overrides:

```typescript tests/evals/evaluators.ts theme={null}
import { verify, Verdict } from '@outputai/core/evals';
import { z } from '@outputai/core';

export const lengthOfOutput = verify(
  {
    name: 'length_of_output',
    input: z.object({ topic: z.string() }),
    output: z.object({ title: z.string(), blog_post: z.string() })
  },
  ({ output, context }) =>
    Verdict.gte(output.blog_post.length, Number(context.ground_truth.min_length ?? 100))
);
```

The ground truth comes from the dataset YAML:

```yaml tests/datasets/stripe_blog.yml theme={null}
name: stripe_blog
input:
  topic: "Stripe the payment processor"
ground_truth:
  notes: "Known good case"
  evals:
    length_of_output:
      min_length: 100
    evaluate_content:
      required_content: "https://stripe.com"
```

Global ground truth fields (like `notes`) are available to all evaluators. Fields under `evals.<evaluator_name>` are merged in for that specific evaluator, overriding globals with the same key.

## Verdict Helpers

The `Verdict` object provides deterministic assertion helpers and LLM result wrappers. All deterministic helpers return results with confidence `1.0`.

### Deterministic Assertions

| Helper | Arguments | Passes when |
| - | - | - |
| `Verdict.equals(actual, expected)` | any, any | `actual === expected` |
| `Verdict.closeTo(actual, expected, tolerance)` | number, number, number | `\|actual - expected\| <= tolerance` |
| `Verdict.gt(actual, threshold)` | number, number | `actual > threshold` |
| `Verdict.gte(actual, threshold)` | number, number | `actual >= threshold` |
| `Verdict.lt(actual, threshold)` | number, number | `actual < threshold` |
| `Verdict.lte(actual, threshold)` | number, number | `actual <= threshold` |
| `Verdict.inRange(actual, min, max)` | number, number, number | `min <= actual <= max` |
| `Verdict.contains(haystack, needle)` | string, string | `haystack.includes(needle)` |
| `Verdict.matches(value, pattern)` | string, RegExp | `pattern.test(value)` |
| `Verdict.includesAll(actual, expected)` | array, array | `actual` contains every element of `expected` |
| `Verdict.includesAny(actual, expected)` | array, array | `actual` contains at least one element of `expected` |
| `Verdict.isTrue(value)` | boolean | `value === true` |
| `Verdict.isFalse(value)` | boolean | `value === false` |

### Manual Verdicts

| Helper | Arguments | Result |
| - | - | - |
| `Verdict.pass(reasoning?)` | string? | Pass with confidence 1.0 |
| `Verdict.partial(confidence, reasoning?, feedback?)` | number, string?, FeedbackArg\[]? | Partial with custom confidence |
| `Verdict.fail(reasoning, feedback?)` | string, FeedbackArg\[]? | Fail with confidence 0.0 |

### LLM Result Wrappers

These wrap LLM judge output into evaluation results with confidence `0.9`:

| Helper | Arguments | Result type |
| - | - | - |
| `Verdict.fromJudge({ verdict, reasoning })` | object | Verdict (pass/partial/fail) |
| `Verdict.score(value, reasoning?)` | number, string? | Number |
| `Verdict.label(value, reasoning?)` | string, string? | String |

## LLM Judge Functions

For subjective evaluation — "is this blog post on-topic?", "rate the quality 0-100" — use the judge functions. They load a `.prompt` file, call the LLM, and return a typed evaluation result.

```typescript tests/evals/evaluators.ts theme={null}
import { verify } from '@outputai/core/evals';
import { judgeVerdict, judgeScore, judgeLabel } from '@outputai/core/llm';
import { z } from '@outputai/core';

const blogInput = z.object({ topic: z.string() });
const blogOutput = z.object({ title: z.string(), blog_post: z.string() });

export const evaluateTopic = verify(
  { name: 'evaluate_topic', input: blogInput, output: blogOutput },
  async ({ input, output, context }) =>
    judgeVerdict({
      prompt: 'judge_topic@v1',
      variables: {
        blog_title: output.title,
        blog_post: output.blog_post,
        required_topic: String(context.ground_truth.required_topic ?? input.topic)
      }
    })
);

export const evaluateQuality = verify(
  { name: 'evaluate_quality', input: blogInput, output: blogOutput },
  async ({ input, output }) =>
    judgeScore({
      prompt: 'judge_quality@v1',
      variables: {
        blog_title: output.title,
        blog_post: output.blog_post,
        topic: input.topic
      }
    })
);

export const evaluateTone = verify(
  { name: 'evaluate_tone', input: blogInput, output: blogOutput },
  async ({ output }) =>
    judgeLabel({
      prompt: 'judge_tone@v1',
      variables: {
        blog_title: output.title,
        blog_post: output.blog_post
      }
    })
);
```

| Function | Expected schema | Returns |
| - | - | - |
| `judgeVerdict` | `{ verdict: 'pass'\|'partial'\|'fail', reasoning: string }` | `EvaluationVerdictResult` |
| `judgeScore` | `{ score: number, reasoning: string }` | `EvaluationNumberResult` |
| `judgeBoolean` | `{ result: boolean, reasoning: string }` | `EvaluationBooleanResult` |
| `judgeLabel` | `{ label: string, reasoning: string }` | `EvaluationStringResult` |

All judge functions accept a `JudgeArgs` object:

| Field | Type | Description |
| - | - | - |
| `prompt` | `string` | Prompt filename (e.g., `'judge_topic@v1'`) |
| `variables` | `Record<string, unknown>` | Template variables |
| `schema` | `ZodType` | Custom output schema (overrides the default) |

Judge `.prompt` files live in the same `tests/evals/` directory as your evaluators:

```yaml tests/evals/judge_topic@v1.prompt theme={null}
---
provider: anthropic
model: claude-haiku-4-5-20251001
temperature: 0
maxOutputTokens: 1000
---

<system>
You are an evaluation judge. Assess whether a blog post is faithfully about the required topic.

Return a JSON object with:
- verdict: "pass" if the blog clearly focuses on the topic, "partial" if it mentions but lacks depth, "fail" if not about the topic
- reasoning: brief explanation of your judgment
</system>

<user>
Required topic: {{ required_topic }}

Blog title: {{ blog_title }}

Blog post:
{{ blog_post }}

Judge whether this blog post is faithfully about the required topic.
</user>
```

## Creating an Eval Workflow

`evalWorkflow()` ties your evaluators together into a workflow that the CLI can run against datasets:

```typescript tests/evals/workflow.ts theme={null}
import { evalWorkflow } from '@outputai/core/evals';
import { evaluateSum } from './evaluators.js';

export default evalWorkflow({
  name: 'simple_eval',
  evals: [
    {
      evaluator: evaluateSum,
      criticality: 'required',
      interpret: { type: 'boolean' }
    }
  ]
});
```

Each entry in the `evals` array defines:

| Field | Type | Default | Description |
| - | - | - | - |
| `evaluator` | Function | — | An evaluator created with `verify()` |
| `criticality` | `'required' \| 'informational'` | `'required'` | Whether failure should fail the case |
| `interpret` | `InterpretConfig` | — | How to convert the evaluator result to a verdict |

A more complete example mixing deterministic and LLM evaluators:

```typescript tests/evals/workflow.ts theme={null}
import { evalWorkflow } from '@outputai/core/evals';
import {
  lengthOfOutput,
  evaluateTopic,
  evaluateQuality,
  evaluateContent,
  evaluateTone
} from './evaluators.js';

export default evalWorkflow({
  name: 'blog_generator_eval',
  evals: [
    {
      evaluator: lengthOfOutput,
      criticality: 'required',
      interpret: { type: 'boolean' }
    },
    {
      evaluator: evaluateTopic,
      criticality: 'required',
      interpret: { type: 'verdict' }
    },
    {
      evaluator: evaluateQuality,
      criticality: 'required',
      interpret: { type: 'number', pass: 0.7, partial: 0.4 }
    },
    {
      evaluator: evaluateContent,
      criticality: 'informational',
      interpret: { type: 'boolean' }
    },
    {
      evaluator: evaluateTone,
      criticality: 'informational',
      interpret: { type: 'string', pass: ['professional', 'informative'], partial: ['casual'] }
    }
  ]
});
```

### Criticality

* **`required`** (default): If this evaluator fails, the entire case fails.
* **`informational`**: Failure is reported but doesn't affect the case verdict. Use for metrics you want to track without gating on.

### Interpret Types

The `interpret` config tells the framework how to convert the raw evaluator result into a pass/partial/fail verdict:

| Type | Config | Pass when | Partial when | Fail when |
| - | - | - | - | - |
| `boolean` | `{ type: 'boolean' }` | `value === true` | — | `value === false` |
| `verdict` | `{ type: 'verdict' }` | `value === 'pass'` | `value === 'partial'` | `value === 'fail'` |
| `number` | `{ type: 'number', pass: 0.7, partial: 0.4 }` | `value >= pass` | `value >= partial` | otherwise |
| `string` | `{ type: 'string', pass: ['a', 'b'], partial: ['c'] }` | `value in pass` | `value in partial` | otherwise |

The `partial` threshold is optional for both `number` and `string` types — omit it to have only pass and fail.

### Case Verdict Aggregation

Each dataset case runs all evaluators. The case-level verdict is determined by:

1. If any **required** evaluator fails → case **fails**
2. Else if any **required** evaluator is partial → case is **partial**
3. Otherwise → case **passes**

Informational evaluators never affect the case verdict.

## Datasets

Datasets are YAML files that live in `tests/datasets/` within your workflow directory. Each file contains one or more test cases — the top-level key is the case name.

```yaml tests/datasets/core_cases.yml theme={null}
basic_sum:
  input:
    values:
      - 1
      - 2
      - 3
      - 4
      - 5
  ground_truth:
    notes: "Simple sum test"

edge_case_empty:
  input:
    values: []
  ground_truth:
    notes: "Empty input should return 0"
```

The key is the case name — no separate `name:` field inside. Every case must have an `input` field; the framework throws if one is missing.

| Field | Required | Description |
| - | - | - |
| `input` | Yes | The workflow input |
| `ground_truth` | No | Expected values for evaluators to check against |
| `last_output` | No | Cached workflow output (used with `--cached` flag) |
| `last_eval` | No | Cached evaluation results from the last run |

Organise files however makes sense for your project — by topic, test suite version, or importance level:

```
tests/datasets/
├── core_cases.yml       # happy path coverage
├── edge_cases.yml       # edge cases and error paths
└── v2_regression.yml    # cases added for v2
```

The `--dataset` flag filters by case name regardless of which file it lives in:

```bash theme={null}
output workflow test my_workflow --dataset basic_sum,edge_case_empty
```

You can write datasets by hand, or generate them from workflow executions using the CLI — see [Dataset Commands](https://docs.output.ai/packages/cli.md#dataset-commands).

### Ground Truth Structure

Ground truth supports global values and per-evaluator overrides:

```yaml theme={null}
ground_truth:
  # Global — available to all evaluators
  notes: "Known good case"
  min_length: 100

  # Per-evaluator overrides
  evals:
    evaluate_topic:
      required_topic: "Stripe the payment processor"
    evaluate_content:
      required_content: "https://stripe.com"
```

When an evaluator runs, the framework merges global ground truth with its evaluator-specific values. Per-evaluator values override globals with the same key.

## Directory Structure

Eval files live alongside your workflow in a `tests/` directory:

```text theme={null}
src/workflows/
└── blog_generator/
    ├── workflow.ts
    ├── steps.ts
    ├── evaluators.ts          # Inline evaluators (for production loops)
    ├── types.ts
    ├── prompts/
    │   └── generate_blog@v1.prompt
    └── tests/
        ├── evals/
        │   ├── workflow.ts    # evalWorkflow() definition
        │   ├── evaluators.ts  # Workflow evaluators (verify + Verdict)
        │   ├── judge_topic@v1.prompt
        │   └── judge_quality@v1.prompt
        └── datasets/
            ├── core_cases.yml
            └── edge_cases.yml
```

The eval workflow file (`tests/evals/workflow.ts`) is discovered automatically by the worker alongside your regular workflow.

## Running Evaluations

Use the CLI to run evaluations and manage datasets. See [CLI Evaluation Commands](https://docs.output.ai/packages/cli.md#evaluation-commands) for the full reference.

```bash theme={null}
# Run evals using cached output (no workflow re-execution)
output workflow test blog_generator --cached

# Run evals with fresh workflow execution and save results
output workflow test blog_generator --save

# Run specific datasets only
output workflow test blog_generator --dataset happy_path,edge_case

# List available datasets
output workflow dataset list blog_generator

# Generate a dataset from a scenario
output workflow dataset generate blog_generator my_scenario --name new_case
```

## API Reference

For complete TypeScript API documentation, see the [Evals Module API Reference](https://output-ai-reference-code-docs.onrender.com/modules/core_src_evals.html).


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.