SpudSuiteHelp Open SpudSuite

Agents

Evals and campaigns

Evals check that a workflow does what you expect, by running it on test cases and scoring what comes out. Campaigns run a workflow over a whole list of targets, such as every repository, within limits you set. Both run on your runners, never in SpudSuite.

Evals

Make one under Build β€Ί Evals. An eval names a workflow, a set of cases, and the graders that score each case.

  • Cases: each has a name, the workflow inputs to run it with, and what you expect. You can also take cases from one of your datasets, or import them from an eval platform's dataset.
  • Graders: each scores a case from 0 to 1, and a case's score is their weighted mean.
Grader How it scores
Expression An expression over outputs, expect, inputs and run: true scores 1, false 0, or give a number
JSON Schema The output must match a schema
Judge model A model places the output on a rubric's levels, worst first
Decision A decision model answers a score or yes-or-no question about it
  • Threshold: a case passes when its score is at least this.
  • Gate: the whole eval run fails if its score falls below a bar (fail_under), or drops too far from a baseline run (max_regression). Handy before shipping a change to a workflow.
  • Variants: compare versions of the workflow side by side.

Choose Run to start it (pick only some cases if you like). A runner takes it on its next check-in, runs each case through the workflow, and grades it. Results show as they come in, worst first, and Export results downloads them. Send each run's traces and scores to Braintrust or Langfuse by adding an eval platform.

An eval run's results
An eval run's results

Replay recorded runs

A recorded run keeps everything it touched: its inputs, each model's answers, each connector's responses, each step's output. Secrets are kept as names only, never their values. An eval can replay recordings instead of running cases live, so it needs no network, no connections and no secrets, costs nothing, and gives the same answer every time. It's the quickest way to check a change to a workflow's steps or expressions.

  1. Record a run: tick Record this run when you start it, or choose Run again and record on a finished run.
  2. When it ends, the run page shows Recorded. Choose Add to an eval. It goes into <workflow>-replay (made for you if there isn't one), or another replay eval of the same workflow.
  3. The new case expects what the run returned. Edit the case's expect to say what should have happened, and pick graders that check it.
  4. Run the eval as usual.

If a changed workflow asks for something the recording doesn't have, such as a connector call with different inputs, that case fails with a tape miss that shows the call. Set replay β€Ί on miss to sequence to answer it with the next recorded call of that kind instead, or list fields to leave out of matching under replay β€Ί ignore, like action:jira.create_issue.params.due.

Steps SpudSuite can't see inside, such as scripts, approvals and Claude Code agents, give back what they gave when the run was recorded. Recordings are kept for 30 days, or for as long as an eval case uses them. Recording needs runners from version 0.1.24; a recorded run waits for one. To review many recordings at once and grow a dataset from them, see Datasets and recordings.

Campaigns

Make one under Operate β€Ί Campaigns. A campaign names a workflow, its targets (a target list, or targets listed in the campaign), and the inputs for each run, written as expressions over the target, like {{ target.repository }}.

Its policy keeps it in check:

  • Concurrency: how many runs at once (5 by default).
  • Runs per hour: a ceiling on the pace.
  • Max failures: pause the campaign after this many failures.
  • Budget: stop spending past an amount.

Choose Launch, optionally with extra inputs for this launch only. A runner resolves the targets and runs the workflow for each. Follow it on the launch's page, where you can Pause, Resume or Cancel it. Cancelling also cancels its runs that are still going.

Launching a campaign
Launching a campaign

Something unclear, or out of date? Spud wants to know: tell us from SpudSuite and we'll fix the page.