SpudSuiteHelp Open SpudSuite

Agents

Datasets and recordings

A dataset is a list of eval cases that several evals can share. Recordings are runs kept whole so an eval can replay them offline. Put the two together and your evals grow from what your workflows really did, one reviewed run at a time.

Record runs

There are three ways to record a run:

  • Tick Record this run when you start one, or choose Run again and record on a finished run.

  • Give a workflow a capture sample, and that share of its runs is recorded. Runs from triggers count too. For one run in ten, add this to the workflow's YAML:

    spec:
      capture:
        sample: 0.1
    

A recording keeps the run's inputs, every model answer and connector response, and each step's output. Secrets are kept as names, never their values. Recordings are deleted after 30 days, unless a dataset or eval case uses them. They count towards your workspace's file storage.

Review recordings

Open Build โ€บ Recordings. To review lists recorded runs that nobody has looked at yet, with what went in and what came out. For each one, choose:

  • Good if it did the right thing. It joins the workflow's dataset as a case that expects exactly what came out.
  • Wrongโ€ฆ if it didn't. Write what it should have returned and choose Save the fix, and it joins the dataset expecting your fix. Choose Just mark it wrong to leave it out.
  • Skip if it's not worth keeping.

Recordings waiting for review
Recordings waiting for review

Cases go into <workflow>-cases, which is made for you the first time. To use another dataset, type its name in Add to dataset first. Reviewed lists the recordings you've already marked.

Datasets

Build โ€บ Datasets lists your datasets. Open one to edit its cases by hand. A case marked recorded replays a recording.

A dataset with a recorded case
A dataset with a recorded case

  • Download JSONL saves the cases as a file, one case a line, ready for git or another tool.
  • Import JSONL adds the cases in a file to the dataset. Each line is a case with at least a name, like {"name": "short text", "inputs": {"text": "hi"}, "expect": {"label": "small"}}.

Use a dataset in an eval

In an eval, pick the dataset under Cases from a dataset. Its cases run along with any written in the eval, as they are each time the eval runs. To replay the recordings instead of running live, give the eval a replay section:

spec:
  workflow: label
  dataset: { name: label-cases }
  replay: { mode: full }
  graders:
    - { name: same-outputs, type: expression, check: "outputs == expect" }

Every case needs a recording to replay. A dataset case you typed in by hand has none, so the eval won't start until you remove it or drop replay.

Check a change against production

To replay your latest recorded runs against the current version of a workflow, use from_runs with replay. Each case expects what that run returned at the time:

spec:
  workflow: label
  from_runs: { limit: 50, since: 7d }
  replay: { mode: full }
  graders:
    - { name: same-outputs, type: expression, check: "outputs == expect" }

Any case that now comes out differently shows you what your change would have done to real traffic. If a changed workflow asks for something the recording doesn't have, that case fails with a tape miss (see Replay recorded runs).

Datasets and replaying production runs need runners from version 0.1.27.

Something unclear, or out of date? Spud wants to know: tell us from SpudSuite and we'll fix the page.