ZeroWidth Caliper

The 45 tools it listed

"Read only" and "Changes data" are the server's own marks. Underlined parameters are required.

1. caliper_datasets_add_items

   Add items to a Caliper dataset

   Appends items to an existing dataset — use this to grow coverage (new edge cases, scenarios from a completed interview) instead of creating a parallel dataset. Same four shapes as caliper_datasets_create (Q&A, chat, sequence, simulated). Existing items and their ratings are untouched.

   datasetId items workspace approvalId

2. caliper_datasets_create

   Create a Caliper dataset

   Creates a dataset of test items — THE FIRST STEP of setting up evaluation for a flow. Four item shapes: Q&A (input + optional expectedOutput, the golden answer); CHAT (messages: the conversation so far as role/content user and assistant messages you write, ending on a user message, + expectedResponse for the next reply; the flow writes only that reply, so every earlier assistant turn stays fixed; THE shape for an agent's conversation, built up by depth: a greeting alone, then two messages, then four, each item testing one more reply with the earlier ones baked in); SEQUENCE (turns: 2-20 scripted user messages the model answers one at a time with its own earlier replies in front of it, + expectedResponse for the final reply, optional expectedBehavior for the whole conversation); SIMULATED (goal + optional persona/disposition/strategy/maxTurns + expectedBehavior; a platform flow plays a person adaptively, Caliper-run evals only; disposition is a preset id like genuine, pressure, confused

   visibility title description items workspace approvalId

3. caliper_datasets_deleteChanges data

   Delete a Caliper dataset

   Deletes a dataset. Evals bound to it stop being runnable (their binding shows datasetOk: false), and reviews/specs over its items lose their source — check caliper_evals_list for evals that reference it and say so before proposing. May return `needs_confirmation`.

   datasetId workspace approvalId

4. caliper_datasets_generate

   Generate a Caliper dataset from a flow (spends credit)

   Creates a NEW dataset of model-written items — Q&A by default, or scripted sequences / simulated people via `shape` — pass flowId and the generator reads the flow's prompt, mode, and schema to write realistic cases for THAT flow; `description` adds guidance (or stands alone when there's no flow). Use this when the user wants test cases fast and has none; prefer caliper_datasets_create with hand-written items when real scenarios are already in hand (Compass pages, a transcript). This SPENDS workspace inference credit (one generator call), so it sits behind the approval gate: say so and expect `needs_confirmation`. Returns the dataset summary; read the items with caliper_datasets_get and tell the user to review them before trusting an eval built on them.

   title flowId description count shape disposition outputsMode expectedStyle presetId datasetDescription visibility workspace approvalId

5. caliper_datasets_generate_items

   Generate more items for a Caliper dataset (spends credit)

   Appends model-written items to an EXISTING dataset, in the style of what's already there (existing items are the few-shot examples; the output shape matches theirs unless overridden). Use to widen coverage when the user says 'more like these' or 'add edge cases'; write them by hand with caliper_datasets_add_items when the scenarios are known. SPENDS workspace inference credit, so it sits behind the approval gate — say so. Existing items and their ratings are untouched.

   datasetId count description shape disposition outputsMode expectedStyle anchorItemId workspace approvalId

6. caliper_datasets_getRead only

   Read a Caliper dataset (items paged)

   One dataset with a page of its items — read this BEFORE editing items (caliper_datasets_update_item needs the item id) and before extending coverage, so you don't add cases that already exist. Items come `limit` at a time (default 25) from `offset`; `total` is the full count. Long fields are cut at ~800 characters with a truncation marker. Get the datasetId from caliper_datasets_list.

   datasetId limit offset workspace

7. caliper_datasets_listRead only

   List Caliper datasets

   Lists datasets in the active workspace. Returns summaries (id, name, item count); items are not included.

   workspace

8. caliper_datasets_remove_itemChanges data

   Remove one item from a Caliper dataset

   Drops one item. Ratings and run results keyed to it are orphaned (kept in history, gone from the dataset). A dataset must keep at least one item. May return `needs_confirmation`.

   datasetId itemId workspace approvalId

9. caliper_datasets_updateChanges data

   Rename / re-describe / re-scope a Caliper dataset

   Changes a dataset's title, description, or visibility. Pass only what changes; items are untouched (use caliper_datasets_add_items / caliper_datasets_update_item / caliper_datasets_remove_item for those). May return `needs_confirmation`.

   datasetId title description visibility workspace approvalId

10. caliper_datasets_update_itemChanges data

    Replace one item in a Caliper dataset

    Rewrites one item's content, keeping its id (ratings and eval results keyed to it stay attached). The item's KIND can't change — a Q&A item stays Q&A, a chat item stays chat, a sequence stays a sequence — so pass the same shape you read from caliper_datasets_get. Raw items, and chat items captured with tool calls in them, can't be edited from here. May return `needs_confirmation`.

    datasetId itemId item workspace approvalId

11. caliper_evals_create

    Create a Caliper eval

    Binds a dataset + rubric to something under test as a repeatable eval — THE LAST SETUP STEP before scoring. Two targets: a Workbench flow (pass flowId; Caliper runs inference itself, then caliper_evals_run scores it), or 'external' (target: 'external'; the user's own model runs elsewhere and their script submits outputs through the public API, usually from CI — see the docs guide 'Run evals in CI'). For an external eval, hand back the eval id and tell the user to create a workspace API key with the ci_evals preset in their workspace settings; keys can't be minted from here. flowStage 'draft' evals the live draft (pre-publish); 'published' (default) evals the latest published revision at run time, or one pinned with flowRevisionId. Reuse an existing rubric from caliper_rubrics_list when one already scores this job.

    visibility title description datasetId rubricId target externalLabel flowId flowStage flowRevisionId scenarioId workspace approvalId

12. caliper_evals_deleteChanges data

    Delete a Caliper eval

    Deletes an eval and stops its schedule. Run history is kept but no longer reachable from the eval. May return `needs_confirmation`.

    evalId workspace approvalId

13. caliper_evals_getRead only

    Read one Caliper eval

    One eval: what it targets (Workbench flow + stage, or external), its dataset and rubric ids, the rubric criteria it scores with (the snapshot taken at creation), schedule + runOnPublish + regression settings, run counts, latest score, and `binding` health (datasetOk / flowOk false = a run would fail at resolution — say so before proposing caliper_evals_run). Get the evalId from caliper_evals_list or caliper_flow_performance.

    evalId workspace

14. caliper_evals_listRead only

    List Caliper evals

    Lists evals in the active workspace with their target config (which Workbench flow, which dataset/rubric). Use to find the evalId for caliper_evals_run.

    workspace

15. caliper_evals_run

    Run a Caliper eval

    Queues a new run of an eval — inference over the dataset, then LLM-judge scoring. THE VERIFY STEP of the improvement loop: after an approved workbench_flows_edit_text, run the eval again and report the score delta vs the previous run. Runs take a while — but you're brought back into THIS conversation automatically with the scores the moment it finishes, so tell the user it's queued and that you'll follow up here; never poll or ask them to check back. Costs workspace LLM budget, so it sits behind the approval gate: may return `needs_confirmation`.

    evalId label scenarioId workspace approvalId

16. caliper_evals_runs_cancelChanges data

    Stop a running Caliper eval run

    Stops a run that is still PENDING / RUNNING / SCORING — the brake on a run that's spending more than expected or was started by mistake. The queue stops at once; an item already handed to the flow finishes on its own timeout. The run settles as FAILED with a 'cancelled' reason and keeps the items it completed. A run that already finished returns run_not_live. May return `needs_confirmation`.

    evalId runId workspace approvalId

17. caliper_evals_runs_getRead only

    Read one eval run (per-item, per-criterion scores)

    One run: status, overall score, and by default the ten lowest-scoring items with their per-criterion scores and the judge's reasoning (`items: all` for every item, `none` for totals only). THE ANSWER to 'why did the score drop' — read this, then name the criterion that slipped and quote the reasoning on the lowest items. Works for runs Caliper ran and for runs submitted from CI (triggeredBy 'ci', usually labeled with a pull request number). Still no polling loops; read it when the user asks.

    evalId runId items limit workspace

18. caliper_evals_runs_item_executionRead only

    Read the flow trace behind one scored item

    What the flow actually DID on one run item: every step in order (what it said, which tools it called with what arguments, what came back) and the final answer, plus status, duration, and cost. Read this when a low score needs explaining beyond the judge's reasoning — a wrong tool call or an empty tool result is usually the cause, and the fix is different from a prompt fix. Pass the run item's `id` (NOT `itemId`) from caliper_evals_runs_get. Items whose outputs were supplied from outside (external evals) have no trace and return not_found. Steps are capped for transport; the run page in Caliper has the full record.

    evalId runId runItemId workspace

19. caliper_evals_runs_listRead only

    List an eval's runs (status + scores)

    Recent runs for one eval, newest first: status (PENDING/RUNNING/SCORING/DONE/FAILED), overall score once DONE, label, and timestamps. THE CHECK-BACK for caliper_evals_run: when the user asks how the run went, read this — cite the run id and score, and compare against the PREVIOUS run's score for the delta. Still don't poll in a loop; check when the user asks or when reporting.

    evalId workspace

20. caliper_evals_updateChanges data

    Update a Caliper eval (metadata, schedule, alerts)

    Changes an eval's title, description, visibility, schedule (interval + time anchor), runOnPublish (queue a run whenever the flow publishes), regressionThreshold (score drop vs the previous run that triggers an alert; null = off), or scenarioId (the scenario runs use). Pass only what changes. The dataset, rubric snapshot, and target are fixed at creation — create a new eval to change those. Scheduled and on-publish runs spend credit on their own, so state that plainly when turning them on. May return `needs_confirmation`.

    evalId title description visibility scheduleInterval scheduleTime scheduleTimezone scheduleDayOfWeek runOnPublish regressionThreshold scenarioId workspace approvalId

21. caliper_flow_performanceRead only

    Eval performance for a Workbench flow

    How a Workbench flow is ACTUALLY doing, with receipts: every Caliper eval targeting the flow, recent runs with scores, the latest run decomposed into per-criterion averages, the score delta vs the previous run, and the worst-scoring items WITH the judge's reasoning. Use this BEFORE claiming a flow works or proposing changes — and cite the runId + scores when you do. The worst items are diagnostic: failures clustered around missing company facts suggest a knowledge gap (consider proposing a Compass interview with the workflow owner) rather than a prompt problem.

    flowId workspace

22. caliper_rubrics_create

    Create a Caliper rubric

    Creates the scoring rubric an eval's LLM judge uses — 1-10 criteria, each scored on a numeric scale (default 1-5), judged pass/fail (kind pass_fail), or checked in code with no judge (kind check: contains, not_contains, matches a regex, valid_json, max_chars, equals_expected). Write criteria about the FLOW'S JOB (accuracy to source material, tone, refusal behavior), not generic 'quality'.

    visibility title description criteria workspace approvalId

23. caliper_rubrics_deleteChanges data

    Delete a Caliper rubric

    Deletes a rubric. Existing evals keep their snapshot of it and keep running; nothing new can bind to it. May return `needs_confirmation`.

    rubricId workspace approvalId

24. caliper_rubrics_getRead only

    Read a Caliper rubric (criteria + scales)

    One rubric with its criteria — each criterion's name, what the judge looks for, and the score scale. Read this to explain a score (which criterion slipped and what it asks for) or before caliper_rubrics_update. Get the rubricId from caliper_rubrics_list or an eval's rubricId.

    rubricId workspace

25. caliper_rubrics_listRead only

    List Caliper rubrics

    Lists the rubrics in the active workspace (id, name, description, criterion count). Check here BEFORE caliper_rubrics_create — reuse an existing rubric's id in caliper_evals_create when one already scores the same job. Read a rubric's criteria with caliper_rubrics_get.

    workspace

26. caliper_rubrics_updateChanges data

    Update a Caliper rubric

    Changes a rubric's title, description, visibility, or replaces its criteria wholesale (pass the FULL list — criteria get fresh ids). Evals snapshot the rubric when they're created, so an existing eval keeps scoring with the criteria it started with; say so, and offer to create a new eval when the criteria change materially. May return `needs_confirmation`.

    rubricId title description visibility criteria workspace approvalId

27. caliper_source_feeds_listRead only

    List a source's live datasets

    Datasets this source keeps adding to as conversations arrive: which dataset, for what (review, eval, spec), the pick it matches, sampling, how many have landed, and whether it's still running or why it stopped.

    sourceId workspace

28. caliper_source_feeds_stopChanges data

    Stop a live dataset

    Stops a source from adding new conversations to a dataset. What it already added stays. Get the feedId from caliper_source_feeds_list. May return `needs_confirmation`.

    sourceId feedId workspace approvalId

29. caliper_sources_listRead only

    List Caliper sources

    Apps sending their agent's traces to Caliper — from their own code, OpenTelemetry, or a published Workbench flow — busiest first: name, how it sends, traces in the last 14 days (per day), and the tools its agent calls most with the share of traces using each. Start here when the user asks how their agent behaves on real traffic; then caliper_sources_over_time for one source.

    workspace

30. caliper_sources_over_timeRead only

    Read a Caliper source over time

    One source day by day: traces, failures, tool calls, lookups that found nothing, tokens, cost, and speed (meanMs; p50/p95 as 'answered within' bucket edges); each tool with its calls and the share of each day's traces that used it; the documents lookups landed on most; models used. Use it to answer 'how often is it calling web search and is that changing' — compare the first and last weeks and name the day it moved. Follow with caliper_traces_list on that day or tool to show the conversations behind it.

    sourceId days workspace

31. caliper_sources_to_dataset

    Save a source's conversations to a dataset

    Saves the conversations behind a pick (a day, a tool, a document, failures — the same narrowing as caliper_traces_list) into a new dataset (newDatasetName) or an existing one (datasetId), newest first up to `limit`. `purpose` decides what each item keeps: review = the agent's reply, tool calls included, for people to rate; eval = the reply becomes the expected answer; spec = only the questions, for people to answer. Set keepAdding to make it live: new matching conversations keep arriving (every one, or 1 in 10 / 1 in 100), up to 1,000. Then offer the next step — caliper_evals_create, or a review in Caliper. May return `needs_confirmation`.

    sourceId day tool document errorsOnly purpose limit datasetId newDatasetName keepAdding workspace approvalId

32. caliper_starters_install

    Install a Caliper starter (dataset + rubric)

    Copies one starter into the workspace as a new dataset and a new rubric the user owns and can edit. Returns both ids; the next step is caliper_evals_create binding them to the flow (or target 'external'). Needs caliper:datasets:write and caliper:evals:write.

    slug workspace approvalId

33. caliper_starters_listRead only

    List Caliper starters (ready-made dataset + rubric pairs)

    The starter sets Caliper ships: prompt injection, system-prompt leaking, over-refusal, personal-data handling, bias under ambiguity. Each is a small original dataset (single messages AND multi-turn build-ups: false memory, fabricated earlier turns, slow escalation) paired with an anchored rubric. Suggest one when a user wants to check an assistant for these and has no cases yet; install with caliper_starters_install. Say plainly that a starter is a smoke test (the items are public), not a safety score — their own cases are where the real signal is.

34. caliper_traces_getRead only

    Read one trace

    One trace: what came in, what went out, and every step in order — model calls (model, tokens, cost, reasoning), tool calls (arguments and result), lookups (query and documents with scores), anything else — each with timing and status. Long values are cut at 2,000 characters. Use it to explain why one conversation went the way it did.

    sourceId traceId workspace

35. caliper_traces_listRead only

    List a source's conversations

    The traces behind a point on a source's chart, newest first, 50 a page: the question and answer (cut short), tools called, time, tokens, cost, status. Narrow by day, tool, document, or failures; page with `before` (the nextBefore from the last page). Open one with caliper_traces_get.

    sourceId day tool document errorsOnly before workspace

36. comments_create

    Comment on an entity

    Posts a comment on a workspace entity — a new thread, or a reply when rootId is given. Use it to leave findings where the discussion already lives (an eval result on the flow being debated, a summary on a long thread). Mention people via mentionedUserIds (from workspace member ids) to ring their notification bell; never mention someone who didn't ask to be pulled in.

    entityKind entityId body rootId mentionedUserIds workspace approvalId

37. comments_listRead only

    Read an entity's comment threads

    Lists the comment threads on one workspace entity (open first, then resolved) with authors and timestamps. Read this before weighing in on contested work — the threads are where disagreement lives before it becomes a decision.

    entityKind entityId workspace

38. comments_resolveChanges data

    Resolve or reopen a thread

    Sets a comment thread's resolved state (rootId = the thread's root comment id). Resolve ONLY when the human asked or the thread's question is demonstrably settled — and say what settled it in a reply first. Reopening is for new evidence.

    rootId resolved workspace approvalId

39. entity_tags_browseRead only

    Browse the workspace's tags

    Without a tag: every tag in use across the workspace with how many entities carry it, most-used first — the vocabulary the team already organizes by. With a tag: everything filed under it across every tool, each with its kind, id, title, and path. Use it to reuse existing labels instead of inventing near-duplicates, and to answer 'show me everything about X' when X is a label.

    tag workspace

40. entity_tags_getRead only

    Read the tags on entities

    Returns the tags on a batch of entities of one kind — the labels galleries organize by. Ids come from the kind's list/get tool or from search_workspace. Use it before entity_tags_set so you replace the full set knowingly, and to answer 'what is this filed under'. Entities the user can't see are omitted.

    entityKind ids workspace

41. entity_tags_setChanges data

    Set an entity's tags

    Replaces the FULL tag set on one entity (an empty list clears it). Read the current tags with entity_tags_get first and pass the merged list — this is not additive. Tags are lowercase letters, numbers, spaces, and hyphens; prefer labels already in use (entity_tags_browse) so the workspace's vocabulary stays small. The id comes from the kind's list/get tool or search_workspace.

    entityKind entityId tags workspace approvalId

42. get_docRead only

    Fetch ZeroWidth doc by slug

    Fetch the full Markdown body of a specific docs page by its slug. Use this after `search_docs` when the user needs the complete content of a page. No authentication required.

    slug

43. list_docsRead only

    List ZeroWidth docs pages

    Enumerate all available docs pages, optionally filtered by product (e.g. 'compass', 'legal', 'overview'). Use this to discover what slugs exist before calling `get_doc`. No authentication required.

    product

44. search_docsRead only

    Search ZeroWidth docs

    Search ZeroWidth product documentation. Returns matching pages with title, slug, public URL, and a query-relevant snippet. Use this when the user asks about a ZeroWidth product (Compass, Workbench, Caliper, Prism, Ledger, Napkin, the assistant), an API behavior, or a policy. No authentication required — the docs corpus is public.

    query limit

45. search_workspaceRead only

    Search the whole workspace

    Finds anything in the workspace in one call — uploaded files (searching the text inside them), folders, Workbench flows, shims, knowledge bases and tasks, Compass pages and changes, Caliper datasets, evals, rubrics, reviews, specs and sources, Ledger entries, Napkin sketches, decks, docs, sheets, diagrams and interfaces. The same search the person uses in Home. Use it FIRST whenever the user mentions something without saying where it lives ('the SOW', 'the dispatch interviews', 'that invoice'). The query is words in any order: each word must appear in the thing's name, its tags, the names of the folders it's filed under, or the text inside it — so 'northwind sow' finds the SOW filed in the Northwind folder, and 'dispatch interview' finds 'Dispatcher interview 1.md'. Results are ranked best first. Each hit carries its id, kind, path (for a link), folderPath (where it's filed, top-down), tags, and a snippet when the words were found inside it rather than in its name — say where a thing l

    q kinds folderId limit workspace