pmstack

Eval Studio for product managers

Find how your AI product fails. Iterate. Raw pattern recognition meets Product Sense.

Read real traces, name the failure modes, and turn the ones that matter into checks you can trust.

Open Eval Studio Try the dental booking sample

Free. Runs in your browser. Your traces stay on your computer.

The funnel of an AI experience for a dental booking assistant. 100 reviewed conversations move left to right through five stages: understand the request, hand off when needed, check the calendar, book or change, reply to the patient. At each stage, failing conversations drop out under a named failure mode, such as 7 that ignored requests for a person, and green chips show success modes, such as repeating the booking back. A bottom row shows the check for each failure mode, where one exists. 64 of 100 reach a good outcome.
How to read it. Each trace is one full conversation or task, with every step the AI took. Green chips are success modes to keep working. Red ribbons are failure modes leaving the funnel, each counted once at the first stage that went wrong. The bottom row shows the check for each failure mode, when it has one: a code check (a rule a computer can test) or an AI judge (a prompt that asks a model for pass or fail).

Pick a sample product

Each sample is a made-up product with realistic traces. It opens in Eval Studio, ready to review, with nothing to upload or install.

How it works

pmstack follows error discovery, the method taught by Hamel Husain and Shreya Shankar. You read before you measure, so every check you build tracks a problem your customers actually hit.

Error discovery in six steps: read traces, write notes, group into failure modes, count by stage, build checks, then test the checks and keep them running. Steps one to three repeat until new failure modes stop appearing. A problem caused by a missing instruction is fixed right away instead of getting a check.

  1. Step 1: Read traces

    See each trace the way your customer saw it.

    In Review traces

  2. Step 2: Write notes

    Note the first thing that went wrong, in plain words.

    In Review traces

  3. Step 3: Group into failure modes

    Sort your notes into a short list of named problems, and name what went well as success modes.

    In Failure modes

  4. Step 4: Count by stage

    See where each failure mode starts and how often, then decide: fix it now, build a check, or keep watching.

    In Funnel

  5. Step 5: Build checks

    Turn each failure mode worth tracking into a code check or an AI judge.

    In Checks

  6. Step 6: Test, then keep running

    Make sure each check agrees with your labels, then run it on every change.

    In Checks and Report

Every product is a different funnel

Eval Studio suggests stages from how your AI works, then shows each trace the way your customer saw it: a text message, a phone call, an email, an answer with sources, a ranked list, or a code review. If none of those fit, build your own view.

Six ways AI products are built, drawn on the same stages from understand to answer, with a red mark where failures usually start in each.

Checks for agents that use tools

When your AI looks up an account, issues a credit, or books a visit, each tool call can go wrong in three ways. Tool call checks ask one question for each.

Three questions for every tool call, shown on one example. The customer asks: The outage took us down for two days. Can I get a credit? The agent calls the tool issue_credit with account_id A-1182 and amount 80. The tool returns status pending_approval. The agent replies: Done! $80 is off your next bill. Policy sits on the call: allowed, verified, confirmed, under the limit? Here it fails, because a credit over $50 needs a supervisor. It is checked with a code check of policy rules. Relevance spans the request and the call: right tool, right details for what they asked? It is checked with a code check that uses an intent map, or with an AI judge. Output grounding spans the tool result and the reply: does the reply match what the tool returned? Here it fails, because a pending credit is shown as done. It is checked with a code check of the values, then an AI judge. Policy rules come from your company. Relevance and grounding problems show up when you read traces.

  1. Policy: Is this call allowed?

    Your company's rules for which tools the agent may use, when, and with what details.

    Code check: policy rules

  2. Relevance: Is it the right call for what the customer asked?

    Right tool, right details, no calls the request didn't need, none it skipped.

    Code check: intent map, or AI judge

  3. Output grounding: Does the reply match what the tool returned?

    No contradicted values, no invented facts, no dropped qualifiers (pending shown as done), no success claimed after a failed call.

    Code check: values, then AI judge

Write your policy rules down up front, then read traces to find the rules you forgot. Confirm relevance and output grounding problems in your own traces, then start from the templates: an enterprise policy file, an intent map, and AI judge prompts.

In Claude Code:/pmstack:tool-policy/pmstack:tool-relevance/pmstack:tool-grounding
Try the support agent sample Read the guide

Use it with Claude Code

The pmstack skills set up your traces, start Eval Studio on your own folder, and help you group notes, write AI judges, and test them. You still make every call on what counts as a failure.

  1. Add the pmstack marketplace
    /plugin marketplace add RyanAlberts/pmstack
  2. Install the plugin
    /plugin install pmstack@pmstack
  3. Start: it asks a few questions and picks the right skill
    /pmstack:start

Or run it on your computer

Node 20 or newer, nothing to install. Point it at your own folder of traces the same way.

git clone https://github.com/RyanAlberts/pmstackcd pmstacknode bin/pmstack.mjs studio examples/quickstart --open