Claude Code adds a formal testing framework for its own plugins and skills
The new claude plugin eval command scores a plugin against realistic prompts with six grader types, compares the result to a no-plugin baseline, and can gate CI on the score.
Anthropic shipped claude plugin eval in Claude Code 2.1.269, released September 11, 2026. It gives plugin and skill authors a built-in way to test whether their plugin improves Claude's behavior. Each eval case pairs a realistic prompt with one or more graders: a regex over the reply, whether a specific tool got called, or a rubric a second model judges against. claude plugin eval init can propose and write an initial suite by asking the author what a good result looks like. Of the six grader types, four (regex, tool_used, tool_order, file_exists) are computed from the transcript at no cost. The other two, llm and baseline, call a judge model and bill against the account running them.
Every case runs twice
By default, every case runs three times with the plugin loaded and three times without it, producing a WITH and a W/OUT score. Anthropic's documentation explains why: "A high score on its own doesn't tell you the plugin helped." Claude might do as well unassisted, so the gap between the two scores, labeled Δ, is what gets attributed to the plugin. A skill whose description doesn't reliably trigger it shows up as a near-zero Δ with the tool_used grader failing, so it can't pass on the strength of the underlying model.
Built for CI
The same command runs unchanged in CI, so a team can gate a plugin or skill change on whether its eval score holds or regresses. Prompts and skill descriptions have mostly shipped without any automated check. If your team builds internal Claude Code plugins, this tells you whether a prompt change helped.