2 posts tagged with "Evals"

A skill that "feels right" can still add nothing, or make the agent worse. Paired evals answer the question with two runs of the same cases: one with the skill, one without. In this part you use the seeded woo-triage-paired-eval dataset to run a paired baseline against woo-request-triage, read the delta, and watch the L5 evidence appear where Part 2 left a named gap.

You now have a working skill — but does it actually work well? And does it keep working when Claude itself gets upgraded or when you tweak the skill? This third, optional part shows how to evaluate a skill systematically: test scenarios, trigger tests, a baseline measurement, and the eval runner that /skill-creator sets up for you. This is the step from Maturity Level 4 ("feels right") to Level 5 ("measurably right").