Hermiq skills tutorial series — Part 3: Measuring skills with paired evals

A skill that "feels right" can still add nothing, or make the agent worse. Paired evals answer the question with two runs of the same cases: one with the skill, one without. In this part you use the seeded woo-triage-paired-eval dataset to run a paired baseline against woo-request-triage, read the delta, and watch the L5 evidence appear where Part 2 left a named gap.

How a paired eval works

An EvalDataset is a named set of cases. Each case is a prompt plus an expectation:

A plain run executes every case once through the agent's real engine path: same impersonation, same model, same tools as a scheduled run. Nothing is delivered to Talk or notifications; results land only on the run record.

A paired run executes everything twice. The with-half exposes the agent's installed skills plus the dataset's linked skills. The without-half detaches the linked skills, in memory only, for that run. Stored installs are never touched, even if the run crashes halfway. The difference between the two pass rates is the baseline delta: the measured contribution of the linked skills.

Kill-switch and budget gates apply exactly as they do to any run, and every half's tokens count against the same budget. There is no separate spend meter for evals.

Step 1: Open the seeded dataset

Open Evaluations in the left navigation, then click woo-triage-paired-eval.

The seed carries three municipality-context cases against the woo-request-triage skill from Parts 1 and 2:

1. A triage prompt scored with contains "termijn": a proper triage names the response deadline.
2. A routing prompt scored with notContains "klacht": a WOO request must not be routed as a complaint.
3. A rubric case at threshold 0.7: the judge checks that the answer names routing, the four-week deadline, and an exemption pre-check.

Verify: the dataset detail page shows the three cases and one linked skill, woo-request-triage.

Step 2: Link and unlink skills on the dataset

The skill link lives on the dataset, in its skillRefs list. That direction matters: one dataset can qualify several skills, and a skill can be measured before it is ever installed on an agent.

1. On the dataset detail page, open the skill panel. It lists the linked woo-request-triage.
2. Link a second skill, for example your pet-store-order-summary from Part 1, then unlink it again. Both are plain object writes; there is no separate link endpoint.

Leave only woo-request-triage linked for the rest of this part.

Verify: the skill panel reflects each link and unlink immediately.

Step 3: Pick the agent's baseline mode

The agent decides how the without-half is constructed, via its evalBaselineMode property on the agent detail page. Open the info affordance next to the property; the explanation you read there is the contract:

With one linked skill the two modes coincide, so leave your agent on joint for this tutorial. Reach for per-skill when several skills share a dataset and you need to know which one earns its place.

Verify: the info affordance on evalBaselineMode explains both modes and the cost consequence in place.

Step 4: Run the paired baseline

1. On the dataset detail page, open the run panel.
2. Pick the agent that carries woo-request-triage (Part 1, step 3).
3. Enable the baseline toggle. It is only available because the dataset has a linked skill; on a dataset without skillRefs a baseline run is refused.
4. Read the cost note next to the toggle: roughly 2x in joint mode. Confirm and run.

The halves execute sequentially, so a paired run takes noticeably longer than a plain run. Rubric cases add a judge call per case per half.

Verify: a new run appears in the dataset's run history with a paired/baseline marking.

Step 5: Read the paired results

Open the completed run.

• Both halves side by side. Every case shows with-result and without-result, with failing cases distinguished in both halves (not by colour alone).
• Pass rates. passRate is the with-half aggregate. baselinePassRate is the without-half.
• The delta. Each linked skill's entry shows passRateWith, passRateWithout, and baselineDelta. A delta of 0.33 means the skill turned one of three failing cases into a pass.
• The regression gate. The with-half pass rate is compared against the previous completed run for this dataset and agent. A drop beyond the threshold records a failed gate. Your first run records not_applicable.

Two outcomes should worry you, and the display is honest about both. A delta near zero means the skill adds nothing on these cases. A negative delta means the agent did better without the skill; over-prescriptive skills do this more often than you would expect.

Verify: the run detail shows passRate, baselinePassRate, and a baselineDelta for woo-request-triage.

Step 6: Watch the L5 evidence arrive

A completed paired run is the only writer of L5 evidence in Hermiq. On completion, each linked skill receives levelEvidence.l5: the dataset id, the pass rate, the baseline delta, the completion timestamp, and the attribution mode. Failed or budget-blocked runs write nothing.

1. Open the woo-request-triage detail page.
2. The eval evidence card now shows the pass rate, the delta, the last-validated timestamp, and a pass-rate trend across this dataset's runs. In joint mode with several linked skills, the card labels the delta as a joint contribution, never as a per-skill marginal.
3. The maturity badge has not moved yet. Evidence and level are decoupled on purpose: run Qualify (Part 2) to fold the new evidence into the level.

The card also offers Run paired eval directly, with the same dataset picker, agent picker, and cost note, so future re-validation starts from the skill.

Verify: after qualifying, the scorecard reports L5 with the evidence values, provided L1 through L4 pass for this skill. If L4 is not attested yet, the scorecard says so; the evidence waits, contiguity holds.

What you get

A measured skill. The badge now rests on a with/without comparison through the real engine, and the scorecard shows when it was last validated. When the skill's content changes, the trend on the eval evidence card tells you whether it is drifting.

Where to go from here

Measurement is the gate everything else passes through. Part 4 closes the loop: skills that capture learnings from their own runs, propose improved versions of themselves, and still never change without your approval.