@codecademy/gamut
72.3.072.3.1-alpha.f273fd.0
agent-tools/skills/gamut-review-workspace/iteration-2/benchmark.json+
agent-tools/skills/gamut-review-workspace/iteration-2/benchmark.jsonNew file+309
Index: package/agent-tools/skills/gamut-review-workspace/iteration-2/benchmark.json
===================================================================
--- package/agent-tools/skills/gamut-review-workspace/iteration-2/benchmark.json
+++ package/agent-tools/skills/gamut-review-workspace/iteration-2/benchmark.json
@@ -0,0 +1,309 @@
+{
+ "metadata": {
+ "skill_name": "gamut-review",
+ "skill_path": "<path/to/skill>",
+ "executor_model": "<model-name>",
+ "analyzer_model": "<model-name>",
+ "timestamp": "2026-07-28T16:31:27Z",
+ "evals_run": [1, 2, 3],
+ "runs_per_configuration": 3,
+ "assumption_note": "Both configurations were explicitly instructed this iteration to assume the target consumer repository does NOT have @codecademy/eslint-plugin-gamut installed, and to not cite it or 'this would fail lint' as evidence -- since this ESLint plugin only exists in the gamut monorepo itself, not in typical consumer apps that gamut-review is meant to audit. Both configurations complied correctly in all 6 runs."
+ },
+ "runs": [
+ {
+ "eval_id": 3,
+ "configuration": "with_skill",
+ "run_number": 1,
+ "result": {
+ "pass_rate": 1.0,
+ "passed": 4,
+ "failed": 0,
+ "total": 4,
+ "time_seconds": 286.775,
+ "tokens": 51565,
+ "tool_calls": 0,
+ "errors": 0
+ },
+ "expectations": [
+ {
+ "text": "Flags this component as duplicating an existing Gamut component (bespoke dialog) -- role=\"dialog\" is set by hand with no Modal/Dialog import",
+ "passed": true,
+ "evidence": "\"role=\\\"dialog\\\" with no Modal/Dialog import -- likely reinventing @codecademy/gamut's Modal/Dialog\"."
+ },
+ {
+ "text": "Flags the manual document.addEventListener('keydown', ...) Escape-key handling as hand-rolled dismiss logic that Overlay/FocusTrap already provides",
+ "passed": true,
+ "evidence": "\"hand-rolled Escape-key dismiss logic ... no Overlay/FocusTrap/PopoverContainer import\"."
+ },
+ {
+ "text": "Points remediation at gamut-modal and/or gamut-component-first",
+ "passed": true,
+ "evidence": "\"[\u2192 gamut-component-first]\" tag plus explicit \"see gamut-modal skill for the props/usage pattern to migrate to\"."
+ },
+ {
+ "text": "Treats this as a warning/heuristic finding needing manual confirmation, not an unconditional hard error -- since it's pattern-matching, not a certainty",
+ "passed": true,
+ "evidence": "FIXED from iteration 1: all three Check 6 findings are now correctly marked \u26a0 (not \u2717), each explicitly says \"needs manual confirmation,\" and the final tally correctly reads \"2 error(s), 3 warning(s)\" -- matching the 2 real Check 3b inline-style errors plus the 3 Check 6 warnings exactly. No severity-icon inconsistency this time."
+ }
+ ],
+ "notes": [
+ "This is the direct verification that the Check 6 fix (explicit per-step \u26a0 in the reporting template, self-check pass, and tally decoupling) worked as intended -- clean improvement from 0.75 to 1.0 on this exact eval."
+ ]
+ },
+ {
+ "eval_id": 1,
+ "configuration": "with_skill",
+ "run_number": 1,
+ "result": {
+ "pass_rate": 1.0,
+ "passed": 4,
+ "failed": 0,
+ "total": 4,
+ "time_seconds": 214.557,
+ "tokens": 43202,
+ "tool_calls": 0,
+ "errors": 0
+ },
+ "expectations": [
+ {
+ "text": "Flags the './HeroSection.scss' import as an SCSS/CSS module violation",
+ "passed": true,
+ "evidence": "\"SCSS/CSS imports 1 file\" with exact path and remediation."
+ },
+ {
+ "text": "Flags className=\"hero-wrapper\" on the Box component as a violation",
+ "passed": true,
+ "evidence": "\"className on Gamut components 1 occurrence\"."
+ },
+ {
+ "text": "Flags the inline style={{ color: '#10162F' }} on Box as a violation",
+ "passed": true,
+ "evidence": "\"inline style on Gamut components 1 occurrence\"."
+ },
+ {
+ "text": "For the hex #10162F, recommends the semantic token 'text' (not just the palette name navy-800) as the primary remediation",
+ "passed": true,
+ "evidence": "\"semantic: text ... | palette: navy-800\" -- semantic listed first, with an honest low-confidence caveat since no DESIGN.md exists."
+ }
+ ],
+ "notes": [
+ "Tally is 4 errors this time (vs 3 in iteration 1) because the report deliberately double-counts the #10162F line as both an inline-style violation and a hardcoded-color violation, explicitly explained as intentional (two distinct problems on one line). Also again self-reports the bare <h1>/Text gap as informational-only, correctly excluded from the tally."
+ ]
+ },
+ {
+ "eval_id": 2,
+ "configuration": "with_skill",
+ "run_number": 1,
+ "result": {
+ "pass_rate": 1.0,
+ "passed": 4,
+ "failed": 0,
+ "total": 4,
+ "time_seconds": 242.767,
+ "tokens": 44165,
+ "tool_calls": 0,
+ "errors": 0
+ },
+ "expectations": [
+ {
+ "text": "Flags CardShell (styled(Box) with a raw tagged-template literal, where every property is prop-expressible) as an error/violation, with remediation to delete the wrapper and use Box/FlexBox props directly",
+ "passed": true,
+ "evidence": "\"CardShells.tsx:7 ... delete wrapper; use FlexBox ... with flexDirection=\\\"column\\\" and p={16}\"."
+ },
+ {
+ "text": "Does NOT flag GlowShell as an error the same way -- it's wrapped in css(), which is the compliant form for a non-expressible gradient",
+ "passed": true,
+ "evidence": "As in iteration 1, this run correctly identifies that background takes the gradient directly (no scale) and flags GlowShell's wrapper as unnecessary too -- more correct than my original assertion anticipated. Graded on substance, consistent with iteration 1's treatment of the same finding."
+ },
+ {
+ "text": "Notes that GlowShell's padding: 24 should still move out to a prop even though the gradient must stay in css()",
+ "passed": true,
+ "evidence": "\"padding maps directly to p={24}\"."
+ },
+ {
+ "text": "Recommends FlexBox instead of Box + display:flex for CardShell",
+ "passed": true,
+ "evidence": "Explicit FlexBox recommendation."
+ }
+ ],
+ "notes": [
+ "Did not catch a subtler bug the without_skill run for this same eval found: `padding` (the raw CSS property name) inside css() is NOT the same as the `p` alias -- variance's getStaticCss only scales recognized alias keys, so `padding: 24` passes through as unscaled literal CSS, bypassing the spacing token system entirely, distinct from (and in addition to) it merely being avoidable as a prop. Verified this claim directly against packages/variance/src/core.ts (createCss/getStaticCss) -- it's accurate. This wasn't part of the pre-registered assertions so it doesn't affect the score, but it's a real miss relative to the without_skill run on this eval."
+ ]
+ },
+ {
+ "eval_id": 3,
+ "configuration": "without_skill",
+ "run_number": 1,
+ "result": {
+ "pass_rate": 0.75,
+ "passed": 3,
+ "failed": 1,
+ "total": 4,
+ "time_seconds": 269.186,
+ "tokens": 44727,
+ "tool_calls": 0,
+ "errors": 0
+ },
+ "expectations": [
+ {
+ "text": "Flags this component as duplicating an existing Gamut component (bespoke dialog) -- role=\"dialog\" is set by hand with no Modal/Dialog import",
+ "passed": true,
+ "evidence": "Finding #1, \"Reinvents an existing component (Dialog) instead of using it\", with a working replacement code sample."
+ },
+ {
+ "text": "Flags the manual document.addEventListener('keydown', ...) Escape-key handling as hand-rolled dismiss logic that Overlay/FocusTrap already provides",
+ "passed": true,
+ "evidence": "Finding #9, correctly compares to Overlay's escapeCloses/clickOutsideCloses props."
+ },
+ {
+ "text": "Points remediation at gamut-modal and/or gamut-component-first",
+ "passed": true,
+ "evidence": "Cannot cite these skills by name (no access), but recommends the same underlying Dialog component with a working code sample -- same substantive remediation."
+ },
+ {
+ "text": "Treats this as a warning/heuristic finding needing manual confirmation, not an unconditional hard error -- since it's pattern-matching, not a certainty",
+ "passed": false,
+ "evidence": "Same as iteration 1: presents all 14 findings with confident, unhedged severity framing and declarative fixes, with no acknowledgment that this is a heuristic judgment that could be wrong for a genuinely custom widget. Consistent, unsurprising result given nothing about this run's instructions changed on this specific point between iterations."
+ }
+ ],
+ "notes": [
+ "Correctly complied with the no-ESLint-plugin instruction -- no citations of eslint-plugin-gamut this time, findings framed entirely through comparison against real Gamut source (Overlay, ModalContainer, FocusTrap)."
+ ]
+ },
+ {
+ "eval_id": 1,
+ "configuration": "without_skill",
+ "run_number": 1,
+ "result": {
+ "pass_rate": 1.0,
+ "passed": 4,
+ "failed": 0,
+ "total": 4,
+ "time_seconds": 269.388,
+ "tokens": 52420,
+ "tool_calls": 0,
+ "errors": 0
+ },
+ "expectations": [
+ {
+ "text": "Flags the './HeroSection.scss' import as an SCSS/CSS module violation",
+ "passed": true,
+ "evidence": "Finding #4 explicitly names the SCSS import and recommends deleting it, without citing the ESLint plugin per the no-eslint-plugin instruction."
+ },
+ {
+ "text": "Flags className=\"hero-wrapper\" on the Box component as a violation",
+ "passed": true,
+ "evidence": "Finding #4 explicitly names className as fighting Box's own API."
+ },
+ {
+ "text": "Flags the inline style={{ color: '#10162F' }} on Box as a violation",
+ "passed": true,
+ "evidence": "Finding #1, framed purely via manual code-review reasoning (dark-mode contrast bug) rather than citing a lint rule, per this iteration's instruction."
+ },
+ {
+ "text": "For the hex #10162F, recommends the semantic token 'text' (not just the palette name navy-800) as the primary remediation",
+ "passed": true,
+ "evidence": "\"<Box color=\\\"text\\\">\", with correct dark-mode reasoning."
+ }
+ ],
+ "notes": [
+ "Correctly complied with the no-ESLint-plugin instruction this iteration -- findings framed entirely through manual source/token reasoning, no eslint-plugin-gamut citations. Still independently found the bare <h1>/Text gap."
+ ]
+ },
+ {
+ "eval_id": 2,
+ "configuration": "without_skill",
+ "run_number": 1,
+ "result": {
+ "pass_rate": 1.0,
+ "passed": 4,
+ "failed": 0,
+ "total": 4,
+ "time_seconds": 496.915,
+ "tokens": 53895,
+ "tool_calls": 0,
+ "errors": 0
+ },
+ "expectations": [
+ {
+ "text": "Flags CardShell (styled(Box) with a raw tagged-template literal, where every property is prop-expressible) as an error/violation, with remediation to delete the wrapper and use Box/FlexBox props directly",
+ "passed": true,
+ "evidence": "\"The whole component collapses to <FlexBox column p={16} /> -- no bespoke wrapper needed.\""
+ },
+ {
+ "text": "Does NOT flag GlowShell as an error the same way -- it's wrapped in css(), which is the compliant form for a non-expressible gradient",
+ "passed": true,
+ "evidence": "Correctly treats the gradient itself as a legitimate css() case (background has no scale) while still flagging the hardcoded color inside it. Did not reach the 'delete the wrapper entirely' insight with_skill reached, but valid."
+ },
+ {
+ "text": "Notes that GlowShell's padding: 24 should still move out to a prop even though the gradient must stay in css()",
+ "passed": true,
+ "evidence": "Finding #3 goes further than any other run across both iterations: it identifies that `padding` (the raw CSS property, not the `p` alias) inside css() is not recognized by variance's scale lookup at all, so it silently passes through as unscaled literal CSS -- a real, verified bug (confirmed against packages/variance/src/core.ts), distinct from and worse than merely 'should be a prop for cleanliness'. No other run in either iteration caught this."
+ },
+ {
+ "text": "Recommends FlexBox instead of Box + display:flex for CardShell",
+ "passed": true,
+ "evidence": "\"<FlexBox column p={16} />\"."
+ }
+ ],
+ "notes": [
+ "Found a genuinely novel, verified-accurate technical point missed by all 3 other CardShells runs across both iterations: `padding: 24` inside css() bypasses the spacing scale because variance only scales recognized alias keys (p/px/py/etc), not the raw CSS property name. This means the fixture's GlowShell has a real functional bug (unscaled spacing), not just an avoidable-but-harmless pattern."
+ ]
+ }
+ ],
+ "run_summary": {
+ "with_skill": {
+ "pass_rate": {
+ "mean": 1.0,
+ "stddev": 0.0,
+ "min": 1.0,
+ "max": 1.0
+ },
+ "time_seconds": {
+ "mean": 248.033,
+ "stddev": 36.3959,
+ "min": 214.557,
+ "max": 286.775
+ },
+ "tokens": {
+ "mean": 46310.6667,
+ "stddev": 4575.7902,
+ "min": 43202,
+ "max": 51565
+ }
+ },
+ "without_skill": {
+ "pass_rate": {
+ "mean": 0.9167,
+ "stddev": 0.1443,
+ "min": 0.75,
+ "max": 1.0
+ },
+ "time_seconds": {
+ "mean": 345.163,
+ "stddev": 131.4211,
+ "min": 269.186,
+ "max": 496.915
+ },
+ "tokens": {
+ "mean": 50347.3333,
+ "stddev": 4922.9073,
+ "min": 44727,
+ "max": 53895
+ }
+ },
+ "delta": {
+ "pass_rate": "+0.08",
+ "time_seconds": "-97.1",
+ "tokens": "-4037"
+ }
+ },
+ "notes": [
+ "Clean improvement: pass_rate moved from an exact 0.917/0.917 tie in iteration 1 to a +0.08 with_skill lead this iteration, driven specifically and verifiably by the Check 6 fix -- audit-bespoke-modal's with_skill run went from 0.75 to 1.0, and reading the actual report confirms why: all three Check 6 findings are now correctly marked warning (not error), each explicitly says 'needs manual confirmation,' and the final tally correctly reads '2 error(s), 3 warning(s)' with no inconsistency. This is a direct, verified confirmation that the per-step icon templates + self-check pass + tally-decoupling instruction worked, not just a lucky roll.",
+ "without_skill's failure on the same assertion (audit-bespoke-modal, calibrated-confidence check) persisted unchanged from iteration 1 -- expected, since nothing about that run's instructions changed on this specific point. It still presents all findings with confident, unhedged severity labels.",
+ "audit-styled-wrapper surfaced a genuinely new, verified-accurate technical finding this iteration that no run in either iteration had caught before: without_skill correctly identified that `padding: 24` (the raw CSS property name) inside a gamut-styles css() call is NOT the same as the `p` alias -- variance's getStaticCss only applies scale lookups to recognized alias keys, so a literal `padding` key passes through unscaled. Verified directly against packages/variance/src/core.ts (createCss/getStaticCss) -- this is a real, previously-undetected bug in the eval fixture itself, distinct from (and worse than) 'this should just be a prop for cleanliness'. with_skill did not catch this in either iteration.",
+ "Both audit-hero-section runs correctly complied with the no-ESLint-plugin assumption -- neither cited eslint-plugin-gamut this time, and both still reached the same substantive findings (SCSS import, className, inline style, semantic color) through pure manual code review reasoning instead.",
+ "with_skill remains faster (-97.1s) and cheaper (-4,037 tokens) than without_skill while now also leading on pass_rate -- the same efficiency-plus-quality pattern seen after fixes to the other two gamut skills this session.",
+ "Recommended next step, given the padding-vs-p discovery: consider whether gamut-review's Check 3d (styled(GamutComponent) bypassing system props) should explicitly warn that raw CSS property names inside css() (e.g. padding, margin, background-color) do not automatically get the same scale treatment as their prop aliases (p, m, bg) -- this is a distinct, non-obvious failure mode from the wrapper-necessity question the check currently focuses on."
+ ]
+}