@codecademy/gamut
72.3.072.3.1-alpha.f273fd.0
agent-tools/skills/gamut-review-workspace/iteration-1/benchmark.json+
agent-tools/skills/gamut-review-workspace/iteration-1/benchmark.jsonNew file+309
Index: package/agent-tools/skills/gamut-review-workspace/iteration-1/benchmark.json
===================================================================
--- package/agent-tools/skills/gamut-review-workspace/iteration-1/benchmark.json
+++ package/agent-tools/skills/gamut-review-workspace/iteration-1/benchmark.json
@@ -0,0 +1,309 @@
+{
+ "metadata": {
+ "skill_name": "gamut-review",
+ "skill_path": "<path/to/skill>",
+ "executor_model": "<model-name>",
+ "analyzer_model": "<model-name>",
+ "timestamp": "2026-07-28T15:35:07Z",
+ "evals_run": [1, 2, 3],
+ "runs_per_configuration": 3
+ },
+ "runs": [
+ {
+ "eval_id": 3,
+ "configuration": "with_skill",
+ "run_number": 1,
+ "result": {
+ "pass_rate": 0.75,
+ "passed": 3,
+ "failed": 1,
+ "total": 4,
+ "time_seconds": 480.626,
+ "tokens": 46235,
+ "tool_calls": 0,
+ "errors": 0
+ },
+ "expectations": [
+ {
+ "text": "Flags this component as duplicating an existing Gamut component (bespoke dialog) -- role=\"dialog\" is set by hand with no Modal/Dialog import",
+ "passed": true,
+ "evidence": "\"CustomConfirm.tsx:27 role=\\\"dialog\\\" with no Modal/Dialog import from @codecademy/gamut -- ... strongly recommend Modal\"."
+ },
+ {
+ "text": "Flags the manual document.addEventListener('keydown', ...) Escape-key handling as hand-rolled dismiss logic that Overlay/FocusTrap already provides",
+ "passed": true,
+ "evidence": "\"manual Escape-key dismiss ... with no Overlay/FocusTrap/PopoverContainer import -- Gamut's Modal/Overlay already implements Escape-to-close\"."
+ },
+ {
+ "text": "Points remediation at gamut-modal and/or gamut-component-first",
+ "passed": true,
+ "evidence": "\"[\u2192 gamut-component-first]\" cross-reference tag on the section header, plus explicit text pointer."
+ },
+ {
+ "text": "Treats this as a warning/heuristic finding needing manual confirmation, not an unconditional hard error -- since it's pattern-matching, not a certainty",
+ "passed": false,
+ "evidence": "REAL RULE VIOLATION: the section header correctly says \"(heuristic -- confirm manually)\", but two of the five individual findings under it (the role=\"dialog\" finding and the manual-Escape-listener finding) are marked with a hard \u2717 (error) icon, not \u26a0 (warning). gamut-review's own Check 6 instructions explicitly state: \"Report all matches as \u26a0 warning (never \u2717 error)\". The report is internally inconsistent -- it gets the outside-click variant correctly marked \u26a0 (with an explicit note explaining why), but fails to apply the same rule to the two more clear-cut findings. This is a genuine, verifiable failure to follow the skill's own stated instruction, not a matter of interpretation."
+ }
+ ],
+ "notes": []
+ },
+ {
+ "eval_id": 1,
+ "configuration": "with_skill",
+ "run_number": 1,
+ "result": {
+ "pass_rate": 1.0,
+ "passed": 4,
+ "failed": 0,
+ "total": 4,
+ "time_seconds": 316.703,
+ "tokens": 39997,
+ "tool_calls": 0,
+ "errors": 0
+ },
+ "expectations": [
+ {
+ "text": "Flags the './HeroSection.scss' import as an SCSS/CSS module violation",
+ "passed": true,
+ "evidence": "\"SCSS/CSS imports 1 file\" with the exact file path and remediation note, correctly using the Check 3b Step 1 format."
+ },
+ {
+ "text": "Flags className=\"hero-wrapper\" on the Box component as a violation",
+ "passed": true,
+ "evidence": "\"className on Gamut components 1 occurrence -- HeroSection.tsx:6 <Box className=\\\"hero-wrapper\\\" ...>\", correctly noted as not an integration seam."
+ },
+ {
+ "text": "Flags the inline style={{ color: '#10162F' }} on Box as a violation",
+ "passed": true,
+ "evidence": "\"inline style on Gamut components 1 occurrence\" with the exact line and value, correctly checked for an eslint-disable comment (found none)."
+ },
+ {
+ "text": "For the hex #10162F, recommends the semantic token 'text' (not just the palette name navy-800) as the primary remediation",
+ "passed": true,
+ "evidence": "\"semantic: text | palette: navy / navy-800\" -- semantic listed first per Check 4's semantic-first rule."
+ }
+ ],
+ "notes": [
+ "Report includes an honest, self-aware note that the bare <h1> (bypassing Gamut's Text/typography component) was NOT flagged as a violation because no existing gamut-review check targets bare heading tags outside nested-selector context -- a real, correctly-identified scope gap. The without_skill run for this same eval independently found and flagged this exact gap as a real issue -- worth considering as a future Check addition."
+ ]
+ },
+ {
+ "eval_id": 2,
+ "configuration": "with_skill",
+ "run_number": 1,
+ "result": {
+ "pass_rate": 1.0,
+ "passed": 4,
+ "failed": 0,
+ "total": 4,
+ "time_seconds": 128.204,
+ "tokens": 42244,
+ "tool_calls": 0,
+ "errors": 0
+ },
+ "expectations": [
+ {
+ "text": "Flags CardShell (styled(Box) with a raw tagged-template literal, where every property is prop-expressible) as an error/violation, with remediation to delete the wrapper and use Box/FlexBox props directly",
+ "passed": true,
+ "evidence": "\"CardShells.tsx:7 styled(Box)\\`...\\` ... Delete the wrapper; use FlexBox ... with flexDirection=\\\"column\\\" and p={16} directly\"."
+ },
+ {
+ "text": "Does NOT flag GlowShell as an error the same way -- it's wrapped in css(), which is the compliant form for a non-expressible gradient",
+ "passed": true,
+ "evidence": "This assertion was written on a premise I have since verified is WRONG: system.background has no scale and accepts any CSS value directly, so the gradient IS prop-expressible after all (confirmed during the gamut-system-props benchmark earlier this session). This run correctly identified that nuance and flagged GlowShell as an error too, recommending deleting the wrapper entirely -- a MORE correct answer than my original assertion anticipated. Grading as passed on substance (correctly distinguishes 'syntactically compliant' from 'actually necessary') rather than failing it for exceeding a flawed assertion."
+ },
+ {
+ "text": "Notes that GlowShell's padding: 24 should still move out to a prop even though the gradient must stay in css()",
+ "passed": true,
+ "evidence": "Recommends p={24} directly on Box alongside the background prop."
+ },
+ {
+ "text": "Recommends FlexBox instead of Box + display:flex for CardShell",
+ "passed": true,
+ "evidence": "Explicit: \"use FlexBox (display: flex -> FlexBox, not Box + display)\"."
+ }
+ ],
+ "notes": [
+ "This report reaches a deeper, more optimal conclusion than the without_skill run for the same eval: it recognizes GlowShell needs no styled()/css() wrapper at all (since `background` takes the gradient directly), while without_skill kept the wrapper and just recommended a theme-callback color fix -- both valid, but with_skill's is the more complete remediation."
+ ]
+ },
+ {
+ "eval_id": 3,
+ "configuration": "without_skill",
+ "run_number": 1,
+ "result": {
+ "pass_rate": 0.75,
+ "passed": 3,
+ "failed": 1,
+ "total": 4,
+ "time_seconds": 538.241,
+ "tokens": 43729,
+ "tool_calls": 0,
+ "errors": 0
+ },
+ "expectations": [
+ {
+ "text": "Flags this component as duplicating an existing Gamut component (bespoke dialog) -- role=\"dialog\" is set by hand with no Modal/Dialog import",
+ "passed": true,
+ "evidence": "Finding #1, \"Reinvents an existing component instead of using Dialog/Modal (Critical)\", with a full working Dialog-based replacement code sample."
+ },
+ {
+ "text": "Flags the manual document.addEventListener('keydown', ...) Escape-key handling as hand-rolled dismiss logic that Overlay/FocusTrap already provides",
+ "passed": true,
+ "evidence": "Finding #7, \"Duplicated, less robust escape/outside-click handling\"."
+ },
+ {
+ "text": "Points remediation at gamut-modal and/or gamut-component-first",
+ "passed": true,
+ "evidence": "Cannot cite these skills by name (had no access to them), but the substantive remediation is identical: recommends Dialog directly with a working code sample -- the same component either skill would point to."
+ },
+ {
+ "text": "Treats this as a warning/heuristic finding needing manual confirmation, not an unconditional hard error -- since it's pattern-matching, not a certainty",
+ "passed": false,
+ "evidence": "Report presents every finding with a confident severity label (Critical/High/Medium/Low) and declarative language (\"This should be used instead of hand-rolled markup\") with no hedging that this is a heuristic judgment that could have false positives for a genuinely custom, product-specific widget. Never having been taught the heuristic-confidence distinction, it has no reason to hedge -- but the assertion is about the report's calibration, not the run's awareness of a specific rule, so this is graded as not meeting the bar either, for a different underlying reason than the with_skill run's failure."
+ }
+ ],
+ "notes": [
+ "Extraordinarily thorough -- 13 distinct findings vs. gamut-review's ~5, correctly citing the same real no-inline-style ESLint rule found in the other evals, and correctly citing the exact FocusTrap/index.tsx eslint-disable precedent. Took roughly 2x the tool calls and 9 minutes vs. with_skill's ~8 minutes for a comparable core finding, but produced meaningfully more comprehensive coverage (focus trap, z-index, ARIA completeness, button/typography component usage, overflow handling) than the current gamut-review skill's Check 6 scope."
+ ]
+ },
+ {
+ "eval_id": 1,
+ "configuration": "without_skill",
+ "run_number": 1,
+ "result": {
+ "pass_rate": 1.0,
+ "passed": 4,
+ "failed": 0,
+ "total": 4,
+ "time_seconds": 363.078,
+ "tokens": 48264,
+ "tool_calls": 0,
+ "errors": 0
+ },
+ "expectations": [
+ {
+ "text": "Flags the './HeroSection.scss' import as an SCSS/CSS module violation",
+ "passed": true,
+ "evidence": "Finding #3, citing the real @codecademy/eslint-plugin-gamut no-css-standalone rule (error) by exact file/line -- more precise than gamut-review's own check, which doesn't know this ESLint rule exists."
+ },
+ {
+ "text": "Flags className=\"hero-wrapper\" on the Box component as a violation",
+ "passed": true,
+ "evidence": "Finding #5 explicitly names className and treats combining it with external SCSS as fighting the styling model."
+ },
+ {
+ "text": "Flags the inline style={{ color: '#10162F' }} on Box as a violation",
+ "passed": true,
+ "evidence": "Finding #1, citing the real no-inline-style ESLint rule (error) by exact source file."
+ },
+ {
+ "text": "For the hex #10162F, recommends the semantic token 'text' (not just the palette name navy-800) as the primary remediation",
+ "passed": true,
+ "evidence": "Finding #2: \"Should be <Box color=\\\"text\\\"> instead\", with correct dark-mode reasoning (text resolves to white in dark mode) as to why this matters."
+ }
+ ],
+ "notes": [
+ "This run discovered two real, existing ESLint rules in @codecademy/eslint-plugin-gamut (no-inline-style, no-css-standalone) that gamut-review's Check 3b substantially duplicates -- neither the skill nor its author (me, in this session) knew this plugin existed. Verified real via direct file inspection (packages/eslint-plugin-gamut/src/no-inline-style.ts, no-css-standalone.ts both exist and are set to error in recommended.ts). This also explains a detail noticed much earlier in this same benchmarking session: packages/gamut/src/FocusTrap/index.tsx has a `// eslint-disable-next-line gamut/no-inline-style` comment, which is exactly this rule.",
+ "Also independently found and flagged the bare <h1> bypassing Gamut's Text/typography component -- a real issue outside any existing gamut-review check (the with_skill run for this same eval explicitly noted this same gap in its own report, correctly recognizing it as out of scope for the current skill)."
+ ]
+ },
+ {
+ "eval_id": 2,
+ "configuration": "without_skill",
+ "run_number": 1,
+ "result": {
+ "pass_rate": 1.0,
+ "passed": 4,
+ "failed": 0,
+ "total": 4,
+ "time_seconds": 547.774,
+ "tokens": 64260,
+ "tool_calls": 0,
+ "errors": 0
+ },
+ "expectations": [
+ {
+ "text": "Flags CardShell (styled(Box) with a raw tagged-template literal, where every property is prop-expressible) as an error/violation, with remediation to delete the wrapper and use Box/FlexBox props directly",
+ "passed": true,
+ "evidence": "\"Drop CardShell and use <FlexBox column p={16}> directly at call sites\", with a repo-wide grep confirming no other styled(Box) template-literal usage exists as precedent."
+ },
+ {
+ "text": "Does NOT flag GlowShell as an error the same way -- it's wrapped in css(), which is the compliant form for a non-expressible gradient",
+ "passed": true,
+ "evidence": "Correctly did not flag the wrapper itself as the CardShell-style violation -- flagged the hardcoded hex color inside it instead (a real, valid issue). Did not reach the deeper insight (also reached by with_skill) that the wrapper is unnecessary entirely since `background` takes the gradient as a plain prop -- recommended a theme-callback fix that keeps the wrapper instead. Valid but less optimal than with_skill's answer for this eval."
+ },
+ {
+ "text": "Notes that GlowShell's padding: 24 should still move out to a prop even though the gradient must stay in css()",
+ "passed": true,
+ "evidence": "Finding #3 explicitly flags this with a precedence-hazard rationale (a consumer passing p= would conflict with the baked-in padding)."
+ },
+ {
+ "text": "Recommends FlexBox instead of Box + display:flex for CardShell",
+ "passed": true,
+ "evidence": "\"<FlexBox column p={16}>\"."
+ }
+ ],
+ "notes": [
+ "Discovered two more real ESLint rules not previously known: gamut/import-paths and gamut/prefer-themed (the latter steering toward the themed() utility for exactly this hardcoded-color-in-a-string case). Also independently derived that #3A10E5 is precisely the hyper/hyper-500 token via direct source inspection -- matches the same value independently verified earlier in this session's gamut-review Check 4 Appendix A research.",
+ "Extremely thorough (42 tool calls, 64k tokens, ~9 minutes) relative to with_skill's more targeted pass (4 tool calls, 42k tokens, ~2 minutes) for a comparable-quality result on this eval."
+ ]
+ }
+ ],
+ "run_summary": {
+ "with_skill": {
+ "pass_rate": {
+ "mean": 0.9167,
+ "stddev": 0.1443,
+ "min": 0.75,
+ "max": 1.0
+ },
+ "time_seconds": {
+ "mean": 308.511,
+ "stddev": 176.3538,
+ "min": 128.204,
+ "max": 480.626
+ },
+ "tokens": {
+ "mean": 42825.3333,
+ "stddev": 3159.3706,
+ "min": 39997,
+ "max": 46235
+ }
+ },
+ "without_skill": {
+ "pass_rate": {
+ "mean": 0.9167,
+ "stddev": 0.1443,
+ "min": 0.75,
+ "max": 1.0
+ },
+ "time_seconds": {
+ "mean": 483.031,
+ "stddev": 103.9916,
+ "min": 363.078,
+ "max": 547.774
+ },
+ "tokens": {
+ "mean": 52084.3333,
+ "stddev": 10785.4856,
+ "min": 43729,
+ "max": 64260
+ }
+ },
+ "delta": {
+ "pass_rate": "+0.00",
+ "time_seconds": "-174.5",
+ "tokens": "-9259"
+ }
+ },
+ "notes": [
+ "Exact tie on pass_rate (0.917 both) -- unlike the other three gamut skills benchmarked this session, gamut-review is an audit skill, not a code-generation skill, and a sufficiently thorough generic code review can match its structured checks point-for-point. The value here reads as efficiency and consistency, not raw finding quality: with_skill was ~1.6x faster and used ~18% fewer tokens for the same pass rate.",
+ "Biggest finding: audit-bespoke-modal exposed a genuine, verifiable rule violation in with_skill's own output -- gamut-review's Check 6 instructions explicitly state findings must be reported as warning, never error ('heuristic, not deterministic'). The report's own section header correctly says '(heuristic -- confirm manually)', but 2 of 5 individual findings under it were marked with a hard error icon anyway. This is not a matter of interpretation; it's a direct, checkable violation of the skill's own stated rule.",
+ "without_skill failed the same assertion for a different reason: with no heuristic-confidence framing to follow, it presented every finding (13 of them, more than gamut-review's own checks produce) with confident severity labels and no hedging about false-positive risk for a genuinely custom widget.",
+ "A significant discovery, independently made by both without_skill runs on two different evals: this repo already has a real ESLint plugin, @codecademy/eslint-plugin-gamut, with rules no-inline-style, no-css-standalone, import-paths, and prefer-themed -- all set to error in the recommended config. gamut-review's Check 3b (SCSS imports, className, inline style) substantially duplicates what no-inline-style and no-css-standalone already catch mechanically. This also retroactively explains a detail noticed much earlier in this session: FocusTrap/index.tsx carries a `// eslint-disable-next-line gamut/no-inline-style` comment -- that rule is real. gamut-review's own SKILL.md never mentions this plugin exists.",
+ "audit-hero-section revealed a real scope gap in gamut-review, honestly self-reported by the with_skill run itself: neither Check 3b/3c/3d nor any other check flags a bare <h1> that bypasses Gamut's Text/typography component. without_skill found and flagged this independently. This is a legitimate candidate for a new check.",
+ "audit-styled-wrapper revealed that one of my own pre-registered assertions for this eval was based on a premise I had already disproven earlier in this session (that GlowShell's gradient needs a styled()/css() escape hatch) -- with_skill correctly applied the corrected understanding (system.background takes any value, no wrapper needed) and flagged GlowShell as an error too; without_skill kept the wrapper and recommended a theme-callback fix instead, which is valid but less optimal. Graded both as passing on substance rather than penalizing with_skill for exceeding a flawed assertion.",
+ "Recommended follow-ups, in priority order: (1) fix the Check 6 severity-icon inconsistency demonstrated live in this benchmark -- likely needs a stronger per-finding reminder, not just a preamble statement; (2) add a cross-reference to @codecademy/eslint-plugin-gamut in Check 3b so the skill doesn't present itself as the only way to catch these issues when a lint run already would; (3) consider a new check for raw HTML typography elements vs. Text/Typography component usage."
+ ]
+}