Methodology v1.0 · public

How we score AI design add-ons

Every skill, DESIGN.md file or MCP server gets the same brief, the same tokens, the same agent and model, and the same automated scorer. Transcripts and generated code are published with every score.

1. The standard brief

The agent builds two pages for Ledgerly, a fictional invoicing SaaS. Together they exercise the things design systems are for: forms, states, density, hierarchy and repeated components.

Page 1 — Workspace settings

  • App header with primary nav and account menu; settings sub-navigation.
  • Form: workspace name, URL with prefix, currency select, invoice prefix with helper text, time-zone select.
  • Three notification toggles with descriptions.
  • Validation error state (“This URL is already taken.”).
  • Danger zone with a destructive action; sticky Cancel / Save bar.

Page 2 — Pricing

  • Headline, subhead, monthly/yearly switch with “Save 20%”.
  • Three plans (Solo €9, Studio €29 recommended, Agency €79) with CTAs.
  • Comparison table (≥ 8 rows) and a 4-question FAQ disclosure.

Constraints: plain HTML + CSS, tokens only, works at 1280 px and 390 px, WCAG 2.2 AA, real copy. Full text: BRIEF.md.

The token set

A deliberately small DTCG token set (tokens.json · tokens.css): 11 colours, 2 font families, 6 type sizes (12–32 px), 9 spacing steps (0–64 px), 5 radii and 2 shadows. Small enough that drifting off-system is easy to detect.

2. Rubric and weights

Design-token adherence30%
Accessibility30%
Consistency20%
AI-slop avoidance20%

Each criterion is scored 0–100 in a headless browser. The weighted total is multiplied by brief completeness (share of required elements present), so skipping half the brief can’t buy a clean score.

Design-token adherence (30%)

70%: share of rendered colours, font sizes, spacing values and radii that exactly match a token. 30%: share of CSS values written as var(--token) rather than hard-coded literals.

Accessibility (30%)

axe-core (WCAG 2.0/2.1/2.2 A + AA and best-practice rules) on both pages at both widths. Deductions per affected node (max 5 per rule): critical 10, serious 6, moderate 3, minor 1. Horizontal overflow at 390 px costs 10.

Consistency (20%)

Counts distinct values across both pages against budgets — font sizes ≤ 6, spacing ≤ 9, radii ≤ 4, colours ≤ 12, shadows ≤ 2, font families ≤ 2, button variants ≤ 4. Each value over budget costs 10 points in that family.

AI-slop avoidance (20%)

Starts at 100; each detected pattern costs 15: large decorative gradients, gradient text, glassmorphism blur, emoji used as icons, nested cards, eyebrow pill above the headline, purple/violet accents not in the tokens, default generic fonts (Inter, Roboto, Arial, Poppins…), lorem ipsum. Our detectors are our own, so no tested add-on’s tooling grades itself.

3. Run protocol

  1. Fresh project folder containing only the brief and tokens. Add-on installed with its documented method, pinned to a commit SHA.
  2. Agent runs headless with one fixed prompt; no human follow-ups. Agent, model ID and version are recorded.
  3. Baseline: identical run with no add-on. We report the score and the lift over baseline.
  4. 3 repetitions per item; we publish the median and the range.
  5. Transcripts, generated files and raw scorer JSON are published with every score.
  6. MCP servers that need a source file (Figma, Penpot, Storybook) get a matching reference file built from the same tokens. Hosted services needing paid plans are only scored when access is available on equal terms.

Calibration

The scorer was checked on two hand-written fixtures: a token-compliant control page (96) and a deliberately sloppy one (15). These are calibration fixtures, not add-on scores, and never appear on the leaderboard.

4. Independence policy

Sponsors never affect scores. Scores come only from the automated scorer and the published protocol.

  • No one can pay for a test, a higher score, a re-run or the removal of a score.
  • Sponsored placements and featured listings are always labelled “Sponsored” or “Featured” and are shown apart from the leaderboard ranking.
  • If a sponsor or affiliate partner also makes a tested add-on (e.g. a design-system platform with an MCP server), we disclose it on that item’s page.
  • Affiliate links are marked and never decide which items we test.
  • Authors may request a re-run once per version; re-runs use the same protocol and both results stay public.
  • Methodology changes are versioned; scores always show the brief and scorer version they used.

5. Known limitations

  • Automated checks measure discipline, not taste. A blinded designer panel score is planned for v1.1.
  • Results depend on the agent and model; scores are labelled with both.
  • Star counts are a GitHub API snapshot and measure popularity only.