Automated accessibility testing: what it can and cannot prove

Coverage by WCAG principle

EN 301 549’s web chapter incorporates WCAG’s four organising principles — Perceivable, Operable, Understandable, Robust — grouped here by the first digit of each criterion’s WCAG number, an objective, publicly documented split, not an editorial one. Automation coverage is uneven across them: structural and interface-level requirements automate more readily than judgement calls about meaning.

What automation cannot prove

These 19 criteria have no automated check at all in our data — not a gap in our tooling specifically, but the honest state of automated accessibility testing generally: judging whether an alternative is truly equivalent, whether an error message is genuinely helpful, or whether a navigation pattern stays predictable is not something a script can decide. Every one of these needs a human reviewer, every time.

• 1.2.3 Audio description or media alternative (pre-recorded)
• 1.2.4 Captions (live)
• 1.2.5 Audio description (pre-recorded)
• 1.3.2 Meaningful sequence
• 1.3.3 Sensory characteristics
• 1.4.5 Images of text
• 1.4.11 Non-text contrast
• 1.4.13 Content on hover or focus
• 2.1.4 Character key shortcuts
• 2.3.1 Three flashes or below threshold
• 2.5.1 Pointer gestures
• 2.5.2 Pointer cancellation
• 2.5.4 Motion actuation
• 3.2.1 On focus
• 3.2.2 On input
• 3.3.1 Error identification
• 3.3.3 Error suggestion
• 3.3.4 Error prevention (legal, financial, data)
• 4.1.3 Status messages

Where our own checks go beyond bare axe-core

11 criteria get an automated check only because Verscala’s own worker code adds one — axe-core’s static-markup rules do not reach real-browser behaviour like keyboard traps, focus order, or resize/reflow at all. Each of these is a heuristic with a named, documented limitation, not a silent guess — the full caveat for each check is on its criterion page.

• 1.2.2 Captions (pre-recorded)
  a11y-video-no-captions
• 1.4.2 Audio control
  a11y-autoplay-media
• 1.4.4 Resize text
  a11y-resize-200
• 1.4.10 Reflow
  a11y-reflow-320
• 2.1.2 No keyboard trap
  a11y-keyboard-trap
• 2.4.3 Focus Order
  a11y-focus-order
• 2.4.5 Multiple ways
  a11y-multiple-ways
• 2.4.6 Headings and labels
  a11y-empty-heading
• 2.4.7 Focus visible
  a11y-focus-invisible
• 3.2.3 Consistent navigation
  a11y-inconsistent-navigation
• 3.2.4 Consistent identification
  a11y-inconsistent-identification

Why Verscala does not publish site scores

With only 62% of EN 301 549’s web criteria having any automated check — and none of those checks proving conformance on their own — a single numeric “accessibility score” for a site would compress a mostly-manual standard into a number automation cannot honestly produce. That is why Verscala’s own scanner reports findings against the rules it actually ran, not a score, and why our methodology and accessibility statement name the same 31/50 figure, not a friendlier one.

How this was made

Every number on this page is computed at build time from data/a11y/en301549-coverage.json — generated from EN 301 549 V3.2.1 ch.9 + axe-core 4.13.0 + worker/lib checks — by a single aggregator script, separate from the one behind our other report; a test recomputes the figures from that same source and fails if the published snapshot drifts. This is the identical dataset that drives our methodology page and the per-criterion pages under /wcag/, read once here as a single narrative instead of a lookup table.