Automated vs Manual Accessibility Testing: What Each One Actually Finds

Every accessibility vendor makes a claim about how much their tool catches, and the numbers in circulation range from 20% to nearly 60%. They cannot all be right — except that, oddly, most of them are. They are measuring different things.

Getting this straight matters more than it sounds. If you believe automation catches 57% of your problems, you will under-invest in the human testing that finds the rest. If you believe it catches 20%, you may skip automation altogether and lose the cheapest, fastest quality gate you have. Neither posture is correct.

The two numbers, and what each one measures

Measured by WCAG success criteria: roughly a third. Deque, who build the axe-core engine that underpins a large share of the industry's tooling, published research finding that automated tests produced results for 16 of the 50 WCAG 2.1 Level AA success criteria — 32%. Independent academic work reached similar territory: a 2013 benchmarking study of six evaluation tools by Vigo, Brown and Conway found coverage of at most 50% of success criteria, with completeness between 14% and 38%, and noted that the tools which detected more also produced more false positives.

Measured by individual defects found on real pages: much higher. The same Deque study found that 57.38% of all issues in their audit dataset — nearly 300,000 issues across more than 13,000 pages, from over 2,000 first-time audits — were detected by automated tests.

Both numbers are honest. The reason they diverge so far is that accessibility failures are not evenly distributed across the standard. A handful of failure types recur enormously often, and those happen to be the mechanical, machine-detectable ones: contrast, missing alt text, missing labels, empty links and buttons. Deque found the top five success criteria accounted for over 78% of all issues discovered. So counting defects weights the total heavily towards what automation is good at, while counting criteria treats "colour contrast" and "meaningful sequence" as equally weighted.

The most useful framing is this: automation finds most of the errors and about a third of the kinds of errors. The remaining criteria are not the rare ones. They include whether your page structure makes sense, whether your custom components behave correctly, whether a process can actually be completed, and whether your alt text says anything useful.

The independent test worth knowing about

The single most illuminating experiment on this question was run by the UK's Government Digital Service in 2017. They built a web page deliberately seeded with 143 known accessibility barriers across 19 categories, then ran ten automated tools at it to see how many each would find.

The best-performing single tool found 41% of the barriers. Google's developer tooling found 17%. Collectively, all ten tools together found 71% — which means 29% of the barriers, 42 of them, were missed by every automated tool tested.

That result has aged well, because the categories automation misses are structural rather than technological. No amount of engine improvement lets a machine decide whether a heading accurately describes the section beneath it.

What automation is genuinely excellent at

It is worth being positive about this, because the case against over-claiming is often mistaken for a case against automation.

Automated testing is fast — a scan covers hundreds of pages in the time a human reviews one. It is consistent, applying identical rules on Monday and Friday, which humans do not. It is repeatable, so it can run on every deploy and catch regressions the day they appear rather than at the next annual audit. And it produces precise, actionable output: the exact element, the exact rule, the exact measured value.

Concretely, automated tools reliably detect: images with no alt attribute, form inputs with no accessible name, colour contrast below threshold where the colours are declared in CSS, empty links and buttons, missing document language, duplicate IDs, invalid ARIA attributes and roles, incorrect heading nesting, missing table headers, positive tabindex values, and video elements with no caption track.

That is a substantial list, and it maps directly onto the failures that dominate real sites. WebAIM's annual million-page analysis finds the same six issues account for the overwhelming majority of detected errors year after year.

What only a human can determine

The other side of the ledger is not a small residue of edge cases. It includes some of the most consequential requirements in the standard.

Whether text alternatives are correct. A scanner sees alt="image" and alt="Quarterly revenue rose 42% to 2.1 million" as equally passing. Only a person knows whether the description serves the image's purpose.

Whether the reading order makes sense. 1.3.2 Meaningful Sequence asks whether the order content is presented in preserves meaning. That requires reading it.

Whether keyboard navigation actually works. A tool can flag a div with a click handler and no key handler. It cannot tell you that your modal returns focus to the wrong place, that your custom dropdown does not respond to arrow keys, or that a sticky header hides the focused element.

Whether a process can be completed. WCAG conformance is claimed for complete processes. Only a human can attempt a checkout with a screen reader and find out where it breaks.

Whether error messages help. Present, announced and comprehensible are three different tests, and machines can only check the first two.

Whether video captions are accurate. A tool sees a caption track and passes. It cannot hear that the auto-generated captions mangled every proper noun.

Whether the new WCAG 2.2 criteria are met. Most of the nine additions are judgement calls: is dragging essential to this function, is this help mechanism in a consistent relative position, does this authentication step require a cognitive function test.

What good practice looks like

The mature approach is not to choose. It is to assign each method the work it is suited to.

Automate continuously, on every page you can. The value of automation is coverage and frequency, not depth. Scanning your whole site weekly catches the regression introduced by last Tuesday's template change, on the page nobody thought to test. This is where accessibility programmes actually decay — not because the initial audit was wrong, but because sites change constantly and nobody re-checks. The UK Government's public sector monitoring found that on retest of 47 previously-monitored websites, only 3 were found to have no issues, despite those organisations having already been told what was wrong.

Manually test the journeys that matter. You cannot hand-test a thousand pages. You can hand-test the ten paths that carry your revenue and your legal exposure: home to product to cart to payment to confirmation; the registration flow; the contact form; the login; the search. Do it with a keyboard, then with a screen reader.

Involve people with disabilities. No amount of specification reading substitutes for watching someone who uses a screen reader daily attempt your checkout. This is the single highest-value accessibility activity most organisations have never done.

Never describe an automated result as compliance. This is the honesty line, and it matters legally as well as ethically. WebAIM state it plainly in every edition of their million-page report: "All automated tools, including WAVE, have limitations — not all conformance failures can be automatically detected. Absence of detected errors does not indicate that a page is accessible or conformant."

A clean automated scan is a real achievement and a useful signal. It means the machine-detectable failures are gone. It is a floor, not a certificate — and any vendor who tells you their scanner makes you compliant is describing something the technology cannot do. The US Federal Trade Commission has already taken action on exactly that kind of claim: in 2025 it ordered a widget vendor to pay $1 million over allegations that its representations about making websites conformant were false, misleading or unsubstantiated. We cover that case in do accessibility overlays work.

*Coverage figures are from Deque Systems' Automated Accessibility Coverage Report, the UK Government Digital Service's 2017 evaluation of ten automated tools against a page with 143 seeded barriers, and Vigo, Brown and Conway (W4A 2013). Retest figures are from the UK Government Digital Service accessibility monitoring report covering 2022 to 2024. axe® and axe-core® are trademarks of Deque Systems, Inc.; ChecksTrue is not affiliated with or endorsed by Deque, WebAIM or the W3C.*