How accurate is Lumtera?
An accessibility checker can go wrong in two ways: it can miss real problems, or it can raise false alarms on things that are fine. We measure both on test pages whose right answers are known. All the numbers on this page were measured on the Lumtera 1.3.0 code in November 2026.
These are our own tests, not an independent audit. None of them is a score for WCAG as a whole, and none says how much of your site's accessibility Lumtera can judge.
The short version
| What we measured | Result |
|---|---|
| W3C ACT failed examples reported, at any severity | 296 of 393. Read this as a generous number: see below. |
| W3C ACT failed examples reported as an error | 121 of 393 |
| W3C ACT passed or inapplicable examples reported as an error by the check that covers them | 1 of 816 |
| Pages that meet WCAG or nearly so: false errors in review mode and the browser engine | 0 on 532 pages, after reading each error left |
| The same pages: false errors from the content check on raw HTML | 8, explained below |
Known failures found
The W3C ACT rules publish small test pages, each marked as passing or failing one rule. We ran Lumtera's content checks and its browser engine on all 1,209 of them.
| On the W3C ACT test cases | Result |
|---|---|
| Failed examples reported, at any severity | 296 of 393 (296 of 352 on rules that have a matching check) |
| Failed examples reported as an error | 121 of 393 |
| Passed or inapplicable examples reported as an error by the matching check | 1 of 816 |
| Passed or inapplicable examples reported as Needs review or a tip by the matching check | 107 of 816 |
Why 296 is generous. It counts a failed example as found when a matching check says anything at all about that page. For 175 of the 296, Lumtera didn't say "this fails": it asked a person to review the page, or gave a tip. A video check that asks someone to confirm captions gets credit on every ACT video rule, whether the captions are wrong or not, and a page with good captions gets the same question. 121 is the number found as definite failures.
Neither figure is a share of all accessibility problems. ACT cases are small pages, each built to test one rule. 41 of the failed examples are on rules Lumtera has no check for.
This method replaces two numbers published earlier: "98% on 21 rules" (130 of 132) looked at a hand-picked set of rules, and "177 of 259" used a mapping that was never committed, so it can't be reproduced or compared. Please don't quote either.
No comparison published
Our per-rule notes record how axe-core did on the same ACT pages in the 2026 rounds, but that comparison hasn't been re-run on the 1.3.0 code, so we don't publish one here. axe-core and Lumtera are built for different jobs, and neither replaces testing by a person.
False alarms on pages that meet WCAG
An error on a page that meets WCAG is a false alarm. Needs review isn't counted as one, because it asks a person to decide.
We ran three routes on 532 pages that meet WCAG 2.2 AA or nearly so. On the final 1.3.0 code we re-ran 524 of them, with one more Twenty Twenty-Five pattern than the table lists, and the error counts didn't change; the 9 WooCommerce pages come from the run on 4 November 2026. An error left on these pages isn't automatically a false alarm, because published examples and themes have real problems too, so each one was read.
| Pages | Content check, raw HTML | Browser engine, with keyboard walk and widgets | Review mode's route |
|---|---|---|---|
| Hand-written conforming pages (9) | 0 errors | 0 errors | 0 errors |
| WAI-ARIA Authoring Practices examples (76) | 32 errors | 0 errors | 26 errors |
| GOV.UK Design System component examples (248) | 61 errors | 2 errors | 61 errors |
| Twenty Twenty-Five: 108 patterns and 7 views (115) | 0 errors | 0 errors | 0 errors |
| Twenty Twenty-Four: 68 patterns and 7 views (75) | 2 errors | 1 error | 2 errors |
| WooCommerce with Twenty Twenty-Five (9) | not run | 0 errors | 1 error |
Result: review mode's route and the browser engine have no false alarms on these pages. The content check on raw HTML has 8.
The 8 are in two W3C pattern examples whose script names a button and hides a field after the page loads. The content check reads the saved markup, so it can't see that. Review mode checks the page as the browser shows it, and reports nothing there.
The other errors are real problems in the pages as published:
- Authoring Practices examples: support-table frames without a
title, and links to definitions that aren't on the page. - GOV.UK examples: published component examples that link to targets the example doesn't contain, such as
href="#1"and the skip link's#test-target-element. This is about the examples, not GOV.UK's live pages. - Twenty Twenty-Four: a Cover image output as an unnamed image, and a core pattern with
#111link text on a black band (1.11:1). - WooCommerce 11.2.1: the closed Mini-Cart drawer is hidden from screen readers, but its close button can still take keyboard focus.
The ARIA errors added in 1.3.0
Two new checks, Custom control cannot be reached with the keyboard and Custom control is missing a required state, are Needs review on raw content, because a script may add the attribute later, and errors on the running page. On the 532 pages they reported nothing at all.
On the ACT passed and inapplicable examples, the new checks raised 21 errors in review mode's route (18, 2 and 1 across three checks). None is on a check matched to that ACT rule. Each is a real problem the example isn't testing, such as <div role="button"> that a keyboard can't reach.
How we ran these tests
Our test scripts and the mapping of ACT rules to checks aren't public, and they aren't in the plugin's zip, so nobody outside MantraBrain can repeat this exact run. The W3C ACT test cases are public, so you can try any of them against Lumtera yourself.
- Pages. All 1,209 ACT test cases; 76 Authoring Practices examples; 248 GOV.UK examples; WordPress with Twenty Twenty-Five, Twenty Twenty-Four and WooCommerce; and 9 hand-written pages covering names, images, forms, contrast, hidden content, right-to-left and East Asian languages, links, media, widgets and tables. Everything was served from 127.0.0.1, with requests to anywhere else blocked.
- Three routes. The content checks (the same code as
wp lumtera checkand the editor's check), the browser engine in headless Chrome, and review mode's own route, which sends a snapshot of the running page to the site. - Scoring. A failed ACT example counts as found when a check matched to that rule reports it: "any" means at any severity, "error" means as an error. A passed or inapplicable example counts as a false alarm when a matched check reports it as an error.
Limits of these tests
- WooCommerce pages didn't go through the raw-content check in this run.
- A real screen reader wasn't part of it. Findings about names and roles come from the page's code.
- Passing these tests says nothing about pages unlike the ones tested, such as a heavily customized page builder layout.
Why "Needs review" isn't a false alarm
When Lumtera is sure, it reports an Error. When a person has to decide, it reports Needs review. There's one exception: an Image block with empty alt text and no sign of being decorative is an error, because that's often a field left empty. Mark the image decorative and it clears. Needs review findings ask a person to look, which is what the confidence rules are for. For example, a field named only by its placeholder text is Needs review, because it may be fine for some forms.
Needs review does cost you time. On the ACT passed and inapplicable examples, 107 of 816 get a review item or a tip from the matching check, and the same question appears whether or not the page turns out to have the problem.
Known limits of the checks
These are left as they are on purpose, or not solved yet:
- Text inside web components (shadow DOM) isn't checked for contrast yet. Web components are rare on WordPress sites.
- Changes a script makes after the page loads, such as a name it adds or a field it hides, can't be seen by the editor's check of saved content. Review mode and the rendered page check see the page as the browser shows it.
- How long an autoplaying clip plays isn't in the page, so a short clip near the end of a long file can still be reported.
- Focus guards, hidden slides, symbols and silent autoplaying videos are reported as Needs review rather than not at all, because only running the page, or hearing it, shows whether they are a problem.
- Text may be in a different language from its tag is an educated guess from letters and common small words, so it's a tip, hidden until you ask for possible issues.
What no automated checker can find
Some WCAG failures need a person, or need the page to be used over time. Lumtera lists them as guided checklist items instead. For example:
- whether alt text, captions and transcripts are right, not just present
- whether reading order, focus order, headings and link text make sense
- meaning carried only by layout, color, shape or sound
- keyboard traps and focus handling that depend on scripts
- whether error messages say what went wrong and how to fix it
- timing, motion and flashing in context
- whether navigation stays consistent across pages (Lumtera Pro's consistency checks compare the pages you check)
With review mode's whole-page checks, Lumtera's automated checks cover 38 of 55 WCAG 2.2 A and AA criteria (1 fully, 37 in part). Covering a criterion means a check looks at it, not that your site meets it. See What automated testing can't do.
Found a false alarm?
Every finding has Why is this flagged? and Report a false positive, which opens a report in the WordPress.org support forum with the details filled in. Nothing is sent from your site until you post it.