Webwright Keyboard Testing Benchmark

Architecture

Three Browser Automation Lanes

npx playwright test

Codified CI keyboard tests, visual regression, axe-core scans. Version-controlled .spec.js files that run in CI.

Primary path · Node.js

agent-browser

Interactive recon via snapshot+ref pattern. One shell call per action, persistent Rust daemon, real CDP keyboard events.

Exploratory · CLI

/webwright:run

LLM generates complete Python Playwright scripts from prose descriptions. Reusable .py artifacts with assertions and screenshots.

Script generation · Python

Do you have a prose description of what to test, but no test script yet?

↓

Yes /webwright:run
(one-shot) or /webwright:craft (reusable)

No, existing test npx playwright test

No, exploring agent-browser

Benchmark

WAI-ARIA APG Correctness

25 /25 — all tasks passed, all checkpoints cleared

Canary: Disclosure Toggle

aria-expanded false→true via locator.press('Enter')

PASS

3a: Dialog Focus Trap

Tab wraps inside dialog, Escape closes, focus returns to trigger. 6/6 checkpoints.

ALL PASS

3b: Tabs Automatic Activation

Click 4 tabs, ArrowRight keyboard nav. aria-selected and panel visibility verified.

ALL PASS

3c: axe-core Injection

CDN injection, axe.run(document), 3 violations with impact levels reported.

PASS

3d: Menu Button (Active Descendant)

Enter, ArrowDown, End, Escape. aria-activedescendant tracked. 13/13 checks.

ALL PASS

3e: ARIA Tree Inspection

aria_snapshot() captured full tree: 1 tablist, 4 tabs, 4 panels, all relationships mapped.

PASS

Performance

Speed Comparison

Script Execution (re-running existing scripts)

Task	Webwright	agent-browser	Ratio
Dialog focus trap	4.4s	1.4s	3.1×
Tabs ARIA state	4.2s	2.6s	1.6×
Menu keyboard nav	3.6s	1.5s	2.4×
Average	4.1s	1.8s	2.3×

Visual: Average Script Execution

Webwright

4.1s

agent-browser

1.8s

Why agent-browser is faster

Persistent Rust daemon with CDP — no Python interpreter startup, no browser launch per run. Reuses the running Chrome instance.

Why Webwright wins on re-runs

Generated .py scripts are reusable — run them 100 times at ~4s each with zero LLM cost. agent-browser must re-compose commands every time.

Total Webwright Workflow (LLM generation + execution)

Task	Total	LLM Gen	Script Exec
Canary (disclosure toggle)	~59s	~55s	~4s
Dialog focus trap	~132s	~128s	~4s
Tabs ARIA state	~68s	~64s	~4s
axe-core injection	~33s	~29s	~4s
Menu keyboard nav	~66s	~62s	~4s
ARIA tree inspection	~36s	~35s	~1s
Average	~66s	~62s	~3.5s

Workflow

Updated Accessibility Lifecycle

plan → generate test scripts → critique plan → perspective audit → revise → implement → test → critique → fix → re-test

generate test scripts — new optional step using Webwright to produce Python Playwright scripts from the planner's output

Quality Gate

Keyboard Event Verification

✓

All scripts use real CDP keyboard events

Every generated script uses page.keyboard.press() or locator.press() — the Playwright Python sync API that dispatches via CDP Input.dispatchKeyEvent. Zero instances of synthetic dispatchEvent(new KeyboardEvent(...)) across all 5 benchmark tasks and 3 saved scripts (dialog, tabs, menu).

KEYBOARD APIS VERIFIED IN GENERATED CODE

✓ page.keyboard.press("Tab")

✓ page.keyboard.press("Enter")

✓ page.keyboard.press("Escape")

✓ page.keyboard.press("ArrowDown")

✓ page.keyboard.press("ArrowRight")

✓ page.keyboard.press("End")

✓ locator.press("Enter")

✓ get_attribute("aria-expanded")