AI QA mobile web testing
Closing the QA Loop With AI Agents Running End-to-End Tests on Mobile Web
Desktop-green is a trap on mobile web. I have merged PRs that passed unit tests and looked fine in Chrome DevTools mobile emulation, then failed in Safari's real bottom bar, Android WebView cookie timing, or a Maestro tap target buried under a sticky header.
Closing the QA loop meant treating device E2E as an automated handoff, not a Friday manual chore.
What "Closed QA Loop" Means
A closed loop has four properties:
- Tests trigger from the same event that humans care about (usually a PR preview URL going live).
- Failures attach machine-readable artifacts to the PR.
- An implementer agent can iterate without you retyping steps.
- Merge stays blocked until defined suites pass or a human waives with reason.
Maestro is the runner I use for iOS and Android flows against our H5 surfaces. The principles transfer to Playwright device farms or Appium if that is your stack.
QA Agent Responsibilities
On my team the QA agents do two jobs:
Run orchestration. Pick suites based on changed paths, wait for preview deploy, execute Maestro on the device matrix, upload logs and screenshots.
Test case maintenance. When a feature adds a new critical path, the agent drafts YAML, runs it once, and opens a small follow-up PR if stable.
They do not replace exploratory testing on weird devices. They eliminate "did anyone run checkout on a real iPhone this sprint?"
Mapping Code Changes to Suites
Path-based routing keeps CI time sane:
# Example routing table (conceptual)
apps/wallet/** -> [wallet_smoke, topup_happy_path]
apps/kyc/** -> [kyc_document_upload]
shared/ui/** -> [shell_navigation_smoke]
The QA agent reads the PR file list, selects suites, and refuses to run the full catalog on a typo fix. DevOps agents confirm preview health before Maestro starts. Racing the deploy caused flaky "connection refused" failures that trained everyone to ignore red builds.
What Gets Attached on Failure
Humans need story. Agents need structure. Both get:
- Failed step name and index
- Screenshot at failure
- Optional screen recording clip
- Preview URL and git SHA
- Last known good run for comparison
Handoff back to implementer:
## Maestro failure
Suite: topup_happy_path
Step 14: assertVisible "Top-up successful"
Observed: loader stuck; network 403 on /api/topup
Devices: iOS pass on retry 1, Android fail consistent
Artifact: ./maestro-output/abc123/
That is enough for an agent to grep the API client, not enough to debate philosophy.
Writing Tests With an Agent Without Flaky Garbage
Agents love brittle selectors. We enforce conventions:
- Prefer accessibility labels and stable test IDs over CSS nth-child.
- No
sleep 5000unless documented with ticket link. - One assertion intent per step where possible.
- Tag flows as
smokevsfull; only smoke blocks merge by default.
When the agent authors a new flow, a human spot-checks the first merge. After that, policy and review agents handle churn.
Mobile Web Specifics That Belong in QA Policy
Safe areas and viewport units. Sticky CTAs overlap tappable regions. Maestro catches it; unit tests do not.
Session and cookie timing. Third-party cookie restrictions and in-app browsers behave differently. Tests must run in WebView profiles we actually ship to.
Performance gates optional but valuable. Some teams attach lab Lighthouse on preview for LCP/TBT regressions. We pair that with perf review agents so QA does not duplicate bundle analysis.
Human-in-the-Loop on QA
I still approve:
- Changing merge-blocking suite lists
- Waiving red E2E on known upstream outages
- First-time flows touching money movement
Everything else loops agent-to-agent until green or retry cap.
FAQ
Why Maestro for mobile web?
Flows read like scripts, run on real devices, and produce artifacts agents can parse. The same ideas work with other runners if your artifact format is consistent.
How long should E2E take on a PR?
Aim for sub-fifteen-minute smoke. Full suites can run nightly.
Do agents fix flaky tests?
They can propose waits or selector fixes. Chronic flakes get quarantined and assigned to humans. Agents do not infinite-retry flakes.
What if preview URLs are auth-gated?
Use short-lived test accounts or signed preview tokens injected via CI secrets. Never paste production credentials into agent chat.
Closing
If mobile web is your product surface, QA loops belong in the PR path, not in a spreadsheet. Wire preview deploy → Maestro → structured failure → implementer retry, and measure time-to-green instead of time-to-manual-QA-scheduled.
That is how you stop shipping "works on my emulator."