Agent method
Vision mode
In Vision mode (“Tesla” mode), an Act run sees the tab only as screenshots, with no accessibility tree, element references or selectors, and acts by clicking, typing, scrolling and dragging at screen coordinates.
Why it matters
Canvases, maps, charts and custom widgets have little usable structure, and a cookie banner can cover the button the page structure says is there. Seeing what the user sees handles both.
How it stays cheap and precise
A run starts with one full screenshot. After that it sends only the changed region, a new full frame when most of the screen changed, or “no visible change”, and the model can zoom in. Coordinates are mapped back across retina displays, browser zoom and models that answer in 0–1000 space.
WebBrain falls back to the accessibility tree around file uploads, for workflows whose completion must be proved, when reading a whole conversation, on WhatsApp Web and Telegram, and when the model can’t take images.
Caveats
- Vision alone is not always enough: small local models often read text better than pixels, and long forms need precise targeting.