Tiny raw LFM 2.5 checkpoints in WebBrain's frozen planner benchmark

We ran two tiny raw Liquid LFM 2.5 checkpoints through WebBrain's frozen browser-agent first-tool benchmark: lfm2.5-230m and lfm2.5-350m served locally from http://localhost:8000. Both runs completed all 100 cases without transport errors. The quality numbers are low, but that is not a surprise: these are tiny raw checkpoints, not WebBrain-tuned browser planners. The useful result is the shape of the mistakes.

What we ran

Both runs used the same frozen May 23, 2026 WebBrain baseline used by the published planner table: Claude Sonnet 4.6's system prompt and 41-tool schema, system hash 5c4fac1387025050. The endpoint was a local OpenAI-compatible server at http://localhost:8000.

node test/llm/run-llamacpp.mjs \
  --base http://localhost:8000 \
  --model lfm2.5-230m \
  --tag 2026-06-26-lfm25-230m-localhost8000 \
  --concurrency 16 \
  --timeout 180000 \
  --no-save-request \
  --freeze test/llm/freeze/baseline-2026-05-23.json

node test/llm/run-llamacpp.mjs \
  --base http://localhost:8000 \
  --model lfm2.5-350m \
  --tag 2026-06-26-lfm25-350m-localhost8000 \
  --concurrency 16 \
  --timeout 180000 \
  --no-save-request \
  --freeze test/llm/freeze/baseline-2026-05-23.json

One naming note: the second local server advertised lfm2.5-350m with root LiquidAI/LFM2.5-350M, so that is the label used here.

Result files:

test/llm/results/2026-06-26-lfm25-230m-localhost8000_chrome_lfm2.5-230m_frozen
test/llm/results/2026-06-26-lfm25-350m-localhost8000_chrome_lfm2.5-350m_frozen

Headline results

The clean positive signal: both models were usable through the OpenAI-compatible API path. There were no request failures, and the 230M run emitted a parsed tool call on every case.

The planner signal is much weaker. Neither model produced an exact match against the benchmark's ideal first call. Both matched the ideal tool name only four times. Against the Sonnet 4.6 first-tool reference, the 350M checkpoint did better than the 230M checkpoint, but both are far below the small tuned and instruction-following models already in the table.

How they failed

The 230M checkpoint is syntactically eager and strategically narrow. It calls tools constantly, but it collapses toward inspection:

That is why it has 100 parsed tool calls but only 19% Sonnet first-tool alignment. It often knows that a browser agent should use a tool, but it does not yet know which first action matches the user's intent. Direct navigation prompts, browser-internal pages, and search tasks frequently became generic page inspection.

The 350M checkpoint is more varied and closer to Sonnet, but less schema-disciplined:

That variety helps it reach 25% Sonnet first-tool alignment, but it also produces more invalid or non-frozen tool names: open_link, open_chat, stop_recording, close_popup, compose, solve_captcha, and delete_all_emails are not part of the frozen 41-tool schema. For a browser-agent planner, that is a fine-tuning problem before it is a reasoning problem.

Context against the older LFM and tiny Qwen rows

The most useful comparison is not with the top of the leaderboard. It is with other small or efficiency-oriented models.

The earlier LFM 2.5-8B-A1B run is still the better Liquid comparison point for actual planner use. It matched Sonnet on 40% of all cases and had 23 ideal tool-name matches. The tiny raw checkpoints do not reach that tier.

Qwen 3.5 0.8B is the sharper tiny baseline. Even with fewer than one billion parameters, it keeps higher valid-tool discipline and materially better Sonnet alignment than either raw tiny LFM run. That does not make the tiny LFMs uninteresting. It does show that raw scale at 230M or 350M is not enough for browser planning without task-specific post-training.

What this means

These runs should not be read as "LFM is bad at tools." They are closer to a pre-fine-tuning diagnostic. WebBrain's benchmark asks a very specific first-turn planning question: given a browser state, a user instruction, and 41 available tools, choose the first tool and arguments. Tiny raw models are not expected to have that policy baked in.

What they do show is encouraging in a narrower way:

• The local serving path works cleanly with WebBrain's OpenAI-compatible runner.
• The 230M model strongly learned the "emit a tool call" behavior, even if it overuses inspection.
• The 350M model has a broader action vocabulary and better Sonnet alignment, but needs schema control.
• Both runs are small enough that fine-tuning experiments should be cheap and fast.

For WebBrain, the next interesting experiment is not to keep benchmarking these raw checkpoints as-is. It is to tune them on browser-planner traces and rerun the same frozen harness. If a 230M or 350M model can be trained out of the get_accessibility_tree default and into reliable schema-following, it becomes a much more interesting on-device planner candidate.

The bottom line: raw tiny LFMs are not ready browser planners. That is fine. They are the kind of models you fine-tune into a planner, and this run gives us a compact before-picture. Liquid's new LFM2.5-230M strengthens that case: if a 230M model can perform as well as the 350M checkpoint while being roughly 33% smaller, it should become the default tiny LFM target for fine-tuning experiments. For no-fine-tuning, raw tiny-model use, though, Qwen 3.5 0.8B remains the de facto standard to beat.

Tags: #LFM25 #LiquidAI #TinyLanguageModels #ToolCalling #BrowserAgent #WebBrain