Proposal / RFC: Benchmark Skyvern's agent-facing interface against the AXI ("Agent eXperience Interface") principles

[gwpl]

(AI 🤝 Greg collab:)

Status: open question for the team or any contributor (myself included) to pick up — no specific action expected
Related: #1420 (human-in-loop / Copilot / trace-driven workflow co-creation) — see "Connection to #1420" below

TL;DR

AXI is a recent write-up by Kun Chen (@kunchenguid, github.com/kunchenguid/axi) proposing 10 design principles for "agent-ergonomic" CLI/tool interfaces, with two reference implementations and ~915 benchmarked runs (Claude Sonnet 4.6 as agent + judge). One reference impl, chrome-devtools-axi, is a browser-automation wrapper — the same domain as Skyvern.

The interesting, falsifiable question for Skyvern: does shaping the interface the agent consumes (CLI/SDK/API responses) along some or all of these principles measurably improve Skyvern's task success, cost, and latency — and if so, which principles, and by how much?

This is meant as a benchmarking hypothesis, not an adoption pitch. The result is useful either way, and the principles are independently testable, so even a subset is a cheap experiment.

Why this might matter for Skyvern specifically

This is a continuation of a thread some of us touched in early 2026 around "better CLI + libraries beyond the MCP era" (context-window cost, easy wrappers, ergonomic scripting). AXI is essentially a rigorous, benchmarked formalization of that intuition, and it shares the same lineage Skyvern users have been pointing at: Anthropic's "code execution with MCP" work, Cloudflare's "Code Mode," the Linearis CLI write-up, and "MCPs are not like other APIs."

Skyvern is increasingly consumed by agents (SDK v1+, CLI, API responses fed back into an LLM loop). Anywhere an agent reads Skyvern output and decides a next step, interface ergonomics translate directly into tokens, turns, latency, and reliability.

The 10 AXI principles (credit: Kun Chen / axi.md)

Efficiency

1. Token-efficient output — TOON-style compact format instead of verbose JSON (claimed ~40% token savings)
2. Minimal default schemas — 3–4 fields per list item by default, opt-in to more via a flag
3. Content truncation — truncate large fields with a size hint + an explicit escape hatch

Robustness
4. Pre-computed aggregates — return totalCount, status summaries, etc., inline to kill round-trips
5. Definitive empty states — explicit "0 results" instead of ambiguous empty output
6. Structured errors & exit codes — idempotent mutations, structured errors to stdout, no interactive prompts, clean exit codes

Discoverability
7. Ambient context — self-install into the agent's session so relevant state is visible before the agent acts
8. Content first — no-args invocation shows live actionable data, not help text
9. Contextual disclosure — append concrete next-step command templates after output
10. Consistent way to get help — concise per-subcommand --help fallback

What the AXI benchmarks show (and the important caveat)

On AXI's own benchmarks, chrome-devtools-axi led on all four metrics (100% success, lowest cost, fewest turns, fastest), with MCP conditions using ~2.3× the input tokens; a separate GitHub benchmark showed up to a 12× cost gap on a complex investigation task.

Caveat worth stating up front: those numbers come from wrapping chrome-devtools-mcp with a generic Claude agent. Skyvern's architecture (visual + DOM, its own planner/executor) is different, so AXI's numbers do not transfer to Skyvern. That gap is exactly why a Skyvern-specific benchmark would be informative rather than assumed.

Concrete, low-cost ways to pick this up (subset-friendly)

Any one of these stands alone:

• Cheapest first experiment: apply principles 1 (compact output) + 4 (pre-computed aggregates) to one high-traffic Skyvern CLI/SDK response surface; A/B the agent loop on a fixed task set; measure tokens / turns / success.
• Same-domain baseline: use chrome-devtools-axi as a comparison point on overlapping browser tasks to sanity-check whether the effect direction holds for Skyvern at all.
• Discoverability slice: test principles 7–9 (ambient context + contextual next-step disclosure) against Skyvern's current CLI ergonomics — this is the slice most relevant to multi-step workflow building.
• Methodology reuse: AXI's benchmark harness and STUDY.md are public; the task taxonomy (single-step / multi-step / investigation / error-recovery) is reusable as a Skyvern eval skeleton even independent of the AXI question.

Connection to #1420

AXI principle 7 ("ambient context" — surfacing state via session hooks before the agent acts) overlaps with the trace/handoff-hook idea in #1420. If Skyvern exposes hooks around human/agent action and workflow-step state, those same hooks could carry an AXI-style compact ambient summary. The two ideas reinforce each other: #1420 is about capturing handoff provenance; this is about how that and other state is shaped for the agent that consumes it.

Asks (all optional)

1. Is AXI already on the team's radar? (If yes, ignore the rest.)
2. Would the team find a Skyvern-specific micro-benchmark of even 2–3 principles useful, or is interface ergonomics out of scope for now?
3. If there's mild interest but no bandwidth, I'm happy to prototype a minimal A/B on one response surface and report numbers here so it stays community-driven and pickup-able by anyone.

---

References: axi.md · github.com/kunchenguid/axi · chrome-devtools-axi · gh-axi · published GitHub study: axi/blob/main/bench-github/published-results/STUDY.md. All principle credit to Kun Chen.