Agent Context · Add file · llms.txt

Qwen GUI Coordinates — MAI-UI / Qwen-UI-Agent Reference

Primary target: Qwen-UI-Agent (MAI-UI). Qwen2.5-VL and Qwen3-VL are covered only as the "legacy path" you also need to support.

Source of truth for this document: github.com/Tongyi-MAI/MAI-UI, files MAI-UI/src/mai_naivigation_agent.py, mai_grounding_agent.py, prompt.py, utils.py, and cookbook/grounding.ipynb. Facts below tagged [SOURCE] were read directly from that code. Facts tagged [PAPER] come from the technical report and are not backed by the shipped code — several of them contradict it. Facts tagged [LEGACY] concern the older Qwen VL models.

Version pin. Read against the default branch, late August 2026. The repo README was rebranded from MAI-UI to Qwen-UI-Agent on 2026-07-30, and src/ changed materially around then — in particular, start_coordinate / end_coordinate normalization (§2.4) is absent from pre-rebrand snapshots. If your checkout predates that, drag coordinates come out in raw 0–999 space and everything in §2.4 is wrong for you. git pull and diff src/ before finalizing a profile. The repo still has zero tagged releases, so there is no version string to pin against — pin a commit SHA.


0. The headline

For MAI-UI / Qwen-UI-Agent the entire coordinate story is three lines [SOURCE]:

SCALE_FACTOR = 999

normalized_x = raw_x / 999          # model emits integers in 0..999
device_x     = normalized_x * screenshot.width

There is no smart_resize in this pipeline. The screenshot is PNG-encoded to base64 and sent as-is. Coordinates are divided by a constant, then multiplied by the original screenshot dimensions. No patch factor, no resized-image space, no inverse resize transform.

This makes MAI-UI dramatically simpler than Qwen2.5-VL, and it means most of what you know about Qwen2.5-VL coordinate handling is not just unnecessary here — applying it will actively break your clicks.


1. Your normalization utility is correct — this is what the reference does

You were second-guessing the "convert everything to normalized coordinates" design. Don't. It is exactly what MAI-UI ships [SOURCE]:

So the canonical space is [0, 1] floats, and the model-space ↔ canonical conversion is a named, isolated step. That is the design. The "two-step" concern is unfounded — the second step (canonical → device) is where crop, rotation, and virtual-display handling live, and you need it regardless.

One detail worth copying: MAI-UI stores normalized coordinates in memory and converts back to int(x * 999) when replaying history into the prompt [SOURCE]. The path is _build_messages() → mem2response(step) → int(point_x * SCALE_FACTOR). The raw prediction string is stored on TrajStep but is not what gets replayed. (A history_responses property does the same re-encoding, but its call site in _build_messages is commented out — dead code.) Keeping the canonical form as the stored form, and re-encoding only at the prompt boundary, is what makes multi-turn history consistent.

That round-trip uses int(), which truncates rather than rounds. A coordinate can drift by 1 unit (~0.1% of screen) per round-trip through history. Harmless for taps, but don't build anything that assumes round-trip stability.

Known bug in HEAD: drag is normalized but not re-encoded

parse_action_to_structure_output guards coordinate, start_coordinate, and end_coordinate — all three get divided by 999. But mem2response guards only if "coordinate" in action_json. Consequence: a drag action is stored normalized, and then replayed into the next turn's prompt as

{"action": "drag", "start_coordinate": [0.43, 0.21], "end_coordinate": [0.43, 0.78]}

instead of [430, 210] / [430, 778]. The model sees history in a different coordinate space than it emits. The parse side was extended for drag; the re-encode side was not.

If you use drag in multi-turn tasks, mirror the start/end guards into your own mem2response equivalent. Symptom if you don't: degraded behaviour on turns following a drag, with no parse error and no exception — the model just gets confused history.


2. The shipped action space [SOURCE]

This is from prompt.py, not the paper. It differs from the paper in ways that will break an executor written from the report.

prompt.py defines four prompts: three navigation variants (below) plus MAI_MOBILE_SYS_PROMPT_GROUNDING, covered separately in §4.

2.1 MAI_MOBILE_SYS_PROMPT (default) and MAI_MOBILE_SYS_PROMPT_NO_THINKING

{"action": "click",         "coordinate": [x, y]}
{"action": "long_press",    "coordinate": [x, y]}
{"action": "type",          "text": ""}
{"action": "swipe",         "direction": "up|down|left|right", "coordinate": [x, y]}   # coordinate OPTIONAL
{"action": "open",          "text": "app_name"}
{"action": "drag",          "start_coordinate": [x1, y1], "end_coordinate": [x2, y2]}
{"action": "system_button", "button": "back|home|menu|enter"}
{"action": "wait"}
{"action": "terminate",     "status": "success|fail"}
{"action": "answer",        "text": "xxx"}

2.2 MAI_MOBILE_SYS_PROMPT_ASK_USER_MCP (Jinja template, used when mcp_tools is passed)

All of the above, plus:

{"action": "ask_user",      "text": "xxx"}
{"action": "double_click",  "coordinate": [x, y]}

…and an injected ## MCP Tools section listing external tool JSON. MCP tools are called with the same <tool_call> envelope but a different name.

2.3 Corrections against the technical report

Claim Reality
[PAPER] swipe was replaced by drag Both exist. And swipe is direction-based ("direction": "up") with an optional coordinate — it is not two-point. drag is the two-point action, and it uses start_coordinate / end_coordinate, not coordinate / coordinate2.
[PAPER] double_click is in the action space Only in the MCP prompt variant. Not in the default prompt.
[PAPER] ask_user is in the action space Only in the MCP prompt variant.
[PAPER] cli_command / bash execution Not present in any shipped prompt. No shell action schema in the repo.
[PAPER] api_call Not present. The nearest thing is the MCP tool escape hatch.
[PAPER] batched actions — one turn returns an ordered action sequence Not supported. See §3.
Not in the paper's table answer — returns a text answer to the user. You must handle it.

If you built an executor from the report's Table 1, it will emit swipe with coordinate2, and the model will never produce that.

2.4 Coordinate arity — 2 or 4 values [SOURCE] (navigation only)

This rule is navigation-only. The grounding parser (parse_grounding_response) accepts length 2 and raises on length 4 (expected 2 values, got {n}). Do not share an arity policy between the two agents.

In parse_action_to_structure_output, every coordinate field is parsed as either a point or a bbox:

len == 2  ->  [x, y]                  use directly
len == 4  ->  [x1, y1, x2, y2]        take the center: ((x1+x2)/2, (y1+y2)/2)
anything else -> raise

This applies to coordinate, start_coordinate, and end_coordinate independently. If your parser only handles length-2 you will crash on a valid model output.


3. One action per turn — no batching [SOURCE]

The parser is:

pattern = r"<thinking>(.*?)</thinking>.*?<tool_call>(.*?)</tool_call>"
match   = re.search(pattern, text, re.DOTALL)

re.search returns the first match only. If the model emits multiple <tool_call> blocks, everything after the first is silently discarded. There is no loop, no findall, no batch executor. The paper's "batched actions in a single model turn" is not in this codebase.

Implication for your framework: do not build batch-execution plumbing for this model. If you want it, you are writing it yourself and changing the parser, and you should expect the model's training distribution to be one-action-per-turn.

Thinking-tag normalization [SOURCE]

The parser also handles reasoning-model output that closes with </think> instead of </thinking>:

if "</think>" in text and "</thinking>" not in text:
    text = text.replace("</think>", "</thinking>")
    text = "<thinking>" + text

Note it prepends an opening tag — the model in that mode does not emit one. Copy this or thinking-mode outputs will fail to parse.


4. The grounding agent uses a different format [SOURCE]

Two agents, two envelopes. Don't share a parser.

Aspect Navigation (MAIUINaivigationAgent) Grounding (MAIGroundingAgent)
Reasoning tag <thinking> <grounding_think>
Payload tag <tool_call> <answer>
Payload {"name": "mobile_use", "arguments": {...}} {"coordinate": [x, y]}
Scale 999 999
History multi-turn, history_n default 3 stateless, single image

If you only need "where is element X" — your earlier question — the grounding agent is the cheaper path: one image, no history, no tool schema, a much shorter system prompt.


5. max_pixels / min_pixels are accepted and ignored [SOURCE]

Both agents' docstrings list max_pixels and min_pixels as valid runtime_conf keys. Neither is ever read. The constructor extracts only temperature, top_k, top_p, max_tokens, and (navigation only) history_n. pil_to_base64() saves the image at full resolution with no resize step anywhere in the call path.

Consequences:

  1. Setting max_pixels in runtime_conf does nothing. If you are tuning it for latency, you are tuning nothing.
  2. The serving layer decides the resolution. Whatever vLLM / your OpenAI- compatible endpoint is configured with is what actually applies. Token budget is a deployment concern here, not an agent config concern.
  3. If you want to reduce tokens, you must downscale the PIL image yourself before calling predict(). Because coordinates are normalized rather than in pixel space, downscaling is safe — normalized output is resolution-independent, so the inverse mapping is unaffected. This is the one place where the normalized convention genuinely buys you something over Qwen2.5-VL.

6. Screen size handling

Because the model outputs normalized coordinates, the transform chain is much shorter than for Qwen2.5-VL:

screenshot (any size)  →  base64 PNG  →  model  →  [0..999]²
                                                      ↓ /999
                                                   [0..1]²
                                                      ↓ × (W, H)
                                              screenshot pixels
                                                      ↓ crop/rotate/scale inverse
                                                   device pixels  →  adb input tap

The reference cookbook multiplies by the dimensions of the image it sent. So the rule is: multiply by the dimensions of the exact image you passed to predict(). Everything after that is your own capture geometry.

Things that still bite you, none of them model-specific:

Since there is no smart_resize, the classic Qwen2.5-VL failure mode — encoding with one resize config and inverting with another — cannot happen on this path. Don't port that defensive code over; it will only add opportunities to introduce an error.


7. Legacy path: Qwen2.5-VL [LEGACY]

Only relevant if your framework must also drive older models. Fundamentally different, and the difference is not cosmetic.

Qwen3-VL sits between the two: patch factor 32, normalized output, but its published range is inconsistently documented (0–1, 0–999, and 0–1000 all appear in official sources) and its cookbooks apply smart_resize for computer_use but not for mobile_use. If you must support it, probe it.


8. Framework shape, revised

Given the above, the profile is small and the important fields are not the ones I would have guessed from the paper.

# profiles/mai-ui.yaml          # VERIFIED against MAI-UI source
id: MAI-UI-8B
family: mai-ui

coordinates:
  space: normalized_scaled
  scale: 999                    # divide by this; do NOT use 1000
  accepts_bbox: true            # len-4 coordinate => take center
  denominator: submitted_image  # multiply by the image you sent, not device

vision:
  resize_owner: none              # no smart_resize in the client
  client_side_downscale: allowed  # safe: output is resolution-independent
  runtime_conf_pixel_keys: ignored  # max_pixels/min_pixels are dead config

protocol:
  reasoning_tag: thinking
  also_accept: think            # normalize </think> -> </thinking>, prepend open tag
  payload_tag: tool_call
  tool_name: mobile_use
  actions_per_turn: 1           # first <tool_call> only; rest discarded

action_schema:
  click:         { coordinate }
  long_press:    { coordinate }
  type:          { text }
  swipe:         { direction, coordinate? }     # direction-based!
  open:          { text }
  drag:          { start_coordinate, end_coordinate }
  system_button: { button }
  wait:          {}
  terminate:     { status }
  answer:        { text }
  # MCP prompt variant only:
  ask_user:      { text }
  double_click:  { coordinate }
# profiles/mai-ui-grounding.yaml
id: MAI-UI-8B-grounding
coordinates: { space: normalized_scaled, scale: 999 }
protocol:
  reasoning_tag: grounding_think
  payload_tag: answer
  payload_shape: { coordinate: [x, y] }
  stateless: true

The three fields that actually matter and that differ across your two target generations are coordinates.space, coordinates.scale, and vision.resize_owner. resize_owner: none vs resize_owner: caller is the single switch that decides whether the inverse-resize machinery runs at all.

Everything downstream of canonical [0, 1] — crop offset, rotation, device scaling, ADB dispatch — is written once and shared. That's the payoff, and it's the same payoff MAI-UI gets from normalizing at the parser boundary.


9. Pre-flight checklist


10. Sources