Qwen GUI Coordinates — MAI-UI / Qwen-UI-Agent Reference
Primary target: Qwen-UI-Agent (MAI-UI). Qwen2.5-VL and Qwen3-VL are covered only as the "legacy path" you also need to support.
Source of truth for this document: github.com/Tongyi-MAI/MAI-UI, files
MAI-UI/src/mai_naivigation_agent.py, mai_grounding_agent.py, prompt.py,
utils.py, and cookbook/grounding.ipynb. Facts below tagged [SOURCE] were
read directly from that code. Facts tagged [PAPER] come from the technical
report and are not backed by the shipped code — several of them contradict
it. Facts tagged [LEGACY] concern the older Qwen VL models.
Version pin. Read against the default branch, late August 2026. The repo
README was rebranded from MAI-UI to Qwen-UI-Agent on 2026-07-30, and src/
changed materially around then — in particular, start_coordinate /
end_coordinate normalization (§2.4) is absent from pre-rebrand snapshots.
If your checkout predates that, drag coordinates come out in raw 0–999 space
and everything in §2.4 is wrong for you. git pull and diff src/ before
finalizing a profile. The repo still has zero tagged releases, so there is no
version string to pin against — pin a commit SHA.
0. The headline
For MAI-UI / Qwen-UI-Agent the entire coordinate story is three lines [SOURCE]:
SCALE_FACTOR = 999
normalized_x = raw_x / 999 # model emits integers in 0..999
device_x = normalized_x * screenshot.width
There is no smart_resize in this pipeline. The screenshot is PNG-encoded
to base64 and sent as-is. Coordinates are divided by a constant, then
multiplied by the original screenshot dimensions. No patch factor, no
resized-image space, no inverse resize transform.
This makes MAI-UI dramatically simpler than Qwen2.5-VL, and it means most of what you know about Qwen2.5-VL coordinate handling is not just unnecessary here — applying it will actively break your clicks.
1. Your normalization utility is correct — this is what the reference does
You were second-guessing the "convert everything to normalized coordinates"
design. Don't. It is exactly what MAI-UI ships [SOURCE]:
parse_action_to_structure_output()divides every coordinate by 999 and storesaction["coordinate"] = [point_x, point_y]in[0, 1].parse_grounding_response()does the same for the grounding agent.cookbook/grounding.ipynbthen multiplies the normalized pair bytest_image.width/test_image.heightto get pixels.
So the canonical space is [0, 1] floats, and the model-space ↔ canonical
conversion is a named, isolated step. That is the design. The "two-step"
concern is unfounded — the second step (canonical → device) is where crop,
rotation, and virtual-display handling live, and you need it regardless.
One detail worth copying: MAI-UI stores normalized coordinates in memory and
converts back to int(x * 999) when replaying history into the prompt
[SOURCE]. The path is _build_messages() → mem2response(step) →
int(point_x * SCALE_FACTOR). The raw prediction string is stored on
TrajStep but is not what gets replayed. (A history_responses property
does the same re-encoding, but its call site in _build_messages is commented
out — dead code.) Keeping the canonical form as the stored form, and
re-encoding only at the prompt boundary, is what makes multi-turn history
consistent.
That round-trip uses int(), which truncates rather than rounds. A
coordinate can drift by 1 unit (~0.1% of screen) per round-trip through
history. Harmless for taps, but don't build anything that assumes
round-trip stability.
Known bug in HEAD: drag is normalized but not re-encoded
parse_action_to_structure_output guards coordinate, start_coordinate,
and end_coordinate — all three get divided by 999. But mem2response
guards only if "coordinate" in action_json. Consequence: a drag action
is stored normalized, and then replayed into the next turn's prompt as
{"action": "drag", "start_coordinate": [0.43, 0.21], "end_coordinate": [0.43, 0.78]}
instead of [430, 210] / [430, 778]. The model sees history in a different
coordinate space than it emits. The parse side was extended for drag; the
re-encode side was not.
If you use drag in multi-turn tasks, mirror the start/end guards into your
own mem2response equivalent. Symptom if you don't: degraded behaviour on
turns following a drag, with no parse error and no exception — the model just
gets confused history.
2. The shipped action space [SOURCE]
This is from prompt.py, not the paper. It differs from the paper in ways
that will break an executor written from the report.
prompt.py defines four prompts: three navigation variants (below) plus
MAI_MOBILE_SYS_PROMPT_GROUNDING, covered separately in §4.
2.1 MAI_MOBILE_SYS_PROMPT (default) and MAI_MOBILE_SYS_PROMPT_NO_THINKING
{"action": "click", "coordinate": [x, y]}
{"action": "long_press", "coordinate": [x, y]}
{"action": "type", "text": ""}
{"action": "swipe", "direction": "up|down|left|right", "coordinate": [x, y]} # coordinate OPTIONAL
{"action": "open", "text": "app_name"}
{"action": "drag", "start_coordinate": [x1, y1], "end_coordinate": [x2, y2]}
{"action": "system_button", "button": "back|home|menu|enter"}
{"action": "wait"}
{"action": "terminate", "status": "success|fail"}
{"action": "answer", "text": "xxx"}
2.2 MAI_MOBILE_SYS_PROMPT_ASK_USER_MCP (Jinja template, used when mcp_tools is passed)
All of the above, plus:
{"action": "ask_user", "text": "xxx"}
{"action": "double_click", "coordinate": [x, y]}
…and an injected ## MCP Tools section listing external tool JSON. MCP tools
are called with the same <tool_call> envelope but a different name.
2.3 Corrections against the technical report
| Claim | Reality |
|---|---|
[PAPER] swipe was replaced by drag |
Both exist. And swipe is direction-based ("direction": "up") with an optional coordinate — it is not two-point. drag is the two-point action, and it uses start_coordinate / end_coordinate, not coordinate / coordinate2. |
[PAPER] double_click is in the action space |
Only in the MCP prompt variant. Not in the default prompt. |
[PAPER] ask_user is in the action space |
Only in the MCP prompt variant. |
[PAPER] cli_command / bash execution |
Not present in any shipped prompt. No shell action schema in the repo. |
[PAPER] api_call |
Not present. The nearest thing is the MCP tool escape hatch. |
[PAPER] batched actions — one turn returns an ordered action sequence |
Not supported. See §3. |
| Not in the paper's table | answer — returns a text answer to the user. You must handle it. |
If you built an executor from the report's Table 1, it will emit swipe with
coordinate2, and the model will never produce that.
2.4 Coordinate arity — 2 or 4 values [SOURCE] (navigation only)
This rule is navigation-only. The grounding parser
(parse_grounding_response) accepts length 2 and raises on length 4
(expected 2 values, got {n}). Do not share an arity policy between the two
agents.
In parse_action_to_structure_output, every coordinate field is parsed as
either a point or a bbox:
len == 2 -> [x, y] use directly
len == 4 -> [x1, y1, x2, y2] take the center: ((x1+x2)/2, (y1+y2)/2)
anything else -> raise
This applies to coordinate, start_coordinate, and end_coordinate
independently. If your parser only handles length-2 you will crash on a valid
model output.
3. One action per turn — no batching [SOURCE]
The parser is:
pattern = r"<thinking>(.*?)</thinking>.*?<tool_call>(.*?)</tool_call>"
match = re.search(pattern, text, re.DOTALL)
re.search returns the first match only. If the model emits multiple
<tool_call> blocks, everything after the first is silently discarded. There
is no loop, no findall, no batch executor. The paper's "batched actions in a
single model turn" is not in this codebase.
Implication for your framework: do not build batch-execution plumbing for this model. If you want it, you are writing it yourself and changing the parser, and you should expect the model's training distribution to be one-action-per-turn.
Thinking-tag normalization [SOURCE]
The parser also handles reasoning-model output that closes with </think>
instead of </thinking>:
if "</think>" in text and "</thinking>" not in text:
text = text.replace("</think>", "</thinking>")
text = "<thinking>" + text
Note it prepends an opening tag — the model in that mode does not emit one. Copy this or thinking-mode outputs will fail to parse.
4. The grounding agent uses a different format [SOURCE]
Two agents, two envelopes. Don't share a parser.
| Aspect | Navigation (MAIUINaivigationAgent) |
Grounding (MAIGroundingAgent) |
|---|---|---|
| Reasoning tag | <thinking> |
<grounding_think> |
| Payload tag | <tool_call> |
<answer> |
| Payload | {"name": "mobile_use", "arguments": {...}} |
{"coordinate": [x, y]} |
| Scale | 999 | 999 |
| History | multi-turn, history_n default 3 |
stateless, single image |
If you only need "where is element X" — your earlier question — the grounding agent is the cheaper path: one image, no history, no tool schema, a much shorter system prompt.
5. max_pixels / min_pixels are accepted and ignored [SOURCE]
Both agents' docstrings list max_pixels and min_pixels as valid
runtime_conf keys. Neither is ever read. The constructor extracts only
temperature, top_k, top_p, max_tokens, and (navigation only)
history_n. pil_to_base64() saves the image at full resolution with no
resize step anywhere in the call path.
Consequences:
- Setting
max_pixelsinruntime_confdoes nothing. If you are tuning it for latency, you are tuning nothing. - The serving layer decides the resolution. Whatever vLLM / your OpenAI- compatible endpoint is configured with is what actually applies. Token budget is a deployment concern here, not an agent config concern.
- If you want to reduce tokens, you must downscale the PIL image yourself
before calling
predict(). Because coordinates are normalized rather than in pixel space, downscaling is safe — normalized output is resolution-independent, so the inverse mapping is unaffected. This is the one place where the normalized convention genuinely buys you something over Qwen2.5-VL.
6. Screen size handling
Because the model outputs normalized coordinates, the transform chain is much shorter than for Qwen2.5-VL:
screenshot (any size) → base64 PNG → model → [0..999]²
↓ /999
[0..1]²
↓ × (W, H)
screenshot pixels
↓ crop/rotate/scale inverse
device pixels → adb input tap
The reference cookbook multiplies by the dimensions of the image it sent.
So the rule is: multiply by the dimensions of the exact image you passed to
predict(). Everything after that is your own capture geometry.
Things that still bite you, none of them model-specific:
- Capture ≠ device resolution. scrcpy, virtual displays, and scaled capture pipelines produce screenshots whose size differs from the panel. Track capture dimensions per session.
- Cropping. If you strip the status bar to save tokens, add the offset back before tapping. Symptom: x correct, y consistently off.
- Rotation. Landscape changes which dimension is which. Capture orientation with the screenshot.
- Downscaling is free here (see §5) but changes nothing about the returned numbers — do not "correct" for it.
Since there is no smart_resize, the classic Qwen2.5-VL failure mode —
encoding with one resize config and inverting with another — cannot happen
on this path. Don't port that defensive code over; it will only add
opportunities to introduce an error.
7. Legacy path: Qwen2.5-VL [LEGACY]
Only relevant if your framework must also drive older models. Fundamentally different, and the difference is not cosmetic.
- Emits absolute pixels in the smart-resized image space, not normalized.
- Patch factor 28.
smart_resizerounds H and W to multiples of 28 and clamps total pixels into[min_pixels, max_pixels]. - Ceiling is
16384 * 28²= 12,845,056 px, which is what ships inpreprocessor_config.json; the recommended range is 256–1280 visual tokens, i.e. 200,704 to 1,003,520 px. Running at the ceiling measurably degrades grounding on UI screenshots. - Inverse mapping:
x_screenshot = x_model / resized_w * orig_w - Reported bug (issue-tracker sourced, not code-verified — QwenLM/Qwen3-VL
issue #1052, version-dependent): passing
max_pixelstoAutoProcessor.from_pretrainedmay not update the processor's internalsize. Mitigation regardless of whether your version is affected: read back actual grid dimensions rather than trusting config. mobile_useschema here usesswipewithcoordinate+coordinate2— the opposite of MAI-UI's direction-based swipe.
Qwen3-VL sits between the two: patch factor 32, normalized output, but its
published range is inconsistently documented (0–1, 0–999, and 0–1000 all
appear in official sources) and its cookbooks apply smart_resize for
computer_use but not for mobile_use. If you must support it, probe it.
8. Framework shape, revised
Given the above, the profile is small and the important fields are not the ones I would have guessed from the paper.
# profiles/mai-ui.yaml # VERIFIED against MAI-UI source
id: MAI-UI-8B
family: mai-ui
coordinates:
space: normalized_scaled
scale: 999 # divide by this; do NOT use 1000
accepts_bbox: true # len-4 coordinate => take center
denominator: submitted_image # multiply by the image you sent, not device
vision:
resize_owner: none # no smart_resize in the client
client_side_downscale: allowed # safe: output is resolution-independent
runtime_conf_pixel_keys: ignored # max_pixels/min_pixels are dead config
protocol:
reasoning_tag: thinking
also_accept: think # normalize </think> -> </thinking>, prepend open tag
payload_tag: tool_call
tool_name: mobile_use
actions_per_turn: 1 # first <tool_call> only; rest discarded
action_schema:
click: { coordinate }
long_press: { coordinate }
type: { text }
swipe: { direction, coordinate? } # direction-based!
open: { text }
drag: { start_coordinate, end_coordinate }
system_button: { button }
wait: {}
terminate: { status }
answer: { text }
# MCP prompt variant only:
ask_user: { text }
double_click: { coordinate }
# profiles/mai-ui-grounding.yaml
id: MAI-UI-8B-grounding
coordinates: { space: normalized_scaled, scale: 999 }
protocol:
reasoning_tag: grounding_think
payload_tag: answer
payload_shape: { coordinate: [x, y] }
stateless: true
The three fields that actually matter and that differ across your two target
generations are coordinates.space, coordinates.scale, and
vision.resize_owner. resize_owner: none vs resize_owner: caller is the
single switch that decides whether the inverse-resize machinery runs at all.
Everything downstream of canonical [0, 1] — crop offset, rotation, device
scaling, ADB dispatch — is written once and shared. That's the payoff, and it's
the same payoff MAI-UI gets from normalizing at the parser boundary.
9. Pre-flight checklist
- Divide by 999, not 1000. A 1000 divisor is a ~0.1% error — small enough to pass casual testing, large enough to miss small targets at screen edges.
- Handle length-4 coordinates by taking the center — navigation only. Grounding raises on length 4; don't "fix" that by accepting bboxes there.
- Mirror the
start_coordinate/end_coordinateguards into your history re-encoder, not just your parser (§1). HEAD gets this wrong. - Confirm your checkout is post-2026-07-30, or drag isn't normalized at all.
- Handle
answerandterminateas terminal actions, not gestures. -
swipecarriesdirection; onlydragcarries two points. - Only parse the first
<tool_call>. - Normalize
</think>→</thinking>and prepend the opening tag. - Don't set
max_pixelsinruntime_confand expect it to do anything. Configure the serving layer instead. - Multiply by the dimensions of the image you actually sent.
- Only enable
ask_user/double_clickwhen using the MCP prompt variant — the model isn't told about them otherwise.
10. Sources
Tongyi-MAI/MAI-UI—src/mai_naivigation_agent.py(SCALE_FACTOR, parser, history round-trip),src/mai_grounding_agent.py,src/prompt.py(the three system prompts and the real action space),src/utils.py,cookbook/grounding.ipynb(normalized × image dims).- Note:
Tongyi-MAI/Qwen-UI-Agentis the report website repo only — PDF, README, assets. No implementation. The code lives underMAI-UI/. - Qwen-UI-Agent technical report, arXiv 2607.28227 — useful for training methodology and benchmarks; do not implement from its Table 1.
- Qwen2.5-VL model cards and cookbooks — legacy
smart_resizepath. bytedance/UI-TARSREADME_coordinates.md— referencesmart_resizeand inverse mapping, for the legacy path only.