# Qwen GUI Coordinates — MAI-UI / Qwen-UI-Agent Reference

**Primary target: Qwen-UI-Agent (MAI-UI).** Qwen2.5-VL and Qwen3-VL are covered
only as the "legacy path" you also need to support.

**Source of truth for this document:** `github.com/Tongyi-MAI/MAI-UI`, files
`MAI-UI/src/mai_naivigation_agent.py`, `mai_grounding_agent.py`, `prompt.py`,
`utils.py`, and `cookbook/grounding.ipynb`. Facts below tagged `[SOURCE]` were
read directly from that code. Facts tagged `[PAPER]` come from the technical
report and **are not backed by the shipped code** — several of them contradict
it. Facts tagged `[LEGACY]` concern the older Qwen VL models.

**Version pin.** Read against the default branch, late August 2026. The repo
README was rebranded from MAI-UI to Qwen-UI-Agent on 2026-07-30, and `src/`
changed materially around then — in particular, `start_coordinate` /
`end_coordinate` normalization (§2.4) is **absent** from pre-rebrand snapshots.
If your checkout predates that, drag coordinates come out in raw 0–999 space
and everything in §2.4 is wrong for you. `git pull` and diff `src/` before
finalizing a profile. The repo still has zero tagged releases, so there is no
version string to pin against — pin a commit SHA.

---

## 0. The headline

For MAI-UI / Qwen-UI-Agent the entire coordinate story is three lines `[SOURCE]`:

```python
SCALE_FACTOR = 999

normalized_x = raw_x / 999          # model emits integers in 0..999
device_x     = normalized_x * screenshot.width
```

**There is no `smart_resize` in this pipeline.** The screenshot is PNG-encoded
to base64 and sent as-is. Coordinates are divided by a constant, then
multiplied by the **original screenshot dimensions**. No patch factor, no
resized-image space, no inverse resize transform.

This makes MAI-UI dramatically simpler than Qwen2.5-VL, and it means most of
what you know about Qwen2.5-VL coordinate handling is not just unnecessary here
— applying it will actively break your clicks.

---

## 1. Your normalization utility is correct — this is what the reference does

You were second-guessing the "convert everything to normalized coordinates"
design. Don't. It is exactly what MAI-UI ships `[SOURCE]`:

- `parse_action_to_structure_output()` divides every coordinate by 999 and
  stores `action["coordinate"] = [point_x, point_y]` in `[0, 1]`.
- `parse_grounding_response()` does the same for the grounding agent.
- `cookbook/grounding.ipynb` then multiplies the normalized pair by
  `test_image.width` / `test_image.height` to get pixels.

So the canonical space is `[0, 1]` floats, and the model-space ↔ canonical
conversion is a named, isolated step. That is the design. The "two-step"
concern is unfounded — the second step (canonical → device) is where crop,
rotation, and virtual-display handling live, and you need it regardless.

One detail worth copying: MAI-UI stores normalized coordinates in memory and
converts **back** to `int(x * 999)` when replaying history into the prompt
`[SOURCE]`. The path is `_build_messages()` → `mem2response(step)` →
`int(point_x * SCALE_FACTOR)`. The raw `prediction` string is stored on
`TrajStep` but is **not** what gets replayed. (A `history_responses` property
does the same re-encoding, but its call site in `_build_messages` is commented
out — dead code.) Keeping the canonical form as the stored form, and
re-encoding only at the prompt boundary, is what makes multi-turn history
consistent.

That round-trip uses `int()`, which truncates rather than rounds. A
coordinate can drift by 1 unit (~0.1% of screen) per round-trip through
history. Harmless for taps, but don't build anything that assumes
round-trip stability.

### Known bug in HEAD: drag is normalized but not re-encoded

`parse_action_to_structure_output` guards `coordinate`, `start_coordinate`,
**and** `end_coordinate` — all three get divided by 999. But `mem2response`
guards **only** `if "coordinate" in action_json`. Consequence: a `drag` action
is stored normalized, and then replayed into the next turn's prompt as

```json
{"action": "drag", "start_coordinate": [0.43, 0.21], "end_coordinate": [0.43, 0.78]}
```

instead of `[430, 210]` / `[430, 778]`. The model sees history in a different
coordinate space than it emits. The parse side was extended for drag; the
re-encode side was not.

**If you use `drag` in multi-turn tasks, mirror the start/end guards into your
own `mem2response` equivalent.** Symptom if you don't: degraded behaviour on
turns following a drag, with no parse error and no exception — the model just
gets confused history.

---

## 2. The shipped action space `[SOURCE]`

This is from `prompt.py`, not the paper. **It differs from the paper in ways
that will break an executor written from the report.**

`prompt.py` defines **four** prompts: three navigation variants (below) plus
`MAI_MOBILE_SYS_PROMPT_GROUNDING`, covered separately in §4.

### 2.1 `MAI_MOBILE_SYS_PROMPT` (default) and `MAI_MOBILE_SYS_PROMPT_NO_THINKING`

```text
{"action": "click",         "coordinate": [x, y]}
{"action": "long_press",    "coordinate": [x, y]}
{"action": "type",          "text": ""}
{"action": "swipe",         "direction": "up|down|left|right", "coordinate": [x, y]}   # coordinate OPTIONAL
{"action": "open",          "text": "app_name"}
{"action": "drag",          "start_coordinate": [x1, y1], "end_coordinate": [x2, y2]}
{"action": "system_button", "button": "back|home|menu|enter"}
{"action": "wait"}
{"action": "terminate",     "status": "success|fail"}
{"action": "answer",        "text": "xxx"}
```

### 2.2 `MAI_MOBILE_SYS_PROMPT_ASK_USER_MCP` (Jinja template, used when `mcp_tools` is passed)

All of the above, **plus**:

```text
{"action": "ask_user",      "text": "xxx"}
{"action": "double_click",  "coordinate": [x, y]}
```

…and an injected `## MCP Tools` section listing external tool JSON. MCP tools
are called with the same `<tool_call>` envelope but a different `name`.

### 2.3 Corrections against the technical report

| Claim | Reality |
| --- | --- |
| `[PAPER]` `swipe` was replaced by `drag` | **Both exist.** And `swipe` is **direction-based** (`"direction": "up"`) with an *optional* coordinate — it is not two-point. `drag` is the two-point action, and it uses `start_coordinate` / `end_coordinate`, **not** `coordinate` / `coordinate2`. |
| `[PAPER]` `double_click` is in the action space | Only in the MCP prompt variant. Not in the default prompt. |
| `[PAPER]` `ask_user` is in the action space | Only in the MCP prompt variant. |
| `[PAPER]` `cli_command` / bash execution | **Not present in any shipped prompt.** No shell action schema in the repo. |
| `[PAPER]` `api_call` | **Not present.** The nearest thing is the MCP tool escape hatch. |
| `[PAPER]` batched actions — one turn returns an ordered action sequence | **Not supported.** See §3. |
| Not in the paper's table | `answer` — returns a text answer to the user. You must handle it. |

If you built an executor from the report's Table 1, it will emit `swipe` with
`coordinate2`, and the model will never produce that.

### 2.4 Coordinate arity — 2 **or** 4 values `[SOURCE]` (navigation only)

**This rule is navigation-only.** The grounding parser
(`parse_grounding_response`) accepts length 2 and **raises on length 4**
(`expected 2 values, got {n}`). Do not share an arity policy between the two
agents.

In `parse_action_to_structure_output`, every coordinate field is parsed as
either a point or a bbox:

```text
len == 2  ->  [x, y]                  use directly
len == 4  ->  [x1, y1, x2, y2]        take the center: ((x1+x2)/2, (y1+y2)/2)
anything else -> raise
```

This applies to `coordinate`, `start_coordinate`, and `end_coordinate`
independently. If your parser only handles length-2 you will crash on a valid
model output.

---

## 3. One action per turn — no batching `[SOURCE]`

The parser is:

```python
pattern = r"<thinking>(.*?)</thinking>.*?<tool_call>(.*?)</tool_call>"
match   = re.search(pattern, text, re.DOTALL)
```

`re.search` returns the **first** match only. If the model emits multiple
`<tool_call>` blocks, everything after the first is silently discarded. There
is no loop, no `findall`, no batch executor. The paper's "batched actions in a
single model turn" is not in this codebase.

**Implication for your framework:** do not build batch-execution plumbing for
this model. If you want it, you are writing it yourself and changing the
parser, and you should expect the model's training distribution to be
one-action-per-turn.

### Thinking-tag normalization `[SOURCE]`

The parser also handles reasoning-model output that closes with `</think>`
instead of `</thinking>`:

```python
if "</think>" in text and "</thinking>" not in text:
    text = text.replace("</think>", "</thinking>")
    text = "<thinking>" + text
```

Note it prepends an opening tag — the model in that mode does not emit one.
Copy this or thinking-mode outputs will fail to parse.

---

## 4. The grounding agent uses a different format `[SOURCE]`

Two agents, two envelopes. Don't share a parser.

| Aspect | Navigation (`MAIUINaivigationAgent`) | Grounding (`MAIGroundingAgent`) |
| --- | --- | --- |
| Reasoning tag | `<thinking>` | `<grounding_think>` |
| Payload tag | `<tool_call>` | `<answer>` |
| Payload | `{"name": "mobile_use", "arguments": {...}}` | `{"coordinate": [x, y]}` |
| Scale | 999 | 999 |
| History | multi-turn, `history_n` default 3 | stateless, single image |

If you only need "where is element X" — your earlier question — the grounding
agent is the cheaper path: one image, no history, no tool schema, a much
shorter system prompt.

---

## 5. `max_pixels` / `min_pixels` are accepted and ignored `[SOURCE]`

Both agents' docstrings list `max_pixels` and `min_pixels` as valid
`runtime_conf` keys. **Neither is ever read.** The constructor extracts only
`temperature`, `top_k`, `top_p`, `max_tokens`, and (navigation only)
`history_n`. `pil_to_base64()` saves the image at full resolution with no
resize step anywhere in the call path.

Consequences:

1. Setting `max_pixels` in `runtime_conf` does nothing. If you are tuning it
   for latency, you are tuning nothing.
2. **The serving layer decides the resolution.** Whatever vLLM / your OpenAI-
   compatible endpoint is configured with is what actually applies. Token
   budget is a *deployment* concern here, not an *agent config* concern.
3. If you want to reduce tokens, you must downscale the PIL image yourself
   before calling `predict()`. Because coordinates are normalized rather than
   in pixel space, **downscaling is safe** — normalized output is
   resolution-independent, so the inverse mapping is unaffected. This is the
   one place where the normalized convention genuinely buys you something over
   Qwen2.5-VL.

---

## 6. Screen size handling

Because the model outputs normalized coordinates, the transform chain is much
shorter than for Qwen2.5-VL:

```text
screenshot (any size)  →  base64 PNG  →  model  →  [0..999]²
                                                      ↓ /999
                                                   [0..1]²
                                                      ↓ × (W, H)
                                              screenshot pixels
                                                      ↓ crop/rotate/scale inverse
                                                   device pixels  →  adb input tap
```

The reference cookbook multiplies by the dimensions of the **image it sent**.
So the rule is: *multiply by the dimensions of the exact image you passed to
`predict()`.* Everything after that is your own capture geometry.

Things that still bite you, none of them model-specific:

- **Capture ≠ device resolution.** scrcpy, virtual displays, and scaled capture
  pipelines produce screenshots whose size differs from the panel. Track
  capture dimensions per session.
- **Cropping.** If you strip the status bar to save tokens, add the offset back
  before tapping. Symptom: x correct, y consistently off.
- **Rotation.** Landscape changes which dimension is which. Capture orientation
  with the screenshot.
- **Downscaling is free here** (see §5) but changes nothing about the returned
  numbers — do not "correct" for it.

Since there is no `smart_resize`, the classic Qwen2.5-VL failure mode —
encoding with one resize config and inverting with another — **cannot happen**
on this path. Don't port that defensive code over; it will only add
opportunities to introduce an error.

---

## 7. Legacy path: Qwen2.5-VL `[LEGACY]`

Only relevant if your framework must also drive older models. Fundamentally
different, and the difference is not cosmetic.

- Emits **absolute pixels in the smart-resized image space**, not normalized.
- Patch factor 28. `smart_resize` rounds H and W to multiples of 28 and clamps
  total pixels into `[min_pixels, max_pixels]`.
- Ceiling is `16384 * 28²` = 12,845,056 px, which is what ships in
  `preprocessor_config.json`; the *recommended* range is 256–1280 visual
  tokens, i.e. 200,704 to 1,003,520 px. Running at the ceiling measurably
  degrades grounding on UI screenshots.
- Inverse mapping:
  `x_screenshot = x_model / resized_w * orig_w`
- Reported bug (**issue-tracker sourced, not code-verified** — QwenLM/Qwen3-VL
  issue #1052, version-dependent): passing `max_pixels` to
  `AutoProcessor.from_pretrained` may not update the processor's internal
  `size`. Mitigation regardless of whether your version is affected: read back
  actual grid dimensions rather than trusting config.
- `mobile_use` schema here uses `swipe` with `coordinate` + `coordinate2` —
  the *opposite* of MAI-UI's direction-based swipe.

Qwen3-VL sits between the two: patch factor 32, normalized output, but its
published range is inconsistently documented (0–1, 0–999, and 0–1000 all
appear in official sources) and its cookbooks apply `smart_resize` for
`computer_use` but not for `mobile_use`. If you must support it, probe it.

---

## 8. Framework shape, revised

Given the above, the profile is small and the important fields are not the ones
I would have guessed from the paper.

```yaml
# profiles/mai-ui.yaml          # VERIFIED against MAI-UI source
id: MAI-UI-8B
family: mai-ui

coordinates:
  space: normalized_scaled
  scale: 999                    # divide by this; do NOT use 1000
  accepts_bbox: true            # len-4 coordinate => take center
  denominator: submitted_image  # multiply by the image you sent, not device

vision:
  resize_owner: none              # no smart_resize in the client
  client_side_downscale: allowed  # safe: output is resolution-independent
  runtime_conf_pixel_keys: ignored  # max_pixels/min_pixels are dead config

protocol:
  reasoning_tag: thinking
  also_accept: think            # normalize </think> -> </thinking>, prepend open tag
  payload_tag: tool_call
  tool_name: mobile_use
  actions_per_turn: 1           # first <tool_call> only; rest discarded

action_schema:
  click:         { coordinate }
  long_press:    { coordinate }
  type:          { text }
  swipe:         { direction, coordinate? }     # direction-based!
  open:          { text }
  drag:          { start_coordinate, end_coordinate }
  system_button: { button }
  wait:          {}
  terminate:     { status }
  answer:        { text }
  # MCP prompt variant only:
  ask_user:      { text }
  double_click:  { coordinate }
```

```yaml
# profiles/mai-ui-grounding.yaml
id: MAI-UI-8B-grounding
coordinates: { space: normalized_scaled, scale: 999 }
protocol:
  reasoning_tag: grounding_think
  payload_tag: answer
  payload_shape: { coordinate: [x, y] }
  stateless: true
```

The three fields that actually matter and that differ across your two target
generations are `coordinates.space`, `coordinates.scale`, and
`vision.resize_owner`. `resize_owner: none` vs `resize_owner: caller` is the
single switch that decides whether the inverse-resize machinery runs at all.

Everything downstream of canonical `[0, 1]` — crop offset, rotation, device
scaling, ADB dispatch — is written once and shared. That's the payoff, and it's
the same payoff MAI-UI gets from normalizing at the parser boundary.

---

## 9. Pre-flight checklist

- [ ] Divide by **999**, not 1000. A 1000 divisor is a ~0.1% error — small
      enough to pass casual testing, large enough to miss small targets at
      screen edges.
- [ ] Handle length-4 coordinates by taking the center — **navigation only**.
      Grounding raises on length 4; don't "fix" that by accepting bboxes there.
- [ ] Mirror the `start_coordinate` / `end_coordinate` guards into your history
      re-encoder, not just your parser (§1). HEAD gets this wrong.
- [ ] Confirm your checkout is post-2026-07-30, or drag isn't normalized at all.
- [ ] Handle `answer` and `terminate` as terminal actions, not gestures.
- [ ] `swipe` carries `direction`; only `drag` carries two points.
- [ ] Only parse the first `<tool_call>`.
- [ ] Normalize `</think>` → `</thinking>` and prepend the opening tag.
- [ ] Don't set `max_pixels` in `runtime_conf` and expect it to do anything.
      Configure the serving layer instead.
- [ ] Multiply by the dimensions of the image you actually sent.
- [ ] Only enable `ask_user` / `double_click` when using the MCP prompt variant
      — the model isn't told about them otherwise.

---

## 10. Sources

- `Tongyi-MAI/MAI-UI` — `src/mai_naivigation_agent.py` (SCALE_FACTOR, parser,
  history round-trip), `src/mai_grounding_agent.py`, `src/prompt.py` (the three
  system prompts and the real action space), `src/utils.py`,
  `cookbook/grounding.ipynb` (normalized × image dims).
- Note: `Tongyi-MAI/Qwen-UI-Agent` is the **report website repo only** — PDF,
  README, assets. No implementation. The code lives under `MAI-UI/`.
- Qwen-UI-Agent technical report, arXiv 2607.28227 — useful for training
  methodology and benchmarks; **do not implement from its Table 1.**
- Qwen2.5-VL model cards and cookbooks — legacy `smart_resize` path.
- `bytedance/UI-TARS` `README_coordinates.md` — reference `smart_resize` and
  inverse mapping, for the legacy path only.
