Testing mobile apps on real devices

You are driving a physical phone or tablet over the network. It has real latency, real animations, and a real keyboard. Nearly every failure in agent-driven mobile testing comes from three things: tapping the wrong pixel, typing before focus lands, and acting before the screen settled. This skill is built around avoiding those.

The loop

Steps 4–7 are one indivisible cycle. Skipping OBSERVE to save a call is how a run ends up tapping a stale screen and reporting a false pass.

Rule 1 — never invent a coordinate

Both platforms give you an authoritative element source. Use it.

Android. device_screenshot returns the image plus a labeled element list with bounds already in tap-coordinate space. Prefer, in order:

1. device_tap_by_text(udid, text | contentDesc | resourceId) — no coordinates at all. It ranks candidates so a Button or EditText beats a passive TextView, and auto-scrolls to find an off-screen target. This is the default choice.
2. Bounds from the screenshot's labeled elements or device_page_source ([L,T][R,B] → tap the center, ((L+R)/2, (T+B)/2)). Pixel-exact, no scaling.
3. Only if the target is invisible to the accessibility tree (image-only icon, custom canvas): estimate from the image — and then you must scale, see below.

iOS. Prefer ios_tap_by_label(udid, label). It works even on zero-area elements and needs no coordinate math. Fall back to ios_find_element → returns a center point, then ios_tap. Coordinates must come from ios_page_source, never from looking at a screenshot.

The Android scaling trap

The screenshot you are shown is downscaled. device_tap expects full-resolution tap-space. If you estimated a point off the image, convert it:

Read both numbers from the footer of the actual screenshot. Never assume a constant — it differs per device. Forgetting this scale is the single most common cause of taps landing in the wrong place. It does not apply to bounds from page source or the labeled-element list, which are already in tap space.

Rule 2 — focus, then type, sequentially

device_tap_by_text (or the tap that focuses a field) must complete before device_type is issued. Never put them in the same parallel batch.

If you fire both at once, the type races the focus change and lands in the previously focused field. Real observed failure: a login test where username and password were concatenated into the username box, and the test reported "typed successfully".

device_type extras that remove a call each:

• clearFirst: true — wipes the field first (reads its length, sends that many deletes)
• pressKey: "ENTER" — submits after typing, for type-then-search flows
• method — leave it on the default keys. Some manufacturer keyboards silently drop @, ., and _ under the ime and shell methods, which quietly corrupts every email and URL you type. Non-ASCII (CJK, emoji) auto-falls back on its own.

Rule 3 — wait for elements, never for time

Polls until it appears and returns coordinates ready to tap. A fixed pause is either slower than it needs to be or flaky on a slow build — usually both across a device pool.

On iOS: ios_scroll_to_element / ios_find_element serve the same role.

If an element is not in device_page_source, it may simply be scrolled off-screen — Android only dumps rendered nodes. Use device_scroll_to_element to bring it in before concluding it is absent. Pass search: to device_page_source to grep the dump case-insensitively across text, content-desc, class, and resource-id — the fastest way to answer "is this on screen at all?".

Installing the app under test

iOS uses the same upload/install path with an .ipa, then ios_launch_app(udid, bundleId).

Already installed? device_list_apps / ios_list_apps to find the identifier, and device_clear_app_data to force a clean first-run state before an onboarding test.

Asserting

An assertion reads the device, not your expectation of it.

• Text on screen → device_page_source(udid, search: "Order #") or device_find_element; iOS → ios_find_element / ios_page_source.
• Correct app in foreground → device_current_app / ios_active_app.
• Something must be absent → search page source and confirm no match. Do not infer absence from a screenshot; an element can be present and off-screen.

Report what you actually observed. "The confirmation screen did not appear within 15s; page source still shows the payment form" is a useful result. "Test passed" without a read is not.

Capturing evidence

• device_screenshot(udid, includeElements: false) — image only, saves tokens on screens you only need to show, not interact with.
• device_record_start / device_record_stop — video of a repro.
• android_video_stream — live view; use this instead of polling screenshots when debugging a flow that is stuck.
• device_locators_for(udid, text | x,y) — ranked, stable locators (id, text, content-desc, accessibility selector, XPath) for the element. Use these when writing the flow up as a real test script, so selectors are not re-derived by hand.

Always release

Holds count against your parallel-device limit. Release on the failure path too.

Input goes straight to the device

Taps and keystrokes are delivered through the device's own input path, not routed through a separate automation framework layer. On iOS the *_hid tools need no automation session at all (ios_hid_status reports availability), and in the device browser webpage_click makes the device tap its own screen — a genuinely trusted touch that satisfies targets which reject synthesized events.

You do not need a framework session to drive a device. One is available when you want framework parity — see the framework-integration skill.

Related skills

Deeper reference

• references/android.md — full Android tool map, gestures, hardware keys, app data, shell access
• references/ios.md — session lifecycle, label-first interaction, hardware buttons, HID vs automation input, files and diagnostics