Your Mobile Suite Isn't Slow, It's Serial

A 400-test Appium suite that takes 90 minutes is not usually 90 minutes of slow tests. It's 90 minutes of one device doing 400 things in a row.

You can shave seconds off individual steps forever and never fix that. The only lever with real leverage is how many devices are running at once — and the reasons teams don't pull it are almost never technical.

The arithmetic nobody does

Take the suite you have. Total wall-clock is roughly:

Two things fall out of that immediately.

One: past a certain point, adding devices stops helping. If your longest single test is 6 minutes, no amount of parallelism gets the suite under 6 minutes. Splitting that one test is worth more than the next four devices.

Two: setup_per_device is paid once per shard, not once per suite. If every shard installs the app, waits for a device to be handed to it, signs in and seeds fixtures, then a 12-way split pays that twelve times. Teams routinely go from 4 shards to 12, watch wall-clock barely move, and conclude parallelism doesn't work for mobile. What actually happened is that setup went from 20% of the run to 60% of it.

Measure both numbers before you buy anything.

Where to cut the suite

Sharding by "test file, alphabetically, into N buckets" is the default in most CI templates and it's the worst one available. Test durations on mobile are wildly uneven — a login test is 20 seconds, a checkout-through-payment test is four minutes — so alphabetical buckets give you one shard that finishes in three minutes and one that runs for twenty. Your suite is as slow as that last shard.

Two better options, in order of effort:

Balance by recorded duration. Keep last run's per-test timings in a JSON file in the repo, sort tests descending, and greedily assign each to whichever shard currently has the least work. Fifteen lines of code, and it typically beats alphabetical splitting by 30–40% on wall-clock:

// shard.mjs — longest-first bin packing
const timings = JSON.parse(fs.readFileSync("test-timings.json", "utf8"));
const tests = allTests.sort((a, b) => (timings[b] ?? 60) - (timings[a] ?? 60));

const shards = Array.from({ length: N }, () => ({ total: 0, tests: [] }));
for (const test of tests) {
  const lightest = shards.reduce((a, b) => (a.total <= b.total ? a : b));
  lightest.tests.push(test);
  lightest.total += timings[test] ?? 60;
}

Unknown tests default to a middle-of-the-road guess so a new test never lands in a shard that's already full.

Split by device requirement instead. Some tests only make sense on specific hardware — the biometric flow, the small-screen layout regression, the tablet split view. Those aren't interchangeable work units, so pin them and balance whatever's left.

The failure that shows up only in parallel

Serial suites hide shared state. Run them twelve at a time and it surfaces within a day.

One login account, twelve sessions. The classic. Test 7 signs in, test 31 signs the same account in elsewhere, the backend invalidates the first session, test 7 fails on a screen that has nothing to do with auth. Every re-run picks a different victim, so it reads as flakiness. Fix it with a pool of accounts leased per shard, not per suite.

Shared fixtures with mutable state. A test that edits the seeded user's profile and a test that asserts on that profile are fine in sequence and a coin flip in parallel. Either seed per shard or make tests create what they assert on.

Rate limits and quotas. Twelve shards hitting the same staging API multiply your request rate by twelve. Sandboxes for payments and SMS are especially quick to start throttling — and a 429 surfaces in your app as a generic error screen, not as "you are being rate limited."

Device state left behind. Cached logins, granted permissions, notification badges, a half-finished onboarding. Serially, test 40 inherits whatever test 39 left; in parallel it inherits whatever some other shard left. Decide explicitly whether each shard starts from a clean install or a warm known state, and enforce it in setup rather than hoping.

None of these are parallelism bugs. Parallelism just stops your suite from accidentally serialising around them.

Why "just add more parallel runs" is usually the expensive answer

On most device clouds, concurrency is the billed unit: you buy N parallel slots and the suite is shaped around N. That has two consequences that get felt long before the invoice does.

The first is queuing. When all slots are busy, jobs wait — so the number that matters isn't your test duration, it's your time-to-first-device during the 4pm rush when everyone merges.

The second is that you start optimising the wrong thing. Once concurrency is scarce, teams cut coverage to fit the slots they bought, and "which tests run on hardware" becomes a budgeting decision rather than a risk decision.

Our answer is a different unit. Devices here are dedicated to you and priced flat per device per month — not metered per parallel run, per minute, or per slot. A device you're paying for is yours whether it's mid-suite or idle at 3am, which means the shard count is something you tune for wall-clock, not something you ration.

It also means state persists between runs. A shard doesn't have to reinstall and re-sign-in against a freshly wiped machine every time, which cuts straight into that setup_per_device term that was eating your gains.

Wiring it into CI

The matrix is the easy part. This is GitHub Actions, but the shape is identical in GitLab CI or Jenkins:

jobs:
  e2e:
    strategy:
      fail-fast: false
      matrix:
        shard: [1, 2, 3, 4, 5, 6]
    steps:
      - uses: actions/checkout@v4
      - run: npm ci
      - run: node shard.mjs --index ${{ matrix.shard }} --total 6 > shard-tests.txt
      - run: npx wdio run wdio.conf.js --spec-file shard-tests.txt
        env:
          GRID_URL: ${{ secrets.ROBOTACTIONS_GRID_URL }}
          GRID_TOKEN: ${{ secrets.ROBOTACTIONS_TOKEN }}
      - uses: actions/upload-artifact@v4
        if: always()
        with:
          name: results-${{ matrix.shard }}
          path: reports/

Three details that matter more than they look:

fail-fast: false — otherwise the first red shard cancels the other five and you get one failure per run instead of the full picture. On a suite you're trying to stabilise, that turns a one-hour debugging loop into a one-day one.

if: always() on the artifact upload — results from failed shards are the ones you actually need.

And merge the reports. Six separate JUnit XMLs in six artifacts is not a test report; run them through a merge step so the PR gets a single pass/fail with a single list of failures.

What to do this week

1. Record per-test durations for one run. You need the data before any of this is a decision.
2. Find your longest single test. That's your floor — split it or accept it.
3. Measure setup_per_device. If it's over ~90 seconds, fixing it beats adding shards.
4. Move from alphabetical to duration-balanced sharding. It's an afternoon and it's free.
5. Then, and only then, raise the shard count.

Most suites have a comfortable 3–4× sitting in steps 2 through 4 before they need a single extra device.

Run your suite across dedicated devices →