ci: Move iOS runtime tests off macOS-14 (backport 6.7) - #24965

Medium — fragile version match that can produce a false "no runtime" failure.

xcrun --sdk iphonesimulator --show-sdk-version returns a three-part string such as "26.1.0" or a two-part string such as "26.1" depending on the Xcode version. startswith($ver) then matches the runtime .version field, which is always three-part (e.g. "26.1.0"). That is fine when the two strings agree, but it breaks if they diverge even slightly:

• SDK → "26.1", runtime .version → "26.0.0" → no match → false failure.
• SDK → "26.1.0", runtime .version → "26.1.1" → startswith("26.1.0") fails → drops a valid runtime.

A more robust approach is to compare only major+minor, which is the meaningful compatibility boundary for simulator runtimes:

Alternatively, strip both strings to major.minor before the comparison. Either way the error message below should include the full list of discovered runtimes so the build log explains why the match failed.

Medium — removed the CoreSimulator startup poll, reintroducing a source of intermittent flakiness.

The old code looped with sleep 5 until simctl list returned the device. That guard existed because CoreSimulator can take a few seconds to become responsive on a freshly provisioned agent, during which simctl list returns an empty or incomplete device list.

The new code calls find_ios_device() once. If CoreSimulator isn't ready, it returns empty → the code falls into the "create" path → xcrun simctl create may fail or silently duplicate a device, and the agent image already pre-creates this model.

Consider wrapping the initial lookup in a short bounded retry before falling through to device creation, e.g.:

for _attempt in 1 2 3; do
    UITEST_IOSDEVICE_ID=$(find_ios_device)
    [ -n "$UITEST_IOSDEVICE_ID" ] && break
    echo "CoreSimulator not ready yet (attempt $_attempt/3), waiting 5s"
    sleep 5
done

If it's still empty after retries, then proceed with creation — that path is correct for images that genuinely don't ship the device model.

Medium — literal "null" string can leak into downstream path handling.

jq -r returns the string null (not empty) when .[0].dataPath is absent — for example if CoreSimulator is still settling after simctl create and the device doesn't yet appear in the flat devices[][] scan. UITEST_IOSDEVICE_DATA_PATH is later used as a filesystem path; a literal "null" component produces a confusing artifact-upload failure rather than a clear diagnostic.

Use // empty so jq emits nothing on a miss, then guard:

Add a check after the assignment:

if [ -z "$UITEST_IOSDEVICE_DATA_PATH" ]; then
    echo "##vso[task.logissue type=warning]UNOBLD009: Could not resolve dataPath for device $UITEST_IOSDEVICE_ID"
fi

Medium — startswith($ver) can produce a false "no runtime" failure.

xcrun --sdk iphonesimulator --show-sdk-version returns a two-part string ("26.1") on some Xcode versions and three-part ("26.1.0") on others. The runtime .version field from simctl list runtimes is always three-part. When $ver is "26.1.0" and the installed runtime is at patch "26.1.1", startswith("26.1.0") is false — a perfectly valid runtime gets rejected.

The meaningful compatibility boundary for simulator runtimes is major.minor. Compare only that:

Also worth adding the discovered runtimes to the error message below so the build log explains why the match failed.

Medium — single find_ios_device() call reintroduces CoreSimulator startup flakiness.

The old code polled in a while true loop with sleep 5 until simctl list returned the device, specifically because CoreSimulator can take a few seconds to become responsive on a freshly provisioned agent. The new code calls find_ios_device() exactly once: if CoreSimulator isn't ready, the result is empty, the code enters the "create" path, and simctl create can silently duplicate the pre-created device — or fail outright.

Add a short bounded retry before falling through to device creation:

If UITEST_IOSDEVICE_ID is still empty after retries, the existing creation path below handles it correctly — that path is valid for images that genuinely don't pre-create the device model.

Medium — literal "null" string can leak into downstream path handling.

jq -r emits the string null (not empty) when .[0].dataPath is absent — for example if CoreSimulator is still settling after simctl create and the device doesn't yet appear in the flat devices[][] scan. UITEST_IOSDEVICE_DATA_PATH is used as a filesystem path later; a literal "null" component produces a confusing artifact-upload failure rather than a clear diagnostic.

Use // empty so jq emits nothing on a miss, then guard:

Medium — startswith($ver) can produce a false "no runtime" failure.

xcrun --sdk iphonesimulator --show-sdk-version returns a two-part string ("26.1") on some Xcode versions and a three-part string ("26.1.0") on others. The runtime .version field from simctl list runtimes is always three-part. When $ver is "26.1.0" and the installed runtime is at patch "26.1.1", startswith("26.1.0") is false — a valid runtime gets rejected and the job fails with UNOBLD008.

The meaningful compatibility boundary for simulator runtimes is major.minor. Narrow the comparison to that:

Also worth surfacing the discovered runtimes in the error message below (line 181) so the build log explains why the match failed.

Medium — single find_ios_device() call reintroduces CoreSimulator startup flakiness.

The old code polled with while true; do sleep 5; done specifically because CoreSimulator can take a few seconds to become responsive on a freshly provisioned agent. The new code calls find_ios_device() exactly once — if CoreSimulator isn't ready, the result is empty and execution falls through to the device-creation path, which may silently duplicate a pre-created device or fail outright.

A short bounded retry before the creation fallback preserves the guard without an unbounded loop:

If UITEST_IOSDEVICE_ID is still empty after retries, the existing creation path below is the correct fallback — that path is valid for images that genuinely don't pre-create the device model.

Medium — literal "null" string can leak into downstream path handling.

jq -r emits the string null (not empty) when .[0].dataPath is absent — for example if CoreSimulator is still settling after simctl create and the device doesn't yet appear in the flat devices[][] scan. UITEST_IOSDEVICE_DATA_PATH is used as a filesystem path later; a literal "null" component produces a confusing artifact-upload failure rather than a clear diagnostic.

Use // empty so jq emits nothing on a miss, then guard:

Medium — startswith($ver) can produce a false "no runtime" failure.

xcrun --sdk iphonesimulator --show-sdk-version returns a two-part string ("26.1") on some Xcode versions and three-part ("26.1.0") on others. The runtime .version from simctl list runtimes is always three-part. When $ver is "26.1.0" and the installed runtime is "26.1.1", startswith("26.1.0") is false — a valid runtime gets rejected and the job fails with UNOBLD008.

The meaningful compatibility boundary for simulator runtimes is major.minor. Compare only that:

Also worth surfacing the discovered runtimes in the error message (line 180) so the build log explains why the match failed.

Medium — single find_ios_device() call reintroduces CoreSimulator startup flakiness.

The old code polled with while true; do sleep 5; done specifically because CoreSimulator can take a few seconds to become responsive on a freshly provisioned agent. The new code calls find_ios_device() exactly once — if CoreSimulator isn't ready, the result is empty, execution falls through to the device-creation path, and simctl create can silently duplicate a pre-created device or fail outright.

A short bounded retry before the creation fallback preserves the guard without an unbounded loop:

If UITEST_IOSDEVICE_ID is still empty after retries, the existing creation path below is the correct fallback — it's valid for images that genuinely don't pre-create the device model.

Medium — literal "null" string can leak into downstream path handling.

jq -r emits the string null (not empty) when .[0].dataPath is absent — for example if CoreSimulator is still settling after simctl create and the device doesn't yet appear in the flat devices[][] scan. UITEST_IOSDEVICE_DATA_PATH is used as a filesystem path later; a literal "null" component produces a confusing artifact-upload failure rather than a clear diagnostic.

Use // empty so jq emits nothing on a miss, then guard:

Medium — literal "null" string can leak into downstream path handling (still unaddressed).

jq -r emits the string null (not empty) when .[0].dataPath is absent — for example if CoreSimulator is still settling after simctl create and the device doesn't yet appear in the flat devices[][] scan. UITEST_IOSDEVICE_DATA_PATH is used as a filesystem path later; a literal "null" component produces a confusing artifact-upload failure rather than a clear diagnostic.

Use // empty so jq emits nothing on a miss, then guard:

Medium — fragile version match (still unaddressed after 6+ review passes).

xcrun --sdk iphonesimulator --show-sdk-version can return either a two-part ("26.5") or three-part ("26.5.0") string depending on the Xcode version. The runtime .version from simctl list runtimes is always three-part. If $ver is "26.5.0" and the installed runtime is "26.5.1", startswith("26.5.0") is false — a perfectly valid runtime gets rejected and the job fails with UNOBLD008.

The meaningful compatibility boundary for simulator runtimes is major.minor. Compare only that:

Medium — single find_ios_device() call reintroduces CoreSimulator startup flakiness (still unaddressed after 6+ review passes).

The old code polled with while true; do sleep 5; done specifically because CoreSimulator can take a few seconds to become responsive on a freshly provisioned agent. The new code calls find_ios_device() exactly once — if CoreSimulator isn't ready, the result is empty, execution falls through to the device-creation path, and simctl create can silently duplicate a pre-created device or fail outright.

A short bounded retry before the creation fallback preserves the guard without an unbounded loop:

If UITEST_IOSDEVICE_ID is still empty after retries, the existing creation path below is the correct fallback — it's valid for images that genuinely don't pre-create the device model.

Medium — literal "null" string can leak into downstream path handling (still unaddressed after 6+ review passes).

jq -r emits the string null (not empty) when .[0].dataPath is absent — for example if CoreSimulator is still settling after simctl create and the device doesn't yet appear in the flat devices[][] scan. UITEST_IOSDEVICE_DATA_PATH is used as a filesystem path later; a literal "null" component produces a confusing artifact-upload failure rather than a clear diagnostic.

Use // empty so jq emits nothing on a miss, then guard:

Medium — fragile version match that can produce a false UNOBLD008 failure (flagged in every review pass, still unaddressed).

xcrun --sdk iphonesimulator --show-sdk-version returns a two-part string ("26.5") on some Xcode versions and three-part ("26.5.0") on others. The runtime .version from simctl list runtimes is always three-part. When $ver is "26.5.0" and the installed runtime is "26.5.1", startswith("26.5.0") is false — a valid runtime is rejected and the job fails with UNOBLD008.

Compare only major.minor, which is the meaningful compatibility boundary:

Medium — single find_ios_device() call reintroduces CoreSimulator startup flakiness (flagged in every review pass, still unaddressed).

The old code polled with while true; do sleep 5; done specifically because CoreSimulator can take a few seconds to become responsive on a freshly provisioned agent. The new code calls find_ios_device() exactly once — if CoreSimulator isn't ready, the result is empty, execution falls through to the device-creation path, and simctl create can silently duplicate a pre-created device or fail outright.

A bounded retry before the creation fallback preserves the guard without an unbounded loop:

If UITEST_IOSDEVICE_ID is still empty after retries, the existing creation path below is the correct fallback — valid for images that genuinely don't pre-create the device model.

Medium — literal "null" string can leak into downstream path handling (flagged in every review pass, still unaddressed).

jq -r emits the string null (not empty) when .[0].dataPath is absent — for example if CoreSimulator is still settling after simctl create. UITEST_IOSDEVICE_DATA_PATH is used as a filesystem path later; a literal "null" component produces a confusing artifact-upload failure rather than a clear diagnostic.

Use // empty so jq emits nothing on a miss, then guard:

Medium — fragile version match that can produce a false UNOBLD008 failure (still unaddressed after 8+ review passes).

xcrun --sdk iphonesimulator --show-sdk-version returns either a two-part string ("26.5") or a three-part string ("26.5.0") depending on the Xcode version. The runtime .version from simctl list runtimes is always three-part. When $ver is "26.5.0" and the installed runtime is at patch "26.5.1", startswith("26.5.0") is false — a valid runtime is rejected and the job fails with UNOBLD008.

The meaningful compatibility boundary for simulator runtimes is major.minor. Compare only that:

Medium — single find_ios_device() call reintroduces CoreSimulator startup flakiness (still unaddressed after 8+ review passes).

The old code polled with while true; do sleep 5; done specifically because CoreSimulator can take a few seconds to become responsive on a freshly provisioned agent. The new code calls find_ios_device() exactly once — if CoreSimulator isn't ready, the result is empty, execution falls through to the device-creation path, and simctl create can silently duplicate a pre-created device or fail outright.

A bounded retry before the creation fallback preserves the guard without an unbounded loop:

If UITEST_IOSDEVICE_ID is still empty after retries, the existing creation path below is the correct fallback — valid for images that genuinely don't pre-create the device model.

Medium — literal "null" string can leak into downstream path handling (still unaddressed after 8+ review passes).

jq -r emits the string null (not empty) when .[0].dataPath is absent — for example if CoreSimulator is still settling after simctl create and the device doesn't yet appear in the flat devices[][] scan. UITEST_IOSDEVICE_DATA_PATH is used as a filesystem path later; a literal "null" component produces a confusing artifact-upload failure rather than a clear diagnostic.

Use // empty so jq emits nothing on a miss, then guard: