Open Radar

 
Summary:
When building and running tests on a slower machine, the test bundle fails to execute. Retrying can fix the issue. But this is particularly problematic when relying on a Continuous Integration service. Flaky test running defeats the purpose of having tests in the first place. Even if you run them on a slow machine, they should still work but just take longer to execute.

Steps to Reproduce:
Preconditions:
Xcode 8.1+
Low resource machine (4GB, single core)

1. Open the attached workspace
2. Run tests
3. If they succeed, clean and run them again until the test bundle fails to execute.

Expected Results:
The test bundle should run every time

Actual Results:
The test bundle fails to execute. Inside the logs we see:

2016-12-21 06:13:08.865 xcodebuild[4070:11608] Connection peer refused channel request for "dtxproxy:XCTestManager_IDEInterface:XCTestManager_DaemonConnectionInterface"; channel canceled <DTXChannel: 0x7fa41e2a2780>
[06:13:08]: ▸ 2016-12-21 06:13:08.868 xcodebuild[4070:11461] Error Domain=IDETestOperationsObserverErrorDomain Code=3 "The operation couldn’t be completed. (DTXProxyChannel error 1.) If you believe this error represents a bug, please attach the log file at /Users/distiller/Library/Developer/Xcode/DerivedData/PlayPlex-cgxrnstnbwmdknfkvwpxgpdcqecz/Logs/Test/E2081E04-E22D-471F-B0CA-813C34FD89CB/Session-PlayPlexUnitTests-2016-12-21_061135-H5xOay.log" UserInfo={NSLocalizedDescription=The operation couldn’t be completed. (DTXProxyChannel error 1.) If you believe this error represents a bug, please attach the log file at /Users/distiller/Library/Developer/Xcode/DerivedData/PlayPlex-cgxrnstnbwmdknfkvwpxgpdcqecz/Logs/Test/E2081E04-E22D-471F-B0CA-813C34FD89CB/Session-PlayPlexUnitTests-2016-12-21_061135-H5xOay.log}

Checking the log file referenced in this error we see:

00:53:18.769 xcodebuild[4049:10877] Starting test session with ID <__NSConcreteUUID 0x7f9eaec6ee70> BB412F85-6BAF-4713-916E-D91CB0BFC424
00:53:18.769 xcodebuild[4049:10877] Checking test manager availability..., will wait up to 120s
00:54:33.980 xcodebuild[4049:10958] Closing socket 17 to testmanagerd for Sim iPhone 6s
00:54:33.988 xcodebuild[4049:11100] Error from testmanagerd: The operation couldn’t be completed. (DTXProxyChannel error 1.) ((null))
00:54:33.989 xcodebuild[4049:10877] Test operation failure: The operation couldn’t be completed. (DTXProxyChannel error 1.)
00:54:33.989 xcodebuild[4049:10877] _finishWithError:Error Domain=IDETestOperationsObserverErrorDomain Code=3 "The operation couldn’t be completed. (DTXProxyChannel error 1.)" UserInfo={NSLocalizedDescription=The operation couldn’t be completed. (DTXProxyChannel error 1.)} didCancel: 1

It seems that the simulator isn't starting in time for testmanagerd to connect to it, and testmanagerd times out.

Version:
macOS 10.12
Xcode 8.1 and 8.2

Notes:
Opening the simulator with a command like:

xcrun instruments -w 'iPhone 6s (10.1)'

And then waiting for 60s+ before attempting to run the tests will reduce the frequency of the issue but it doesn't eliminate it entirely.

Configuration:
Hardware:
Single core
4GB RAM
Non-SSD HD