XCTest.framework not producing errors with the XCTestErrorCodeFailureWhileWaiting code

Originator:brian.s.croom
Number:rdar://25604155 Date Originated:07-Apr-2016
Status:Open Resolved:
Product:Developer Tools Product Version:Xcode 7.3
Classification:Bug Reproducible:Always
 
Summary:
XCTest.framework defines an error code `XCTestErrorCodeFailureWhileWaiting` which is documented as follows:

Indicates that a failure assertion was raised while waiting in `-waitForExpectationsWithTimeout:handler:`.

However, as far as I can tell, this error is not actually produced in this scenario. Instead, when a failure assertion is raised while waiting for expectations, the method continues to wait until the expectation is fulfilled or timeout occurs.

Steps to Reproduce:
1. Clone the following repository and open the XCTestFailureWhileWaitingBug.xcodeproj project: https://github.com/briancroom/RadarSamples (Note: I have also attached a .zip archive containing this project)
2. Perform a Test action on the XcodeTestFailureWhileWaitingBug scheme. This will execute an XCTestCase subclass with a method which causes a test assertion failure to occur while waiting for expectations.
3. Note the console output printing the error code passed to the handler block provided when waiting for expectations. 

Expected Results:
I would expect that the error passed to the handler has a code of 1, corresponding to `XCTestErrorCodeFailureWhileWaiting`, and that this error would be provided almost immediately.

Actual Results:
Instead, the `-waitForExpectationsWithTimeout:handler:` method continues to wait until the timeout occurs, at which point it calls the handler with the `XCTestErrorCodeFailureWhileWaiting` error code.

Regression:
I do not know whether this behavior has changed or if it has always behaved this way.

Notes:
It is very possible that there are other situations in which this error actually is produced. If that turns out to be the case, it may indicate that the documentation for this error code value should be updated to more clearly explain how it is used. I encountered this issue while working to add the XCTest error constants to swift-corelibs-xctest, and noticing that that library does not currently produce this error either.

Comments


Please note: Reports posted here will not necessarily be seen by Apple. All problems should be submitted at bugreport.apple.com before they are posted here. Please only post information for Radars that you have filed yourself, and please do not include Apple confidential information in your posts. Thank you!