XCTestObservation crashes if removing an observer during a notification

Originator:nicolas.bouilleaud
Number:rdar://23591437 Date Originated:2015-11-18
Status:Closed Resolved:2016-01-12
Product:Developer Tools Product Version:7.1 (7B91b)
Classification:Crash Reproducible:Always
 
XCTestObservationCenter crashes if we try to remove a test observer from within "testCaseDidFinish" call.

Steps to Reproduce:
See attached minimal sample project. We simulate a "BackendTest" that uses a fake "BackendMock". It uses XCTestObservation to automatically stop mocking after the test finishes. This pattern is very useful, for example to simulate all network requests with an NSURLProtocol during tests.

Expected Results:
Do not crash and unregister the "BackendMock".

Actual Results:
Crash:

<unknown>:0: error: BackendTest : *** Collection <__NSSetM: 0x1002b5e70> was mutated while being enumerated.
(
	0   CoreFoundation                      0x00007fff96407ae2 __exceptionPreprocess + 178
	1   libobjc.A.dylib                     0x00007fff8d58973c objc_exception_throw + 48
	2   CoreFoundation                      0x00007fff9640737c __NSFastEnumerationMutationHandler + 300
	3   XCTest                              0x0000000100034801 -[XCTestObservationCenter _testCaseDidStop:] + 432
	4   XCTest                              0x000000010001dc55 -[XCTestCaseRun stop] + 85
	5   XCTest                              0x0000000100022869 -[XCTestCase performTest:] + 669
	6   XCTest                              0x0000000100020458 -[XCTestSuite performTest:] + 377
	7   XCTest                              0x0000000100020458 -[XCTestSuite performTest:] + 377
	8   XCTest                              0x0000000100020458 -[XCTestSuite performTest:] + 377
	9   XCTest                              0x000000010000e822 __25-[XCTestDriver _runSuite]_block_invoke + 51
	10  XCTest                              0x0000000100033807 -[XCTestObservationCenter _observeTestExecutionForBlock:] + 611
	11  XCTest                              0x000000010000e76b -[XCTestDriver _runSuite] + 408
	12  XCTest                              0x000000010000f31a -[XCTestDriver _checkForTestManager] + 696
	13  XCTest                              0x0000000100056c69 _XCTestMain + 628
	14  xctest                              0x0000000100001dc9 xctest + 7625
	15  libdyld.dylib                       0x00007fff8b3905ad start + 1
	16  ???                                 0x0000000000000005 0x0 + 5
)


Version:
Xcode Version 7.1 (7B91b), OS X 10.11.2 (15C40a)

Notes:
The exception (__NSSetM was mutated while being enumerated) seems to indicate that the solution isn’t complicated. I’m a bit surprised by this crash, actually. When else would be a right time to unregister a test observer?

Configuration:
Any Mac running Xcode

Attachments:
'XCTestObservationRadar.zip' was successfully uploaded.

Comments

Fixed in Xcode 7.3 beta

yay

By nicolas.bouilleaud at Jan. 12, 2016, 9:08 a.m. (reply...)

Attachments

http://cl.ly/0f3s3d03353L

By nicolas.bouilleaud at Nov. 18, 2015, 10:49 a.m. (reply...)

Please note: Reports posted here will not necessarily be seen by Apple. All problems should be submitted at bugreport.apple.com before they are posted here. Please only post information for Radars that you have filed yourself, and please do not include Apple confidential information in your posts. Thank you!