Open Radar

 
Summary:
In order to build a custom DSL atop the XCTest framework, I created a subclass of XCTestCase that overrides +[XCTestCase testInvocations]. The overridden method uses the objc/runtime.h function class_addMethod to add methods to the test case at runtime, and then returns an array of invocations that execute the new methods.

This works like a charm. The latest version of the XCTest framework that is shipped with Xcode 6 displays the name of the selector in the test gutter. There is one problem, however: when the selector name changes in a subsequent test run, it is displayed alongside the old selector name. The old selector name is displayed with a gray check mark, indicating that the test was not executed.

Ideally, instead of a gray check mark, Xcode should determine that the test in question no longer exists, and remove the selector name from the test navigator altogether.

Steps to Reproduce:
1. Download the source code for the following project from GitHub: https://github.com/modocache/Quick
2. Open the Quick.xcodeproj file in the latest version of Xcode
3. Run the unit tests for the "Quick" scheme. See that the test navigator displays a series of passing tests.
4. Open QuickTests/Matchers/ContainSpec.swift, and change the following line:
```
it("says it expected subject to contain expected") {
```

...to:

```
it("renamed example") {
```

5. Save the file, then run the unit tests again.

Expected Results:
The test navigator displays a passing test result for a selector containing the string "renamed_example". The result for a selector named "says_it_expected_subject_to_contain_expected" is not displayed.

Actual Results:
The test navigator displays a passing test result for a selector containing the string "renamed_example". It also displays a result for a selector named "says_it_expected_subject_to_contain_expected". This result is displayed with a gray checkmark, indicating the test did not run.

See the attached screenshot for an example.

Closing Xcode clears all selector names from the test navigator.

Version:
Xcode Version 6.0 (6A215l) & OS X 10.9.3 (13D65)

Notes:
It appears Xcode looks at methods defined in the source code when determining what to display in the test navigator. Closing Xcode and then reopening it "reloads" these methods.

Would it be possible to reload the methods immediately prior to when the test suite is run? I think that would solve my problem. In my case, all test methods are defined at runtime, so a "reload" immediately prior to a test run would clear out all methods, thus "refreshing" the test navigator.

Configuration:


Attachments: