Tests that failed recently should be prioritized on subsequent tests

Originator:jalkut
Number:rdar://31867264 Date Originated:27-Apr-2017 01:54 PM
Status:Open Resolved:
Product:Developer Tools Product Version:Version 8.3.2 (8E2002)
Classification:UI/Usability Reproducible:Always
 
Summary:
It is common when developing unit tests to write a failing test method, proceed to write code that conforms to the test code, and then to repeat the process. This configuration is particularly effective when the "Break on Test Failure" option is set, so that execution stops immediately in the focused test.

Xcode could streamline this process by ordering tests in order such that the test suites with tests that have failed most recently would be run first, shortening the period of time between "Build & Test" and interacting with the test and tested code of interest.

Although Xcode's shortcuts for testing only the selected method or suite are handy for this, it would be clever and welcome if Xcode automatically intuited that some test cases and suites are more likely to be of interest to the developer than others.


Steps to Reproduce:
1. Open attached FailingTestsFirst.xcodeproj
2. Build and Test once, triggering a failing testSecondC() method in FailingTestsSecondTests.swift
3. Look at the test log and observe that the tests have been run in order:

testFirstA()
testFirstB()
testFirstC()
testSecondA()
testSecondB()
testSecondC()

4. Now "Build and Test" again.


Expected Results:
Knowing that testSecondC() has failed recently, it would make sense for Xcode to run it before all the others:

testSecondC()
testFirstA()
testFirstB()
testFirstC()
testSecondA()
testSecondB()

If it's too complicated to reorder tests on a test-by-test basis, then it would be sufficient to reorder such that test *suites* are ordered by recency of failure among their tests.


Observed Results:
The run order remains the same regardless of previous failures. This is no big deal in this trivial example where tests take microseconds to run, but in a large test target choosing to implicitly order the running of tests in "failed recently" order would eliminate a lot of unnecessary delay, without forcing users to remember to run tests on a more granular level.


Version:
Version 8.3.2 (8E2002)


Notes:



Configuration:


Attachments:

Comments

I'll add that for this to work really well, it needs to prioritize even a passing test method, if it has failed recently ... this would ensure that a working test that had previously failed, edited to make it fail again, would still be prioritized in the run order in spite of having succeeded in the preceding test run.


Please note: Reports posted here will not necessarily be seen by Apple. All problems should be submitted at bugreport.apple.com before they are posted here. Please only post information for Radars that you have filed yourself, and please do not include Apple confidential information in your posts. Thank you!