Unit tests always run in the same order, but should be randomized

Originator:steipete
Number:rdar://34876681 Date Originated:08-Oct-2017 10:42 PM
Status:Open Resolved:
Product:Developer Tools Product Version:
Classification:Security Reproducible:Always
 
Summary:
This is a duplicate of radar #34873492

Unit tests should be able to be run based on a random order, to prevent application state from leaking between tests. Ideally, the tests could be run with a seed that is logged, and can be re-used to generate the same order. This isn’t possible in Xcode but it’s pretty integral to the way that developers in Ruby and JavaScript test their code.

For example, consider two tests that test code that reads and writes UserDefaults. If the tests are always run in the same order, it’s possible to get false positives order get different results when running only a single test.


Steps to Reproduce:
 

Expected Results:
 

Actual Results:
 

Version:
 

Notes:

Comments


Please note: Reports posted here will not necessarily be seen by Apple. All problems should be submitted at bugreport.apple.com before they are posted here. Please only post information for Radars that you have filed yourself, and please do not include Apple confidential information in your posts. Thank you!