Open Radar

 
Summary:
Benchmark tests should probably not cohabit in the same bundle as unit tests. Xcode’s templates should set them up in a separate bundle, with the scheme configured to run them in release config automatically—there’s not much point in testing & measuring an unoptimized build when your users are going to be running an optimized one.

Additionally they probably shouldn’t be run at the same time as the unit tests by default, since you’re probably going to want a high number of iterations to get better data out of the tests, and that would be counterproductive in the unit tests that you want to run with every build.

I think they should be treated as a sort of integration test, and that a bundle should exist for these in the default templates, and that you should run them with a different key shortcut than the ⌘U used for unit tests. Xcode bots should probably run them all, tho.


Steps to Reproduce:
N/A

Expected Results:
N/A

Actual Results:
N/A

Regression:
N/A

Notes:
N/A