Open Radar

 
Summary:
XCTest is a good baseline for unit testing frameworks, and one that I am very happy to see Apple actively developing. The adoption of better testing facilities at the IDE level is, I think, the best way to encourage better testing practices within the community, and that can only be good for the users, the developers, and for Apple. Therefore, bravo!

At the same time, there are opportunities for experiments to be undertaken that would rightly be outside the scope of XCTest but which could be integrated with Xcode by third-party developers without investment in a full-fledged IDE plugin (and I for one would be happy to avoid writing plugins against a non-public plugin API).

For example, I am writing an open source Objective-C unit testing framework called Lagrangian[1] with a couple of novel experiments; the tests are interleaved with your code, but are compiled out of release builds altogether; they have automatically-derived[2] human-readable names; and they are available within debug builds and could be integrated with e.g. a menu to run and inspect them at runtime in the app.

Here are some things that would ease the integration of custom testing frameworks or otherwise improve the experience of testing in Xcode. Some of these are more involved, but I’ve tried to focus on ideas that would seem not to require the Xcode team to do all of the legwork; i.e. changes that wouldn’t imply support for third-party testing frameworks, but rather would just lend a little more leniency and openness to them, enabling motivated developers to build and modify their own testing solutions while enjoying the excellent UI affordances in Xcode 5. There’s no end to what third parties can and will do to leverage every last tool you give them, and we’ll be ever the more grateful to you for it :) Without further adieu:

1. Running the tests: I have made a “dummy” test target template which points TEST_HOST at a command-line tool distributed with the testing framework and some other options to trick Xcode into running Lagrangian tests. While this works acceptably, the requirement of adding a unit test bundle target which does not actually contain any code (or even a compilation phase) is inconvenient at best. Instead, I would like:
	1.1. Some combination of build setting and scheme setting or similar mechanism to indicate to Xcode that unit tests exist *without* requiring the addition of a dummy unit test bundle target. For example, a checkbox in the scheme indicating the existence of testing intrinsic to the built target would suffice; or perhaps the presence of some build setting.
	1.2. Some build setting or similar mechanism to indicate to Xcode how it should trigger the tests. For example, a TEST_TOOL build setting could be added which would name a command-line tool to execute (with Xcode’s usual inclusion of build settings in its environment!); Other Testing Flags could then be passed to it, allowing the developer to specify any other information that would be needed to trigger a build correctly, e.g. “run tests against such and such a server.” (Xcode’s testing integration is useful for far more than merely unit tests!)

2. Listing the tests: With Lagrangian, the unit tests are not methods on classes subclassing a known test suite class. Instead, they exist as constructor functions triggering the creation of objects to run the tests, named with human-readable strings. The test navigator will list Lagrangian test case names for a moment when the tests are run, but they disappear immediately afterwards. Ideally, I would like to be able to provide a tool or some other mechanism to list all the tests associated with a target offline (i.e. without the tests being run), but in the absence of that, it would go a long, long ways to simply continue displaying the tests which were discovered via the console output when running; re-running the tests would then replace the list shown in the test navigator. I expect this would cover a wide variety of testing frameworks which already provide output in some way compatible with SenTestingKit’s.

3. Interacting with the tests: With XCTest, you get little green checkmarks beside passing tests in the margin at left, and little red x’s beside failing ones. Clicking these runs the tests. I would love to be able to get these with Lagrangian by e.g. logging source references[3], but it doesn’t seem to be possible since my test cases are not methods on some class, and thus Xcode can’t find and annotate them. This would also seem to require some mechanism for Xcode to indicate which specific tests should be run; adding this to an environment variable would be more than sufficient for third-party developers to grab and use for our own nefarious purposes.

4. Display of individual tests: When Xcode 4 was introduced, the parser for test output was made significantly stricter. You could no longer monkeypatch SenTestingKit to print human readable (if somewhat garbled) sentences from your test case names; they wouldn’t be recognized as being tests. With Lagrangian I am using arbitrary (and intended to be human-readable) strings as test case names (and now automatically generated ones!), meaning that I have to butcher the formatting; instead of “array.count should equal 1.” I have to print “-[SuiteName array_count_should_equal_1_] if I want Xcode to show the test in the logs. I think that it would be much better if the parser were made more lenient. I would like to be able to print a log message including the file and line on which the test case starts and an almost-arbitrary string to display as the test’s name. For example (integrating the ideas from #3 above):

	/path/to/file.m:123: test case started: …everything up to the newline is effectively the test case name

Alternatively, allowing me to use the existing format but with anything between the quotes, and even better, handling escaping of single quotes correctly, would be a reasonable measure to have Xcode print nice, human-readable test case names in the log without requiring sweeping changes to the log formats parsed. Requiring this (slightly relaxed) format but allowing the format described above as an optional add-on indicating the name and location of the test would be an excellent combination.

5. Display of assertion failures: When a test assertion fails and the message includes colons, Xcode prints everything from *the last colon* to the end of the line. This is bad at both ends: a message with a colon (e.g. ‘"important note about error:" %@, error’) has useful information stripped from Xcode’s display. Altering the parser to collect everything after “/…/file.m:123: error:” on would avoid losing this information. On the other end of the string, an error message logging an array will generally only show (, not anything useful about the array’s contents. Altering the parser to attempt to balance parentheses, or to accumulate results up to the next assertion failure, test case end, or NSLog-style message would allow Xcode to display the contents of the array—vital context!— in the little bubbles in the editor. This would be *magnificently* helpful when reading and writing tests.

6. Run tests frequently: I would like to have the option of running my tests any time the code compiles. Do it speculatively: try to build and run my tests in the background while I’m working on the code. Run the tests for this file any time I save it. And so on. This isn’t strictly to do with non-Xcode testing, but I think it would be an extremely useful addition to the toolbox. 

7. Continuous Integration of much of the above: In particular, I would like Xcode bots and xcodebuild to know about build settings for running tests with custom tooling. Thus, my tests can be run on the CI server for everyone’s benefit.


In conclusion, I am so glad to see the Xcode team putting so much focus on unit testing. I hope some of these ideas or similar ones could help catalyze this even further.

Thank you for your time and consideration!

Best regards,
Rob


Steps to Reproduce:
N/A

Expected Results:
N/A

Actual Results:
N/A

Regression:
N/A

Notes:

1: https://github.com/robrix/Lagrangian
2: This is in a branch which I’m working on at the moment, and may not actually be represented on github as of the time of writing. Nevertheless, it is the intended workflow, and the existing develop branch allows arbitrary strings to serve as test names.
3: e.g. “/Absolute/Or/Maybe/SRCROOT/Relative/Path/To/Source/File.m:123: test name goes here”