Test Coverage reports from xcodebuild CLI ignore coverage from UI tests

Originator:sinophilic
Number:rdar://29033348 Date Originated:10/31/2016
Status:Duplicate of 28900146 (Open) Resolved:
Product:Developer Tools Product Version:
Classification: Reproducible:yes
 
Summary:
A project with both Unit Tests and UI Tests will generate code coverage when ran in the XCode IDE. If the same project is run using `xcodebuild test` from CLI, code coverage from the Unit Tests will exist but coverage from UI Tests will be missing

Sample project with UI Tests and Unit Tests, showing a difference in coverage from `xcrun llvm-cov report` between the project built via IDE and via CLI

https://github.com/iwllyu/UITestCoverageBug 

thelvis4 found a discrepancy between how the two building processes differ - https://github.com/nakiostudio/xcov/issues/22#issuecomment-232014641

Steps to Reproduce:
1. install XCode 8.1 with Swift 3
2. clone the UITestCoverageBug project from https://github.com/iwllyu/UITestCoverageBug 
3. Open the project in XCode and run the tests. See both Unit and UI tests get run. See both `testedViaUnitTest` and `testedViaUITest` are covered
4. in terminal, run `xcrun llvm-cov report` (see github for details)
5. check that coverage for ViewController.swift is 80%
6. in terminal, run `xcodebuild test` for project (see github for details)
7. in terminal, run `xcrun llvm-cov report` (see github for details)
8. check that coverage for ViewController.swift is now 40%

Expected Results:
coverage for ViewController.swift is 80%

Actual Results:
coverage for ViewController.swift is now 40%

Version:
XCode 8.1
OSX 10.11.5

Notes:


Configuration:


Attachments:

Comments

This is still occurring in Xcode 9.2

By alan.grubb at Jan. 17, 2018, 8:31 p.m. (reply...)

Issue appears resolved in Xcode 9

https://github.com/iwllyu/UITestCoverageBug/issues/4

By cory at June 7, 2017, 10:27 p.m. (reply...)

Please note: Reports posted here will not necessarily be seen by Apple. All problems should be submitted at bugreport.apple.com before they are posted here. Please only post information for Radars that you have filed yourself, and please do not include Apple confidential information in your posts. Thank you!