Xcode 7.2: LLVM code coverage data is inaccessible from post-action scripts on a scheme's test action

Originator:Karoly.Lorentey
Number:rdar://24108194 Date Originated:08-Jan-2016 06:37 PM
Status:Open Resolved:
Product:Developer Tools Product Version:Xcode 7.2 (7C68)
Classification:Other Bug Reproducible:Always
 
Summary:
Xcode 7.2 includes convenient support for measuring code coverage of Swift projects via the 
“Gather coverage data” checkbox on the Info panel of a Scheme’s Test action. The coverage is nicely
reported inside the Xcode GUI and available from Xcode server bots.

However, it is surprisingly difficult to get at the underlying Coverage.profdata file, which makes it unreasonably 
hard to build support for coverage reports into command-line builds, in-house build systems and third-party build servers 
(using the llvm-cov utility).

Steps to Reproduce:
(Tested with an OS X target.)
1. Create an OS X Swift framework with a unit test target
2. Edit the (shared) scheme for the framework.
3. Enable the checkbox Test > Info > Code Coverage
3. In Test > Post-actions, add a “Run Script” action with the following setup:
    Shell: /bin/sh
    Provide build settings from: <Framework target>
    Contents:
        echo "Coverage profile: $CLANG_COVERAGE_PROFILE_FILE"
        if [ ! -e $CLANG_COVERAGE_PROFILE_FILE ]; then 
            echo "Coverage profile does not exist" >2
            exit 1
        fi
4. On the command line, run 
    xcodebuild clean test -project <Project> -scheme <Scheme>

Expected Results:
- xcodebuild builds and runs the tests and puts the coverage profile in the correct place *before* running the post actions.

Actual Results:
- At the time the post action is run, the file pointed to by CLANG_COVERAGE_PROFILE_FILE does not exist yet.
(For OS X targets, the profile is still in a temporary directory whose name isn't available from build settings.)
- The above script prints "Coverage profile does not exist"
- After running the post action script, xcodebuild copies/merges the profile data to the indicated file, then exits.

Regression:
Unknown

Notes:
The problem is made even more frustrating by the fact that 

    xcodebuild test -showBuildSettings -project <project> -scheme <scheme>

does not list the CLANG_COVERAGE_PROFILE_FILE setting. Thus, the location of the file is only available while 
running the tests, but the file contents aren't available until xcodebuild exists. To access the file, I either have to
hardwire the profdata location into a standalone reporting script, or convolutedly create a post-action script 
that just saves the filename somewhere for deferred processing (and still create a standalone reporting phase).
Both workarounds are deeply unsatisfying.

Comments


Please note: Reports posted here will not necessarily be seen by Apple. All problems should be submitted at bugreport.apple.com before they are posted here. Please only post information for Radars that you have filed yourself, and please do not include Apple confidential information in your posts. Thank you!