Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

std::process::exit() on MSVC bypasses writing LLVM InstrProf counters to profraw file #77553

Open
richkadel opened this issue Oct 4, 2020 · 1 comment
Labels
A-code-coverage Area: Source-based code coverage (-Cinstrument-coverage) A-process Area: `std::process` and `std::env` C-bug Category: This is a bug. O-windows-msvc Toolchain: MSVC, Operating system: Windows

Comments

@richkadel
Copy link
Contributor

richkadel commented Oct 4, 2020

This bug does not occur on Linux or MacOS.

Note that I observed this unexpected behavior when compiling on Windows, targeting MSVC.

I have not confirmed the behavior on Windows targeting GNU.

The minimal reproducible example (MRE) here shows the problem even when returning a success status code (0), and while a Rust program will generate a 0 status for programs that return a Result::OK or a (), and a 1 status for programs that return Result::Err, as far as I know, we need to call std::process::exit(...) to return any other status code.

Therefore, any program that requires the ability to return a status other than 0 or 1 may get empty, or worse, incomplete, coverage results when testing coverage under MSVC.

I tried this code:

fn main() {
    std::process::exit(0);
}
$ build/x86_64-pc-windows-msvc/stage1/bin/rustc.exe -Zinstrument-coverage basic.rs
$ LLVM_PROFILE_FILE=basic.profraw ./basic.exe
$ ls -l basic.profraw
-rw-r--r-- 1 richkadel Domain Users 0 Oct  4 15:32 basic.profraw

I expected to see this happen: The same result as on Linux and MacOS, and the same result that I get if the call to std::process::exit(0) is removed. In both cases, the basic.profraw file size is greater than 0. In the working cases, the non-empty basic.profraw file can be used with llvm-profdata and llvm-cov tools to generate and view coverage reports.

Here is the expected result on Linux, for example:

$ build/x86_64-unknown-linux-gnu/llvm/bin/llvm-profdata merge -sparse basic.profraw -o basic.profdata
$ build/x86_64-unknown-linux-gnu/llvm/bin/llvm-cov show --instr-profile=basic.profdata basic
    1|      1|fn main() {
    2|      1|    std::process::exit(0);
    3|      1|}

Instead, this happened: The basic.profraw file size is 0 (as shown in the example above). LLVM coverage reports show no results.

Meta

./build/x86_64-unknown-linux-gnu/stage1/bin/rustc --version --verbose
rustc 1.49.0-dev
binary: rustc
commit-hash: unknown
commit-date: unknown
host: x86_64-unknown-linux-gnu
release: 1.49.0-dev
LLVM version: 11.0
@richkadel richkadel added the C-bug Category: This is a bug. label Oct 4, 2020
@jonas-schievink jonas-schievink added the O-windows-msvc Toolchain: MSVC, Operating system: Windows label Oct 4, 2020
richkadel added a commit to richkadel/rust that referenced this issue Oct 5, 2020
This is a combination of 18 commits.

Commit #2:

Additional examples and some small improvements.

Commit #3:

fixed mir-opt non-mir extensions and spanview title elements

Corrected a fairly recent assumption in runtest.rs that all MIR dump
files end in .mir. (It was appending .mir to the graphviz .dot and
spanview .html file names when generating blessed output files. That
also left outdated files in the baseline alongside the files with the
incorrect names, which I've now removed.)

Updated spanview HTML title elements to match their content, replacing a
hardcoded and incorrect name that was left in accidentally when
originally submitted.

Commit #4:

added more test examples

also improved Makefiles with support for non-zero exit status and to
force validation of tests unless a specific test overrides it with a
specific comment.

Commit #5:

Fixed rare issues after testing on real-world crate

Commit #6:

Addressed PR feedback, and removed temporary -Zexperimental-coverage

-Zinstrument-coverage once again supports the latest capabilities of
LLVM instrprof coverage instrumentation.

Also fixed a bug in spanview.

Commit #7:

Fix closure handling, add tests for closures and inner items

And cleaned up other tests for consistency, and to make it more clear
where spans start/end by breaking up lines.

Commit #8:

renamed "typical" test results "expected"

Now that the `llvm-cov show` tests are improved to normally expect
matching actuals, and to allow individual tests to override that
expectation.

Commit #9:

test coverage of inline generic struct function

Commit #10:

Addressed review feedback

* Removed unnecessary Unreachable filter.
* Replaced a match wildcard with remining variants.
* Added more comments to help clarify the role of successors() in the
CFG traversal

Commit #11:

refactoring based on feedback

* refactored `fn coverage_spans()`.
* changed the way I expand an empty coverage span to improve performance
* fixed a typo that I had accidently left in, in visit.rs

Commit #12:

Optimized use of SourceMap and SourceFile

Commit #13:

Fixed a regression, and synched with upstream

Some generated test file names changed due to some new change upstream.

Commit #14:

Stripping out crate disambiguators from demangled names

These can vary depending on the test platform.

Commit #15:

Ignore llvm-cov show diff on test with generics, expand IO error message

Tests with generics produce llvm-cov show results with demangled names
that can include an unstable "crate disambiguator" (hex value). The
value changes when run in the Rust CI Windows environment. I added a sed
filter to strip them out (in a prior commit), but sed also appears to
fail in the same environment. Until I can figure out a workaround, I'm
just going to ignore this specific test result. I added a FIXME to
follow up later, but it's not that critical.

I also saw an error with Windows GNU, but the IO error did not
specify a path for the directory or file that triggered the error. I
updated the error messages to provide more info for next, time but also
noticed some other tests with similar steps did not fail. Looks
spurious.

Commit #16:

Modify rust-demangler to strip disambiguators by default

Commit #17:

Remove std::process::exit from coverage tests

Due to Issue rust-lang#77553, programs that call std::process::exit() do not
generate coverage results on Windows MSVC.

Commit #18:

fix: test file paths exceeding Windows max path len
@wesleywiser wesleywiser added the A-code-coverage Area: Source-based code coverage (-Cinstrument-coverage) label Oct 22, 2021
@wucke13
Copy link

wucke13 commented Jan 7, 2023

I believe that I see this bug on Linux, contradicting the

This bug does not occur on Linux or MacOS.

claim. Further info here: taiki-e/cargo-llvm-cov#235

Anything I can do to help?

@workingjubilee workingjubilee added the A-process Area: `std::process` and `std::env` label Jul 22, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
A-code-coverage Area: Source-based code coverage (-Cinstrument-coverage) A-process Area: `std::process` and `std::env` C-bug Category: This is a bug. O-windows-msvc Toolchain: MSVC, Operating system: Windows
Projects
None yet
Development

No branches or pull requests

5 participants