[klee-dev] Packaging klee on Nix - gtest broken?

[Julian Büning]

Hi Morgan,

nice to see your packaging efforts for KLEE!

I recently ran into some issues with more recent versions of Google Test 
when building KLEE (and running unit tests). I just opened a PR that 
addresses these: https://github.com/klee/klee/pull/1458

Among these issues is one that I image you may also have run into (as I 
assume your package will not be built against Google Test 1.7.0), but it 
differs quite a bit from the issue that you linked. Thus, I will go 
ahead and describe what I experienced (hoping you can tell me if that 
matches what you saw).

When building KLEE with Google Test 1.7.0 and running the unit tests, I 
get 36 successfully passed tests. When instead using a newer Google Test 
version, like 1.11.0, I get the same number of passed tests, but the 
following 10 unresolved tests in addition:

Unresolved Tests (10):
   KLEE Unit tests :: ./AssignmentTest/Running main() from 
/some/absolute/path/to/gtest_main.cc
   KLEE Unit tests :: ./DiscretePDFTest/Running main() from 
/some/absolute/path/to/gtest_main.cc
   KLEE Unit tests :: ./ExprTest/Running main() from 
/some/absolute/path/to/gtest_main.cc
   KLEE Unit tests :: ./RNGTest/Running main() from 
/some/absolute/path/to/gtest_main.cc
   KLEE Unit tests :: ./RefTest/Running main() from 
/some/absolute/path/to/gtest_main.cc
   KLEE Unit tests :: ./SearcherTest/Running main() from 
/some/absolute/path/to/gtest_main.cc
   KLEE Unit tests :: ./SolverTest/Running main() from 
/some/absolute/path/to/gtest_main.cc
   KLEE Unit tests :: ./TimeTest/Running main() from 
/some/absolute/path/to/gtest_main.cc
   KLEE Unit tests :: ./TreeStreamTest/Running main() from 
/some/absolute/path/to/gtest_main.cc
   KLEE Unit tests :: ./Z3SolverTest/Running main() from 
/some/absolute/path/to/gtest_main.cc

For each of these "tests" I see some earlier output like this:

UNRESOLVED: KLEE Unit tests :: ./AssignmentTest/Running main() from 
/some/absolute/path/to/gtest_main.cc (1 of 46)
******************** TEST 'KLEE Unit tests :: ./AssignmentTest/Running 
main() from /some/absolute/path/to/gtest_main.cc' FAILED 
********************
Unable to find '[  PASSED  ] 1 test.' in gtest output:

Running main() from /some/absolute/path/to/gtest_main.cc
Note: Google Test filter = Running main() from 
/some/absolute/path/to/gtest_main.cc
[==========] Running 0 tests from 0 test cases.
[==========] 0 tests from 0 test cases ran. (0 ms total)
[  PASSED  ] 0 tests.

********************

The last 3 lines look similar to the output in the issue you linked. But 
this is simply the output of Google Test when there are no `TEST()`s 
next to `main()` in an executable. The rest stems from a different 
problem (detailed below).

Is this also the issue you ran into? If yes, maybe you want try the 
patches from the PR I linked above. If not and are you having a 
different problem, maybe you could try to provide some more details? 
Then I will try and see if can help resolve them.

--- BEGIN: More details ---

The issue we see above actually stems from llvm-lit, not from Google 
Test itself. Starting from 1.8.1, Google Test's gtest_main.cc uses 
`__FILE__` [1] instead of a fixed string [2] to output a line like this:
 > Running main() from /some/absolute/path/to/gtest_main.cc

To determine which tests exist, llvm-lit will call each executable with 
the `--gtest_list_tests` argument. However, the (usually) first line 
will be the above "Running main()" output. To skip this, each line is 
compared to "Running main() from gtest_main.cc" [3], which is a fixed 
string assuming the behavior of 1.8.0 and before.

Hence, the line with path will be recorded as a test, and result in a 
corresponding call to the test executable with `--gtest_filter` set 
accordingly. As there is no test that matches the given pattern, we see 
the output shown above. As it does not include the expected "[  PASSED 
] 1 test." line, it is counted as unresolved.

[1] 
https://github.com/google/googletest/blob/release-1.8.1/googletest/src/gtest_main.cc

[2] 
https://github.com/google/googletest/blob/release-1.8.0/googletest/src/gtest_main.cc

[3] 
https://github.com/llvm/llvm-project/blob/llvmorg-13.0.0/llvm/utils/lit/lit/formats/googletest.py#L60-L64

--- END:   More details ---

Looking forward to your answer!

Best,
Julian



On 1/1/22 01:28, Morgan wrote:
> Hey there,
> 
> I like Klee and have been trying to package it in nixpkgs so more
> people can reproducibly use it without resorting to things like setup
> scripts or Docker. Here are the cmake flags I'm using:
> 
> https://github.com/NixOS/nixpkgs/pull/153014/files#diff-cb8d40a4e82c0c50ce6ec4031c12e06a4dac4bded86b9f01afcb2b4f22532dbbR46
> 
> Everything works including the system tests, which is a very good
> sign. However, I'm having trouble with the unit tests that resembles
> this problem:
> 
> https://github.com/google/googletest/issues/2157
> 
> Has anyone else run into this?
> 
> Thanks!
> Morgan
> 
> _______________________________________________
> klee-dev mailing list
> klee-dev at imperial.ac.uk
> https://mailman.ic.ac.uk/mailman/listinfo/klee-dev
>