UB Canaries: A collection of C/C++ programs that detect undefined
behavior exploitation by compilers.
To run all tests, type:
run-canaries
For a complete list of command line options:
run-canaries --help
Each directory documents and tests an expectation: something a developer might
-- perhaps unreasonably! -- expect the compiler to do when faced with undefined
behavior. For example, the "uninitialized-variable" directory tests the
expectation that an uninitialized scalar variable will consistently return some
value that is legal for the variable's type.
If any test program in a directory does not display the expected behavior (for a
given compiler version + flags), then that compiler is considered to violate
that expectation.
The toplevel script run-canaries tests the expectations for a
specified collection of compilers and compiler flags. For now, you
need to edit that file to change the set of compilers and flags.
The output of run-canaries is, for each compiler / flag / canary
combination, a bit that tells us whether that compiler has been
observed to exploit that particular UB (in other words, whether it has
been observed violating the expectation). So here, for example, clang
has been seen to exploit signed integer overflows and uninitialized
variables, but not signed left shifts:
clang -O3 signed-left-shift 0
clang -O3 signed-integer-overflow 1
clang -O3 uninitialized-variable 1
Guidelines for writing tests:
-
Each test program should be entirely contained (except for standard header
files) in a single compilation unit. -
An expectation should only be tested by looking at a program's stdout, never
by looking at its assembly code or observing its memory usage or execution
time. -
A test program foo.c may have one or more outputs corresponding to the
"expected" case where the compiler does not exploit that UB. If there is one
such file it should be called foo.output. If there are multiple files they
should be called foo.output1, foo.output2, etc. If the actual output does not
match any of these files, the compiler is assumed to have exploited the UB. -
Every test program must test only a single UB. In other words, each test
program is written in a dialect of C that is completely standard except that a
single behavior (signed integer overflow or whatever) is actually defined
instead of undefined. -
Reliance on implementation-defined behavior is unavoidable, but please avoid
gratuitous reliance such as assuming a particular size for int or long. It is
OK to use the fixed-width types such as int32_t. -
A tricky issue to how much to expose to the optimizer and how much to hide.
There are no particular good rules of thumb that I am aware of, you just have
to try different things. -
An easier issue is how to hide from the optimizer. I suggest introducing a
dependency on argc or on the value loaded from a volatile. Tests may assume
that argc == 1.