[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]
Date: Mon, 1 Jul 2013 22:58:57 +0200
From: Szabolcs Nagy <nsz@...t70.net>
To: musl@...ts.openwall.com
Subject: Re: Request for volunteers
* Rich Felker <dalias@...ifal.cx> [2013-06-30 23:51:17 -0400]:
> This could involve tasks like:
>
> - Reading the git log and making a list of noteworthy bugs to add
> regression tests for.
>
> - Determining (perhaps via automated coverage tools) major code we
> lack any testing for.
>
> - Cross-checking glibc bug tracker and/or glibc tests for issues that
> should be checked in musl too.
i was thinking about this and a few category of tests:
functional:
black box testing of libc interfaces
(eg input-output test vectors)
tries to achieve good coverage
regression:
tests for bugs we found
sometimes the bugs are musl specific or arch specific so
i think these are worth keeping separately from the general
functional tests (eg same repo but different dir)
static:
testing without executing code
eg check the symbols and types in headers
(i have tests that cover all posix headers only using cc)
may be the binaries can be checked in some way as well
static-src:
code analyzis can be done on the source of musl
(eg sparse, cppcheck, clang-analyzer were tried earlier)
(this is different from the other tests and probably should
be set up separately)
metrics:
benchmarks and quality of implementation metrics
(eg performance, memory usage the usual checks but even
ulp error measurements may be in this category)
other tools:
a coverage tool would be useful, i'm not sure if anyone
set up one with musl yet (it's not just good for testing,
but also for determining what interfaces are actually in
use in the libc)
clever fuzzer tools would be nice as well, but i don't know
anything that can be used directly on a library (maybe with
small effort they can be used to get better coverage)
as a first step i guess we need to start with the functional
and regression tests
design goals of the test system:
- tests should be easy to run even a single test in isolation
(so test should be self contained if possible)
- output is a report, failure cause should be clear
- the system should not have external dependencies
(other than libc, posix sh, gnu make: so tests are in .c files with
simple buildsystem or .sh wrapper)
- the failure of one test should not interfere with other tests
(so tests should be in separate .c files each with main() and
narrow scope, otherwise a build failure can affect a lot of tests)
- the test system should run on all archs
(so arch specific and implementation defined things should be treated
carefully)
- the test results should be robust
(failures are always reported, deterministically if possible)
- tests should leave the system in clean state
(or easily cleanable state)
some difficulties:
- some "test framework" functionality would be nice, but can be
problematic: eg using nice error reporting function on top of stdio
may cause loss of info because of buffering in case of a crash
- special compiler or linker flags (can be maintained in makefile
or in the .c files as comments)
- tests may require special environment, filesystem access, etc
i'm not sure what's the best way to manage that
(and some tests may need two different uid or other capabilities)
- i looked at the bug history and many bugs are in hard to
trigger cornercases (eg various races) or internally invoke ub
in a way that may be hard to verify in a robust way
- some tests may need significant support code to achieve good
coverage (printf, math, string handling close to 2G,..)
(in such cases we can go with simple self-contained tests without
much coverage, but easy maintainance, or with something
sophisticated)
does that sound right?
i think i can reorganize my libc tests to be more "scalabe"
in these directions..
Powered by blists - more mailing lists
Confused about mailing lists and their use?
Read about mailing lists on Wikipedia
and check out these
guidelines on proper formatting of your messages.
