Openwall GNU/*/Linux - a small security-enhanced Linux distro for servers
[<prev] [next>] [thread-next>] [day] [month] [year] [list]
Date: Mon, 28 Sep 2015 09:39:31 -0500
From: jfoug <jfoug@...nwall.net>
To: john-dev@...ts.openwall.com
Subject: Proposal for --test-full logic

Solar,

I know this has been a pet project which you have spear-headed for 
GSOC.  I would like to propose this change in logic.

currently, -test and -test-full both perform tests, and then perform 
benchmarks. I would propose that for -test-full, we remove that 
benchmark logic, and instead use the optional param to give indication 
for more tests to be run, where some of the 'deeper' tests may not be 
fully fixed in all formats.

So   --test-full or --test-full=0 would run a specific level. This would 
be all tests which we would 'expect' to run without errors (possibly 
with a format or 2 in whitelist).  This would be very useful for adding 
to our CI environments, to use this vs using the --test=0 which is used 
today.

Then,  --test-full=1 would be all the tests from --test-full=0, and then 
some new tests, which may not have had all the bugs worked out of the 
formats yet. This would be very useful allowing someone to easily 
continue to develop improvements to the formats which are failing the 
test, but be controlled by the optional switch, so it would not impact 
the robot runs on the CI's.

There could be as many numbers as were needed.  They could be done in an 
independent manner, or in a sequential manner, so --test-full=3 may be 
--test-full=0 and --test-full=3 or, it could be 0, 1, 2 and 3.  I really 
do not know which would be better, but if it was sequential, then 
someone could simply do --test-full=99999 and pretty much know that they 
will test ALL possible test-full cases.

At some point, the logic would get fixed for all formats, and that 
'extra deep' test code could then be moved into the --test-full=0 to be 
a default check.  From that point on, we would expect no formats at all 
to be allowed to have that particular defect.


I think this logic would greatly help to automate the CI tools to force 
developers to submit code or build new formats in a proper without 
allowing 'known' latent problems.    But also joining benchmarking into 
the --test-full logic, really does not seem to fit well.

Powered by blists - more mailing lists

Your e-mail address:

Powered by Openwall GNU/*/Linux - Powered by OpenVZ