Date: Mon, 28 Sep 2015 09:39:31 -0500 From: jfoug <jfoug@...nwall.net> To: john-dev@...ts.openwall.com Subject: Proposal for --test-full logic Solar, I know this has been a pet project which you have spear-headed for GSOC. I would like to propose this change in logic. currently, -test and -test-full both perform tests, and then perform benchmarks. I would propose that for -test-full, we remove that benchmark logic, and instead use the optional param to give indication for more tests to be run, where some of the 'deeper' tests may not be fully fixed in all formats. So --test-full or --test-full=0 would run a specific level. This would be all tests which we would 'expect' to run without errors (possibly with a format or 2 in whitelist). This would be very useful for adding to our CI environments, to use this vs using the --test=0 which is used today. Then, --test-full=1 would be all the tests from --test-full=0, and then some new tests, which may not have had all the bugs worked out of the formats yet. This would be very useful allowing someone to easily continue to develop improvements to the formats which are failing the test, but be controlled by the optional switch, so it would not impact the robot runs on the CI's. There could be as many numbers as were needed. They could be done in an independent manner, or in a sequential manner, so --test-full=3 may be --test-full=0 and --test-full=3 or, it could be 0, 1, 2 and 3. I really do not know which would be better, but if it was sequential, then someone could simply do --test-full=99999 and pretty much know that they will test ALL possible test-full cases. At some point, the logic would get fixed for all formats, and that 'extra deep' test code could then be moved into the --test-full=0 to be a default check. From that point on, we would expect no formats at all to be allowed to have that particular defect. I think this logic would greatly help to automate the CI tools to force developers to submit code or build new formats in a proper without allowing 'known' latent problems. But also joining benchmarking into the --test-full logic, really does not seem to fit well.
Powered by blists - more mailing lists
Powered by Openwall GNU/*/Linux - Powered by OpenVZ