Follow @Openwall on Twitter for new release announcements and other news
[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]
Date: Sun, 16 Dec 2018 18:50:05 +0100
From: Solar Designer <>
Subject: Re: Anyone knows what does this fuzzer message mean?

On Sat, Dec 15, 2018 at 10:08:25PM -0200, Claudio Andr?? wrote:
> The format seems to be working fine (to me).
> ```
> $ "$JtR" --fuzz --format=sha512crypt
> Fuzzing: sha512crypt, crypt(3) $6$ (rounds=5000) [SHA512 128/128 AVX 2x]...
> Warning: excessive partial hash collisions detected
>    Completed
> All 1 formats passed fuzzing test!
> ```

The message comes from the loader when it encouters too many (by
default, more than 1000) same binary_hash()es, which slows down its dupe
hash check and is likely to slow down actual cracking as well (if the
same or a similar binary_hash*() is used within a salt, and the
collisions are seen within the salt as well).  During fuzzing, this is
normal, because the fuzzer alters our test vector hashes in many simple
ways, resulting in a set of fake hashes that are not random-looking.

I've just used this loader hack to see the actual collision counts:

+++ b/src/loader.c
@@ -1229,6 +1229,7 @@ static void ldr_load_pw_line(struct db_main *db, char *line)
                                if (++collisions <= LDR_HASH_COLLISIONS_MAX)
+                               continue;
                                if (john_main_process) {
                                        if (format->params.binary_size)
@@ -1250,6 +1251,12 @@ static void ldr_load_pw_line(struct db_main *db, char *line)
                        } while ((current_pw = current_pw->next_hash));
+                       static int prevmax = 0;
+                       if (collisions > prevmax) {
+                               fprintf(stderr, "collisions=%d\n", collisions);
+                               prevmax = collisions;
+                       }
                        if (current_pw) continue;

For sha512crypt, the max collision count was over 3000.  For some other
formats like raw-md5 and phpass, it was around 100.  So these collisions
do occur for many (maybe all) formats, but only cross the threshold for
some.  I suggest we suppress these warnings with the following patch:

diff --git a/src/john.c b/src/john.c
index c7181c8..fd15f9e 100644
--- a/src/john.c
+++ b/src/john.c
@@ -1694,6 +1694,13 @@ static void john_run(void)
 #ifdef HAVE_FUZZ
        if (options.flags & FLG_FUZZ_CHK || options.flags & FLG_FUZZ_DUMP_CHK) {
+ * Suppress dupe hash check because fuzzed ones often result in too many
+ * partial hash collisions.
+ */
+               options.loader.flags |= DB_WORDS;
+               list_init(&single_seed); /* Required for DB_WORDS */
                ldr_init_database(&database, &options.loader);
                exit_status = fuzz(&database);

Alternatively, we could export the skip_dupe_checking flag (currently
local to a function in loader.c) and set it from fuzz_test().

Or we could set a flag in the fuzzer and check it from the loader.


Powered by blists - more mailing lists

Confused about mailing lists and their use? Read about mailing lists on Wikipedia and check out these guidelines on proper formatting of your messages.