Follow @Openwall on Twitter for new release announcements and other news
[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]
Message-ID: <000a90f7758ee3744476f488fc7e79b5@smtp.hushmail.com>
Date: Mon, 16 Mar 2015 17:58:05 +0100
From: magnum <john.magnum@...hmail.com>
To: john-dev@...ts.openwall.com
Subject: Re: [GSoC] building JtR for MIC

On 2015-03-15 19:14, Solar Designer wrote:
> On Sun, Mar 15, 2015 at 01:12:48PM +0800, Lei Zhang wrote:
>> It got killed again when running smchapv2-naive. I guess it ran out of memory again?
> 
> Perhaps.  Can you try running the mschapv2-naive self-test on its own,
> not along with the rest?  Something like:
> 
> ./john --test --format=mschapv2-naive
> 
> If it runs fine, then the problem was likely caused by memory allocation
> by the formats that were tested previously.  Some of them intentionally
> allocate memory such that it can't be freed.  This lowers overhead for
> when john is run for cracking, with just one specific format, but it
> results in increased total memory usage for a many-formats self-test.
> We should only be making such non-freed allocations for things that are
> small.  For larger allocations, formats should free the memory in their
> done() methods.  Perhaps some formats allocate "too much" memory (at
> least when the number of threads is this large) and don't free it - and
> perhaps you should identify them and change them to free the memory.

This has been on my to-do list for some time. The "worst" formats use an
OMP_SCALE of eg. 128K. Even at just 16K and with a PLAINTEXT_LENGTH of
125, the key buffer alone is nearly half a gigabyte for 240 threads.

I just committed fixes for a good number of the worst offenders.
Hopefully this will fix the problem already but we should fix some more
later. And Lei, if you see a particular format still causing troubles,
feel free to fix that and make a PR.

Basically the below is what's involved for each format:

commit db1f7e7d6a5b0c75baf878f810f6da1bbb0dc79b
Author: magnum <john.magnum@...hmail.com>
Date:   Mon Mar 16 15:51:13 2015 +0100

    Use malloc/free for buffers; Yet another bunch of formats.

diff --git a/src/sapH_fmt_plug.c b/src/sapH_fmt_plug.c
index a260cbc..f76859a 100644
--- a/src/sapH_fmt_plug.c
+++ b/src/sapH_fmt_plug.c
@@ -154,8 +154,16 @@ static void init(struct fmt_main *self)
 	omp_t *= OMP_SCALE;
 	self->params.max_keys_per_crypt *= omp_t;
 #endif
-	saved_plain = mem_calloc_tiny(sizeof(*saved_plain) *
self->params.max_keys_per_crypt, MEM_ALIGN_NONE);
-	crypt_key = mem_calloc_tiny(sizeof(*crypt_key) *
self->params.max_keys_per_crypt, MEM_ALIGN_WORD);
+	saved_plain = mem_calloc(self->params.max_keys_per_crypt,
+	                         sizeof(*saved_plain));
+	crypt_key   = mem_calloc(self->params.max_keys_per_crypt,
+	                         sizeof(*crypt_key));
+}
+
+static void done()
+{
+	MEM_FREE(crypt_key);
+	MEM_FREE(saved_plain);
 }

 static int valid(char *ciphertext, struct fmt_main *self)
@@ -658,7 +666,7 @@ struct fmt_main fmt_sapH = {
 		tests
 	}, {
 		init,
-		fmt_default_done,
+		done,
 		fmt_default_reset,
 		fmt_default_prepare,
 		valid,


magnum


Powered by blists - more mailing lists

Confused about mailing lists and their use? Read about mailing lists on Wikipedia and check out these guidelines on proper formatting of your messages.