[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]
Date: Tue, 15 Sep 2015 19:32:06 +0200
From: magnum <john.magnum@...hmail.com>
To: john-dev@...ts.openwall.com
Subject: Re: Judy array
On 2015-09-15 17:22, Solar Designer wrote:
> On Tue, Sep 15, 2015 at 09:45:51AM +0200, magnum wrote:
>> On 2015-09-15 03:03, Solar Designer wrote:
>>> magnum - testing this stuff, I see that pot sync is a major bottleneck.
>>
>> Does it look like pot sync is slower than processing "our own" cracks?
>
> Yes, it's a lot slower than processing of the process' own cracks.
> However, I didn't compare it against processing of john.pot records on
> loading.
>
>> It shouldn't, it uses all the same gears in the box. Of course, with
>> -fork=8 it will spend 8x more time processing cracks,
>
> It appeared way worse than 8x in my testing. With a large pot file
> buffer, the processes would become unresponsive for tens of seconds and
> would move way down in "top", leaving idle CPU time (weird). Oh, maybe
> that's because you acquire a lock on the pot file while reading it?
> You shouldn't need to: you may instead lock, save the file size in a
> variable, unlock, and then read up to the recorded size (it'd be a whole
> number of pot file records then). Or you may read without locking at
> all, as long as you ignore the last partial line (easy to detect: if
> it's not containing a colon, it's not complete enough for pot sync).
This crossed my mind too today. Aquiring a read lock is a Bad Idea[tm]
if I ever had one. I will look into that.
>> but in real-life it's not a problem. This is a contrived test.
>
> A contrived test, yes, but if we manage to handle even contrived cases
> like this well, we might end up improving handling of more common cases
> as well.
I did not really mean otherwise. But *completely* disabling pot sync for
unsalted hashes would have some serious drawbacks (eg. 14 processes
chugging along with incremental mode forever even though all hashes were
already cracked).
Anyway: I was probably using a too good computer for my tests so far,
but one thing I saw clear: The one option "ReloadAtCrack = Y" is by far
the worst offender here. I have committed a change so it defaults to
being disabled. Even if read locks etc. can mitigate some of the
regression, that option is overkill: The remaining options are the
important ones.
Another thing is that -enc:raw (which you did not mention using at all)
will speed things up considerably. Maybe I should give up and
comment-out the default "DefaultInternalEncoding" line in john.conf. It
would have to come bundled with a bunch of documentation changes too though.
I also committed all your changes (except john.conf), with the prefetch
stuff wrapped in "#ifdef CRACKER_PREFETCH" (which is not defined).
Furthermore I committed a "best64" ruleset from Hashcat. The rules using
unsupported commands were commented out, and one "f" rule was added to
make it exactly 64.
Finally I re-introduced a source() in MD4 and NT formats. I will do so
for the SHA formats too.
All the above just passed the build-bots' tests, so I'm committing to
bleeding right now.
magnum
Powered by blists - more mailing lists
Confused about mailing lists and their use?
Read about mailing lists on Wikipedia
and check out these
guidelines on proper formatting of your messages.
