[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]
Date: Sat, 10 Sep 2016 13:17:53 +0200
From: Patrick Proniewski <patpro@...pro.net>
To: john-users@...ts.openwall.com
Subject: Re: possible memory leak on FreeBSD?
Hi,
On 09 sept. 2016, at 01:36, Solar Designer wrote:
> On Thu, Sep 08, 2016 at 08:42:38AM +0200, patpro@...pro.net wrote:
>> I'm running JtR (JohnTheRipper-bleeding-jumbo 20160728) on FreeBSD 10.1-RELEASE, and I'm experiencing some nasty memory problem with some settings.
>> I'm cracking huge passwords dump (10s of millions records), and my current pot file is about 4.3 GB. The server has 16 GB ram (but also runs other softwares).
>>
>> For example, --incremental will apparently very slowly consume memory on this server. I can't make really sure about this, but I can see the consumed swap size slowly increase overnight. Under normal usage, this server never swap a single bit.
>> It becomes blatant when I use --fork=4 with --incremental: the memory is exhausted in about 10-30 minutes and swap piles up. If I don't kill john, the box ends up crashing (swap exhaustion on ZFS is not good). Oddly, top output does not show a real increase in john's memory usage while free memory on host is depleting.
>> Same goes with --loopback --fork=4, even with a smaller pot file.
>>
>> Other attack modes like --wordlist are OK.
>
> The way "--fork" works, there's initially a lot of data sharing between
> the 4 processes, but the more passwords they crack, the less sharing
> there remains. Thus, their combined memory usage will in fact increase
> when John is running and is successfully cracking passwords. With
> password hash counts like yours, such increases can easily be in the
> gigabytes. My guess is that incremental mode was somehow more effective
> at getting you more cracks (that were not already in john.pot) than
> wordlist mode, or maybe you didn't use "--fork" with wordlist mode.
I've used --fork with both modes, but I can't remember the guess/second of the wordlist mode. You are right about the fact that incremental is way more effective that wordlist. I'm pretty sure I never achieved 10 M cracked passwords in 15 seconds using wordlist mode. Incremental is very impressive.
The strange thing is that `top` does not reflect this increase in memory, or at least this increase doesn't add up to the memory consumption I'm seeing. But may be I'm missing something here.
> Given that you're close to bumping into your total RAM size, I recommend
> that you get most passwords cracked when running without "--fork" (e.g.,
> for a few hours or a day) and then re-add the "--fork=4" when the passwords
> are no longer getting cracked this frequently. Unfortunately, there's
> no easy way to continue a non-forked session with "--fork" added, so
> some processing time will be lost, but at least you'll hopefully bypass
> the issue you're running into now.
That's what I do, and even later when the remaining password file is small enough I can switch to GPU cracking (currently only using hashcat on windows :( )
> (And it does sound like you need more
> RAM to efficiently crack all of your passwords at once.)
buying a 16 GB RAM upgrade is on my to-do list, but this particular RAM is hard to find.
> You should also use "--save-memory=1" (but not higher), which might help
> a little bit (but likely not enough, hence the above primary suggestion).
>
> Incremental mode may also need more memory as it runs, but not much more.
> Specifically, it defers allocation of per character position tables for
> the maximum length until that length is actually reached. This is not a
> leak, but just a deferred allocation, so that some runs can benefit from
> lower memory usage. However, this allocation is on the order of 100 MB
> (or four times that, for "--fork=4") and not gigabytes, so is probably
> unrelated to what you're seeing.
Thank you very much for all those info!
patpro
Powered by blists - more mailing lists
Confused about mailing lists and their use?
Read about mailing lists on Wikipedia
and check out these
guidelines on proper formatting of your messages.
