Date: Sat, 10 Sep 2016 13:17:53 +0200 From: Patrick Proniewski <patpro@...pro.net> To: john-users@...ts.openwall.com Subject: Re: possible memory leak on FreeBSD? Hi, On 09 sept. 2016, at 01:36, Solar Designer wrote: > On Thu, Sep 08, 2016 at 08:42:38AM +0200, patpro@...pro.net wrote: >> I'm running JtR (JohnTheRipper-bleeding-jumbo 20160728) on FreeBSD 10.1-RELEASE, and I'm experiencing some nasty memory problem with some settings. >> I'm cracking huge passwords dump (10s of millions records), and my current pot file is about 4.3 GB. The server has 16 GB ram (but also runs other softwares). >> >> For example, --incremental will apparently very slowly consume memory on this server. I can't make really sure about this, but I can see the consumed swap size slowly increase overnight. Under normal usage, this server never swap a single bit. >> It becomes blatant when I use --fork=4 with --incremental: the memory is exhausted in about 10-30 minutes and swap piles up. If I don't kill john, the box ends up crashing (swap exhaustion on ZFS is not good). Oddly, top output does not show a real increase in john's memory usage while free memory on host is depleting. >> Same goes with --loopback --fork=4, even with a smaller pot file. >> >> Other attack modes like --wordlist are OK. > > The way "--fork" works, there's initially a lot of data sharing between > the 4 processes, but the more passwords they crack, the less sharing > there remains. Thus, their combined memory usage will in fact increase > when John is running and is successfully cracking passwords. With > password hash counts like yours, such increases can easily be in the > gigabytes. My guess is that incremental mode was somehow more effective > at getting you more cracks (that were not already in john.pot) than > wordlist mode, or maybe you didn't use "--fork" with wordlist mode. I've used --fork with both modes, but I can't remember the guess/second of the wordlist mode. You are right about the fact that incremental is way more effective that wordlist. I'm pretty sure I never achieved 10 M cracked passwords in 15 seconds using wordlist mode. Incremental is very impressive. The strange thing is that `top` does not reflect this increase in memory, or at least this increase doesn't add up to the memory consumption I'm seeing. But may be I'm missing something here. > Given that you're close to bumping into your total RAM size, I recommend > that you get most passwords cracked when running without "--fork" (e.g., > for a few hours or a day) and then re-add the "--fork=4" when the passwords > are no longer getting cracked this frequently. Unfortunately, there's > no easy way to continue a non-forked session with "--fork" added, so > some processing time will be lost, but at least you'll hopefully bypass > the issue you're running into now. That's what I do, and even later when the remaining password file is small enough I can switch to GPU cracking (currently only using hashcat on windows :( ) > (And it does sound like you need more > RAM to efficiently crack all of your passwords at once.) buying a 16 GB RAM upgrade is on my to-do list, but this particular RAM is hard to find. > You should also use "--save-memory=1" (but not higher), which might help > a little bit (but likely not enough, hence the above primary suggestion). > > Incremental mode may also need more memory as it runs, but not much more. > Specifically, it defers allocation of per character position tables for > the maximum length until that length is actually reached. This is not a > leak, but just a deferred allocation, so that some runs can benefit from > lower memory usage. However, this allocation is on the order of 100 MB > (or four times that, for "--fork=4") and not gigabytes, so is probably > unrelated to what you're seeing. Thank you very much for all those info! patpro
Powered by blists - more mailing lists
Confused about mailing lists and their use? Read about mailing lists on Wikipedia and check out these guidelines on proper formatting of your messages.