Date: Fri, 9 Sep 2016 01:36:12 +0200 From: Solar Designer <solar@...nwall.com> To: john-users@...ts.openwall.com Subject: Re: possible memory leak on FreeBSD? On Thu, Sep 08, 2016 at 08:42:38AM +0200, patpro@...pro.net wrote: > I'm running JtR (JohnTheRipper-bleeding-jumbo 20160728) on FreeBSD 10.1-RELEASE, and I'm experiencing some nasty memory problem with some settings. > I'm cracking huge passwords dump (10s of millions records), and my current pot file is about 4.3 GB. The server has 16 GB ram (but also runs other softwares). > > For example, --incremental will apparently very slowly consume memory on this server. I can't make really sure about this, but I can see the consumed swap size slowly increase overnight. Under normal usage, this server never swap a single bit. > It becomes blatant when I use --fork=4 with --incremental: the memory is exhausted in about 10-30 minutes and swap piles up. If I don't kill john, the box ends up crashing (swap exhaustion on ZFS is not good). Oddly, top output does not show a real increase in john's memory usage while free memory on host is depleting. > Same goes with --loopback --fork=4, even with a smaller pot file. > > Other attack modes like --wordlist are OK. The way "--fork" works, there's initially a lot of data sharing between the 4 processes, but the more passwords they crack, the less sharing there remains. Thus, their combined memory usage will in fact increase when John is running and is successfully cracking passwords. With password hash counts like yours, such increases can easily be in the gigabytes. My guess is that incremental mode was somehow more effective at getting you more cracks (that were not already in john.pot) than wordlist mode, or maybe you didn't use "--fork" with wordlist mode. Given that you're close to bumping into your total RAM size, I recommend that you get most passwords cracked when running without "--fork" (e.g., for a few hours or a day) and then re-add the "--fork=4" when the passwords are no longer getting cracked this frequently. Unfortunately, there's no easy way to continue a non-forked session with "--fork" added, so some processing time will be lost, but at least you'll hopefully bypass the issue you're running into now. (And it does sound like you need more RAM to efficiently crack all of your passwords at once.) You should also use "--save-memory=1" (but not higher), which might help a little bit (but likely not enough, hence the above primary suggestion). Incremental mode may also need more memory as it runs, but not much more. Specifically, it defers allocation of per character position tables for the maximum length until that length is actually reached. This is not a leak, but just a deferred allocation, so that some runs can benefit from lower memory usage. However, this allocation is on the order of 100 MB (or four times that, for "--fork=4") and not gigabytes, so is probably unrelated to what you're seeing. Alexander
Powered by blists - more mailing lists
Confused about mailing lists and their use? Read about mailing lists on Wikipedia and check out these guidelines on proper formatting of your messages.