Date: Mon, 3 Mar 2014 06:42:16 +0400 From: Solar Designer <solar@...nwall.com> To: john-dev@...ts.openwall.com Subject: Re: Reload pot file magnum - On Sat, Mar 01, 2014 at 05:09:35PM +0100, magnum wrote: > Simplest but slowest alternative would be to drop the database, and more > or less reload it (from both input hash files and pot files) just like > an initial load. Ouch. I wouldn't even consider this approach. Nor the re-exec. > A faster and safer solution would be to just re-process pot file using > existing functions. We miss the opportunity to reload the input files > containing hashes to crack but that was never my main goal anyway. The > worst problem seems to be the database used during initial load is not > exactly the same as the one ultimately used. Perhaps that doesn't > necessarily matter? I don't understand what you mean by "the database used during initial load is not exactly the same as the one ultimately used". Can you please clarify which aspect(s) you're referring to here? I'd like to comment on this, but as it is I am just confused. > I'm not sure but I might need to implement an alternative version of > ldr_load_pot_file() to begin with. Anyway the hardest part for me now is > deciding what needs to be done - and when it's appropriate to do so. I > think this processing could take place right after crk_password_loop() > with no side effects. > > I'd appreciate any hints you can come up with! For --fork, my tentative plan was to introduce a new mem_alloc_shared() function or such, which would allocate memory from mmap's that would be shared between the child processes. Such allocations would occur before fork(). We'd use mem_alloc_shared() to allocate the database structures that are or may be updated when a cracked hash is removed. And we'd need to introduce a mutex on such removals - perhaps place it within a shared allocation of this kind too. That would be a weird mix of fork(), partially shared memory, and a mutex (from OpenMP? from pthreads?), but it might just work and not even look too ugly - I thought I'd try and then decide whether to keep this in a release or consider this a funny experiment and throw it away. Since this is a bit more operating system specific than the rest of JtR is, I think it'll need to be optional (compile-time). Unfortunately, I haven't yet found time for this experiment since my work on 1.8.0. Re-reading the pot is more generic since it also supports MPI and independent invocations of john (e.g., if someone manually invokes john with multiple wordlists one after another while also running it in incremental mode, the incremental run's john would remove the hashes cracked by the wordlist runs). So even if the shared memory approach above would happen to work well for --fork, I realize there may be demand for re-reading the pot anyway. So you may implement that in jumbo, and leave the shared memory for me to eventually experiment with, or you may try the shared memory thing yourself if you like. Oh, and when you re-read, you can start reading from a previously recorded offset (the last re-reads pot file size). Then it may actually be fast. Thanks! Alexander
Powered by blists - more mailing lists
Confused about mailing lists and their use? Read about mailing lists on Wikipedia and check out these guidelines on proper formatting of your messages.