Date: Sun, 15 Jul 2012 00:22:35 +0400 From: Solar Designer <solar@...nwall.com> To: john-dev@...ts.openwall.com Subject: Re: additional memory usage questions, and the db_password structures On Sat, Jul 14, 2012 at 02:34:19PM -0500, jfoug wrote: > There are other areas of this which can have significant impacts on memory > usage, for large count of input (i.e. reduce this overhead and improve > scalability of JtR). > > > > These 2 are the next_hash pointer, and words pointer. > > > > For these 2, the words pointer is only used for single runs. So why not > build a structure that is shorter than the 'original', OR mem_alloc_tiny the > db_password items sizeof (list*) less bytes when running in non-single mode? We're already doing that. We also do it for the "login" pointer when memory saving is enabled. See pw_size in ldr_load_pw_line(). > If that information even used other than in a single run? The words list is specific to single crack mode. > As for the other pointer (the next_hash), couldn't that be moved outside of > each object, and simply be an array of db_password pointers?? My > understanding was this is used in salted only, and only when there are > multiple salts, thus keeping the ability to walk the salts. However, why > does each and every data object require this pointer? It may simply be a > miss understanding on my part, of just what the next_hash is. Yes, you seem to misunderstand what next_hash is. BTW, I edited the comment on it in my patch introducing source(). What we can in fact do is reduce this to 1 pointer. We only use either "next" or "next_hash" during cracking, not both at once. See the big if/else in crk_password_loop(). We use both in the loader, but we can try to come up with an algorithm that would not require that. At the very least, one of these can be moved to the cold part of the structure. > However, on a 64 bit system, these 2 pointers are 16 bytes PER candidate of > memory used. And for a non-salted (or single salt??) run, that is not > single, this memory is fully wasted, I think. This applies equally to saltless and salted hashes, regardless of salt count. The 16 bytes are per hash loaded. Anyhow, there's no pressing need to work on these enhancements right now, so I don't intend to. I might revisit this after the end of summer. My work on prepare() and source() was not so much for us to gain these enhancements now, but rather to allow for further changes to the formats interface (unrelated to these changes, but actually desirable before the end of summer) without having the core and jumbo trees each deviate from a common base in their own directions. With the changes I got in, I hope this can be avoided - that is, I can experiment with further formats interface changes in a revision of core (not to be committed yet), whereas bleeding will use the same base interface. Alexander
Powered by blists - more mailing lists
Confused about mailing lists and their use? Read about mailing lists on Wikipedia and check out these guidelines on proper formatting of your messages.