john-dev - .bss size in -jumbo

Openwall GNU/*/Linux - a small security-enhanced Linux distro for servers
[<prev] [next>] [thread-next>] [day] [month] [year] [list]
Date: Mon, 28 Nov 2011 05:07:29 +0400
From: Solar Designer <solar@...nwall.com>
To: john-dev@...ts.openwall.com
Subject: .bss size in -jumbo

magnum, Jim -

.bss in 1.7.8-jumbo-8 grew to 140 MB in a 64-bit build with OpenMP:

$ size ../run/john
   text    data     bss     dec     hex filename
 873185  104696 140226432       141204313       86a9b59 ../run/john

The main culprits are:

$ size *.o | sort -rnk 3 | head
   8318    1664 55837824        55847806        3542b7e NETNTLMv2_fmt_plug.o
   6431    1088 27918480        27925999        1aa1def NETLMv2_fmt_plug.o
   6230    1088 27525334        27532652        1a41d6c mskrb5_fmt_plug.o
   8438    1472 10420400        10430310         9f2766 MSCHAPv2_fmt_plug.o
   7500    1664 10420379        10429543         9f2467 NETNTLM_fmt_plug.o
   6038    1184 2883721 2890943  2c1cbf NETLM_fmt_plug.o
   4267     992 1573005 1578264  181518 NETSPLITLM_fmt_plug.o
  14169    1472  713100  728741   b1ea5 mscash1_fmt_plug.o
   3239     800  589872  593911   90ff7 crc32_fmt_plug.o
   4332     704  270656  275692   434ec XSHA512_fmt.o

I think at least top 5 of these (or maybe 7 or 9) should be changed to
allocate their memory dynamically, like 1.7.9 does in MD5_fmt.c and
BSDI_fmt.c.  (1.7.9's DES_fmt.c and c3_fmt.c also use dynamic memory
allocation, but in format-specific ways.  BF_fmt.c's saved_key[] is
relatively small (the number of buffered keys is low because this hash
is so slow), and BF_std.c puts its larger data structures on thread
stacks (just one instance per thread).)

Since there are no format destructors - I mean, we have init(), but
not the opposite - the entire 140 MB will end up getting allocated by a
full --test run anyway, but at least that address space won't be
allocated during normal cracking runs.  Also, systems that can't afford
140 MB will be able to run the tests individually (not all tests in one
invocation).  So this will be an improvement.

Of course, on typical systems this 140 MB is just address space and not
even virtual memory unless/until someone runs the full --test.  So this
cleanup won't matter too much.  Yet it'd be nice to make it if we can do
that without performance impact and without much effort (e.g., limiting
this to top 5 only).  Also, this may allow for scalability to much
higher thread counts.

Format destructors may be added as a next step.

Alexander