Date: Thu, 30 May 2013 12:17:28 +0400 From: Solar Designer <solar@...nwall.com> To: announce@...ts.openwall.com, john-users@...ts.openwall.com Subject: [openwall-announce] John the Ripper 1.8.0 Hi, I've just released John the Ripper 1.8.0, available from the usual place: http://www.openwall.com/john/ Available are a source code tarball (compressed with gzip or xz) and an extra charset files tarball (xz only). Besides the source code, included in the main tarball are 3 .chr files that are arguably most essential, whereas the extras tarball contains 7 additional .chr files. The default john.conf provides pre-defined incremental modes for all 10 of these files. Although only JtR 1.8.0 source code and .chr files are available at this time (to be used mostly on Unix-like systems, where building from source is customary and convenient), we do intend to release Windows build(s) of JtR 1.8.x, a jumbo based on 1.8.x, and a Pro based on 1.8.x at a later time. Meanwhile, magnum is regularly sync'ing the bleeding-jumbo branch on GitHub to be based on the latest core tree (which means 1.8.0 at the moment). Much of the work towards this release was sponsored under Rapid7's Magnificent7 program as previously announced here: http://www.openwall.com/lists/announce/2012/10/17/1 Here are the Magnificent7 sponsored changes, with their short descriptions as extracted from doc/CHANGES: * Revised the incremental mode to let the current character counts grow for each character position independently, with the aim to improve efficiency in terms of successful guesses per candidate passwords tested. Here are some test results of the old (1.7.9's) vs. new incremental mode, using the same training and test sets for both versions: http://www.openwall.com/lists/john-dev/2013/05/18/26 http://www.openwall.com/lists/john-dev/2013/05/19/1 http://www.openwall.com/lists/john-dev/2013/05/21/7 These show an improvement in the number of passwords cracked (for the same number of candidate passwords tested) ranging from 1% to 91% (that is, up to almost twice more passwords cracked) for different points in time. Greater improvements are seen at 10 thousand to 100 million candidates tested. Then the old version slowly catches up and by 1e12 candidates tested the new version's advantage decreases to only 1%, which is nevertheless significant. Tests on other samples show somewhat different results, but the new version generally does stay ahead. I'd like to thank magnum and bartavelle for running additional tests, which first helped spot a problem and then confirmed that, after my fix, the new incremental mode consistently outperformed the old one in their tests as well. * Revised the pre-defined incremental modes, as well as external mode filters that are used to generate .chr files. * Added makechr, a script to (re-)generate .chr files. As of version 1.8.0, pre-defined incremental modes are "ASCII" (all 95 printable ASCII characters), "LM_ASCII" (for use on LM hashes), "Alnum" (all 62 alphanumeric characters), "Alpha" (all 52 letters), "LowerNum" (lowercase letters plus digits, for 36 total), "UpperNum" (uppercase letters plus digits, for 36 total), "LowerSpace" (lowercase letters plus space, for 27 total), "Lower" (lowercase letters), "Upper" (uppercase letters), and "Digits" (digits only). The supplied .chr files include data for lengths up to 13 for all of these modes except for "LM_ASCII" (where password portions input to the LM hash halves are assumed to be truncated at length 7) and "Digits" (where the supplied .chr file and pre-defined incremental mode work for lengths up to 20). The charset files for "ASCII", "LM_ASCII", and "Digits" are included in the main tarball. The remaining 7 are available in the extras tarball. The new compile-time defaults for JtR permit for incremental mode lengths of up to 24, and for the entire 8-bit character set (excluding only the NUL character). In other words, you may generate additional .chr files with data for longer passwords and/or non-ASCII characters while using the normal JtR build that is compatible with the supplied .chr files. (The suggested limit of 13, except for digits, is based on our testing, though. It usually does not make sense to go for higher lengths with incremental mode.) You may also use "Extra = ..." in the configuration file to add more characters manually, including non-ASCII characters. * Enhanced the status reporting to include four distinct speed metrics (g/s, p/s, c/s, and C/s). The new status reporting is illustrated in this posting: http://www.openwall.com/lists/john-dev/2013/04/28/1 The new status line may include: successful guess count ("g"), session duration (in the D:HH:MM:SS format for days, hours, minutes, and seconds), progress indicator (percent done and optionally pass number out of the total number of passes), up to four speed metrics ("g/s", "p/s", "c/s", and "C/s"), and the current (range of) candidate password(s) being tested (John is often able to test multiple candidate passwords in parallel for better performance, hence a range). The four speed metrics are as follows: g/s is successful guesses per second (so it'll stay at 0 until at least one password is cracked), p/s is candidate passwords tested per second, c/s is "crypts" (password hash or cipher computations) per second, and C/s is combinations of candidate password and target hash per second. Versions of John prior to 1.8.0 displayed only the C/s rate (calling it c/s). When you restore a pre-1.8.0 session with version 1.8.0 or newer, only the g/s and C/s rates will be displayed, because the older .rec file format lacked information needed to compute p/s and c/s. * Added the "--fork=N" and "--node=MIN[-MAX]/TOTAL" options for trivial parallel and distributed processing. Naturally, these are documented in doc/OPTIONS. "--fork=N" is only available on Unix-like systems. It is an easy way to make use of multiple CPUs or CPU cores - you simply specify the number of John processes that you'd like to run. You may use "--fork" as an alternative to OpenMP, for formats currently lacking OpenMP support, or/and along with OpenMP (e.g., on a machine with 64 logical CPUs you might choose to run with "--fork=8" for 8 processes and use OpenMP to run 8 threads per process). You may use "--fork" along with "--node" to use multiple machines while also running multiple John processes per machine. For example, to use two similar 8-core machines you may run "--fork=8 --node=1-8/16" on one of the machines and "--fork=8 --node=9-16/16" on the other. You may also use "--node" separately from "--fork". Please refer to doc/OPTIONS for a lot more detail on these new options: http://www.openwall.com/john/doc/OPTIONS.shtml * In the external mode compiler, treat character literals as unsigned. This is to make 8-bit character literals specified in external modes always behave as expected. Previously, they were problematic if naively put e.g. into a revision of the Keyboard external mode: http://www.openwall.com/lists/john-users/2012/02/26/2 Now this problem has been addressed both in John itself and in the revised Keyboard external mode (in case it is used on an older version of John, and 8-bit characters are introduced into it). * Renamed many of the formats. John the Ripper 1.8.0's valid format names are descrypt, bsdicrypt, md5crypt, bcrypt, LM, AFS, tripcode, dummy, and crypt (and many more are added in jumbo). We no longer misuse just "des" or just "md5" to refer to the crypt(3) flavors; this made some sense in 1990s, before raw MD5 started to be misused for password hashing in web apps. However, with today's (still) widespread use of raw MD5 in web apps (and especially in leaked password hash dumps) our use of "md5" to refer to md5crypt was causing confusion. * Updated the documentation. Naturally. :-) Other recent changes unrelated to Magnificent7 are: * Relaxed the license for many source files to cut-down BSD. * Relaxed the license for John the Ripper as a whole from GPLv2 (exact version) to GPLv2 or newer with optional OpenSSL and unRAR exceptions. 80% of John the Ripper 1.8.0 source files (in the src/ directory) are now available under a cut-down BSD license (which is obviously compatible with the GNU GPL, letting me release the entire thing under GNU GPL v2 or newer). Other/older changes (also unrelated to Magnificent7) that I made since version 1.7.9 are as follows: * Enhanced the support for DES-based tripcodes by making use of the bitslice DES implementation and supporting OpenMP parallelization. An equivalent change, providing for much faster tripcode cracking than JtR's earlier proof-of-concept implementation did, has been in jumbo for a while, although it originated from the core tree - and is now properly released "officially". * Implemented bitmaps for fast initial comparison of computed hashes against those loaded for cracking. This is applied before hash table lookups, and it allows for the use of smaller hash tables (thereby saving memory) while achieving the same or greater speed that larger hash tables previously did. The speed increase is due to improved locality of reference (where only the smaller bitmap is accessed all the time, whereas the larger hash table behind it is only accessed for a percentage of comparisons and additionally it is smaller than it would otherwise need to be). * Tuned the bitmap and hash table sizes and thresholds based on testing on saltless hashes on a Core 2'ish CPU. This provides a substantial performance improvement when cracking large numbers of fast hashes. This change is a reason why in the incremental mode tests referenced above the new version achieved the same number of candidate passwords tested much sooner than the old version did. (To remove this effect on our incremental modes' comparison, we compared their successful guess counts at the same numbers of candidates tested, not at the same wall clock time.) * When cracking LM hashes, don't store the ASCII encodings of the hashes in memory, but instead reconstruct them from the binary hashes for writing into john.pot when a password gets cracked. This saves some memory, and the same functionality is also usable for other saltless hashes in jumbo. * With 32-bit x86 builds and at least MMX enabled, the "two hashes at a time" code for bcrypt is now enabled for GCC 4.2 and newer. This change is made based on benchmark results for different builds made with different versions of GCC on CPUs ranging from Pentium 3 to Core i7. Unfortunately, there's a known performance regression with this change on Atom. Previously, this code was only enabled for x86-64 and/or OpenMP-enabled builds. This is faster bcrypt cracking on some old and new computers running 32-bit operating systems or VMs for whatever reason. * The formats interface has been enhanced to better support GPU implementations (in jumbo), as well as fast hashes on multi-CPU systems (not yet made use of). 1.8.0 was a good time for revising the formats interface, which we'll make use of in jumbo shortly. * Assorted minor corrections to Cygwin builds were made. These are essentially the same corrections that were included in Windows builds of 1.7.9 (but were not yet included in 1.7.9 proper). * Fixed a bug in the Keyboard external mode (uninitialized variables on "--restore" or when minlength is greater than 1). * Enhanced the generic crypt(3) format to handle possible NULL returns from crypt() and crypt_r(). Oops. These are fixed now. * Assorted other changes have been made. OK, that was enough for an announcement. Feedback is welcome on the john-users list, as usual. I'd like to thank Rapid7 and their Magnificent7 program for enabling me to dedicate more time to implementing many of these changes and bringing them to release quality. Alexander
Powered by blists - more mailing lists
Confused about mailing lists and their use? Read about mailing lists on Wikipedia and check out these guidelines on proper formatting of your messages.