[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]
Date: Mon, 21 Jan 2013 15:46:33 -0600
From: "jfoug" <jfoug@....net>
To: <john-dev@...ts.openwall.com>
Subject: RE: How to use SHA-2 in JtR code
From: magnum Sent: Monday, January 21, 2013 13:57
>
>Dhiru, all,
>
>I had to fix several new formats using SHA-2. We can't use OpenSSL for them
right away,
>
>Hopefully Jim will chime in if I missed or confused something.
what magnum wrote was pretty comprehensive. The only thing I would add
(part was already in his message), is.
1. always use #include "sha.h" and #include "sha2.h" (also md5.h and
md4.h). This puts the onus on JtR's code, it knows the #defines, and how it
should include, or optionally NOT include these libraries. The magic that
happens behind the curtain, to determine what code to use, is consistent,
and is housed within these 2 files owned by JtR. However, including the
system oSSL headers, breaks this 'magic'. Thus, for people building on
different systems, if there are formats using the system oSSL headers, the
build will likely break on some build platforms.
2. I am not overly sure about the 'extra' OpenSSL stuff (like AES, etc). If
we are building with the primatives coming from sha.c and sha2.c (i.e. the
internal JtR versions), will these oSSL items even work???
As for code produced by my sha2 vs oSSL, the 32 bit stuff (224/256) works
pretty well, on both 32 and 64 bit builds. The 64 bit stuff (384/512) is
dismal on 32 bit builds, but pretty good on 64 bit builds. oSSL uses SIMD
for 32 bit builds, for the 64 bit sha2 stuff (384/512). Since it is all
done using 64 bit registers, a 'generic' 64 bit sha2 sucks. The SIMD code
in oSSL is nowhere need what it could be. In JtR, our SIMD code does
parallel work. The oSSL does not, so would only run 1/2 (1/4???) as fast as
we should be able to get it. I think for sha2 (64 bit), any SSE2/4/SIMD
code, would only be able to do MMX_COEF 2, since these are 64 bit registers.
Thus we should be able to get 2x improvement over oSSL. However, my generic
code does none of that, it was pure C, and should not depend upon CPU.
As a side note, it is pretty easy to use grep to see just what files 'break'
the above rules. Then we can have a list of formats which may need to be
changed, for portability reasons.
$ grep 'openssl/sha.h' *.h
gladman_sha1.h:#include <openssl/sha.h>
gpg2john.h:#include <openssl/sha.h>
lowpbe.h:#include <openssl/sha.h>
$ grep 'openssl/sha.h' *.c
cuda_rawsha512_fmt.c:#include <openssl/sha.h>
cuda_xsha512_fmt.c:#include <openssl/sha.h>
kwallet_fmt_plug.c:#include <openssl/sha.h>
lowpbe.c:#include <openssl/sha.h>
mozilla_fmt.c:#include <openssl/sha.h>
mozilla2john.c:#include <openssl/sha.h>
mscash2_fmt_plug.c:#include <openssl/sha.h>
mysql_netauth_fmt_plug.c:#include <openssl/sha.h>
odf_fmt_plug.c:#include <openssl/sha.h>
office_fmt_plug.c:#include <openssl/sha.h>
oldoffice_fmt_plug.c:#include <openssl/sha.h>
opencl_odf_aes_fmt.c:#include <openssl/sha.h>
opencl_odf_fmt.c:#include <openssl/sha.h>
opencl_office2007_fmt.c:#include <openssl/sha.h>
opencl_office2010_fmt.c:#include <openssl/sha.h>
opencl_office2013_fmt.c:#include <openssl/sha.h>
opencl_rar_fmt.c:#include <openssl/sha.h>
opencl_rar_fmt.c:#include <openssl/sha.h>
opencl_rar_fmt.c:#include <openssl/sha.h>
opencl_rawsha512_fmt.c:#include <openssl/sha.h>
opencl_sxc_fmt.c:#include <openssl/sha.h>
opencl_xsha512_fmt.c:#include <openssl/sha.h>
putty_fmt_plug.c:#include <openssl/sha.h>
rar_fmt.c:#include <openssl/sha.h>
rar_fmt.c:#include <openssl/sha.h>
rar_fmt.c:#include <openssl/sha.h>
sha1_gen_fmt_plug.c:#include <openssl/sha.h>
sxc_fmt_plug.c:#include <openssl/sha.h>
wpapsk_fmt.c:#include <openssl/sha.h>
However, this is by itself, not perfect. I looked at mscash2_fmt_plug.c
(since I have had my hands all over that one). Here is the code:
#ifndef HAVE_OPENSSL
#include "sha.h"
#include "md4.h"
#else
#include <openssl/sha.h>
#include <openssl/md4.h>
#endif
So this one does have some #define 'magic' within the C file. However, this
one should be changed to simply be:
#include "sha.h"
#include "md4.h"
to let the magic happen within the sha.h and md4.h files (owned by JtR) to
happen properly.
On a similar tangent, we should make sure that all code which needs stdint
or stdbool uses: #include "stdint.h" or #include "stdbool.h", and NOT use
the <> include symantics, which would skip the headers we have IN JtR. This
allows proper building on systems that do not properly have stdint/stdbool
(i.e. if not __STDC__ __STDC_VERSION__
Jim.
Powered by blists - more mailing lists
Confused about mailing lists and their use?
Read about mailing lists on Wikipedia
and check out these
guidelines on proper formatting of your messages.
