[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]
Date: Tue, 7 Sep 2010 10:07:38 +0400
From: Solar Designer <solar@...nwall.com>
To: john-users@...ts.openwall.com
Subject: rants (was: NTLM and OpenCL 1.0)
rembrandt,
OK, let me be rude as well, for the benefit of the project. ;-)
Seriously, though, I am not offended nor inconvenienced by your posting
at all. I just have difficulty making use of it as constructive
criticism, for the reasons I outline below. Maybe my reply will result
in more-easily-usable constructive criticism and specific suggestions
from you and/or others.
On Tue, Sep 07, 2010 at 05:58:29AM +0200, rembrandt wrote:
> On Tue, 7 Sep 2010 02:27:23 +0400
> Solar Designer <solar@...nwall.com> wrote:
<moderator hat on><nitpick>
Why, you did not actually quote anything I wrote. ;-) You also misused
an existing thread for your rants, which were better posted into their
own (new) thread.
</nitpick></moderator hat off>
> DO YOU WILL INCLUDE THE PATCHES into the "official" distribution... y/n
Which ones? All? Most? Then no. On the other hand, you might have
noticed (or didn't you?) that the jumbo patch became a lot more official
lately - it is even PGP-signed. Also, you might have noticed that I
included a lot more stuff into the jumbo patch, including some that was
bound to break things, and it did (recall the matching salt detection
issue affecting at least the Oracle and PIX hashes). So I was somewhat
more open to and aggressive in merging the patches. Not enough? Maybe.
I am more willing to accept criticism for not spending enough time on
JtR since, let's say, version 1.6 released in Dec 1998. The project
could definitely evolve much faster if I were spending more time on it.
At the same time, with the relatively little time that I did put into
the project, I don't think I should have spent that time substantially
differently - such as on merging all patches into the official codebase,
without having the time and means to make that code as clean and
reliable as I'd like the official JtR to be.
By having the jumbo patch (and other patches), I am giving everyone (and
myself) the choice: use feature-restricted but reliable and portable
code, or use feature-rich but less reliable and less portable code.
Yes, I am actually making use of this choice myself. I sometimes use
the official JtR, and sometimes JtR with the jumbo patch or with other
patches. Even during the contest, I was using 3 different builds of JtR
for this reason. I understand that it would have been better to have
one perfect version/build instead, but I just did not have time to make
one yet. ;-) ...and if I did, we would have even more/newer/better
patches that would still exist as patches for a while. Just when I
thought I had merged almost everything into jumbo, Simon came up with
more of his SSE intrinsics patches, which are very nice and desirable,
yet difficult or unreasonable to merge even into jumbo at this time.
So another patch on top of jumbo to maintain? Or a few days of my time
to achieve the same functionality and performance in a way suitable for
the official JtR? I just don't have that time to put into that now, and
the priorities for JtR should probably be different anyway (e.g., the
contest highlighted mostly other issues).
> You wont include anything new into the base-codebase...???
Not true. I am including stuff that I had time to (re-)write. Without
putting a lot of time into the project, this happens to be relatively
little code - but there was in fact new and changed code getting into
the official JtR. Recall the generic crypt(3) support and (limited)
OpenMP support for the extra-slow hashes in 1.7.6. Recall the wordlist
rules engine enhancements in 1.7.4 and 1.7.5 - many of which were
requested by the community, including some by JimF. I could let JimF
implement some of them and they would likely end up in jumbo. Instead,
I dedicated some time to implementing them myself and they're official.
Of course, I could/should have done a lot more.
> All those patches wont help. Did anybody tested it under OpenBSD
So what do you propose? Break portability of the official releases by
integrating non-portable code just to hopefully force users to submit
portability corrections? To some extent, this would work. But it could
also result in JtR in fact becoming less portable and staying that way.
We see both kinds of things happen with other projects. I prefer to do
this sort of things with the jumbo patch. If you (and others) like, I
am also open to the idea of starting an entire "community edition" or
whatever we call it - with the difference being that it'd be a separate
repository and tarballs ready to be built, not just patches. A drawback,
though, is that such a tree would likely deviate from the official and
reliable and portable JtR even further, making it harder to merge things
into the official tree (if this is ever to be done - not necessarily by
me and for official release, but maybe for other uses). Simply
providing jumbo-patched tarballs avoids this drawback indeed, but does
it make much of a difference compared to what we have now?
> (you remmeber our SSE-register talk years ago?!
> And still I was right *smiles* :) ).
You were right in exactly one thing: 16 SSE registers with AMD64. I have
no idea if you knew or if you guessed that, and in fact I don't recall
if you even stated it explicitly. You also made a lot of other claims,
which made me treat your guesses/claims as not trustworthy. (I am being
sincere, not rude. Really.)
I admit there was a time (early 2006?) when I thought that AMD64
expanded only the regular register file from 8 to 16 registers (I had
read about that, I've seen the code, etc.), not the SSE one (I did not
see this mentioned anywhere, and I did not read a full architecture
manual). In fact, they did both. When I seriously got to look into
this (before May 2006), of course I found this out and implemented the
proper bitslice DES code (in May 2006).
> So seriously: OpenCL -> GOOD! COOL! *for Windows, Mac, Linux! Not any
> BSD so far! It is the issue of the BSD! Not building a front....
> they do all care for themself and can't code a shit... if you analyze
> them partly. heyho Theo.. ;)) but if it will get handled like all those
> "patches" it will have no effect Alex... INCLUDE them into the base!
> You need to include them somehow... *my oppinion* :)
I find your arguments self-contradictory. You both want the
cutting-edge stuff integrated, which will definitely not work on
OpenBSD right away (speaking of your reference to the brand new OpenCL
patch, which does not even improve performance yet), but you also want
it to work there. Where would this get us?
I am more willing to buy Minga's arguments in favor of us working on the
rulesets. There are clear shortcomings of the rulesets currently
included with JtR, there are specific improvements to be made, this will
actually result in more passwords getting cracked in realistic uses of
JtR, and all of this is do-able in a realistic amount of time. So this
should be a priority. Not what you describe.
> Apply the algos *jumbo, as a hint! not all but liekly most coomon..
OK, "most common" - this is becoming realistic. I do in fact intend to
clean up and merge Alain's NTLM code. Just a matter of my time.
> remmeber stories of you.... wont say more, back then u tpold me about
> it... come on bro...
I am not sure what you're referring to here. It'd be better if you do
"say more" - just not rants and almost-irrelevant references, but
specific reminders, suggestions, requests.
> are the codes to badly?! >THEN COMMENT about it...
And then what?.. I am already maybe-rightly criticized for "ridiculing"
people (although this was never my intent!) when they post
naively-implemented wordlist rules. If that's how people treat my
recommendations on how to write their rules better, then how would they
treat my "complaints" about their contributed code?.. That said, I did
also point out specific issues with the code on some occasions. In many
cases, the authors of the code are well-aware of its limitations (such
as of portability issues), but that does not change much.
For example, I am aware of some issues with my own -omp-des-4 and
-omp-des-7 patches, where neither is strictly better than the other
(they have pros and cons) and neither is getting merged into the
official JtR just yet. There are all sorts of issues "like that" and
others with contributed patches as well. Just pointing them out will
rarely magically get them fixed, although, once again, I did point
problems out on many occasions - and I also made a lot of fixes to code
in the jumbo patch myself. And sometimes pointing problems out does
help - e.g., Alain's work on making his assembly file patches hopefully
work with Solaris' assembler is a result of my report of the problem to
him. (Thanks, Alain!)
> but providing a patch is fucked! most people test their stuff just on
> linux.. they're blind....).. and the codes... and make a little
> test-lap.. and if you need something i will provide it, or somebody
> else...
I do test the jumbo patch code on non-Linux systems on some occasions -
e.g., there was a round of fixes for problems discovered on
NetBSD/sparc64, which I have installed on a normally-powered-off machine
at home. I don't do it for every update of the patch, though. That
would distract me from other work on JtR too much, given how little time
I have for it.
Also, "test" is one thing, "fix all issues" is another. I was aware of
the NTLM assembly code vs. Solaris assembler issue for 2-3 years,
without getting around to fixing it. (Sorry!) Apparently, some recent
Solaris systems are not affected, though - somehow Robert managed to
avoid this issue during his Solaris builds. Also, no one actually
reported this issue (I saw it myself), and no one requested it getting
fixed during all this time (so few people want the jumbo patch working
on older versions of Solaris?)
> Maybe today I got totaly drunken... who knows... but hell Alex... I
> think I am right. What is your oppion or the oppinion of any other
> "core" developer worth if no single patch gets included into the
> base...?!
I understand that this might frustrate code contributors, and there will
likely be an exception for the NTLM code soon.
I care about having things in the official JtR done right to a greater
extent than I do about not frustrating a contributor - that is, I would
not merge a patch just to avoid frustrating someone. With some
contributions, I ended up reimplementing things in somewhat different
ways - e.g., recall Simon's patch for faster processing of successful
guesses and JimF's patch for larger hash tables. I did not merely apply
those patches, but I implemented the equivalent functionality (in this
case, the performance enhancements) in ways similar in spirit but
different in code. Simon's and JimF's code contributions played an
important role of proving the need for such changes and letting them get
tested before I reimplemented them for inclusion. I see no reason for
Simon and JimF to be frustrated that their exact code did not get in,
but some replacement code did. Rather, they should be happy for having
contributed to this end result, which I am grateful to them for.
> And to all developers: test your shit also at BSDs (and if possible
> MAC!)! Most of you seam to be Linux *loosers* *yes, provocating, I know
This is contradictory to your other requests/suggestions - being open to
contributions, merging them all, trying not to frustrate contributors.
Then you ask the contributors not to post their stuff until they've
tested it on lots of systems, some of which they wouldn't want to touch?
> Deeply appreaciating your work and the work fo ANYBODY who sends and
> IMPROVES JOHN(!);
Thank you.
Alexander
Powered by blists - more mailing lists
Confused about mailing lists and their use?
Read about mailing lists on Wikipedia
and check out these
guidelines on proper formatting of your messages.
