Date: Tue, 28 Feb 2012 09:10:17 -0600 From: "jfoug" <jfoug@....net> To: <john-dev@...ts.openwall.com> Subject: RE: Re: [JtR PATCH] Support rar's -p mode by spawning external unrar process. >From: Dhiru Kholia > >On Tue, Feb 28, 2012 at 2:37 AM, magnum wrote: >> >>We would still have this -p mode problem but with more power under the >>hood someone might get the idea to fix that. If I got it right, this is >> only about quickly verifying or rejecting the decrypted data, just like >> Jim did with pkzip. I bet we can find some situations where we can >> reject early, if we just get the full CRC in place first. > >In RAR format, there is no CRC stored for the compressed (decrypted) >data. So there is no quick way to verify / reject it. The CRC stored >is for the original uncompressed file. The 'quick' rejection on pkzip was by design of the algo. There was a 1 byte checksum (or 2 byte checksum) that was located in 1 of 2 places. Thus, if there were multiple files in the zip file, all with the same PW, just that checksum can be used to eliminate almost all invalid passwords. Even if there was only 1 file in the zip blob, this checksum would eliminate at least 255 out of 256 candidates. However, in zip, there was not 'quick' way either, to fully check. The crc was for the original unencrypted, then uncompressed data. There were some quick rejection techniques that worked 'here' also, and I think these would work fine for rar (NOTE, I have not looked into the rar format, so am not 100% sure). One method was to decrypt small amount of data, then use the pkzip algorithm, which has 'some' data validation within it (i.e. there ARE byte streams which are not valid). The check data is mostly 'invalid' cases. Thus, if you decrypt some, run inflate on what you have, and it 'works', then you decrypt some more, test again, and if that works, simply continue forward, until either an error in the data is detected (meaning the PW is correct), or you inflate the whole blob, and the crc matches (which is a perfect match). To do this, I used the example code of zlib, but rewrote it. I want to do nothing more than spin through the data, 'looking' for where zlib would emit an error. I was not storing bytes, or doing much else. Thus my code is much thinner than the zlib, and runs quite a bit faster, and does a hell of a lot less. But it does have a very high probability of detecting that a password is invalid, after only checking a small amount of decrypted data. The problem with this method (same problem you get with rar), is you have to have the original file available, or at least 'enough' to get a very high probablity. You cannot pack much on a JtR input line. A tiny file 'can' be put on an input line, all in one blob. But a larger one, cannot. I made the line have the ability to 'link' to the original file. That way, JtR can open the original file, and get the entire blob, when it needs to do a full test. I do not know the algo for rar. I would assume that the encryption is post compression. Is that the case? I would certainly assume so, since if encryption is done prior to compression, then there IS no compression. One thing I also looked at, in 'speedup' within the pkzip, is to look for certain file-magic values. That has a lot of potential. If you have 2 compress Excel files, or group of gif or jpeg files, then you easily only have to decrypt/decompress a tiny bit of each file, to know that this password is very likely. I am just tossing out some of the things I worked on in the 'old' pkzip crypt format I worked on. Now for uncompressing rar, would the 7z lib be able to be used? Would zlib (I think not). As a faster klunky method (until something better is done), one could get 7z library hooked in, vs spawning out to the unrar process. This should be faster (possibly quite a bit faster), since there would be no process startup/tear down. It will depend upon how efficiently the unrar and the 7z lib can detect the wrong password. If they detect it pretty quickly, then the startup/teardown of the process may be a very large portion of the runtime in the spawning model. I am glad others are digging in, and having fun. I simply do not have time available right now, to invest in this code. The job, and dealing with family health issues have my time pretty much used up. Jim.
Powered by blists - more mailing lists
Confused about mailing lists and their use? Read about mailing lists on Wikipedia and check out these guidelines on proper formatting of your messages.