Date: Fri, 17 Aug 2012 10:19:48 +0300 From: Milen Rangelov <gat3way@...il.com> To: john-dev@...ts.openwall.com Subject: Re: RAR early reject Hey that's great! May I borrow it for my project? I think that check can even be done in OpenCL provided that we do AES decryption on GPU. This would eliminate needless memory transfers and host-side checks. To further improve it, 2-level host checks can be implemented, first against prebuffered initial chunk from the file, then the next one against the whole packed file which involves file i/o. Regards On Fri, Aug 17, 2012 at 7:04 AM, magnum <john.magnum@...hmail.com> wrote: > Milen, > > Thanks to JimF's pkzip code and some crucial hints from Pavel Semjanov, > I finally nailed the early reject we wanted! Have a look at this: > > commit 541ee9fee8f785179b9757a79a7b7d8d9ed6a6cf > Author: magnum <john.magnum@...com> > Date: Fri Aug 17 04:33:16 2012 +0200 > > "RAR: Finally, quite effective early rejection! Verified with almost > 70,000 test files, with no false rejects. And we are rejecting over 96% > of the candidates without resorting to a slow full check (which in turn > may reject semi-early)" > > > We can now crack -p archives at virtually the same speed as -hp ones, > almost regardless of size. Now I'll just need to find out why my kernels > run at half the speed they should. I tried using Sayantan's SHA1 code, > but there was no difference at all. I need to do something really clever > with the key stretching loop. > > magnum > > > On 2012-05-16 00:21, Milen Rangelov wrote: > > True. > > > > If I understand correctly, it should never go beyond 63 as no rar > archiver > > I've seen allows it. I think a similar check can be done for maxmb > (winrar > > imposes a limit of 128MB). This helps. But we definitely need some early > > reject in the LZ case. > > > > On Wed, May 16, 2012 at 12:59 AM, magnum <john.magnum@...hmail.com> > wrote: > > > >> On 05/15/2012 09:05 AM, Milen Rangelov wrote: > >>> I am reaching the same conclusion. I was able to do another check for > the > >>> PPM part, order should never be more than 63 (as this is the archiver > >>> limit) and this is safe I think. > >> > >> Do you mean max_order in ppm_decode_init() right after reading it from > >> rar_get_char()? Or did you test somewhere else? > >> > >> I also found from original unpack.cpp that filter_pos in add_vm_code() > >> should never be > 1024 but I haven't ever seen it happen. Maybe I'll > >> drop that test again even though it's there in the original code (for > >> some reason it was not in clamav's code). BTW I only recently realized > >> how easy it is to compare the original .cpp code with clamav's code. > >> They really just ported it. The original code has more comments. > >> > >> We are currently rejecting 93% of the data on average, compared to > >> reading all of the data each time. That may sound good, but for a 500 MB > >> file we still need to decrypt and read 3.6 MB on average. > >> > >> magnum > >> > >> > > > > > Content of type "text/html" skipped
Powered by blists - more mailing lists
Confused about mailing lists and their use? Read about mailing lists on Wikipedia and check out these guidelines on proper formatting of your messages.