[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]
Date: Fri, 17 Aug 2012 10:19:48 +0300
From: Milen Rangelov <gat3way@...il.com>
To: john-dev@...ts.openwall.com
Subject: Re: RAR early reject
Hey that's great!
May I borrow it for my project? I think that check can even be done in
OpenCL provided that we do AES decryption on GPU. This would eliminate
needless memory transfers and host-side checks. To further improve it,
2-level host checks can be implemented, first against prebuffered initial
chunk from the file, then the next one against the whole packed file which
involves file i/o.
Regards
On Fri, Aug 17, 2012 at 7:04 AM, magnum <john.magnum@...hmail.com> wrote:
> Milen,
>
> Thanks to JimF's pkzip code and some crucial hints from Pavel Semjanov,
> I finally nailed the early reject we wanted! Have a look at this:
>
> commit 541ee9fee8f785179b9757a79a7b7d8d9ed6a6cf
> Author: magnum <john.magnum@...com>
> Date: Fri Aug 17 04:33:16 2012 +0200
>
> "RAR: Finally, quite effective early rejection! Verified with almost
> 70,000 test files, with no false rejects. And we are rejecting over 96%
> of the candidates without resorting to a slow full check (which in turn
> may reject semi-early)"
>
>
> We can now crack -p archives at virtually the same speed as -hp ones,
> almost regardless of size. Now I'll just need to find out why my kernels
> run at half the speed they should. I tried using Sayantan's SHA1 code,
> but there was no difference at all. I need to do something really clever
> with the key stretching loop.
>
> magnum
>
>
> On 2012-05-16 00:21, Milen Rangelov wrote:
> > True.
> >
> > If I understand correctly, it should never go beyond 63 as no rar
> archiver
> > I've seen allows it. I think a similar check can be done for maxmb
> (winrar
> > imposes a limit of 128MB). This helps. But we definitely need some early
> > reject in the LZ case.
> >
> > On Wed, May 16, 2012 at 12:59 AM, magnum <john.magnum@...hmail.com>
> wrote:
> >
> >> On 05/15/2012 09:05 AM, Milen Rangelov wrote:
> >>> I am reaching the same conclusion. I was able to do another check for
> the
> >>> PPM part, order should never be more than 63 (as this is the archiver
> >>> limit) and this is safe I think.
> >>
> >> Do you mean max_order in ppm_decode_init() right after reading it from
> >> rar_get_char()? Or did you test somewhere else?
> >>
> >> I also found from original unpack.cpp that filter_pos in add_vm_code()
> >> should never be > 1024 but I haven't ever seen it happen. Maybe I'll
> >> drop that test again even though it's there in the original code (for
> >> some reason it was not in clamav's code). BTW I only recently realized
> >> how easy it is to compare the original .cpp code with clamav's code.
> >> They really just ported it. The original code has more comments.
> >>
> >> We are currently rejecting 93% of the data on average, compared to
> >> reading all of the data each time. That may sound good, but for a 500 MB
> >> file we still need to decrypt and read 3.6 MB on average.
> >>
> >> magnum
> >>
> >>
> >
>
>
>
Content of type "text/html" skipped
Powered by blists - more mailing lists
Confused about mailing lists and their use?
Read about mailing lists on Wikipedia
and check out these
guidelines on proper formatting of your messages.
