Follow @Openwall on Twitter for new release announcements and other news
[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]
Date: Mon, 16 Jul 2012 15:37:08 +0400
From: Solar Designer <solar@...nwall.com>
To: john-dev@...ts.openwall.com
Subject: Re: problem with disc space for shared files in MJohn

Aleksey -

On Sun, Jul 15, 2012 at 03:24:18PM +0400, Aleksey Cherepanov wrote:
> I heard that some users have about 40gb of wordlists individually.
> Currently it would be a problem if MJohn would copy all files to the
> server.

I think those huge-wordlist runs should be separate from MJohn
currently, if MJohn is such that it'd require wordlist uploads (I don't
know whether and why it does, I haven't been watching the discussions
around it closely enough).

> Problems are
> 1) exhausted disk space,
> 2) exhausted traffic limit,
> 3) just slow.
> 
> Hence the questions:
> Alexander, could contest server have more space?

A little bit more, yes.  Say up to 100 GB total for the 2-day contest is
possible on the current server.  It has 2x146 GB SAS disks in RAID-1.

It would be nice to be able to easily exclude unimportant yet large
files from backups.  Can you have them placed in some subdirectories of
a fixed name (we typically use the name "nobackup" for this purpose)?

I think MJohn shouldn't be tied to a particular server and its capacity,
though.  It should be a generally useful tool.  I understand the need to
try it out on our current contest server in the upcoming contest, though.

> Would not we exhaust traffic limit?

There's no traffic limit on the current server, except for whatever the
100 Mbps switch port imposes.

> Possible improvements:
> - compress files (I expect wordlists to give good ratio),
> - drop git and use something much easier that allows to download one
> file instead of the full repository (I'll do that for other reasons
> too: for instance staging of files to check checksums could be done
> easier),
> - allow attacks with just sha1 instead of the real files: user does not
> upload files but shares sha1, so he could stop and restart this
> attack on his own cores while other users could check they do not
> run the same attacks, but for modification of attack users should ask
> the owner (also it is useful because not all users want to share their
> files),
> - use other file hosting and/or distributed system for that (many
> servers and/or torrent (though this needs our own tracker)).

The SHA-1 approach makes sense to me.  Maybe replace actual files with
SHA-1s when a certain size threshold is exceeded (e.g., 100 MB).

Alexander

Powered by blists - more mailing lists

Confused about mailing lists and their use? Read about mailing lists on Wikipedia and check out these guidelines on proper formatting of your messages.