Date: Mon, 16 Jul 2012 15:37:08 +0400 From: Solar Designer <solar@...nwall.com> To: john-dev@...ts.openwall.com Subject: Re: problem with disc space for shared files in MJohn Aleksey - On Sun, Jul 15, 2012 at 03:24:18PM +0400, Aleksey Cherepanov wrote: > I heard that some users have about 40gb of wordlists individually. > Currently it would be a problem if MJohn would copy all files to the > server. I think those huge-wordlist runs should be separate from MJohn currently, if MJohn is such that it'd require wordlist uploads (I don't know whether and why it does, I haven't been watching the discussions around it closely enough). > Problems are > 1) exhausted disk space, > 2) exhausted traffic limit, > 3) just slow. > > Hence the questions: > Alexander, could contest server have more space? A little bit more, yes. Say up to 100 GB total for the 2-day contest is possible on the current server. It has 2x146 GB SAS disks in RAID-1. It would be nice to be able to easily exclude unimportant yet large files from backups. Can you have them placed in some subdirectories of a fixed name (we typically use the name "nobackup" for this purpose)? I think MJohn shouldn't be tied to a particular server and its capacity, though. It should be a generally useful tool. I understand the need to try it out on our current contest server in the upcoming contest, though. > Would not we exhaust traffic limit? There's no traffic limit on the current server, except for whatever the 100 Mbps switch port imposes. > Possible improvements: > - compress files (I expect wordlists to give good ratio), > - drop git and use something much easier that allows to download one > file instead of the full repository (I'll do that for other reasons > too: for instance staging of files to check checksums could be done > easier), > - allow attacks with just sha1 instead of the real files: user does not > upload files but shares sha1, so he could stop and restart this > attack on his own cores while other users could check they do not > run the same attacks, but for modification of attack users should ask > the owner (also it is useful because not all users want to share their > files), > - use other file hosting and/or distributed system for that (many > servers and/or torrent (though this needs our own tracker)). The SHA-1 approach makes sense to me. Maybe replace actual files with SHA-1s when a certain size threshold is exceeded (e.g., 100 MB). Alexander
Powered by blists - more mailing lists
Confused about mailing lists and their use? Read about mailing lists on Wikipedia and check out these guidelines on proper formatting of your messages.