[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]
Date: Wed, 22 Apr 2015 10:17:46 +0300
From: Shinnok <admin@...nnok.com>
To: john-dev@...ts.openwall.com
Subject: Re: Johnny: core/jumbo differences
After reading your quite good breakdown and your question regarding show stoppers, I think Johnny could potentially differentiate itself from CLI usage in the following ways:
1. Makes session management a breeze
2. Makes concurrent cracking management a breeze(on a single machine)
3. Makes wordlist management easier(mulitple, concat, dedup, reseed from pots, etc..)
4. Makes hash management(inclusion, exclusion, dedup, concat, merge, recognition) easier
5. Simplifies the path towards testing several attacks modes quickly(rules, formats, modes)
See my other thoughts embedded bellow.
> On Apr 22, 2015, at 12:08 AM, Aleksey Cherepanov <lyosha@...nwall.com> wrote:
>
> On Tue, Apr 21, 2015 at 05:18:22PM +0300, Aleksey Cherepanov wrote:
>> I've tried to write down exhaustive list of differences between john
>> core and john jumbo. I don't know everything, so feel free to comment.
>>
>> This list does not cover usage of the differences in Johnny. I'll make
>> a follow up with ideas.
>
> Some ideas for all those options, not prioritized. I am not sure what
> should fall under "jumbo support" and "*2john". What options are
> needed in johnny to be not so limited for jumbo?
>
> What are show-stoppers that prevent johnny from being used much?
>
> After working through jumbo options, I don't think that jumbo is very
> different: there are 3 new cracking modes, there are a lot of options
> for tweaking, there is --list= interface to improve interactive gui
> features. That's all, is not it?
>
> Having -fork, I'd say Johnny lacks ability to attack one file with
> several attacks at the same time, attack multiple files with one/many
> attacks (BTW john can attack several files in one invocation: like
> `john *.pw`). Also I fing inconvenient the way newly cracked passwords
> are shown: user have to monitor the log page. It is not very different
> from cmd line john (or at least it differs from my expectations from
> the gui).
I think this answers my question regarding multiple john instances on a single machine question in the other thread.
I gather it that there's a need to attack a hash set(or multiple sets) using different attacks at the same time in a hope that one will yield confirmation or results sooner than another, something like profiling for the correct attack, is this correct?
>
> BTW I found quite confusing: when short wordlist finishes its work,
> just start/pause buttons switches, there is no big banner "the attack
> is finished!!!!".
>
> If the log is so important then it may be shown all the time as a
> small region with 1-5 lines of the log. The log may be not from cli
> but from events to show only really important things to the user.
>
> (There is no border between fantasies and really important features in
> this text.)
>
>
>> * Common CLI options without differences
>
>> * Common CLI Options
>>
>> The format:
>> core option
>> jumbo option
>> comments
>>
>> ----------------------------------------
>>
>> --single "single crack" mode
>>
>> --single[=SECTION] "single crack" mode
>>
>> It is possible to choose Rules section to run with single mode.
>> [List.Rules:Single] is the default in both versions (so --single is
>> equivalent to --single=Single; though parameters are not case
>> sensitive).
>
> and
>
>> --rules enable word mangling rules for wordlist mode
>>
>> --rules[=SECTION] enable word mangling rules for wordlist modes
>>
>> $ ./core-1.8.0/run/john --rules=NT --wordlist=~/d/t.pw ~/d/t.pw
>> Extra parameter for option: "--rules=NT"
>>
>> Core does not allow rules section to be specified. Both versions
>> default to [List.Rules:Wordlist]
>
> Currently johnny does not allow to choose rules section for single
> mode and wordlist mode. `john --list=rules` can be used to get the
> list of rules sections. Though single mode has additional rules, I am
> not sure how they affect wordlist mode and if rules for wordlist mode
> are good for single mode.
>
>> ----------------------------------------
>>
>> --wordlist=FILE --stdin wordlist mode, read words from FILE or stdin
>>
>> --wordlist[=FILE] --stdin wordlist mode, read words from FILE or stdin
>> --pipe like --stdin, but bulk reads, and allows rules
>>
>> $ ./core-1.8.0/run/john --wordlist ~/d/t.pw
>> Option requires a parameter: "--wordlist"
>>
>> The default wordlist is used by default in jumbo:
>> [Options]
>> # Wordlist file name, to be used in batch mode
>> Wordlist = $JOHN/password.lst
>>
>> Also jumbo has --stdin and --pipe options to read candidates from
>> standard input.
>
> It may be possible to put john's default wordlist as the default for
> this option in johnny. It may be useful to remember wordlists used by
> user between restarts of johnny.
Agree.
>
>> ----------------------------------------
>>
>> --show show cracked passwords
>>
>> --show[=LEFT] show cracked passwords [if =LEFT, then uncracked]
>>
>> In jumbo --show can print remaining hashes but it drops dupes (when
>> there is 1 canonical hash for 2 users and the hash is in different
>> forms and/or there are different gecos).
>>
>> ----------------------------------------
>>
>> --salts=[-]N load salts with[out] at least N passwords only
>>
>> --salts=[-]COUNT[:MAX] load salts with[out] COUNT [to MAX] hashes
>
> That's an option to choose hashes. Like other options to choose hashes
> to be loaded, it may be implemented as filter for table view.
>
>> ----------------------------------------
>>
>> --format=NAME force hash type NAME: descrypt/bsdicrypt/md5crypt/
>> bcrypt/LM/AFS/tripcode/dummy/crypt
>>
>> --format=NAME force hash of type NAME. The supported formats can
>> be seen with --list=formats and --list=subformats
>>
>> Format list is much larger in jumbo, also it is possible to specify
>> hash type for generic crypt through --subformat option.
>
> List of format names may be loaded on the fly with `john
> --list=formats`. Though usually it is not needed to specify the
> format. There are several cases it is needed:
> - file contain hashes of different formats, user wants to attack
> different part of file (running john against file does mean that
> all hashes are under attack),
> - hashes are loaded incorrectly: they may be loaded as several formats
> and the default format is not what user wants,
> - to choose format's implementation (regular vs -ng, cpu vs -gpu, and
> others).
>
>
>
>> * Jumbo-only CLI options
>>
>> --loopback[=FILE] like --wordlist, but fetch words from a .pot file
>
> It seems to be a separate mode. It may be implemented as a checkbox
> for wordlist mode.
>
>> --dupe-suppression suppress all dupes in wordlist (and force preload)
>
> It affects performance only. It may be made as a setting. Or some more
> clever heuristic may be implemented: use dupe suppression only for
> slow hashes.
>
>> --prince[=FILE] PRINCE mode, read words from FILE
>> --mask=MASK mask mode using MASK
>> --markov[=OPTIONS] "Markov" mode (see doc/MARKOV)
>
> The cracking modes not supported by johnny yet. Rexgen is one too.
>
> It may be useful to make "feature testing" determing capabilities of
> john in use.
>
>> --pot=NAME pot file to use
>> --list=WHAT list capabilities, see --list=help or doc/OPTIONS
>> --help print usage summary, just like running the command
>> without any parameters
>> --config=FILE use FILE instead of john.conf or john.ini
>
>
>
>> --mem-file-size=SIZE size threshold for wordlist preload (default 5 MB)
>
> It may be done as a setting.
>
>> --progress-every=N emit a status line every N seconds
>> --crack-status emit a status line whenever a password is cracked
>
> It may be useful for johnny. Though there may be buffering.
>
>> --log-stderr log to screen instead of file
>
> It may be helpful to see log in output with option to write found
> passwords to log because this way it possible to connect the passwords
> with rules used.
I think the best interface for Johnny is john's stdout and stdin. Monitoring files is not that efficient for our use case.
>
>> --costs=[-]C[:M][,...] load salts with[out] cost value Cn [to Mn] for
>> tunable cost parameters, see doc/OPTIONS
>> (comma separated list of values/ranges per param.)
>
> Another hash selector.
>
>> --field-separator-char=C use 'C' instead of the ':' in input and pot files
>
> This option allows to load hashes that are separated not with colon.
> There are formats that use or allow colon as part of them. Johnny
> won't open such files nicely.
>
> Hm, theoretically there may be a hash with all possible characters
> alone on a line in input file... Not sure what then.
>
>> --max-run-time=N gracefully exit after this many seconds
>
> It may be showed to user to limit the work.
>
>> --stress-test[=TIME] loop self tests forever
>
> Johnny may have a mode/page for benchmarks.
Agreed. It was part of the original design.
>
>> --encoding=NAME input encoding (eg. UTF-8, ISO-8859-1). See also
>> doc/ENCODING and --list=hidden-options.
>> --input-encoding=NAME input encoding (alias for --encoding)
>> --internal-encoding=NAME encoding used in rules/masks (see doc/ENCODING)
>> --target-encoding=NAME output encoding (used by format, see doc/ENCODING)
>
> Encoding stuff.
>
>> --format=CLASS valid classes: dynamic, cpu
>> --subformat=FORMAT pick a benchmark format for --format=crypt
>> --mkpc=N request a lower max. keys per crypt
>> --min-length=N request a minimum candidate length
>> --max-length=N request a maximum candidate length
>> --fix-state-delay=N performance tweak, see doc/OPTIONS
>> --nolog disables creation and writing to john.log file
>> --bare-always-valid=C if C is 'Y' or 'y', then the dynamic format will
>> always treat bare hashes as valid
>> --keep-guessing try more candidates for cracked hashes (ie. search
>> for plaintext collisions)
>> --regen-lost-salts=N regenerate lost salts (see doc/OPTIONS)
>> --mkv-stats=FILE "Markov" stats file (see doc/MARKOV)
>> --reject-printable reject printable binaries
>> --verbosity=N change verbosity (1-5, default 3)
>> --skip-self-tests skip self tests
>
> There are options to tweak john. There may be "advanced user" checkbox
> to show all of them.
>
> I am not sure that it is good to put some of such options to settings
> page because it seems natural to separate johnny's settings and john's
> options to be applied always. Also john has options in the config file.
>
> Hm, some options have settings in config file:
> # Emit a status line whenever a password is cracked (this is the same as
> # passing the --crack-status option flag to john). NOTE: if this is set
> # to true here, --crack-status will toggle it back to false.
> CrackStatus = N
This should be the priority order for options:
1. Any option set in the GUI(default or explicit) and thus translated to an argument for john cli - first tier
2. Any option not set/used in the GUI should use the default from john's config file - second tier
3. Any option not set in both GUI and config file is trusted to be handled nice by john - third tier
Is there any reason to suspect that john might not respect this?
>
>
>> * Config file
>>
>> In Jumbo config files may include other files. Core john has only one
>> file.
>>
>> John built from sources searches config file (and other files) in run/
>> directory, john installed from package most usually looks into ~/.john
>> and /etc/john . While distros usually use john core, there distros
>> with jumbo. Also this behaviour may depend on distro.
>>
>> With jumbo, you can get path to the folder with --list=build-info option:
>> $ jumbo/run/john --list=build-info
>> [...]
>> $JOHN is jumbo/run/
>> [...]
>
> The setting to use with --log-stderr in trick mentioned above:
> # Write cracked passwords to the log file (default is just the user name)
> LogCrackedPasswords = N
>
>
>> * Utilities and tools
>>
>> In core
>> john - john
>> mailer - "script to send mail to all users whose passwords got cracked."
>> makechr - utility to make .chr files (incremental mode) from current .pot
>> relbench - statistics for benchmarks
>> unafs
>> unique - filter to remove dupes in wordlist saving order
>> unshadow
>>
>> unafs and unshadow may fall into *2john category.
>>
>>
>> BTW Jumbo has bash auto-completions.
>>
>>
>> In jumbo there are various utilities:
>>
>> *2john converters (including symlinks):
>>
>> These tools usually read file(s) and print hashes to stdout in a form
>> john understands them. Some utilities perform quite non-trivial
>> parsing using pulled-in third party libraries.
>>
>> 1password2john.py
>> 7z2john.py
> [...]
>> wpapcap2john
>> zip2john
>>
>> undrop
>
> These utilities may be used to "import" files. So to crack a .7z file
> with password you don't apply john to the file, you make a hash using
> 7z2john.py and then apply john on the hash.
>
> It may be tricky to choose the tool automatically. Though some files
> have simple connection between their extensions and the tools (.7z
> files have .7z on the end, though not all files are so).
>
> The other way to pick the tools is to get the file's format by
> magic/signature. On Linux, it is possible to call `file` utility.
> Though it may be unsafe (though running the utilities on user's files
> may be unsafe).
>
> The utilities are written in various languages. While johnny does not
> have anything to do with it, such zoo may be a problem to package john
> (e.g. to prepare a bundle john + johnny + all dependencies).
In order to simplify this we can make the distinction between a classic john password file and import functionality crystal clear for us and the users.
1. Open password file(only opens john format passwd)
2. Import file, attempts to recognize a file format and upon success drive that through a *2john or internal parsing module then use the classic format output of that.
For file format recognition, we could use extensions and QMimeDatabase and match that to a Johnny internal map of mime types and 2john scripts, anything beyond that, including calling file or relying on Linux magic might be overkill. Either way, the import dialog should only suggest a certain 2john module mapping and allow the user to correct before proceeding.
>
>> There are more tools:
>>
>> base64conv - raw/hex/base64(mime/crypt/cryptBS) converter
>> genincstats.rb - make statistics from .inc file
>> leet.pl - rules generator for leet speak
>> netscreen.py - netscreen hasher
>> hextoraw.pl - unhex for pass_gen.pl
>> pass_gen.pl - dynamic formats hasher
>> sha-dump.pl - ldap dumper
>> sha-test.pl - ldap sha1 hasher
>> calc_stat - some statistics for wordlist
>>
>> benchmark-unify
>> # John the Ripper benchmark output conversion tool, revision 1
>>
>> netntlm.pl
>> # The purpose of this script is to aid with cracking a LM/NTLM
>> # challenge/response set, when part of the password is known. It
>> # was written with John's NetLM/NetNTLM formats and "halflmchall"
>> # Rainbow Tables in mind.
>>
>> unrule.pl
>> # Extract basewords from list of plains. Based on an embryo from epixoip.
>> # ./unrule.pl < rockyou.lst > basewords.lst
>>
>> (I don't know at all)
>> genmkvpwd
>> mkvcalcproba
>> SIPdump
>> tgtsnarf
>> cprepair
>
> They does not seem very important now.
>
>
> Thanks!
>
> --
> Regards,
> Aleksey Cherepanov
Powered by blists - more mailing lists
Confused about mailing lists and their use?
Read about mailing lists on Wikipedia
and check out these
guidelines on proper formatting of your messages.
