[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]
Date: Sun, 21 Mar 2021 17:57:02 +0100
From: Solar Designer <solar@...nwall.com>
To: john-users@...ts.openwall.com
Subject: Re: Setting node values in heterogenous systems.
On Sun, Mar 21, 2021 at 11:03:19AM +0100, Micha?? Majchrowicz wrote:
> Recently there has been (IMHO) significant changes to johns fork/node
> code: https://github.com/openwall/john/issues/4609
A better reference is: https://github.com/openwall/john/issues/4584
Issue #4584 is about the actual new feature. Issue #4609, which you
refer to, is merely about an early implementation bug, which has since
been fixed.
> Before this patch fork and node values had to match each other. Which
Specifically, when using the "--fork" and "--node" options together, the
number specified in "--fork" had to match the range of numbers allocated
to this invocation in "--node".
For example, "--fork=32 --node=200-231/333" worked, but "--fork=32
--node=200-263/333" was not allowed.
> made it every hard to use fork/node in heterogenous systems as size of
> work per node had to vary from node to node. With this new patch you
> can set node value to any number divisible by fork number.
For example, "--fork=32 --node=200-263/333" now also works, allocating
two virtual node numbers per forked process.
> With a
> little of math it's possible to calculate node values for nodes which
> computational power varies significantly. I attach the script that
> uses very naive approach for finding node values that match node
> hashing speeds within 2% (it can be easily adjusted in code). If you
> run the tool without arguments it shows:
>
> ./calc.py
> Usage: calc.py number_of_threads1 power1 number_of_threads2 power2
>
> Power can be expressed by any number of same magnitude. For example I
> have two nodes, one with 10 cores at my disposal which has speed of
> ~100MH/s and second with 32 cores and speed of around 275MH/s. I pass
> those values to script and get:
>
> ./calc.py 10 100 32 275
> Faster power: 275 Slower power: 100
> Faster threads: 32 Slower threads: 10
> Found: 192 192.5
> Node: 192 70
>
> So in order to match power of this two nodes I have to use 70 nodes
> for slower node and 192 for faster node. With those settings both
> nodes finish their tasks at similar eta.
Suggested improvements to the script:
Easy:
1. Print the specific combinations of "--fork" and "--node" options to
use (assuming these are the only systems), not just the virtual node
counts. These options could then be directly copied on the command
lines of the systems. (If there are more systems, then the user of this
script would need to make adjustments.)
2. Find and report the best match, not the first with error within 2%.
3. With the above, don't fail hard if the error of the best match is
higher than 2%. Just print what it is.
4. Cleaner source code style, copyright+license, comment on what the
script is for, contribution via a pull request.
Difficult:
5. Support finding these numbers for more than 2 systems at once. (This
goal might be conflicting with always finding the best match, as this
could become time-consuming and result in too high virtual node counts.)
Thanks,
Alexander
Powered by blists - more mailing lists
Confused about mailing lists and their use?
Read about mailing lists on Wikipedia
and check out these
guidelines on proper formatting of your messages.
