[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]
Date: Wed, 11 Apr 2012 07:27:59 +0400
From: Solar Designer <solar@...nwall.com>
To: john-dev@...ts.openwall.com
Subject: Re: JtR:Multi-GPU Setups
Samuele -
On Tue, Apr 10, 2012 at 12:58:44PM +0200, Samuele Giovanni Tonon wrote:
> I hope with this one to give you some good advice and extend those advice to
> all the people interested in GPU computing for john.
Thank you for posting this! I think we/you should turn this into a wiki
page like:
http://openwall.info/wiki/john/GPU-setup
(it does not exist yet).
> ===
> = Single GPU hints
> ===
>
> As already told by many people AMD/ATI need to work an Xserver running
Confirmed.
> AND a user logged in
With typical distros, yes, although I think the only thing that's
actually needed is that the program (where we use OpenCL) is allowed to
talk to the X server. I think this has to do with /usr/sbin/atieventsd
and /etc/ati/authatieventsd.sh, which simply runs some xauth commands.
Anyway, for my testing today I simply configured autologin in
/etc/lightdm/lightdm.conf as suggested at the URL you posted (thanks!)
> and use 'ssh -X' (X11 protocol enabled in sshd)
Not confirmed. Moreover, if this command is used when connecting from a
remote machine, it will prevent things from working because our program
(such as JtR) will be talking to the remote machine's X server instead
of to the one with the AMD GPU in it. In my testing today, when
connecting from a remote machine, simple "ssh" works, "ssh -X" does not
(the AMD GPU is not seen with "clinfo" and "john --platform=list", then).
> this means that if that card is your only card but you plan to work on that
> server from remote (ssh), you must set up a lightdm session with an autologin
> this should help you in doing that:
> http://askubuntu.com/questions/51086/how-do-i-enable-auto-login-in-lightdm
This has helped, thanks.
> Nvidia on the other hand doesn't need any xserver running,
In my testing today, although the X server does not have to be running
at the moment in order for the Nvidia GPU to be usable (both CUDA and
OpenCL), it does need to have run at least once since the system was
booted up. I think it initializes the driver or/and the hardware in
some relevant way - and leaves them initialized even upon exit.
> so you can use OpenCL through a normal ssh.
OpenCL works through "normal" SSH (without X forwarding) for both AMD
and Nvidia for me.
> both nvidia and amd gives you libraries for opencl and the driver to use it, that
> driver is the proprietary one; if you find yourself stuck with some errors it's most likely
> because you are using open source driver (radeon, vesa) instead of proprietary (nvidia, fglrx)
Yes, when I ran X with an AMD-only xorg.conf (no Nvidia driver
explicitly mentioned there), somehow I was also getting the free nouveau
driver loaded (along with fglrx). This prevented me from loading the
proprietary nvidia driver later (and somehow the nouveau driver would
refuse to be rmmod'ed because of what looked like a circular dependency
between nouveau-related modules). So I added "blacklist nouveau" to
/etc/modprobe.d/blacklist.conf, which prevented this annoyance.
Another issue was that the new Catalyst 12.3 driver
(amd-driver-installer-12-3-x86.x86_64.run) turned out to be too buggy(?)
for me (with my 7970) - more so than the previous beta I was using
(amd-driver-installer-8.921-x86.x86_64.run), so I had to revert to the
latter. This was unexpected, so it took me a while to figure out - at
first, I suspected that I broke something else. The symptoms were that
the AMD GPU wouldn't be listed in "clinfo", the X desktop would have
various artifacts (lots of them!), and when I tried to run a game
(TORCS) it failed with a GLX related error. Simply downgrading the
driver made all of these go away at once. To be fair: I am doing this
on Ubuntu 12.04 Beta, which is not officially supported by AMD yet; it
is possible that things would be better with the new driver on a
supported distro.
Even with the older driver, I still get occasional X server segfaults on
its startup, which are cured only by rebooting the system (rmmod'ing
fglrx is not enough).
> What card to choose: i suggest move to ati hd 6xxxx family which is a good tradeoff of
> price per speed; ati hd 5xxxx family are good too but maybe a bit outdated and with different
> "bandwith" pci speed from cpu -> gpu and vice versa.
I think the bandwidth is mostly irrelevant, but I agree with the
recommendation - the newer cards are a better target for
development/optimization.
> Nvidia are generally more expensive than amd
Not quite, but they're about 3 times slower for what we're doing.
> but they offer a bit more stability of driver and issue .
Yes, this matches my experience so far.
> Everybody atm is "going wild" for the amd hd 7970; i still don't know if it's worth getting one, i'd wait a bit more
It's slower than 5970 or 6990 (dual-GPU cards) while being priced
similarly. So 6990 is a better thing to buy in terms of performance per
dollar. In fact, cheaper cards like 5870/5850 or 6970/6950 are even
better in that respect (and many 6950's are BIOS-upgradable to 6970),
but of course they're slower.
7970 is good for some of us to have because it's a new architecture,
which we need to target as well. Also, it is slightly more energy
efficient.
Oh, and indeed it is the fastest single GPU card, so if we only have
code supporting one GPU and we don't want to bother running multiple
instances manually, then 7970 has advantage over dual-GPU cards. On the
other hand, it encourages laziness in implementing better multi-GPU
support in JtR. ;-)
> ===
> = CPU mode
> ===
>
> if you have a AMD card and AMD cpu you might change your OpenCL code to either run on CPU or on GPU.
> while this can help on tracking problem debug with gdb
> http://whcomputing.wikispaces.com/NCIT.7.OpenCL.Optimisation
Thank you for this link!
> it could lead to some misbehaviours due to using a different architecture
And that's a good thing - it may help produce more portable OpenCL code.
> ===
> = Dual GPU system:
> ===
>
> As already told in other messages this at the moment is a bit messy: linux distribution are not yet
> ready with packages for nvidia and AMD so mixing both drivers and libraries could lead you to strange
> side effects.
In fact, my conclusion so far is that it's best not to use whatever
facilities a distro might provide to manage the proprietary drivers
because we're likely to end up needing to install different revisions of
those drivers anyway and then it's best for the distro's package manager
and scripts to be completely unaware of the drivers than for them to be
aware of some version that is no longer there.
> Nothing to worrry about, but you could end spending a lot of time trying to figure out which correct
> order of libraries to specify when using nvidia and which one to use when using AMD.
Yes, and getting X to work with both drivers may also trigger issues.
On my system, I can run X with nvidia only or with fglrx only, but when
I try with both at once (custom xorg.conf, which I think is valid), the
X server segfaults at startup (in fact, in what looks like exactly the
same way that is occasionally triggered by fglrx alone - perhaps there's
simply a bug in fglrx or an incompatibility with the X server version).
So to get both cards initialized after system bootup, I first run with
xorg.conf for Nvidia (the nvidia driver), then terminate the X server,
then run it with xorg.conf for AMD (fglrx). I suppose I could instead
run two X servers at once (and keep them running).
> Having dual AMD or dual Nvidia is simpler but not so useful: at the moment jtr still need a good interface
> design for GPU to have it scale with more than one GPU so at the moment is not so *needed* to have more
> than one GPU.
You're speaking of fast hashes only. JtR is already perfectly capable
of fully loading a GPU (or multiple GPUs with multiple instances) for
slow hashes.
> I hope i have clarified a bit more the situation of OpenCL developing and current jtr opencl status.
Your posting was helpful, thanks again! I think I've provided some
helpful info for the new wiki page as well.
user@...l:~/john/magnum-jumbo/src$ ../run/john -pla=list
Platform #0 name: NVIDIA CUDA
Platform version: OpenCL 1.1 CUDA 4.2.1
Device #0 name: GeForce GTX 570
Device vendor: NVIDIA Corporation
Device type: GPU (LE)
Device version: OpenCL 1.1 CUDA
Driver version: 295.33
Global Memory: 1279 MB
Global Memory Cache: 240 KB
Local Memory: 48 KB (Local)
Max clock (MHz) : 1600
Max Work Group Size: 1024
Parallel compute cores: 15
Platform #1 name: AMD Accelerated Parallel Processing
Platform version: OpenCL 1.1 AMD-APP (844.4)
Device #0 name: Tahiti
Device vendor: Advanced Micro Devices, Inc.
Device type: GPU (LE)
Device version: OpenCL 1.1 AMD-APP (844.4)
Driver version: CAL 1.4.1658 (VM)
Global Memory: 3072 MB
Global Memory Cache: 0 bytes
Local Memory: 32 KB (Local)
Max clock (MHz) : 0
Max Work Group Size: 256
Parallel compute cores: 32
Device #1 name: AMD FX(tm)-8120 Eight-Core Processor
Device vendor: AuthenticAMD
Device type: CPU (LE)
Device version: OpenCL 1.1 AMD-APP (844.4)
Driver version: 2.0
Global Memory: 7966 MB
Global Memory Cache: 16 KB
Local Memory: 32 KB (Global)
Max clock (MHz) : 1400
Max Work Group Size: 1024
Parallel compute cores: 8
("1400 MHz" for the CPU is actually the minimum clock rate, because the
CPU was almost idle when that was measured.)
Alexander
Powered by blists - more mailing lists
Confused about mailing lists and their use?
Read about mailing lists on Wikipedia
and check out these
guidelines on proper formatting of your messages.
