[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]
Date: Sat, 13 Jul 2013 04:14:03 -0700
From: Jeremi Gosney <epixoip@...dshell.nl>
To: john-dev@...ts.openwall.com
Subject: Re: new dev box wishes
On 7/13/2013 2:52 AM, marcus.desto wrote:
>> IIUC this box is colocated in a datacenter. No respectable datacenter
>> will allow you to rack up anything water cooled. They will laugh you out
>> of the building.
> Ok. Have you ever heard about Cray? They used to build systems using liquid cooling for more than 20 years.
Are you really trying to compare Cray's liquid cooling solutions to
aftermarket consumer-grade water cooling solutions? There's clearly a
HUGE difference between the two.
Also, we're talking about a colocation facility. You're not going to
find a Cray cluster in a colo facility... But even if some company /did/
want to colo a Cray cluster, that would still be a hell of a lot
different from someone bringing in a home-built system with a water
cooling kit asking for 4U of rack space, especially if the customer is
looking to rent space in a shared rack.
Surely you understand the difference.
>> Water cooling was a passing fad. It's expensive, it's a major pain in
>> the ass, and when it fails, it fails spectacularly. All of the serious
>> password crackers who have flirted with water cooling have gone back to
>> air cooling.
> I never used liquid cooling, but friends who/did do told me about their experiences, which are positive with one
> exception: if your liquid cooling leaks, then it really turns into pain in the butt.
You seem to be relying on a lot of second-hand information to make such
a recommendation. We have a lot of experience building dense GPU
clusters specifically build for password cracking, and I'm telling you,
there's a reason no one in this space does it anymore.
Water cooling is very risky, especially when you have $2k - $4k worth of
GPUs in each system. It's not "if" it leaks, it's "when" it leaks.
Anyone who has been doing water cooling for at least a few years, if
they're honest with you, will likely tell you they have experienced at
least one leak, if not two or three. But, it's hard to find someone
who's actually been doing water cooling that long, because most abandon
it after a year or two. Most people who tell you water cooling is
awesome and worry-free are those who have been doing it for less than
two years.
Water cooling is also very high maintenance. Water cooled systems
require a very watchful eye to regularly inspect for leaks, and to
ensure the reservoirs are topped off. This is highly inappropriate for
remote systems. They also need to be completely torn down, flushed, and
thoroughly cleaned at least once every six months, because the water
blocks and reservoirs will get all gummed up. When you tear the system
down for cleaning, you also need to replace all of the seals and
o-rings, because they do break down over time, and they never quite seal
right once they've been used, removed, and replaced. They're essentially
one-time use items. The hoses also need to be replaced at least
annually, because they will also start to break down over time.
Overall, water cooling is one of those things that most people try
because it looks cool. You see all those neat tubes and fluorescent
additives and blacklights that all the cool kid are using, and you just
/have/ to try it. Then you realize how much of a pain in the ass it is
to maintain, and if you forego the maintenance, you realize how much it
sucks when you experience the joys of catastrophic failure and you're
out several thousand dollars. It's just not worth the risk, the
headaches, or the money.
And trust me, GPUs can be cooled just fine on air. We've spent hundreds
of thousands of dollars on hardware over the years, and we have built
several very large and very dense GPU clusters. If water were the way to
go, we'd still be doing it.
- epixoip.
Powered by blists - more mailing lists
Confused about mailing lists and their use?
Read about mailing lists on Wikipedia
and check out these
guidelines on proper formatting of your messages.
