Date: Sat, 13 Jul 2013 04:14:03 -0700 From: Jeremi Gosney <epixoip@...dshell.nl> To: john-dev@...ts.openwall.com Subject: Re: new dev box wishes On 7/13/2013 2:52 AM, marcus.desto wrote: >> IIUC this box is colocated in a datacenter. No respectable datacenter >> will allow you to rack up anything water cooled. They will laugh you out >> of the building. > Ok. Have you ever heard about Cray? They used to build systems using liquid cooling for more than 20 years. Are you really trying to compare Cray's liquid cooling solutions to aftermarket consumer-grade water cooling solutions? There's clearly a HUGE difference between the two. Also, we're talking about a colocation facility. You're not going to find a Cray cluster in a colo facility... But even if some company /did/ want to colo a Cray cluster, that would still be a hell of a lot different from someone bringing in a home-built system with a water cooling kit asking for 4U of rack space, especially if the customer is looking to rent space in a shared rack. Surely you understand the difference. >> Water cooling was a passing fad. It's expensive, it's a major pain in >> the ass, and when it fails, it fails spectacularly. All of the serious >> password crackers who have flirted with water cooling have gone back to >> air cooling. > I never used liquid cooling, but friends who/did do told me about their experiences, which are positive with one > exception: if your liquid cooling leaks, then it really turns into pain in the butt. You seem to be relying on a lot of second-hand information to make such a recommendation. We have a lot of experience building dense GPU clusters specifically build for password cracking, and I'm telling you, there's a reason no one in this space does it anymore. Water cooling is very risky, especially when you have $2k - $4k worth of GPUs in each system. It's not "if" it leaks, it's "when" it leaks. Anyone who has been doing water cooling for at least a few years, if they're honest with you, will likely tell you they have experienced at least one leak, if not two or three. But, it's hard to find someone who's actually been doing water cooling that long, because most abandon it after a year or two. Most people who tell you water cooling is awesome and worry-free are those who have been doing it for less than two years. Water cooling is also very high maintenance. Water cooled systems require a very watchful eye to regularly inspect for leaks, and to ensure the reservoirs are topped off. This is highly inappropriate for remote systems. They also need to be completely torn down, flushed, and thoroughly cleaned at least once every six months, because the water blocks and reservoirs will get all gummed up. When you tear the system down for cleaning, you also need to replace all of the seals and o-rings, because they do break down over time, and they never quite seal right once they've been used, removed, and replaced. They're essentially one-time use items. The hoses also need to be replaced at least annually, because they will also start to break down over time. Overall, water cooling is one of those things that most people try because it looks cool. You see all those neat tubes and fluorescent additives and blacklights that all the cool kid are using, and you just /have/ to try it. Then you realize how much of a pain in the ass it is to maintain, and if you forego the maintenance, you realize how much it sucks when you experience the joys of catastrophic failure and you're out several thousand dollars. It's just not worth the risk, the headaches, or the money. And trust me, GPUs can be cooled just fine on air. We've spent hundreds of thousands of dollars on hardware over the years, and we have built several very large and very dense GPU clusters. If water were the way to go, we'd still be doing it. - epixoip.
Powered by blists - more mailing lists
Powered by Openwall GNU/*/Linux - Powered by OpenVZ