Zarathustra
Cloudless
- Joined
- Jun 19, 2019
- Messages
- 4,398
- Reaction score
- 4,738
The Z Build(s) 3.0 - Go Big and/or Go Home
- Thread starter Zarathustra
- Start date
Zarathustra
Cloudless
- Joined
- Jun 19, 2019
- Messages
- 4,398
- Reaction score
- 4,738
I decided it was time to replace the temporary CPU cooler in the workstation with the water block. I started this on a Friday evening after work, knowing that at this point I have crossed the river.
Absolutely everything I do on this project takes longer than I expect, and I had to be up and running Monday morning for work.
As usual, the cats aren't helping, constantly wanting to get into my little "workshop" with incessant crying for my attention. It is both cute and very distracting at the same time.
I swear they are just as socially dependent as dogs...
Anyway, I installed the block, and the PCIe slot passthroughs:
And then moved on to installing the rackmount rails, which was not straight forward at all.
I have installed many rackmount rails in my life, but these were confusing, and did not have any instructions.
Luckily some dude on youtube had my back:
Inner rails go on the side of the case:
Mid and outer rails slide over them as follows (once installed in the rack):
Unfortunately I seem to have forgotten to take pics when installing the mid and outer rails into the rack. That took 12 arms and reaching in unforgiving ways, so taking pictures was the last thing on my mind.
Moving on to installing the watertight conduit to try to tame the medusas hair of wires this project has resulted in.
Uh oh, many may have become too long. This box is going to be cramped...
Absolutely everything I do on this project takes longer than I expect, and I had to be up and running Monday morning for work.
As usual, the cats aren't helping, constantly wanting to get into my little "workshop" with incessant crying for my attention. It is both cute and very distracting at the same time.
I swear they are just as socially dependent as dogs...
Anyway, I installed the block, and the PCIe slot passthroughs:
And then moved on to installing the rackmount rails, which was not straight forward at all.
I have installed many rackmount rails in my life, but these were confusing, and did not have any instructions.
Luckily some dude on youtube had my back:
Inner rails go on the side of the case:
Mid and outer rails slide over them as follows (once installed in the rack):
Unfortunately I seem to have forgotten to take pics when installing the mid and outer rails into the rack. That took 12 arms and reaching in unforgiving ways, so taking pictures was the last thing on my mind.
Moving on to installing the watertight conduit to try to tame the medusas hair of wires this project has resulted in.
Uh oh, many may have become too long. This box is going to be cramped...
Zarathustra
Cloudless
- Joined
- Jun 19, 2019
- Messages
- 4,398
- Reaction score
- 4,738
I decided to replace the rubber gasket in the hole in the back of the box with a clamping fitting to prevent accidental ripping out of wires and bending of pins. Since the hole was a little large, this resulted in needing a set of reducing washers:
Which brings us to another thing I forgot to take pictures of.
The watertight conduit coming off of the roll had a shocking amount of curved "memory". I needed to get rid of that memory to make this work.
I cut off some lengths I wanted to use, drilled some holes in the top and bottom of each for zip ties, and used them to hang the lengths of conduit from the ceiling, and hang a dumb bell from the bottom. Then I broke out the heat gun and gently warmed the lengths of conduit, slowly straightening them.
It was a pain in the butt, but it sort of worked.
After that it was time to install them. It's still pretty messy looking, but we are getting there:
Well, this is going to be a cable management nightmare when I plug it all in.
And this is where I once again realized, that because I was rushing I forgot to take pictures of the hooking up of the last bit of tubing runing from the reservoir through a pair of Koolance QD3's to the DDC pump on the top of the bracket, out the back, through a second pair of Koolance QD3's to the case, and then back out, through a third pair of QD3's and back up to the reservoir.
I'll get better pictures of this once everything is done and cleaned up.
It was time to fill the system with the actual coolant this time.
It took almost exactly 8 liters of XSPC Pure clear coolant:
With the system full of coolant, it was now time to crawl in behind the rack, lay down on the floor and try to figure out some assemblance of wire management in the box, while getting everything plugged in where it goes in order to fire the system up and have it actually cool the workstation for the first time.
It was during this process that I realized that while the two DDC pumps attached to the radiators for the radiator portion of the loop were working fine, the DDC pump that was supposed to flow coolant to the workstation was dead as a doornail. It would not power up at all.
I guess it is time for troubleshooting again...
This is exactly what I didn't need to happen. It was Sunday evening. I needed my workstation Monday morning...
Which brings us to another thing I forgot to take pictures of.
The watertight conduit coming off of the roll had a shocking amount of curved "memory". I needed to get rid of that memory to make this work.
I cut off some lengths I wanted to use, drilled some holes in the top and bottom of each for zip ties, and used them to hang the lengths of conduit from the ceiling, and hang a dumb bell from the bottom. Then I broke out the heat gun and gently warmed the lengths of conduit, slowly straightening them.
It was a pain in the butt, but it sort of worked.
After that it was time to install them. It's still pretty messy looking, but we are getting there:
Well, this is going to be a cable management nightmare when I plug it all in.
And this is where I once again realized, that because I was rushing I forgot to take pictures of the hooking up of the last bit of tubing runing from the reservoir through a pair of Koolance QD3's to the DDC pump on the top of the bracket, out the back, through a second pair of Koolance QD3's to the case, and then back out, through a third pair of QD3's and back up to the reservoir.
I'll get better pictures of this once everything is done and cleaned up.
It was time to fill the system with the actual coolant this time.
It took almost exactly 8 liters of XSPC Pure clear coolant:
With the system full of coolant, it was now time to crawl in behind the rack, lay down on the floor and try to figure out some assemblance of wire management in the box, while getting everything plugged in where it goes in order to fire the system up and have it actually cool the workstation for the first time.
It was during this process that I realized that while the two DDC pumps attached to the radiators for the radiator portion of the loop were working fine, the DDC pump that was supposed to flow coolant to the workstation was dead as a doornail. It would not power up at all.
I guess it is time for troubleshooting again...
This is exactly what I didn't need to happen. It was Sunday evening. I needed my workstation Monday morning...
Zarathustra
Cloudless
- Joined
- Jun 19, 2019
- Messages
- 4,398
- Reaction score
- 4,738
...and yes.
I do realize there are only 7 bottles in that picture. I don't know what happened to the 8th one. I misplaced it somewhere during my mixing and filling
Zarathustra
Cloudless
- Joined
- Jun 19, 2019
- Messages
- 4,398
- Reaction score
- 4,738
Alright, some troubleshooting later I found the problem.
One of the crimps on my custom cabling was bad.
...so now I had to crawl on the floor and crimp laying on my side. Fun fun....
Now to get the external loop to trigger when the workstation powers on.
I have an opto-isolated relay triggered power strip I bought a while back for a different project that I wound up not needing, so I decided to use that.
Opto-isolated relays are really cool. They prevent accidentally sending 120VAC up the control signal where it shouldn't go by electrically isolating the two sides.
Inside of the hosuing there is a sealed relay unit. Inside of it the control signal lights up a little LED light. When the mains voltage AC side sees the light from that LED, it knows to turn on the power switch.
The opto-isolated relay did not have an integrated bleeder resistor to sink residual charge during shutoff, so I included one in my wiring.
The reason for this is that when the 12v DC power is switched off there is some residual in the circuit, and having a resistor in parallel helps drain this so that when you shut off, you don't have either a slow shutoff, or a shutoff that fluctuates on and off.
For 12v in this use case 4.7kOhm is the perfect size.
I chose to use the same fan cable that powers and controls the Workstation DDC pump with a splitter to trigger the relay.
So, now whenever the Workstation powers up, the pump gets power, and when the pump gets power, it triggers the relay, and the whole cooling system comes to life.
Right now it is only a single wire coming from the workstation:
12vdc + ground -> 4k7ohm resistor between 12vdc and ground -> relay
Pretty simple.
It gets ever so slightly more complicated when we add the second pc, but not much.
At that point the 12VDC line will look like this:
Code:
PC1 12VDC -> diode \
----> \ -----------------------> Relay + terminal
PC2 12VDC -> diode / \
4k7 ohm Resistor
PC1 Ground \ /
-------------> / -----------------------> Relay - Terminal
PC2 Ground /Please bear with me. I'm an engineer, but I am not an electrical engineer, and drawing circuits, especially in ASCII is not my expertise.
The diodes serve to make sure that power can only go in one direction, so when one power supply is accidentally slightly higher in voltage than the other, we don;t ahve power flowing the wrong way, causing problems.
Anyway. More on this later when we get to Phase 2.
At this point it was 4AM and I had work the next day. I was tired, and probably not very productive that day, but at least my workstation was up and running!
Next up we'll have to Phase 1 final pics, and some initial performance tests.
The performance tests will be of limited value, as in Phase 1 we are only cooling a single CPU, but to be fair, it is a pretty hot CPU, so it should tell us something.
Last edited:
Brian_B
Forum Posting Supreme
- Joined
- May 28, 2019
- Messages
- 8,455
- Reaction score
- 7,151
Interesting. I've often seen arresting diodes on coils to prevent inductive kick, but first time I've seen a bleeder used like this external to equipment. I do like how you are using the diodes to make the 12V PSUs redundant parallel supplies.
Zarathustra
Cloudless
- Joined
- Jun 19, 2019
- Messages
- 4,398
- Reaction score
- 4,738
Alright, first off, here are some semi- finished (could still use some cleanup, especially the rack cable management which is a bit of a rats nest) Phase 1 pictures.
...yeah, this is obviously the back of the rack, not very visible, but it is a rats nest of fiber optic, Ethernet and power cables I really ought to do something about.
We're not done yet. Phase 2 (the game machine) is yet to come. But this is a major milestone, so I decided it deserved some pics
...yeah, this is obviously the back of the rack, not very visible, but it is a rats nest of fiber optic, Ethernet and power cables I really ought to do something about.
We're not done yet. Phase 2 (the game machine) is yet to come. But this is a major milestone, so I decided it deserved some pics
Zarathustra
Cloudless
- Joined
- Jun 19, 2019
- Messages
- 4,398
- Reaction score
- 4,738
Alright. Some numbers.
First off, the flow through the radiators is actually a little bit better than my previous pessimistic figures. I think during the first fill I forgot that I had to tell the Aquaero which type of flow meter was connected for the calibration data.
So I get 232 L/h through the radiators. This is a bit lower than I had predicted with the two DDC's (I had calculated 400 l/h) but it is ~1.02 GPM, so its not terrible.
I spent some time putting together a dashboard in Aquasuite so I could get some cool charts and graphs.
All of the testing is done with just the single Threadripper 3960x in the loop. Now keep in mind that this is a quite substantial CPU. It idles using more power than most CPU's use at full load
Power output per the sensors in the package is about 80W at idle on the desktop, and about 287w at full load.
So for this test I manually cranked up the fans and the pumps *except the workstation loop pump, which is controlled by the motherboard) to max, and then decided to see where everything settled in,
First we have idle on the desktop:
Delta T between the coolant and ambient is 0.8C at 80W. Not bad.
I should note how I measure Delta T here. As you may recall there are three CaliTemp temperature sensors in the loop. One just below the reservoir for each pump intake. The theory here is that if a pump is starting to ingest warmer water, we want to make sure that doesn't happen.
I use the hottest of the three Calitemp sensors as being representative of the loop temperature. It is the worst case. They have thus far been within about half a degree of each other, but given our very low delta T that's actually quite significant
There are two ambient temperature sensors. They are the traditional 10kΩ NTC Thermistors (with a beta of 3950) most PC thermal probe headers use. (you might have seen the wires for them dangling in the pictures in the previous post) They are good, but not quite as well calibrated as Aquacomputers Calitemp sensors, so to make sure I was comparing apples to apples, I let the system sit overnight, then powered on the loop without a PC load, and calibrated the ambient temperature sensors to the average of the three Calitemp sensors using a linear offset.
So, the measured Delta T is the hottest of the three Calitemp sensors in the loop minus the average of the two ambient sensors.
Anyway, time for the load test.
For this test I wanted to get the most heat out of the Threadripper to see how the loop would perform. I know I'd come nowhere near stressing it like the combination of the Ryzen 9 9950X3d and RTX 5090 will later, but I wanted to give it the most I could.
So I ran Prime95 (well, the linux version, mprime) with small FFT's.
A standard Prim95 run uses mixed large and small FFT's. The Large FFT's have a lot of RAM activity and stress the RAM, whereas the small FFT's sit mostly in the cache and really allow the CPU to be hammered and as hot as it is going to get. In my case, 287W, as the Threadripper is at its power limit. (which is technically supposed to be 280w, but I guess it can go slightly over)
A 1.5C Delta T at full load. Not bad.
As you can see, this time the workstation pump was up to full blast.
Incredibly, at this load, the hottest CCD only hit just under 60C (like 59.8C) which is unheard of for a Zen2 Threadripper. A Small FFT run on a Threadripper is usually going to thermal throttle with air or even AIO cooling. You are only ever going to see anything below max temps on a very beefy custom loop. I guess that is what I have built
Alright, so the above looks pretty good, but lets keep in mind that I have only loaded it up with just north of 280W thus far.
The game machine 9950X3D can put out between 150W and 200W of heat, and the RTX5090 can get up to 600W, so even if the workstation is powered off, and nothing is going on in the background, this system needs to handle 750W-800W which is a much bigger challenge.
And if I have the workstation crunching something in the background, we are talking almost 1100w.
So, it is not over yet.
Which is why I am thinking about some performance enhancements. It may seem a bit silly, but my goal here is to keep the 5090 at 40C or under at full load in order to maximize boost clocks at all times. That's not going to be easy, and I need every little bit of cooling performance I can get.
Radiator pumps:
It looks like when I was ordering these pumps I inadvertently got the cost reduced 8W Chinese copy versions of the DDC, not the real deal at 15W - 18W. The Chinese brands (like Freezemod, Bykski, etc.) sell them side by side, and they are visually indistinguishable from each other.
I may wind up needing to upgrade these to actual Laing/Xylem PWM 4.2 DDC pumps. Time will tell.
Unfortunately since the pumps I bought are knockoff DDC's that use a cost reduced shaft design, I won't be able to use the same pump bodies or tops I currently have, which means there will need to be some modifications to the plastic brackets on the radiators. (ugh)
I'm not going to jump to conclusions though. I am going to wait until I can load it up with the actual game machine before I decide. THe metric here will be the temperature in each of the reservoirs. If both reservoirs are roughly even temperature wise, then the ~1GPM of flow will be enough, and I won't have to take any further action. If the reservoir that has the RTX5090 attached to it is much hotter than the reservoir that has the 9950X3d attached to it, then I know that the flow is insufficient to keep up with things.
Fans:
It turns out that in my research for this project, I totally miseed that Watercool has a special version of the Noctua NF-A20 they designed this radiator around. They call it the Noctua NF-A20 HS PWM. (The HS part - presumably high speed - being the key)
Standard Noctua NF-A20 fans range from ~350rpm (lowest PWM duty cycle) to ~800rpm (highest PWM duty cycle). The Watercool HS versions are essentially the same fan, but with a beefier motor that makes them spin between ~500rpm and ~1200rpm.
The latter are the fans I should have bought, but instead I installed all standard speed NF-A20's.
This is especially true since my radiators are "wall" mounted, and thus could definitely benefit from the much greater static pressure when forcing air through them.
The thing is, Noctua doesn't even acknowledge that these HS variant fans exist on their webpage.
It turns out these HS variants of the NF-A20 are not stocked anywhere I could find in the U.S.
So I ordered some from Watercool in Germany. This time with tariffs
I have the faster fans now, but I don't plan on installing them until after I get initial test temperatures with the game machine up and running, because I want to see the before and after temp difference.
Side note: Does anyone know anyone looking to buy 18 200mm Noctua fans?
First off, the flow through the radiators is actually a little bit better than my previous pessimistic figures. I think during the first fill I forgot that I had to tell the Aquaero which type of flow meter was connected for the calibration data.
So I get 232 L/h through the radiators. This is a bit lower than I had predicted with the two DDC's (I had calculated 400 l/h) but it is ~1.02 GPM, so its not terrible.
I spent some time putting together a dashboard in Aquasuite so I could get some cool charts and graphs.
All of the testing is done with just the single Threadripper 3960x in the loop. Now keep in mind that this is a quite substantial CPU. It idles using more power than most CPU's use at full load
Power output per the sensors in the package is about 80W at idle on the desktop, and about 287w at full load.
So for this test I manually cranked up the fans and the pumps *except the workstation loop pump, which is controlled by the motherboard) to max, and then decided to see where everything settled in,
First we have idle on the desktop:
Delta T between the coolant and ambient is 0.8C at 80W. Not bad.
I should note how I measure Delta T here. As you may recall there are three CaliTemp temperature sensors in the loop. One just below the reservoir for each pump intake. The theory here is that if a pump is starting to ingest warmer water, we want to make sure that doesn't happen.
I use the hottest of the three Calitemp sensors as being representative of the loop temperature. It is the worst case. They have thus far been within about half a degree of each other, but given our very low delta T that's actually quite significant
There are two ambient temperature sensors. They are the traditional 10kΩ NTC Thermistors (with a beta of 3950) most PC thermal probe headers use. (you might have seen the wires for them dangling in the pictures in the previous post) They are good, but not quite as well calibrated as Aquacomputers Calitemp sensors, so to make sure I was comparing apples to apples, I let the system sit overnight, then powered on the loop without a PC load, and calibrated the ambient temperature sensors to the average of the three Calitemp sensors using a linear offset.
So, the measured Delta T is the hottest of the three Calitemp sensors in the loop minus the average of the two ambient sensors.
Anyway, time for the load test.
For this test I wanted to get the most heat out of the Threadripper to see how the loop would perform. I know I'd come nowhere near stressing it like the combination of the Ryzen 9 9950X3d and RTX 5090 will later, but I wanted to give it the most I could.
So I ran Prime95 (well, the linux version, mprime) with small FFT's.
A standard Prim95 run uses mixed large and small FFT's. The Large FFT's have a lot of RAM activity and stress the RAM, whereas the small FFT's sit mostly in the cache and really allow the CPU to be hammered and as hot as it is going to get. In my case, 287W, as the Threadripper is at its power limit. (which is technically supposed to be 280w, but I guess it can go slightly over)
A 1.5C Delta T at full load. Not bad.
As you can see, this time the workstation pump was up to full blast.
Incredibly, at this load, the hottest CCD only hit just under 60C (like 59.8C) which is unheard of for a Zen2 Threadripper. A Small FFT run on a Threadripper is usually going to thermal throttle with air or even AIO cooling. You are only ever going to see anything below max temps on a very beefy custom loop. I guess that is what I have built
Alright, so the above looks pretty good, but lets keep in mind that I have only loaded it up with just north of 280W thus far.
The game machine 9950X3D can put out between 150W and 200W of heat, and the RTX5090 can get up to 600W, so even if the workstation is powered off, and nothing is going on in the background, this system needs to handle 750W-800W which is a much bigger challenge.
And if I have the workstation crunching something in the background, we are talking almost 1100w.
So, it is not over yet.
Which is why I am thinking about some performance enhancements. It may seem a bit silly, but my goal here is to keep the 5090 at 40C or under at full load in order to maximize boost clocks at all times. That's not going to be easy, and I need every little bit of cooling performance I can get.
Radiator pumps:
It looks like when I was ordering these pumps I inadvertently got the cost reduced 8W Chinese copy versions of the DDC, not the real deal at 15W - 18W. The Chinese brands (like Freezemod, Bykski, etc.) sell them side by side, and they are visually indistinguishable from each other.
I may wind up needing to upgrade these to actual Laing/Xylem PWM 4.2 DDC pumps. Time will tell.
Unfortunately since the pumps I bought are knockoff DDC's that use a cost reduced shaft design, I won't be able to use the same pump bodies or tops I currently have, which means there will need to be some modifications to the plastic brackets on the radiators. (ugh)
I'm not going to jump to conclusions though. I am going to wait until I can load it up with the actual game machine before I decide. THe metric here will be the temperature in each of the reservoirs. If both reservoirs are roughly even temperature wise, then the ~1GPM of flow will be enough, and I won't have to take any further action. If the reservoir that has the RTX5090 attached to it is much hotter than the reservoir that has the 9950X3d attached to it, then I know that the flow is insufficient to keep up with things.
Fans:
It turns out that in my research for this project, I totally miseed that Watercool has a special version of the Noctua NF-A20 they designed this radiator around. They call it the Noctua NF-A20 HS PWM. (The HS part - presumably high speed - being the key)
Standard Noctua NF-A20 fans range from ~350rpm (lowest PWM duty cycle) to ~800rpm (highest PWM duty cycle). The Watercool HS versions are essentially the same fan, but with a beefier motor that makes them spin between ~500rpm and ~1200rpm.
The latter are the fans I should have bought, but instead I installed all standard speed NF-A20's.
This is especially true since my radiators are "wall" mounted, and thus could definitely benefit from the much greater static pressure when forcing air through them.
The thing is, Noctua doesn't even acknowledge that these HS variant fans exist on their webpage.
It turns out these HS variants of the NF-A20 are not stocked anywhere I could find in the U.S.
So I ordered some from Watercool in Germany. This time with tariffs
I have the faster fans now, but I don't plan on installing them until after I get initial test temperatures with the game machine up and running, because I want to see the before and after temp difference.
Side note: Does anyone know anyone looking to buy 18 200mm Noctua fans?
Zarathustra
Cloudless
- Joined
- Jun 19, 2019
- Messages
- 4,398
- Reaction score
- 4,738
I figured it was absolutely necessary, because even really good PSU's are not perfect, +12VDC can range up and down by a few tenths of a volt, and if one PSU is skewing high, and one low, you could have some reverse power flow going on, and god knows what that might do to the PSU witht he lower 12v output and that entire system by extension.
Again, I am not an electrical engineer, but Iunderstand this is essentially what is called a "Discrete OR Gate" or a "Diode-OR Circuit" (or sometimes even a "Diode-OR Power Multiplexer).
To make things easier I bought myself one of these small boards with diodes and terminal blocks. I'm tired of soldering custom wires.
From my research on the topic, for my voltage (12v), low current and desired low leakage, is that 1N4007 diodes are the perfect choice here.