There's a lot of stuff going around about datacenters, so I decided to do a quick tour yesterday of some of the datacenters in the Salt Lake Valley.
-
@msbellows here's the unrolled thread: https://mastoreader.io?url=https%3A%2F%2Fc.im%2F%40msbellows%2F116557139885627239
Next time, kindly set the visibility to 'Mentioned people only' and mention only me (@mastoreaderio). This ensures we avoid spamming others' timelines and threads unless you intend for others to see the unrolled thread link as well.
Thank you!
@mastoreaderio
Well, this is handy
@msbellows -
Next, an even smaller datacenter, that just about anyone in #SLC has seen! This is XMission, a local Internet Service Provider that's been running since 1993, so one of The Ancients in Internet time. It's on a very busy part of 4th South, and if you've been by at night, you've seen the big LED display on the front of the building that they put various animations on.
One of the things that I *think* is probably in this building is SLIX: https://slix.net/traffic/ - this is an Internet Exchange Point (IXP), where various carriers meet up to exchange traffic without it having to travel long distances. These are often run as a sort of community infrastructure - it's in the best interests of all networks involved to connect to each other so that they can do their jobs more efficiently.
SLIX is fairly small (according to their own data they carry ~200Gbps, with some spikes up to 1Tbps). There are about 40 networks that meet there: https://slix.net/participants/ . Funny story, when I first got Google Fiber at my house, I was getting routed through California to get to the University of Utah campus just a few miles away. I pinged a guy I know who pinged a guy he knows who ... learned that some of the participants in SLIX didn't have their routes set up right. A config change later, and not only me, but basically everyone on any commercial ISP in the Salt Lake Valley had much more direct routes to campus!
@ricci SLIX lists their organizational address publicly on PeeringDB, so you are indeed correct.
https://www.peeringdb.com/org/8808(I guess it's possible that this is just the org's "business address," but I *highly* doubt it.)
-
@ricci SLIX lists their organizational address publicly on PeeringDB, so you are indeed correct.
https://www.peeringdb.com/org/8808(I guess it's possible that this is just the org's "business address," but I *highly* doubt it.)
@CiscoJunkie In this case their business address is adjacent to their datacenter. But yeah I don't see why they'd have it anywhere else, they have a datacenter
and it's in a fairly fiber-rich area -
There's a lot of stuff going around about datacenters, so I decided to do a quick tour yesterday of some of the datacenters in the Salt Lake Valley. Some are indeed quite large, but there are a bunch of smaller ones too - and they are not always where you think!
All of these are publicly known, and you can find them (and ones in your own area) at https://www.datacentermap.com/ .
Let's start with a datacenter that I go by all the time! It's across the street from my grocery store in downtown #SLC. It's listed as a colocation facility; datacenters are famously secretive about who their tenants are, but we can guess that it probably hosts servers belonging to nearby businesses, especially ones that want their storage, etc. nearby, but don't want to have to maintain a secure, cooled room. Given the number of banks that have headquarters nearby, I'd bet at least some of them are customers.
This is a fairly little guy, with apparently 16k square feet of floorspace and 1.6MW of power.
@ricci Great thread, Rob! I've been working in datacenters for decades but I still found it educational. Thank you!
-
@ricci Great thread, Rob! I've been working in datacenters for decades but I still found it educational. Thank you!
@fivetonsflax I'm glad you liked it! Feel free to add anything if you want
-
Next, an even smaller datacenter, that just about anyone in #SLC has seen! This is XMission, a local Internet Service Provider that's been running since 1993, so one of The Ancients in Internet time. It's on a very busy part of 4th South, and if you've been by at night, you've seen the big LED display on the front of the building that they put various animations on.
One of the things that I *think* is probably in this building is SLIX: https://slix.net/traffic/ - this is an Internet Exchange Point (IXP), where various carriers meet up to exchange traffic without it having to travel long distances. These are often run as a sort of community infrastructure - it's in the best interests of all networks involved to connect to each other so that they can do their jobs more efficiently.
SLIX is fairly small (according to their own data they carry ~200Gbps, with some spikes up to 1Tbps). There are about 40 networks that meet there: https://slix.net/participants/ . Funny story, when I first got Google Fiber at my house, I was getting routed through California to get to the University of Utah campus just a few miles away. I pinged a guy I know who pinged a guy he knows who ... learned that some of the participants in SLIX didn't have their routes set up right. A config change later, and not only me, but basically everyone on any commercial ISP in the Salt Lake Valley had much more direct routes to campus!
-
Here's what I hope your takeaway from this thread will be: datacenters come in many sizes, have many uses, and are not necessarily where you'd expect. The impact they have locally depends on how they're powered, how they're cooled, what they're used for, who owns them, and how big they are. It's worth looking at all of these things when considering whether a datacenter project is a good idea or not.
@ricci naive question, wouldn't building large amounts of solar panel be more energy efficien
-
Here's what I hope your takeaway from this thread will be: datacenters come in many sizes, have many uses, and are not necessarily where you'd expect. The impact they have locally depends on how they're powered, how they're cooled, what they're used for, who owns them, and how big they are. It's worth looking at all of these things when considering whether a datacenter project is a good idea or not.
@ricci Thank you for the overview.
What I don't understand is, why build data centers in areas with warmer climates, when colder ones would be... well, easier to cool?Aren't economic and ecologic incentives aligned here?
Data centers for compute in particular (as opposed as, for response time) don't need to be in any particular geographical area anyway, do they? -
@ricci naive question, wouldn't building large amounts of solar panel be more energy efficien
@gerbrandvd I don't know the exact math on this, unfortunately. What I do know though is that you'd need both solar *and* storage, in a setting like this where they're generating all of their own power on-site, they'd need to generate far more power than they need during the brightest hours of the day, then use it overnight and/or when it's overcast.
Then there's also the fact that one generates far less power from solar during the winter when the days are shorter and the angle of the sun in the sky is less favorable (and that you'd have to clear the panels of snow).
Solar might be a reasonably good match for the cooling part of the load; you can sometimes get away with using outside air when it's cool enough (winter and sometimes at night) but would be a lot harder to make work for the actual computing load, since that's going to run 24/7/365 (esp. if this is used for AI training)
-
@ricci Thank you for the overview.
What I don't understand is, why build data centers in areas with warmer climates, when colder ones would be... well, easier to cool?Aren't economic and ecologic incentives aligned here?
Data centers for compute in particular (as opposed as, for response time) don't need to be in any particular geographical area anyway, do they?@phairupegiont You are correct! It's generally easier to cool things down when it's colder outside! Here in northern Utah some datacenters - with far less power density than this one - are able to just use outside air to cool for a good chunk of the year. With the kinds of heat loads generated by warehouses of GPUs; well, I suspect their cooling needs are indeed lower in the winter, but they probably need active cooling all the time anyway.
There are some datacenters in Finland that even use cold seawater as part of their cooling systems!
So, why build in places that get hot part of the year? Well, if you are willing and able to use water for evaporative cooling, that's pretty effective in very dry environments - and can be cheap depending on the cost of water. Sometimes, the availability of power is a big thing too - in this case, there is an existing natural gas pipeline running through the valley that they intend to tap. For some kinds of datacenters, it's important to be near your users - though that's less important for AI training, which is what this one would likely be used for.
-
This is the biggest datacenter I visited - campus, actually. This facility is in West Jordan, near the South Valley Regional Airport. It's big enough that I have to post several pictures to get you a real sense of the size (but it's not the biggest datacenter in Utah.)
What you're looking at here is three buildings that, together, have a power capacity that's reported (depending on the source) to be around 160 MW (put a pin in that number too.)
Two of these buildings are multi-tenant (the ones with the flat white roofs) like the others we've seen.
That third one in the back, with all of the cooling towers on top, has supposedly been built for a single hyperscaler, and is supposedly something like an 80-100MW building. Which hyperscaler? That information is not public. That's a whole lot of cooling on the roof (which is reported to be water-free), so my money would be on this being an AI data center.
In these pictures, you can see more electrical infrastructure. Bringing that much power into one place takes a lot of wires.
@ricci interesting to me that the (presumably) higher-density facility is taller (multi-story). I've noticed that with other new-built high-density facilities.
to save ground space? maybe there is water cooling involved and it is helpful to have that equipment above/below servers? or high ceilings help with thermal engineering?
-
Here's what I hope your takeaway from this thread will be: datacenters come in many sizes, have many uses, and are not necessarily where you'd expect. The impact they have locally depends on how they're powered, how they're cooled, what they're used for, who owns them, and how big they are. It's worth looking at all of these things when considering whether a datacenter project is a good idea or not.
@ricci great thread, thanks!
-
@ricci interesting to me that the (presumably) higher-density facility is taller (multi-story). I've noticed that with other new-built high-density facilities.
to save ground space? maybe there is water cooling involved and it is helpful to have that equipment above/below servers? or high ceilings help with thermal engineering?
@bnewbold yeah I dunno! In this particular campus, that building seems to have consumed all remaining space on the lot, so it *could* just be an issue of the older ones not being as space constrained, but it also could be a fundamentally different design. My assumption (based only on trends, not any special knowledge) is that this new one also takes the cooling loop all the way to the chip - I don't know what that does with optimal layouts
-
Here's what I hope your takeaway from this thread will be: datacenters come in many sizes, have many uses, and are not necessarily where you'd expect. The impact they have locally depends on how they're powered, how they're cooled, what they're used for, who owns them, and how big they are. It's worth looking at all of these things when considering whether a datacenter project is a good idea or not.
@ricci great thread, Rob. I worked for one of the big cloud providers and got to spent quite a bit of time with the datacenter team before I retired. I was blown away by the innovations being implemented to reduce power and cooling requirements. All of that is moot now. We were talking about the potential of 25kW racks. Now they’ve completely blown past that with 100kW racks. It’s insane.
-
Here's what I hope your takeaway from this thread will be: datacenters come in many sizes, have many uses, and are not necessarily where you'd expect. The impact they have locally depends on how they're powered, how they're cooled, what they're used for, who owns them, and how big they are. It's worth looking at all of these things when considering whether a datacenter project is a good idea or not.
@ricci great thread, it really puts things in perspective!
-
@mjd Frankly, I think it's entirely implausible that it will get built as advertised. I'm not sure that the demand is actually there for as many datacenter projects as have been announced. I think it's a very good bet that many or even most of them won't get built out to the size they've discussed. I think the game here is to make big announcements to try to grab headlines and capital before someone else does, and before demand collapses. Is this one of the ones that might actually get built? No idea.
One likely pivot, if the datacenter doesn't get built, or gets built at a much smaller size, is that they switch to being a private power plant with a bunch of land where they don't have to follow state or county land-use regulations (this is what MIDA is for). That would likely mean bringing in other energy-intensive industries; they have more or less said this in county commission meetings. There's a chance that this is actually far worse, as datacenters (if they use low-water cooling) actually use less water and don't produce as much ground pollution as many industrial land uses.
-
-
R relay@relay.infosec.exchange shared this topic
-
Here's what I hope your takeaway from this thread will be: datacenters come in many sizes, have many uses, and are not necessarily where you'd expect. The impact they have locally depends on how they're powered, how they're cooled, what they're used for, who owns them, and how big they are. It's worth looking at all of these things when considering whether a datacenter project is a good idea or not.
I was on the local news about this, lol my office is such a mess
Closed-loop cooling systems save water, but can be a drain on electricity - KSLTV.com
While closed-loop cooling systems, like the one being touted for a large data center in Box Elder County can save lots of water, they often use more electricity in return, which can impact the environment in other ways, according to Dr. Ricci, a professor in the University of Utah's school of computing.
KSLTV.com (ksltv.com)
-
The reason I went on this little tour was to put in perspective the proposed Stratos datacenter project in Box Elder County, UT.
Stratos is supposedly designed to eventually reach a size of 9 GW. That is more than double the 4 GW that the entire state of Utah currently uses. The entire campus is supposed to be big enough that, for comparison, it would fill over 10% of the Salt Lake Valley, as shown in this image (which I didn't make).
That last datacenter campus? At ~160 MW, those three buildings put together are designed for a load about 1/55th the size of Stratos. That 300 MW natural gas power station we saw in the background? Stratos is supposed to generate its own power on-site, so it will need 30 of those things. (Or maybe more - remember PUE?)
In terms of carbon output, this thing is designed to be an absolute monster.
There's not much getting around that. They have handwaved about including solar and/or wind, but without anything concrete, we should assume this is a whole lot of carbon.
How about water?
Well, that's harder to tell, given all the vagaries and "if"s in the public information so far.
Remember, a datacenter has to get rid of a lot of heat. A datacenter that is generating its own energy on-site has to get rid of *far* more heat.
In the desert West, the most *energy* efficient way of getting rid of heat in the hot summer months is evaporative cooling: you boil water. This has, historically, been a major way of cooling both natural gas plants and datacenters, as well as homes, etc.
The same reason why this works well in the west is the same reason why it's problematic: we have very dry air, so evaporative cooling is very effective, but having dry air is connected to the fact that we don't have much water to begin with.
There *are* ways to air-cool natural gas turbines, and there *are* ways to cool datacenters that are not evaporative cooling. They are more *water* efficient. But they are less *power* efficient, which means, in this context, burning even more natural gas.
The backers of Stratos claim that they are trying to get some very new, high-tech gas turbines that operate without water cooling, or at least with very little. That does assuage some water concerns. But their language is very hedge-y - they're trying, they hope to jump in line for the limited supply of them, etc.
They also claim they will use "closed loop" water systems for cooling the datacenter. There are several things this *could* mean, and we need to know more in order to actually understand it. Most cooling systems for datacenters and even large buildings have a closed loop of water (or another coolant) for moving heat around. That's because we cannot *make* cold, we can only *move* heat. In some datacenters, this cold loop comes into the room, where it's used to cool air, which is blown across the servers. In higher-power-density datacenters, the coolant loop comes all the way to the individual rack in order to cool the air right before it enters the servers. In the most high-tech datacenters (which Stratos would likely be), it comes all the way *inside* the server, directly exchanging heat with the hot bits like CPUs and GPUs.
Coolant in these kinds of systems circulates, it's closed, you can generally consider the coolant loop to consume very little to no water after it's been filled.
But: you still have to make the heat go away somehow. This is where Stratos *might* use evaporative cooling. Or they might opt for one of the more expensive, less energy efficient dry systems. Saying "we have a closed loop" only tells us *part* of the story!
Here's what we know: the Stratos people have secured 13,000 acre-feet of water rights. In numbers that mean more to most of us, that's about 4 billion gallons per year.
They *claim* that's far more water than they need, and they won't use most of it.
But: if they don't manage to get their air-cooled gas turbines (which, in addition to being less efficient, also cost more), or decide to go with some evaporative cooling for the datacenter (because it's cheaper and uses less power), they could very easily use that much water. We are very much in a "trust me" situation, and it's not clear that we *should* trust what developers say when they are trying to get permits. We need to get independent studies and binding contracts.
For those who aren't locals, you might not be aware, but: the Great Salt Lake is shrinking. People are trying (not hard enough, probably) to save it. Not just because hey, what would we call our city without it, but also because the lakebed is full of chemicals we'd rather not be breathing in, thanks.
Stratos would not literally pull water out of the lake (which it is quite close to). But: the water rights they have obtained are in the watershed of the lake. So: if they use the water rights they have obtained, they might well contribute to the drying up of the lake.
The point here is that: they are hoarding water rights that they claim they will not use - the more reasonable bet is to assume they will use them; we need a study by actual hydrologists to understand whether using the water would accelerate the lake's demise.
And, you will notice that I have not even touched on a ton of *other* issues, such as:
1) Is there actually demand for all of these computers?
2) Would it be a good idea to fill this demand even if it does exist?
3) Can we build enough computers to fill this thing in a reasonable time anyway?
4) How far will this project get before the AI bubble pops, and will it leave anyone other than the investors holding the bag?
5) If it does get fully built, what other resources (like more water rights) might they go after?
6) Is it a wise idea to provide huge tax breaks to companies that expect to be highly profitable?
7) This is being done though the Military Installation Development Authority - what's the actual military connection here?
Regardless of whether it's wet or dry, is dumping this much heat into one valley a good idea?
9) There's no way that burning that much natural gas doesn't raise gas and electricity prices.
10) Can we trust the developers' numbers for how many jobs this will create locally?Just to name a few.
@ricci that is absurd.
I've used a a couple and toured a couple more of the US's largest supercomputer facilities, each of which manages to live in a single normal-sized building. These things run simulations of the universe. My stuff could take hours, maybe a day to run, but I know other stuff running there took weeks or months, on thousands of nodes. The facility I've worked with the most, NERSC, serves about 11k users for scientific research.
I struggle to imagine what you could possibly do with the scale of compute proposed at Stratos, even if it served the entire population of the US.
-
I was on the local news about this, lol my office is such a mess
Closed-loop cooling systems save water, but can be a drain on electricity - KSLTV.com
While closed-loop cooling systems, like the one being touted for a large data center in Box Elder County can save lots of water, they often use more electricity in return, which can impact the environment in other ways, according to Dr. Ricci, a professor in the University of Utah's school of computing.
KSLTV.com (ksltv.com)
@ricci
Interesting! So do you use distilled water for the closed loop?
