There's a lot of stuff going around about datacenters, so I decided to do a quick tour yesterday of some of the datacenters in the Salt Lake Valley.
-
@Dougfir In county council meetings, they've claimed they are going to build some hotels for contractors and restaurants, etc. Probably on the parcel of land they got right off I-84. But they seem to expect that on-site staff (which I think they are likely overestimating to make it look more attractive) will live in Brigham City, Snowville, etc.
The area already has a similar problem with the rocket plant at Promontory point. Both my brothers did internships there, and they had to get up super early to take a company bus out there from Brigham City.
@ricci
A lot of the goldmines in Nevada are remote so there are busses running crews back and forth from towns all the time.
I still don't believe their handwaving about being able to source that much power generation capacity that quickly. -
@ricci beside using electricity and water, data centers contribute to heating up the local environment . Curious to know how much effect the large ones will have...
https://arxiv.org/abs/2603.20897@lpryszcz Yep, even if you are energy-efficient at shedding heat, you are still shedding heat!
‘So much worse than I even thought’: Utah’s ‘hyperscale’ data center could create massive heat island near Great Salt Lake
Skeptics of the proposed hyperscale data center in Box Elder County are sweating about a lot more than its energy demands and potential toll on water supplies.
The Salt Lake Tribune (www.sltrib.com)
I think one of the things going on here is that the assumption is that 10x as big is "only" 10x as bad, but scales that large certainly have the possibility of qualitative changes that we might not have a good understanding of (and which we should not just take the developers' word on)
-
@ricci
A lot of the goldmines in Nevada are remote so there are busses running crews back and forth from towns all the time.
I still don't believe their handwaving about being able to source that much power generation capacity that quickly.@Dougfir yep, it seems extremely unlikely, and I'm not inclined to take the word of another guy who plays a businessman on TV
-
Here's what I hope your takeaway from this thread will be: datacenters come in many sizes, have many uses, and are not necessarily where you'd expect. The impact they have locally depends on how they're powered, how they're cooled, what they're used for, who owns them, and how big they are. It's worth looking at all of these things when considering whether a datacenter project is a good idea or not.
@ricci This is REALLY thoughtful and informative; thank you. (And it's worth saving/sharing even outside Mastodon, so: hey, @mastoreaderio ! Unroll!)
-
@ricci This is REALLY thoughtful and informative; thank you. (And it's worth saving/sharing even outside Mastodon, so: hey, @mastoreaderio ! Unroll!)
@msbellows here's the unrolled thread: https://mastoreader.io?url=https%3A%2F%2Fc.im%2F%40msbellows%2F116557139885627239
Next time, kindly set the visibility to 'Mentioned people only' and mention only me (@mastoreaderio). This ensures we avoid spamming others' timelines and threads unless you intend for others to see the unrolled thread link as well.
Thank you!
-
@msbellows here's the unrolled thread: https://mastoreader.io?url=https%3A%2F%2Fc.im%2F%40msbellows%2F116557139885627239
Next time, kindly set the visibility to 'Mentioned people only' and mention only me (@mastoreaderio). This ensures we avoid spamming others' timelines and threads unless you intend for others to see the unrolled thread link as well.
Thank you!
@mastoreaderio
Well, this is handy
@msbellows -
Next, an even smaller datacenter, that just about anyone in #SLC has seen! This is XMission, a local Internet Service Provider that's been running since 1993, so one of The Ancients in Internet time. It's on a very busy part of 4th South, and if you've been by at night, you've seen the big LED display on the front of the building that they put various animations on.
One of the things that I *think* is probably in this building is SLIX: https://slix.net/traffic/ - this is an Internet Exchange Point (IXP), where various carriers meet up to exchange traffic without it having to travel long distances. These are often run as a sort of community infrastructure - it's in the best interests of all networks involved to connect to each other so that they can do their jobs more efficiently.
SLIX is fairly small (according to their own data they carry ~200Gbps, with some spikes up to 1Tbps). There are about 40 networks that meet there: https://slix.net/participants/ . Funny story, when I first got Google Fiber at my house, I was getting routed through California to get to the University of Utah campus just a few miles away. I pinged a guy I know who pinged a guy he knows who ... learned that some of the participants in SLIX didn't have their routes set up right. A config change later, and not only me, but basically everyone on any commercial ISP in the Salt Lake Valley had much more direct routes to campus!
@ricci SLIX lists their organizational address publicly on PeeringDB, so you are indeed correct.
https://www.peeringdb.com/org/8808(I guess it's possible that this is just the org's "business address," but I *highly* doubt it.)
-
@ricci SLIX lists their organizational address publicly on PeeringDB, so you are indeed correct.
https://www.peeringdb.com/org/8808(I guess it's possible that this is just the org's "business address," but I *highly* doubt it.)
@CiscoJunkie In this case their business address is adjacent to their datacenter. But yeah I don't see why they'd have it anywhere else, they have a datacenter
and it's in a fairly fiber-rich area -
There's a lot of stuff going around about datacenters, so I decided to do a quick tour yesterday of some of the datacenters in the Salt Lake Valley. Some are indeed quite large, but there are a bunch of smaller ones too - and they are not always where you think!
All of these are publicly known, and you can find them (and ones in your own area) at https://www.datacentermap.com/ .
Let's start with a datacenter that I go by all the time! It's across the street from my grocery store in downtown #SLC. It's listed as a colocation facility; datacenters are famously secretive about who their tenants are, but we can guess that it probably hosts servers belonging to nearby businesses, especially ones that want their storage, etc. nearby, but don't want to have to maintain a secure, cooled room. Given the number of banks that have headquarters nearby, I'd bet at least some of them are customers.
This is a fairly little guy, with apparently 16k square feet of floorspace and 1.6MW of power.
@ricci Great thread, Rob! I've been working in datacenters for decades but I still found it educational. Thank you!
-
@ricci Great thread, Rob! I've been working in datacenters for decades but I still found it educational. Thank you!
@fivetonsflax I'm glad you liked it! Feel free to add anything if you want
-
Next, an even smaller datacenter, that just about anyone in #SLC has seen! This is XMission, a local Internet Service Provider that's been running since 1993, so one of The Ancients in Internet time. It's on a very busy part of 4th South, and if you've been by at night, you've seen the big LED display on the front of the building that they put various animations on.
One of the things that I *think* is probably in this building is SLIX: https://slix.net/traffic/ - this is an Internet Exchange Point (IXP), where various carriers meet up to exchange traffic without it having to travel long distances. These are often run as a sort of community infrastructure - it's in the best interests of all networks involved to connect to each other so that they can do their jobs more efficiently.
SLIX is fairly small (according to their own data they carry ~200Gbps, with some spikes up to 1Tbps). There are about 40 networks that meet there: https://slix.net/participants/ . Funny story, when I first got Google Fiber at my house, I was getting routed through California to get to the University of Utah campus just a few miles away. I pinged a guy I know who pinged a guy he knows who ... learned that some of the participants in SLIX didn't have their routes set up right. A config change later, and not only me, but basically everyone on any commercial ISP in the Salt Lake Valley had much more direct routes to campus!
-
Here's what I hope your takeaway from this thread will be: datacenters come in many sizes, have many uses, and are not necessarily where you'd expect. The impact they have locally depends on how they're powered, how they're cooled, what they're used for, who owns them, and how big they are. It's worth looking at all of these things when considering whether a datacenter project is a good idea or not.
@ricci naive question, wouldn't building large amounts of solar panel be more energy efficien
-
Here's what I hope your takeaway from this thread will be: datacenters come in many sizes, have many uses, and are not necessarily where you'd expect. The impact they have locally depends on how they're powered, how they're cooled, what they're used for, who owns them, and how big they are. It's worth looking at all of these things when considering whether a datacenter project is a good idea or not.
@ricci Thank you for the overview.
What I don't understand is, why build data centers in areas with warmer climates, when colder ones would be... well, easier to cool?Aren't economic and ecologic incentives aligned here?
Data centers for compute in particular (as opposed as, for response time) don't need to be in any particular geographical area anyway, do they? -
@ricci naive question, wouldn't building large amounts of solar panel be more energy efficien
@gerbrandvd I don't know the exact math on this, unfortunately. What I do know though is that you'd need both solar *and* storage, in a setting like this where they're generating all of their own power on-site, they'd need to generate far more power than they need during the brightest hours of the day, then use it overnight and/or when it's overcast.
Then there's also the fact that one generates far less power from solar during the winter when the days are shorter and the angle of the sun in the sky is less favorable (and that you'd have to clear the panels of snow).
Solar might be a reasonably good match for the cooling part of the load; you can sometimes get away with using outside air when it's cool enough (winter and sometimes at night) but would be a lot harder to make work for the actual computing load, since that's going to run 24/7/365 (esp. if this is used for AI training)
-
@ricci Thank you for the overview.
What I don't understand is, why build data centers in areas with warmer climates, when colder ones would be... well, easier to cool?Aren't economic and ecologic incentives aligned here?
Data centers for compute in particular (as opposed as, for response time) don't need to be in any particular geographical area anyway, do they?@phairupegiont You are correct! It's generally easier to cool things down when it's colder outside! Here in northern Utah some datacenters - with far less power density than this one - are able to just use outside air to cool for a good chunk of the year. With the kinds of heat loads generated by warehouses of GPUs; well, I suspect their cooling needs are indeed lower in the winter, but they probably need active cooling all the time anyway.
There are some datacenters in Finland that even use cold seawater as part of their cooling systems!
So, why build in places that get hot part of the year? Well, if you are willing and able to use water for evaporative cooling, that's pretty effective in very dry environments - and can be cheap depending on the cost of water. Sometimes, the availability of power is a big thing too - in this case, there is an existing natural gas pipeline running through the valley that they intend to tap. For some kinds of datacenters, it's important to be near your users - though that's less important for AI training, which is what this one would likely be used for.
-
This is the biggest datacenter I visited - campus, actually. This facility is in West Jordan, near the South Valley Regional Airport. It's big enough that I have to post several pictures to get you a real sense of the size (but it's not the biggest datacenter in Utah.)
What you're looking at here is three buildings that, together, have a power capacity that's reported (depending on the source) to be around 160 MW (put a pin in that number too.)
Two of these buildings are multi-tenant (the ones with the flat white roofs) like the others we've seen.
That third one in the back, with all of the cooling towers on top, has supposedly been built for a single hyperscaler, and is supposedly something like an 80-100MW building. Which hyperscaler? That information is not public. That's a whole lot of cooling on the roof (which is reported to be water-free), so my money would be on this being an AI data center.
In these pictures, you can see more electrical infrastructure. Bringing that much power into one place takes a lot of wires.
@ricci interesting to me that the (presumably) higher-density facility is taller (multi-story). I've noticed that with other new-built high-density facilities.
to save ground space? maybe there is water cooling involved and it is helpful to have that equipment above/below servers? or high ceilings help with thermal engineering?
-
Here's what I hope your takeaway from this thread will be: datacenters come in many sizes, have many uses, and are not necessarily where you'd expect. The impact they have locally depends on how they're powered, how they're cooled, what they're used for, who owns them, and how big they are. It's worth looking at all of these things when considering whether a datacenter project is a good idea or not.
@ricci great thread, thanks!
-
@ricci interesting to me that the (presumably) higher-density facility is taller (multi-story). I've noticed that with other new-built high-density facilities.
to save ground space? maybe there is water cooling involved and it is helpful to have that equipment above/below servers? or high ceilings help with thermal engineering?
@bnewbold yeah I dunno! In this particular campus, that building seems to have consumed all remaining space on the lot, so it *could* just be an issue of the older ones not being as space constrained, but it also could be a fundamentally different design. My assumption (based only on trends, not any special knowledge) is that this new one also takes the cooling loop all the way to the chip - I don't know what that does with optimal layouts
-
Here's what I hope your takeaway from this thread will be: datacenters come in many sizes, have many uses, and are not necessarily where you'd expect. The impact they have locally depends on how they're powered, how they're cooled, what they're used for, who owns them, and how big they are. It's worth looking at all of these things when considering whether a datacenter project is a good idea or not.
@ricci great thread, Rob. I worked for one of the big cloud providers and got to spent quite a bit of time with the datacenter team before I retired. I was blown away by the innovations being implemented to reduce power and cooling requirements. All of that is moot now. We were talking about the potential of 25kW racks. Now they’ve completely blown past that with 100kW racks. It’s insane.
-
Here's what I hope your takeaway from this thread will be: datacenters come in many sizes, have many uses, and are not necessarily where you'd expect. The impact they have locally depends on how they're powered, how they're cooled, what they're used for, who owns them, and how big they are. It's worth looking at all of these things when considering whether a datacenter project is a good idea or not.
@ricci great thread, it really puts things in perspective!
