hah, I worked in a basement for what I came to realize was organized crime (I left after 6 months) and everything in the racks had to be mounted a couple feet off the floor because the parking lot would leak into the server room during very heavy rain.

We had multiple sump pumps on marine batteries that would pump the water up and outside where it likely just drained back in eventually.

We had a roof leak that poured gallons of rainwater onto three racks of servers in our data center once. That was ... exciting.

Coincidentally, that's also when we discovered our environment monitoring software was complete shit.

Given the state of the world I think there are engineers everywhere that will get to live through this same experience in the next decade or so.

One question, since it was an exchange server, how could you tell it wasn't functioning?

They say LLMs have been deleting production for decades...

It is wild that people turn this shit loose in their VPC, though...

...Sounds like before any LLM was ever used that PocketOS had abysmal design, development, and deployment procedures which created many gaps that were super easy for an agent with too many permissions to exploit.

How could running against your staging environment ever impact production like this unless you were already vulnerable to this exact scenario through human mistakes?

A lot of these "LLMs deleted my entire production environment" posts are just CYA from company leadership that their infrastructure wasn't a steaming pile of shit to begin with.