I have 384TB of ECC DDR4 across two blades with 4 CPUs for a combined core count of 96.
-
@agowa338 @cursedsql @bob_zim @SecurityWriter
I also would lean towards it being GB, although 384 GB does seem quite modest for what I assume is quite a high performance SAN, given it's all solid state.
I once worked on a mid range combined NAS/SAN head that topped out at 1TB for the high-end model. That wasn't just connected to the CPUs, it was also in caches and buffers for other chips in the data path.
That was a few years ago, and I can imagine a high end system might have a lot more, but 384TB does sound excessive, especially if there's only 192 SSDs hanging off it. It might be possible to load the entire array into RAM in that case.@GerardThornley @agowa338 @bob_zim @SecurityWriter yes that's why I figured it was still credible because anyone who has a 384 tb solid state san might be rich enough to back it entirely in ram
-
@GerardThornley @agowa338 @bob_zim @SecurityWriter yes that's why I figured it was still credible because anyone who has a 384 tb solid state san might be rich enough to back it entirely in ram
@GerardThornley @agowa338 @bob_zim @SecurityWriter also if they were 8tb instead of 2tb it would just be like a huge working set
-
I have 384TB of ECC DDR4 across two blades with 4 CPUs for a combined core count of 96.
It powers a fully populated 192 disk solid state SAN.
I was told it was old and in need of replacing, but apparently now it’s worth more than the GDP of the UK.
Can’t afford to run it (or hear my thoughts when in the vicinity)… but I can sit atop it like a fucking dragon.
And I will.
@SecurityWriter@infosec.exchange I'm imagining the dragon hoard as a pile of equipment that refuses to be thrown out. Who am I kidding, that was my office before we started having kids.
-
@GerardThornley @agowa338 @bob_zim @SecurityWriter also if they were 8tb instead of 2tb it would just be like a huge working set
@cursedsql @agowa338 @bob_zim @SecurityWriter I don't know what's typical for these things with solid state, but with spinning rust (and a few years ago) large arrays typically didn't use drives much bigger than about 600GB. The preference would be for more drives, rather than larger. The reason for that was to do with failure rates, rebuild times and bandwidth.
The maths might have changed with the technology, but I'd suggest that if you're using SSDs then your focus is probably response time and bandwidth rather than storage density, so I'd expect smaller rather than larger drives. -
Well he only said "DDR4", not that it is used as the systems memory. And PCIe add-on cards for ramdisks exist, sooo
@agowa338 Cards like that exist, but they don’t hold thousands of DIMMs.
-
@cursedsql @agowa338 @bob_zim @SecurityWriter I don't know what's typical for these things with solid state, but with spinning rust (and a few years ago) large arrays typically didn't use drives much bigger than about 600GB. The preference would be for more drives, rather than larger. The reason for that was to do with failure rates, rebuild times and bandwidth.
The maths might have changed with the technology, but I'd suggest that if you're using SSDs then your focus is probably response time and bandwidth rather than storage density, so I'd expect smaller rather than larger drives.@GerardThornley @cursedsql @bob_zim @SecurityWriter
Or you want to place it in an environment where it has to deal with heavy vibrations. Like on a moving trolly or in a vehicle or ... there are multiple reasons for this. It may even just be because you need high random IO speeds...
And the sizing also depends on what you're using it for. Like e.g. if you get your data in to the system in infrequent busts but at multiple TB/s and you've to cache it until it is synced even to SSDs, well
-
@GerardThornley @cursedsql @bob_zim @SecurityWriter
Or you want to place it in an environment where it has to deal with heavy vibrations. Like on a moving trolly or in a vehicle or ... there are multiple reasons for this. It may even just be because you need high random IO speeds...
And the sizing also depends on what you're using it for. Like e.g. if you get your data in to the system in infrequent busts but at multiple TB/s and you've to cache it until it is synced even to SSDs, well
@GerardThornley @cursedsql @bob_zim @SecurityWriter
(The later was an example from scientific environments. I think it was CERN but I'm not sure...)
-
@agowa338 Cards like that exist, but they don’t hold thousands of DIMMs.
@bob_zim But PCIe lane splitters and extenders also exist. And I don't know what the highest archivable density of these cards currently is.
I so far have only had one old one in my hands and seen them in slides in class at my job training about 10 years ago (they were mentioned as accelerator cards primarily used for things like MS Dynamics and SAP databases)...
-
@GerardThornley @cursedsql @bob_zim @SecurityWriter
Or you want to place it in an environment where it has to deal with heavy vibrations. Like on a moving trolly or in a vehicle or ... there are multiple reasons for this. It may even just be because you need high random IO speeds...
And the sizing also depends on what you're using it for. Like e.g. if you get your data in to the system in infrequent busts but at multiple TB/s and you've to cache it until it is synced even to SSDs, well
@agowa338 @cursedsql @bob_zim @SecurityWriter Yep, those are also possibilities. I described what I think is most probable given the information available and scenarios I've seen, but yeah, there are reasons it might be a less typical setup, or my knowledge might be out of date.
-
@GerardThornley @cursedsql @bob_zim @SecurityWriter
(The later was an example from scientific environments. I think it was CERN but I'm not sure...)
@agowa338 @cursedsql @bob_zim @SecurityWriter Yeah, that's sounds pretty plausible for things like the LHC experiments.
-
@agowa338 @cursedsql @bob_zim @SecurityWriter Yeah, that's sounds pretty plausible for things like the LHC experiments.
@agowa338 @cursedsql @bob_zim @SecurityWriter For the vehicle scenario, I know modern trains have a lot of sensors on, and I don't know the sampling rate, but they do only have short windows to upload the data they're in their terminal station. I'm not sure I can believe them needing TBs, though.
On the other hand, a F1 team would probably combine transport (though presumably powered down) with need for high bandwidth and low latency.
-
@bob_zim But PCIe lane splitters and extenders also exist. And I don't know what the highest archivable density of these cards currently is.
I so far have only had one old one in my hands and seen them in slides in class at my job training about 10 years ago (they were mentioned as accelerator cards primarily used for things like MS Dynamics and SAP databases)...
@agowa338 It’s more about the physical space for the cards. Most hold eight DIMMs. Holding 3072 would take 384 cards. That’s nearly a full rack just for the RAM cards, not counting the persistent drives. No way would you run that with even two entire blade frames, let alone two blades.
-
@agowa338 It’s more about the physical space for the cards. Most hold eight DIMMs. Holding 3072 would take 384 cards. That’s nearly a full rack just for the RAM cards, not counting the persistent drives. No way would you run that with even two entire blade frames, let alone two blades.
Hmm true. Makes it quite unlikely.
However I want to push back on one thing. It would still be theoretically possible, as PCIe fabrics and such also exist.
But yea, probably not then. I didn't do the math for the space requirements....
-
Don't forget to mention AI
️@stekopf @SecurityWriter I don't think you have to, it's all they can think of anyway nowadays
-
I have 384TB of ECC DDR4 across two blades with 4 CPUs for a combined core count of 96.
It powers a fully populated 192 disk solid state SAN.
I was told it was old and in need of replacing, but apparently now it’s worth more than the GDP of the UK.
Can’t afford to run it (or hear my thoughts when in the vicinity)… but I can sit atop it like a fucking dragon.
And I will.
@SecurityWriter 384TB of RAM is a lot. Perhaps you meant GB?
-
Don't Breath Fire !!!! you evil dragon !!!
@moppi @SecurityWriter we're not all like that
-
Hmm true. Makes it quite unlikely.
However I want to push back on one thing. It would still be theoretically possible, as PCIe fabrics and such also exist.
But yea, probably not then. I didn't do the math for the space requirements....
@agowa338 Also worth noting 128 GB DDR4 DIMMs are almost universally “load-reduced”, or LR-DIMMs. Those have significantly higher latency than normally expected from RAM, though still lower than persistent storage. For low-latency RAM, you’re generally limited to 64 GB per DIMM, so double everything.
Then you have the PCIe cabling. Each card needs at least two dedicated lanes, otherwise contention wrecks your latency and you may as well use flash. That’s 768 or 1536 twinax bundles.
Lots of RAM is possible, but it takes *bonkers* amounts of space and some very exotic hardware.
-
I have 384TB of ECC DDR4 across two blades with 4 CPUs for a combined core count of 96.
It powers a fully populated 192 disk solid state SAN.
I was told it was old and in need of replacing, but apparently now it’s worth more than the GDP of the UK.
Can’t afford to run it (or hear my thoughts when in the vicinity)… but I can sit atop it like a fucking dragon.
And I will.
@SecurityWriter I never thought about quantifying the perching potential of my computer hardware ...
I have a physical treasure trove, of a sort!
-
I have 384TB of ECC DDR4 across two blades with 4 CPUs for a combined core count of 96.
It powers a fully populated 192 disk solid state SAN.
I was told it was old and in need of replacing, but apparently now it’s worth more than the GDP of the UK.
Can’t afford to run it (or hear my thoughts when in the vicinity)… but I can sit atop it like a fucking dragon.
And I will.
@SecurityWriter I'm not gonna lie, I'm a little turned on.....
More than a little even.
-
I have 384TB of ECC DDR4 across two blades with 4 CPUs for a combined core count of 96.
It powers a fully populated 192 disk solid state SAN.
I was told it was old and in need of replacing, but apparently now it’s worth more than the GDP of the UK.
Can’t afford to run it (or hear my thoughts when in the vicinity)… but I can sit atop it like a fucking dragon.
And I will.
@SecurityWriter@infosec.exchange king of the ram
