a lot of "AI deleted their prod data and backups" and not enough "they had their backups in the same volume as their prod data"

[opny721@mastodon.uno]

@lzg stupid prompt from unprepared people. Back in time, newbies with root access to prod have done the same over and over.

[the_wub@mastodon.social]

@phil_stevens @lzg We need to remember that unless they routinely tested their back ups by restoring them to a server that mirrored their production system accurately then there is no way that they could say that the had any back ups in the first place.

As the AI agent was allowed to delete everything including the alleged back ups we now have no way of checking if the back ups could have been used to restore their system.

[phil_stevens@mastodon.nz]

@the_wub @lzg Schrödinger's disaster recovery plan.

[wouter@pleroma.debian.social]

@lzg
I mean, "AI broke prod" is the modern version of "computer says no"...

[kanamauna@sauropods.win]

@lzg

AI is sounding more and more like a big dog that eats homework.

[endlessmason@hachyderm.io]

@jaykass @lzg
It costs a lot to measure output, so you can just pretend you know what the output is and what it looks like

The added bonus is that you can simply say "aah yes, they are 20% more productive and that's worth it" even when the "industry" recommends consuming half your salary worth of AI tokens and by any actual "legacy" metric you can see that people are indeed not doing more work when they spend half their day chatting to the Automatic Dumbass

[grovewest@mstdn.social]

@lzg I would love to see the prompt someone entered that caused AI to delete the backups. And I can imagine AI replying ‘Okay, you asked for it’.

[tadnaff@pnw.zone]

. @lzg I'm ok with drone striking guys as well. Take away the "suck" knob and see what other tools are in the box.

I was going to use AI to do the needful, but this is somehow better:

[teflontrout@beige.party]

@lzg

"the backups were on the same volume so when they were wiped the backups were too"

and what did we learn about backup data storage? No no, I mean back in school, what was the first thing you learned about backup data storage

[endolexi@social.vivaldi.net]

@lzg also true where AI is pushing out freelancing artists, I guess. They aren't employed anywhere in the first place, but in this case I readily blame countless guys + businessed, who rather prompt crappy AI music, paintings etc. than hiring artists.

[kc@social.coop]

I feel this is one of those everybody is wrong kind of situations that has the air of OVH putting their customer backups on the same site, if not the same room, that was only found out by the customers when their data centre went up in flames

OVHcloud condamné à payer 250 000 € à des clients pour perte de données dans l'incendie de son centre de données à Strasbourg | Actualite Cloud

OVHcloud, le fournisseur de services de cloud français, a été condamné à payer 250 000 euros à deux entreprises ayant perdu des données critiques lors de

Actualité Cloud – IA, Cloud Souverain, FinOps et Sécurité (ActualitéCloud) (actualitecloud.com)

[drewtowler@mas.to]

@lzg
And not enough "they decided to touch AI".

[reiddragon@fedi.catto.garden]

@lzg We're talking about people who think a chatbot is sentient, the shoddy backup strategy was the least of their concerns

[jpetazzo@hachyderm.io]

@lzg I suppose that you meant "they had their backups in the same cloud account, and the credentials they were using had delete permission on said backups, and they didn't know about or didn't set up a safety net with lifecycle rules or whatever", which amounts to the same but is alas a mistake that's much easier to make (I'd bet that a huge majority of folks out there, when provisioning infra, use credentials that also have permission to destroy backups!)

[dalias@hachyderm.io]

@lzg The whole selling point is that it lets those humans pretend they weren't behind it.

[literatesavant@mastodon.au]

@lzg I have deleted plenty of prod data in my time, I didn't even NEED AI to do that!

[david_chisnall@infosec.exchange]

@lzg

If I understood their report correctly, this was a platform issue not theirs. They had backups as a separate abstraction from volumes, but a policy that deleting a volume also deleted the backups of a volume.

I don’t think it’s unreasonable to assume that, if a platform offers ‘backups of X’ as a feature then deleting X does not delete the backups (at least, not immediately. Deleting backups after some configurable grace period would be fine).