It turns out GenAI code changes are causing serious incidents and outages at Amazon with "high blast radius" https://www.ft.com/content/7cab4ec7-4712-4137-b602-119a44f771de

[timotimo@peoplemaking.games]

@cwebber "While generative AI does have its uses, especially in specialized fields like medical research [...]"

surely they did not mean generative AI? certainly they mean the more traditional neural networks and other machine learning approaches that have already been used successfully before the LLM hype took off?

[deech@mastodon.social]

@cwebber @KimPerales I'll bet $100 the incidents would have happened even with sign off. No one wants to review slop.

[threkk@mastodon.social]

@cwebber @KimPerales None saw this coming

[sandrockcstm@orbitalstation.one]

@cwebber I love that Amazon just admitted that high-risk changes were pushed to production by Jr Engineers with no review by Senior staff. There's so much to unpack here.

[aaron@social.caskey-demaret.se]

@cwebber re. Tom's Hardware article: Why must every story about how genAI sucks end with "While GenAI is definitely useful in some circumstances..." or something similar? Stories about how a Coal plant poisoned a river don't end in "While it's true that coal plants generate useful electricity..." as though that is something that needs reinforcing.

[jennyfluff@chitter.xyz]

@cwebber @KimPerales AmazonSlopService

[abhayakara@mastodon.nl]

@deech @cwebber @KimPerales

Ah, but then there would have been a senior engineer to blame, and it would no longer be the LLM looking bad...

[ox1de@cyberplace.social]

@cwebber @KimPerales interesting senior devs weren't reviewing them

[cwebber@social.coop]

@Ox1de @KimPerales A bunch of people are saying this, but it's common for mid-level engineers to do code reviews also normally

[agentultra@types.pl]

@KimPerales@toad.social @deech @cwebber not a matter of “if” it would happen in my mind, but when.

Human review is a limited control gap in the system of pushing changes to a production system and has limited utility in preventing errors.

Most empirical studies I’ve read seem to agree that human code review has a marginal effect on error rates that disappears of the reviewer had read a couple hundred SLOC in the last hour.

CEOs aren’t engineers and most software developers aren’t either, IMO. Liability is very screwy in our industry.

Making developers sign off on changes they can’t even hope to vet is a disaster waiting to happen.

[ox1de@cyberplace.social]

@cwebber @KimPerales yeah, not were I worked, we developed applications for electrical, safe-shutdown and fire protection engineers for the nuclear industry and were regular audited by the NRC.

I get it's a different situation, or is it? Doesn't amazon have government contracts that deal with safety and civilian lives?

[mnf@mastodon.social]

@twipped @cwebber https://archive.is/wXvF3

[alice@lgbtqia.space]

@cwebber Amazon, WTF, Seriously?? (AWS)

@KimPerales

[zwol@masto.hackers.town]

@arichtman @deech @cwebber @KimPerales Yeah I can't remember the official psychology term either but it's real. Same phenomenon as it being hard to notice typos. And, in fact, same phenomenon as drivers not seeing bikers and walkers, which may be a better way to highlight the safety concerns here.

[stephanie@thetransagenda.gay]

@twipped@twipped.social @cwebber@social.coop Does this work? https://archive.is/hLd8X