so the take-away from this is that all of this agentic stuff backend is just begging the LLM to please, please not fuck up?

[fen@zoner.work]

@eniko@mastodon.gamedev.place there's a reason why Most Engineers don't recognize Software Engineering as an actual engineering discipline, and the current crop isn't doing anything to convince them that should change.

[cxberger@mastodon.boiler.social]

@eniko gotta preach the TESCREAL gospel or the Basilisk will get ya

[jimfl@hachyderm.io]

@fen @eniko A while back, I was renting an ADU in the backyard of an aeronautical engineer. We got to talking about our respective careers (like you do) and when I told him what I did, he said, “oh, you’re a software guy.”

Pretty accurate.

[jedimb@mastodon.gamedev.place]

@eniko A vast mass of bad code held together by duct tape (and prayers).

[bovaz@misskey.social]

@alice@mk.nyaa.place @eniko@mastodon.gamedev.place because right now brain goes brap, this post just made me realize that "you're absolutely right" can be shortened to YAR.

Pirates are onto something good.

[foxyoreos@gulp.cafe]

@eniko YES!!!

I have been yelling about this in different places for 4 years. Before they really took off I used to do hobby red-teaming around GPT-3 and 4, and there was this moment of realizing that none of the security for this stuff fucking works, at all, and yet no one wanted to admit that, it was just layers of programmers going "what if there's a second LLM though", and stuffing their heads in the sand, and NOTHING HAS CHANGED.

[foxyoreos@gulp.cafe]

@eniko the only difference now is that the media has bought into it and now we have "agentic" browsers and all this shit and literally all of it is fundamentally impossible to secure - and the entire tech space (not entire but like, you know) has gone, "but what if we pretend it's fine tho".

4 years ago I was arguing with programmers about this and they were like "well by the time it gets access to your emails, this will be fixed."

It was not fixed.

[foxyoreos@gulp.cafe]

@eniko we have replaced solid software security with the equivalent of the Google SEO wars. We have expanded phishing attacks so that now they work on your computer itself, not just on you.

We are so unbelievably fucked, none of this can be used in a sensitive environment.

And the overwhelming consensus from researchers is that this is impossible to solve, all you can do is beg for the computer not to fuck everything up in increasingly desperate ways.

[foxyoreos@gulp.cafe]

@eniko (this is also why "a chat prompt" is the wrong way to measure energy costs for these things, because the only way to get them to do anything halfway useful is to burn tokens like a forest fire. Not sure about the output? Run an entirely separate LLM to check! Run it 3 times and average the results! Run it in a loop until the test passes! Oops, the format was wrong, run it again and see what happens.

[rootwyrm@weird.autos]

@eniko @catsalad this isn't even a religious cult, it's a fucking *cargo* cult.

[silvermoon82@wandering.shop]

@eniko
Also, "please don't do crimes".

[foxyoreos@gulp.cafe]

@eniko and I actually think that this is why "agents" are when this slop took off for programmers. Because it lets that happen in the background where you don't have to see your shame.

But like.. this is also why even assuming they're true, the "single query is like running your microwave for a second" shit is so disingenuous. No one is doing a single query.

They are leaving the microwave running 24x7, which turns out is actually quite bad for the environment!

[foxyoreos@gulp.cafe]

@eniko measuring LLM energy usage per query is like measuring gas efficiency "per rotation of your car's wheel"

[malwareminigun@infosec.exchange]

@eniko maybe this is a hot take but I think having employees can be a lot like this which is why it doesn’t raise red flags for managers