There's one very important thing I would like everyone to try to remember this week, and it is that AI companies are full of shit
-
@mnl when a mafia boss walks into a shop and talks about how much of a shame it would be if something happened to the place, that's also not an empty threat. That's the whole point. You can choose to pay them off, or not. What you absolutely do not do is run to all of your neighbors and redeliver the same threat
@jenniferplusplus true, I hope that's not what I'm doing when I say "there's something to this and you need to pay attention to the impact of LLMs on security", even if I think anthropic is run by dangerous clowns (like you have mythos, and also your other stuff is maybe the most broken software I've ever used
) -
@jenniferplusplus this is maybe more what i'm reacting to. don't dismiss this stuff too quickly and bathe yourself in false comfort. If you are working on software, there's a reasonable chance these things can do a significant chunk of your job better than you. That they can't necessarily do it all, or do so for an extravagant amount of resources doesn't change that. I also don't want to sound contrarian, I know I might be a bit too autistic in my communication style (and I'm just as frustrated and anxious and exhausted like the rest of us).
@mnl @jenniferplusplus you seem fucking exhausting and have a long history on your public profile of AI boosterism so it’s not surprising that your response to both my and Jennifer’s posts is bland hype that doesn’t respond to any of the facts we’ve put forth
oh we’ll be left behind if we don’t adopt this terrible crap? good. leave us the fuck alone.
-
There's one very important thing I would like everyone to try to remember this week, and it is that AI companies are full of shit
Only rarely do their claims actually bear scrutiny, and those are only the mildest of claims they make.
So, anthropic is claiming that their new, secret, unreleased model is hyper competent at finding computer security vulnerabilities and they're *too scared* to release it into the wild.
Except all the AI companies have been making the same hypercompetence claims about literally every avenue of knowledge work for 3+ years, and it's literally never true. So please keep in mind the highly likely possibility that this is mostly or entirely bullshit marketing meant to distract you from the absolute garbage fire that is the code base of the poster child application for "agentically" developed software
You may now resume doom scrolling. Thank you
@jenniferplusplus@hachyderm.io do they give a false positive rate? That seems like a relevant statistic here
-
@jenniferplusplus@hachyderm.io do they give a false positive rate? That seems like a relevant statistic here
@dangerdyke
️I wouldn't believe them if they did
-
@dangerdyke
️I wouldn't believe them if they did
@jenniferplusplus@hachyderm.io same. But I bet its a big number, is what I'm sayin
-
So here's the other thing that bothers me about all this. Regardless of the eventual results, this thing they're doing is *incredibly* resource intensive. They routinely spend billions of dollars on training these models, and billions more on operating them. It's not simple to parse out what fraction of that is directly attributable to the massive scale vuln finder/fabricator. But for the sake of argument lets just pick a plausible number, and call it 50-100 million dollars.
What could we have gotten for 50-100 million dollars of sponsorship for security audits? Prior to this, the largest single investment into FOSS security I'm aware of was the 2015 audit of openssl, after the heartbleed incident. It's hard to find precise costs for that, but I found a few sources estimating 1.2 million dollars, and that is arguably the most security critical piece of software in the world.
But suddenly there's 100x more resources available to do this work, now that producing the artifact can be done with stolen labor? Now that they can externalize the cost of false positives onto the already mostly unpaid maintainers of these projects? Even if their claims are true, which we have no reason to believe and very good reason not to, it's still a travesty
@jenniferplusplus They want to get rid of us. The price doesn't matter.
-
@budududuroiu @jenniferplusplus Let's talk about JavaScript. Have you ever looked at your browser's developer console? On any major website on the planet, there are 8 trillion errors in every one. Two-thirds of them are vulnerabilities, but none of them are exploitable or matter for anything at all. That is what is being found.
Those kinds of errors I've been reviewing, all the ones Daniel's been reviewing too, and I'm seeing it over and over. "Yes, okay, technically that is the buffer overrun, but it doesn't matter because you can't ever get to it!"
@Sempf @budududuroiu @jenniferplusplus
Yes, that is Javascript culture
In other cultures clean builds are mandatory
Impossible, or way too hard, in the fragmented browser world.
That said: that is a chilling excuse to allow a buffer over run. The technical term is "famous last words"
-
There's one very important thing I would like everyone to try to remember this week, and it is that AI companies are full of shit
Only rarely do their claims actually bear scrutiny, and those are only the mildest of claims they make.
So, anthropic is claiming that their new, secret, unreleased model is hyper competent at finding computer security vulnerabilities and they're *too scared* to release it into the wild.
Except all the AI companies have been making the same hypercompetence claims about literally every avenue of knowledge work for 3+ years, and it's literally never true. So please keep in mind the highly likely possibility that this is mostly or entirely bullshit marketing meant to distract you from the absolute garbage fire that is the code base of the poster child application for "agentically" developed software
You may now resume doom scrolling. Thank you
Please check out https://stopgenai.com
-
@jenniferplusplus this is maybe more what i'm reacting to. don't dismiss this stuff too quickly and bathe yourself in false comfort. If you are working on software, there's a reasonable chance these things can do a significant chunk of your job better than you. That they can't necessarily do it all, or do so for an extravagant amount of resources doesn't change that. I also don't want to sound contrarian, I know I might be a bit too autistic in my communication style (and I'm just as frustrated and anxious and exhausted like the rest of us).
> If you are working on software, there's a reasonable chance these things can do a significant chunk of your job better than you
No. They cannot.
But they can make me much better at my job, which is why I use them.
-
There's one very important thing I would like everyone to try to remember this week, and it is that AI companies are full of shit
Only rarely do their claims actually bear scrutiny, and those are only the mildest of claims they make.
So, anthropic is claiming that their new, secret, unreleased model is hyper competent at finding computer security vulnerabilities and they're *too scared* to release it into the wild.
Except all the AI companies have been making the same hypercompetence claims about literally every avenue of knowledge work for 3+ years, and it's literally never true. So please keep in mind the highly likely possibility that this is mostly or entirely bullshit marketing meant to distract you from the absolute garbage fire that is the code base of the poster child application for "agentically" developed software
You may now resume doom scrolling. Thank you
@jenniferplusplus Two things can be true at once:
1. The field of LLMs is fill of grifters and scammers
2. LLMs are a revolutionary technology that will change information processing considerably
