I've seen people claiming - with a straight face - that mechanical refactoring is a good use-case for LLM-based tools.
-
@adingbatponder yes, but why? Which packages where taking so long? Firefox has almost 4 millions of lines of Rust and it takes only a few minutes to build them.
@gabrielesvelto No clue. At the time it was chrome that pushed it into silly territory. But this was inside a flake. All I know was when it was refactored it was able to use 32 processors instead of only 2.
-
@a how so? Now you don't need a person to run that particular exploit for years, you can just poison an LLM so that whenever someone generates a sufficiently large sequence of commits the exploit can be injected in them directly. No user intervention and it can be done at scale. And it can be done in closed-source codebases too, it's just a matter of someone using a bot on them.
@gabrielesvelto @a You are correct, LLMs have made this exploit many times easier to execute.
-
And it's crucial to remember what happened during the xz compromise: a chain of seemingly innocuous commits where malicious behavior was hidden, then triggered by changing a single character in a generated file. A SINGLE CHARACTER. If you truly believe you can catch that by manually reviewing thousands upon thousands of machine-generated commits obtained via black-box training data I'm sorry, but you're being extremely naive.
@gabrielesvelto that incident example of Metamorphic Malware?
-
@gabrielesvelto@mas.to it is also worth remembering that the xz incident happened WITHOUT LLMs involved, so you comparison is not a very good one
@a @gabrielesvelto no it's actually an extremely well-made point. if we were (almost) unable to detect something like that in a FOSS project (not in the code, in a debug object mind you) then where do we get off introducing the black box which increases complexity a thousand times and claim we can still quality-control the final product. not to mention it took someone years to gain influence within the project vs a model that just scrapes public code indiscriminately
-
@a @gabrielesvelto no it's actually an extremely well-made point. if we were (almost) unable to detect something like that in a FOSS project (not in the code, in a debug object mind you) then where do we get off introducing the black box which increases complexity a thousand times and claim we can still quality-control the final product. not to mention it took someone years to gain influence within the project vs a model that just scrapes public code indiscriminately
@silhouette@dumbfuckingweb.site @gabrielesvelto@mas.to who said this already hadn't happened before the advent of LLMs? you detected ONE, you don't know how many you haven't
-
@a @gabrielesvelto no it's actually an extremely well-made point. if we were (almost) unable to detect something like that in a FOSS project (not in the code, in a debug object mind you) then where do we get off introducing the black box which increases complexity a thousand times and claim we can still quality-control the final product. not to mention it took someone years to gain influence within the project vs a model that just scrapes public code indiscriminately
@silhouette @a @gabrielesvelto most people (by volume AND mass) using LLMs are doing so because they do not have the skills necessary to produce the code in question (they "have the skill to read it" but if you've ever tried reimplementing a compsci research paper without just copying their code as-is you know instinctively that's not the same thing), which means that they are unlikely to tell well-crafted malicious code from legitimate code, knowing that both achieve their results
this is implying they even do review it at all rather than simply relegate this to an agent that only checks if it matches the acceptance criteria (just like a real product manager!), which obviously immediately fails -
@silhouette@dumbfuckingweb.site @gabrielesvelto@mas.to who said this already hadn't happened before the advent of LLMs? you detected ONE, you don't know how many you haven't
@a @gabrielesvelto I don't follow, are you agreeing with me or... what?
-
@a @gabrielesvelto I don't follow, are you agreeing with me or... what?
@silhouette@dumbfuckingweb.site @gabrielesvelto@mas.to I'm not, I'm saying that the xz is a bad example for several reasons, including the fact that (and this was my last point) it is one known case among an unknown number of total cases
-
@silhouette@dumbfuckingweb.site @gabrielesvelto@mas.to I'm not, I'm saying that the xz is a bad example for several reasons, including the fact that (and this was my last point) it is one known case among an unknown number of total cases
@a @gabrielesvelto I still don't follow your line of argument here. You are saying that there are currently an unknown number of potential vulnerabilities in human-generated FOSS code, so we should... hook it up to the complexity generator?
-
@a @gabrielesvelto I still don't follow your line of argument here. You are saying that there are currently an unknown number of potential vulnerabilities in human-generated FOSS code, so we should... hook it up to the complexity generator?
@silhouette@dumbfuckingweb.site @gabrielesvelto@mas.to The argument sounds more like "I know a guy who almost died for peanut allergy, so we should prohibit the peanut production". Yes it is possible. It was also possible in the past. My point is that the use of LLMs doesn't change much the landscape in that regard.
-
@silhouette@dumbfuckingweb.site @gabrielesvelto@mas.to The argument sounds more like "I know a guy who almost died for peanut allergy, so we should prohibit the peanut production". Yes it is possible. It was also possible in the past. My point is that the use of LLMs doesn't change much the landscape in that regard.
@gabrielesvelto@mas.to @silhouette@dumbfuckingweb.site of course, you can do whatever you want, I just think if you are going to criticize the use of LLMs there are better arguments that are less convoluted.
️ -
@csepp @gabrielesvelto Doesn't look like lua really has a good binding to libclang but if you used Python you could use the same libraries that clang-format/tidy do. They're using the actual llvm parser and give you an API to manipulate the AST.
@crazyeddie @gabrielesvelto I'll look into this, I couldn't find many up to date refactoring examples, but looking at the docs it should be possible to get something going. I think I've come across it when I was researching tools for my refactor but the lack of examples turned me off, since I had no idea how much work I'd have to put into it.
-
And it's crucial to remember what happened during the xz compromise: a chain of seemingly innocuous commits where malicious behavior was hidden, then triggered by changing a single character in a generated file. A SINGLE CHARACTER. If you truly believe you can catch that by manually reviewing thousands upon thousands of machine-generated commits obtained via black-box training data I'm sorry, but you're being extremely naive.
@gabrielesvelto this also has the same problem which keeps antivirus software in a Red Queen's race: the attacker has access to the same tools and can tune their attack until it passes before targeting you. It’ll be highly effective against specific obtrusive patterns but that only stops lazy attackers.
-
@silhouette@dumbfuckingweb.site @gabrielesvelto@mas.to The argument sounds more like "I know a guy who almost died for peanut allergy, so we should prohibit the peanut production". Yes it is possible. It was also possible in the past. My point is that the use of LLMs doesn't change much the landscape in that regard.
@a @gabrielesvelto @silhouette "people die from peanut allergy so maybe it isn't such a great idea to introduce machines that have a 0.1% probability of introducing a peanut in every single item in the supermarket" is a pretty good point
-
I think there's an important clarification to be made about LLM usage in coding tasks: do you trust the training data? Not your inputs, those are irrelevant, I mean the junk that the major vendors have dredged from the internet. Because I'm 100% positive that any self-respecting state-sponsored actor is poisoning training data as we speak by... simply publishing stuff on the internet.
@gabrielesvelto after using a few of the LLMs to generate #powerShell code, i don't trust any of them.
-
I think there's an important clarification to be made about LLM usage in coding tasks: do you trust the training data? Not your inputs, those are irrelevant, I mean the junk that the major vendors have dredged from the internet. Because I'm 100% positive that any self-respecting state-sponsored actor is poisoning training data as we speak by... simply publishing stuff on the internet.
@gabrielesvelto LLMs the average internet response to a query, which includes coding ones.
And paraphrasing Carlin: realize how bad average code is, and realize that half the code is worse than that
-
R relay@relay.infosec.exchange shared this topic
