I've seen people claiming - with a straight face - that mechanical refactoring is a good use-case for LLM-based tools.
-
@gabrielesvelto prompt-injections
The project is closed source, and we don't have places where we randomly include text files, if someone IN THE COMPANY manages to introduce malicious code, imho they'd just infect gradle instead of hoping on someone running an LLM to trigger something (other than devs having access to only what they need). State sponsored hackers specifically are really not in my list of things I can defend from, be it from LLMs or whatever introduced attacks
@fourlastor you don't need to do anything special to be a target of state-sponsored actors if your rely on an LLM for your coding tasks. State-sponsored actors have almost certainly poisoned the training data of major commercial LLMs, you don't need to add anything yourself. Remember, these things are trained on anything that's dredged from the internet. *Anything*. Do you really trust what happens within the model? Remember the xz compromise? It can now be done automatically *at scale*.
-
I've seen people claiming - with a straight face - that mechanical refactoring is a good use-case for LLM-based tools. Well, sed was developed in 1974 and - according to Wikipedia - first shipped in UNIX version 7 in 1979. On modern machines it can process files at speeds of several GB/s and will not randomly introduce errors while processing them. It doesn't cost billions, a subscription or internet access. It's there on your machine, fully documented. What are we even talking about?
I think there's an important clarification to be made about LLM usage in coding tasks: do you trust the training data? Not your inputs, those are irrelevant, I mean the junk that the major vendors have dredged from the internet. Because I'm 100% positive that any self-respecting state-sponsored actor is poisoning training data as we speak by... simply publishing stuff on the internet.
-
@adingbatponder why did the project take so long to build?
@gabrielesvelto Well that is what rust seems to be like. I used a lot of packages incl. browser and screen grabbing tools which took ages to build. Like 20 mins. (It was inside a nixos flake though.)
-
I think there's an important clarification to be made about LLM usage in coding tasks: do you trust the training data? Not your inputs, those are irrelevant, I mean the junk that the major vendors have dredged from the internet. Because I'm 100% positive that any self-respecting state-sponsored actor is poisoning training data as we speak by... simply publishing stuff on the internet.
Any blogger can poison the LLMs.
I hacked ChatGPT and Google's AI - and it only took 20 minutes
I found a way to make AI tell you lies – and I'm not the only one.
(www.bbc.com)
-
I think there's an important clarification to be made about LLM usage in coding tasks: do you trust the training data? Not your inputs, those are irrelevant, I mean the junk that the major vendors have dredged from the internet. Because I'm 100% positive that any self-respecting state-sponsored actor is poisoning training data as we speak by... simply publishing stuff on the internet.
And it's crucial to remember what happened during the xz compromise: a chain of seemingly innocuous commits where malicious behavior was hidden, then triggered by changing a single character in a generated file. A SINGLE CHARACTER. If you truly believe you can catch that by manually reviewing thousands upon thousands of machine-generated commits obtained via black-box training data I'm sorry, but you're being extremely naive.
-
@gabrielesvelto Well that is what rust seems to be like. I used a lot of packages incl. browser and screen grabbing tools which took ages to build. Like 20 mins. (It was inside a nixos flake though.)
@adingbatponder yes, but why? Which packages where taking so long? Firefox has almost 4 millions of lines of Rust and it takes only a few minutes to build them.
-
And it's crucial to remember what happened during the xz compromise: a chain of seemingly innocuous commits where malicious behavior was hidden, then triggered by changing a single character in a generated file. A SINGLE CHARACTER. If you truly believe you can catch that by manually reviewing thousands upon thousands of machine-generated commits obtained via black-box training data I'm sorry, but you're being extremely naive.
@gabrielesvelto@mas.to it is also worth remembering that the xz incident happened WITHOUT LLMs involved, so you comparison is not a very good one
-
I've seen people claiming - with a straight face - that mechanical refactoring is a good use-case for LLM-based tools. Well, sed was developed in 1974 and - according to Wikipedia - first shipped in UNIX version 7 in 1979. On modern machines it can process files at speeds of several GB/s and will not randomly introduce errors while processing them. It doesn't cost billions, a subscription or internet access. It's there on your machine, fully documented. What are we even talking about?
@gabrielesvelto "people are using this inadequate and problematic tool for a job, so let me suggest they use this different completely inadequate tool instead."
Speaking of unfortunate painful experience, using grep and sed at scale for mechanical refactoring very much randomly introduces mistakes into a codebase. I beg developers to use *at least* syntax-aware tools for mechanical refactoring jobs -
@gabrielesvelto@mas.to it is also worth remembering that the xz incident happened WITHOUT LLMs involved, so you comparison is not a very good one
@a how so? Now you don't need a person to run that particular exploit for years, you can just poison an LLM so that whenever someone generates a sufficiently large sequence of commits the exploit can be injected in them directly. No user intervention and it can be done at scale. And it can be done in closed-source codebases too, it's just a matter of someone using a bot on them.
-
@a how so? Now you don't need a person to run that particular exploit for years, you can just poison an LLM so that whenever someone generates a sufficiently large sequence of commits the exploit can be injected in them directly. No user intervention and it can be done at scale. And it can be done in closed-source codebases too, it's just a matter of someone using a bot on them.
@gabrielesvelto@mas.to you didn't need an LLM for xz, that is how
-
@fourlastor you don't need to do anything special to be a target of state-sponsored actors if your rely on an LLM for your coding tasks. State-sponsored actors have almost certainly poisoned the training data of major commercial LLMs, you don't need to add anything yourself. Remember, these things are trained on anything that's dredged from the internet. *Anything*. Do you really trust what happens within the model? Remember the xz compromise? It can now be done automatically *at scale*.
@gabrielesvelto and ok, but what is the *actual* scenario you're imagining? because my coding tasks go as such when I use LLMs:
1. I have 10-15 classes that need to change the way we do X from Y to Z
2. I prompt the LLM, telling it "change A,B,C so that they use Z instead of Y"
3. I review the code, fixing mistakes as I see them
1/x because post length limits -
@gabrielesvelto and ok, but what is the *actual* scenario you're imagining? because my coding tasks go as such when I use LLMs:
1. I have 10-15 classes that need to change the way we do X from Y to Z
2. I prompt the LLM, telling it "change A,B,C so that they use Z instead of Y"
3. I review the code, fixing mistakes as I see them
1/x because post length limits@gabrielesvelto
The code change is frankly pretty simple, we're talking of stuff on the level of "migrate Book so instead of using function calls, uses annotations for ABC, update the call sites", we're not talking about "change this complex piece of code so that it does complex ABC in another complex XYZ way". The realm of errors is "I know that Foo doesn't work well by itself and needs extra care" -
@gabrielesvelto
The code change is frankly pretty simple, we're talking of stuff on the level of "migrate Book so instead of using function calls, uses annotations for ABC, update the call sites", we're not talking about "change this complex piece of code so that it does complex ABC in another complex XYZ way". The realm of errors is "I know that Foo doesn't work well by itself and needs extra care"@gabrielesvelto anything that goes over the bar of "this is stupid but boring" goes into the "I'll do it by hand because if anything I need to learn how it works before touching it"
-
I've seen people claiming - with a straight face - that mechanical refactoring is a good use-case for LLM-based tools. Well, sed was developed in 1974 and - according to Wikipedia - first shipped in UNIX version 7 in 1979. On modern machines it can process files at speeds of several GB/s and will not randomly introduce errors while processing them. It doesn't cost billions, a subscription or internet access. It's there on your machine, fully documented. What are we even talking about?
@gabrielesvelto Just the other day I saw a goddamn professor claiming that we need to teach chatbots to reason in order for them to do math. As if we haven't had calculators that actually work every time for like 450 years. It's insane.
-
@adingbatponder yes, but why? Which packages where taking so long? Firefox has almost 4 millions of lines of Rust and it takes only a few minutes to build them.
@gabrielesvelto No clue. At the time it was chrome that pushed it into silly territory. But this was inside a flake. All I know was when it was refactored it was able to use 32 processors instead of only 2.
-
@a how so? Now you don't need a person to run that particular exploit for years, you can just poison an LLM so that whenever someone generates a sufficiently large sequence of commits the exploit can be injected in them directly. No user intervention and it can be done at scale. And it can be done in closed-source codebases too, it's just a matter of someone using a bot on them.
@gabrielesvelto @a You are correct, LLMs have made this exploit many times easier to execute.
-
And it's crucial to remember what happened during the xz compromise: a chain of seemingly innocuous commits where malicious behavior was hidden, then triggered by changing a single character in a generated file. A SINGLE CHARACTER. If you truly believe you can catch that by manually reviewing thousands upon thousands of machine-generated commits obtained via black-box training data I'm sorry, but you're being extremely naive.
@gabrielesvelto that incident example of Metamorphic Malware?
-
@gabrielesvelto@mas.to it is also worth remembering that the xz incident happened WITHOUT LLMs involved, so you comparison is not a very good one
@a @gabrielesvelto no it's actually an extremely well-made point. if we were (almost) unable to detect something like that in a FOSS project (not in the code, in a debug object mind you) then where do we get off introducing the black box which increases complexity a thousand times and claim we can still quality-control the final product. not to mention it took someone years to gain influence within the project vs a model that just scrapes public code indiscriminately
-
@a @gabrielesvelto no it's actually an extremely well-made point. if we were (almost) unable to detect something like that in a FOSS project (not in the code, in a debug object mind you) then where do we get off introducing the black box which increases complexity a thousand times and claim we can still quality-control the final product. not to mention it took someone years to gain influence within the project vs a model that just scrapes public code indiscriminately
@silhouette@dumbfuckingweb.site @gabrielesvelto@mas.to who said this already hadn't happened before the advent of LLMs? you detected ONE, you don't know how many you haven't
-
@a @gabrielesvelto no it's actually an extremely well-made point. if we were (almost) unable to detect something like that in a FOSS project (not in the code, in a debug object mind you) then where do we get off introducing the black box which increases complexity a thousand times and claim we can still quality-control the final product. not to mention it took someone years to gain influence within the project vs a model that just scrapes public code indiscriminately
@silhouette @a @gabrielesvelto most people (by volume AND mass) using LLMs are doing so because they do not have the skills necessary to produce the code in question (they "have the skill to read it" but if you've ever tried reimplementing a compsci research paper without just copying their code as-is you know instinctively that's not the same thing), which means that they are unlikely to tell well-crafted malicious code from legitimate code, knowing that both achieve their results
this is implying they even do review it at all rather than simply relegate this to an agent that only checks if it matches the acceptance criteria (just like a real product manager!), which obviously immediately fails
