"A recent 2026 empirical study titled "Beyond Code Snippets: Benchmarking LLMs on Repository-Level Question Answering" (published on arXiv/ResearchGate) explicitly tested LLMs on codebase comprehension.
-
"A recent 2026 empirical study titled "Beyond Code Snippets: Benchmarking LLMs on Repository-Level Question Answering" (published on arXiv/ResearchGate) explicitly tested LLMs on codebase comprehension. The researchers concluded that high performance often "results from verbatim reproduction of Stack Overflow answers rather than genuine reasoning." " https://www.researchgate.net/publication/403262523_Beyond_Code_Snippets_Benchmarking_LLMs_on_Repository-Level_Question_Answering
@codinghorror I have witnessed cloud based LLMs giving me authoritative-sound answers, when asked questions about macOS xnu kernel code, which came from Linux constructions, and were completely irrelevant. Generative AI is often useless and not fit for its advertised purpose. The limitations are structural and well known to ML researchers, and Noam Chomsky called it already years ago. I wish they'd just jog on and stop bothering real people with what is essentially a shell game.
-
@codinghorror I have witnessed cloud based LLMs giving me authoritative-sound answers, when asked questions about macOS xnu kernel code, which came from Linux constructions, and were completely irrelevant. Generative AI is often useless and not fit for its advertised purpose. The limitations are structural and well known to ML researchers, and Noam Chomsky called it already years ago. I wish they'd just jog on and stop bothering real people with what is essentially a shell game.
@codinghorror The really sad part of this was that the LLMs were caught directly in the act of fabricating erroneous output. I had a Git sparse-checkout of xnu directly in front of me. The technical matter related to TCP interactive behaviour.
-
@codinghorror The really sad part of this was that the LLMs were caught directly in the act of fabricating erroneous output. I had a Git sparse-checkout of xnu directly in front of me. The technical matter related to TCP interactive behaviour.
@bms48 it's better for blank page ideation, and mashups / galactic brain fuzz testing in my opinion, and should always be double-checked by a human
-
R relay@relay.infosec.exchange shared this topic
-
@bms48 it's better for blank page ideation, and mashups / galactic brain fuzz testing in my opinion, and should always be double-checked by a human
@codinghorror I gots no problem with da one-shotting da boilerplate! But the actual useful application is a far cry from what Jensen, who pretends to be everyone's friend, wants you to do the "tokenmaxxing" for.
-
E em0nm4stodon@infosec.exchange shared this topic
-
"A recent 2026 empirical study titled "Beyond Code Snippets: Benchmarking LLMs on Repository-Level Question Answering" (published on arXiv/ResearchGate) explicitly tested LLMs on codebase comprehension. The researchers concluded that high performance often "results from verbatim reproduction of Stack Overflow answers rather than genuine reasoning." " https://www.researchgate.net/publication/403262523_Beyond_Code_Snippets_Benchmarking_LLMs_on_Repository-Level_Question_Answering
@codinghorror I'm genuinely curious how much of the hype is an army of mid level engineers all building the same five web apps as the other guy.
-
R relay@relay.mycrowd.ca shared this topic
-
@codinghorror So ... roughly on par with normal developers then?
@henryk @codinghorror I used to think the "just copying from Stack Overflow" jokes developers made were, you know, jokes. Then I watched all my coworkers embrace LLMs, and I was forced to conclude that they weren't joking.
So, depressing agreement.
-
"A recent 2026 empirical study titled "Beyond Code Snippets: Benchmarking LLMs on Repository-Level Question Answering" (published on arXiv/ResearchGate) explicitly tested LLMs on codebase comprehension. The researchers concluded that high performance often "results from verbatim reproduction of Stack Overflow answers rather than genuine reasoning." " https://www.researchgate.net/publication/403262523_Beyond_Code_Snippets_Benchmarking_LLMs_on_Repository-Level_Question_Answering
@codinghorror I made a code going against "the recommended way of doing it" in several ways for certain task. It was a very conscious decision, these violations are needed to make something new I want. Recently I handed the code to my student and asked her to read and understand it. She handed me what she said was a written summary of her notes. It was a summary of the average recommended solutions on stack overflow, absolutely no mention of my very specific anti standard choices or code.
-
@codinghorror I'm genuinely curious how much of the hype is an army of mid level engineers all building the same five web apps as the other guy.
@brianowen @codinghorror This is exactly what it is. This is exactly what the web dev industry has been for decades. Millions of LoC of garbage to justify prices for what should be an easy in-house job using an existing CMS with minimal or no code and should be as easy as using Excel.
-
@brianowen @codinghorror This is exactly what it is. This is exactly what the web dev industry has been for decades. Millions of LoC of garbage to justify prices for what should be an easy in-house job using an existing CMS with minimal or no code and should be as easy as using Excel.
@dalias @brianowen @codinghorror there are certainly many doing just that, but i'm probably not alone in doing something completely different, my vibe coded app for my own personal use is an openXR remote display for my quest3, in rust, with a desktop (linux/macos) agent capturing, encoding and streaming to it, using rust on linux, swift on macos, and python to wrap things.
it was done in weeks what would have taken me months/years, assuming i would have found the time/motivation to even try.
-
"A recent 2026 empirical study titled "Beyond Code Snippets: Benchmarking LLMs on Repository-Level Question Answering" (published on arXiv/ResearchGate) explicitly tested LLMs on codebase comprehension. The researchers concluded that high performance often "results from verbatim reproduction of Stack Overflow answers rather than genuine reasoning." " https://www.researchgate.net/publication/403262523_Beyond_Code_Snippets_Benchmarking_LLMs_on_Repository-Level_Question_Answering
@codinghorror
So, they're all like the AI on LinkedIn that will do a "smart" search for me that takes 100 times longer to give me the exact same list as the normal search option does? Because it probably just runs the search and fails a thousand process calls before just giving me the search? -
@dalias @brianowen @codinghorror there are certainly many doing just that, but i'm probably not alone in doing something completely different, my vibe coded app for my own personal use is an openXR remote display for my quest3, in rust, with a desktop (linux/macos) agent capturing, encoding and streaming to it, using rust on linux, swift on macos, and python to wrap things.
it was done in weeks what would have taken me months/years, assuming i would have found the time/motivation to even try.
@dalias @brianowen @codinghorror (and i want to be very clear this is not my code, it's really codex/openai, i barely glanced at some of it, let alone write anything other than prompts, i'm a dev with years of experience, not the best in the world for sure, but i know how to code, here i've been doing a PM's job, and not a very competent one, the tool had to cater to my whims and half ideas, and did quite well at that)
-
"A recent 2026 empirical study titled "Beyond Code Snippets: Benchmarking LLMs on Repository-Level Question Answering" (published on arXiv/ResearchGate) explicitly tested LLMs on codebase comprehension. The researchers concluded that high performance often "results from verbatim reproduction of Stack Overflow answers rather than genuine reasoning." " https://www.researchgate.net/publication/403262523_Beyond_Code_Snippets_Benchmarking_LLMs_on_Repository-Level_Question_Answering
@codinghorror@infosec.exchange
I'M Shocked! - Futurama GIF - Shocker Shocked Futurama - Discover & Share GIFs
The perfect Shocker Shocked Futurama Animated GIF for your conversation. Discover and Share the best GIFs on Tenor.
Tenor (tenor.com)
-
"A recent 2026 empirical study titled "Beyond Code Snippets: Benchmarking LLMs on Repository-Level Question Answering" (published on arXiv/ResearchGate) explicitly tested LLMs on codebase comprehension. The researchers concluded that high performance often "results from verbatim reproduction of Stack Overflow answers rather than genuine reasoning." " https://www.researchgate.net/publication/403262523_Beyond_Code_Snippets_Benchmarking_LLMs_on_Repository-Level_Question_Answering
@codinghorror theft en masse as a business model
-
"A recent 2026 empirical study titled "Beyond Code Snippets: Benchmarking LLMs on Repository-Level Question Answering" (published on arXiv/ResearchGate) explicitly tested LLMs on codebase comprehension. The researchers concluded that high performance often "results from verbatim reproduction of Stack Overflow answers rather than genuine reasoning." " https://www.researchgate.net/publication/403262523_Beyond_Code_Snippets_Benchmarking_LLMs_on_Repository-Level_Question_Answering
@joe More of an FYI for this repost in case you’re curious. (It’s mentioned in the abstract.) https://macaw.social/@mergesort/116444049426350678
-
@joe More of an FYI for this repost in case you’re curious. (It’s mentioned in the abstract.) https://macaw.social/@mergesort/116444049426350678
@mergesort sounds like a good opportunity for a one-up paper to try it again with the newer models. would be interesting to see what difference the "reasoning" really makes
-
@codinghorror I gots no problem with da one-shotting da boilerplate! But the actual useful application is a far cry from what Jensen, who pretends to be everyone's friend, wants you to do the "tokenmaxxing" for.
@bms48 turns out far too many humans are pretty goddamned lazy and will ship the prototype. How do we change this?
-
@mergesort sounds like a good opportunity for a one-up paper to try it again with the newer models. would be interesting to see what difference the "reasoning" really makes
@joe Agreed! I’m genuinely always in favor of repeating research like this given how fast the models are moving. Even the non-reasoning models are dramatically better today so I’d love to run an experiment on them too, it’s just concerning to me when 1-2 year old outdated material becomes considered a source of truth.
-
"A recent 2026 empirical study titled "Beyond Code Snippets: Benchmarking LLMs on Repository-Level Question Answering" (published on arXiv/ResearchGate) explicitly tested LLMs on codebase comprehension. The researchers concluded that high performance often "results from verbatim reproduction of Stack Overflow answers rather than genuine reasoning." " https://www.researchgate.net/publication/403262523_Beyond_Code_Snippets_Benchmarking_LLMs_on_Repository-Level_Question_Answering
@codinghorror I have yet to have an LLM tell me to RTFM and then end the conversation.
-
@bms48 turns out far too many humans are pretty goddamned lazy and will ship the prototype. How do we change this?
@codinghorror @bms48 change incentives to be for long term not quarterly. Give people doing work more autonomy to set their own standards. Possibly UBI will enable this shift in perspective from eeking out a paycheck to professional/citizen/human responsibility/opportunity.
-
@codinghorror I have yet to have an LLM tell me to RTFM and then end the conversation.
@rjohnston I've never had that happen to me, personally, but I have pretty good resting bitch face to be fair.
