<![CDATA["A recent 2026 empirical study titled "Beyond Code Snippets: Benchmarking LLMs on Repository-Level Question Answering" (published on arXiv/ResearchGate) explicitly tested LLMs on codebase comprehension.]]>"A recent 2026 empirical study titled "Beyond Code Snippets: Benchmarking LLMs on Repository-Level Question Answering" (published on arXiv/ResearchGate) explicitly tested LLMs on codebase comprehension. The researchers concluded that high performance often "results from verbatim reproduction of Stack Overflow answers rather than genuine reasoning." " https://www.researchgate.net/publication/403262523_Beyond_Code_Snippets_Benchmarking_LLMs_on_Repository-Level_Question_Answering

]]>https://board.circlewithadot.net/topic/e568d96b-66c0-48b6-8967-e7ec04a72e1f/a-recent-2026-empirical-study-titled-beyond-code-snippets-benchmarking-llms-on-repository-level-question-answering-published-on-arxiv-researchgate-explicitly-tested-llms-on-codebase-comprehension.RSS for NodeFri, 15 May 2026 00:33:23 GMTTue, 21 Apr 2026 13:05:53 GMT60<![CDATA[Reply to "A recent 2026 empirical study titled "Beyond Code Snippets: Benchmarking LLMs on Repository-Level Question Answering" (published on arXiv/ResearchGate) explicitly tested LLMs on codebase comprehension. on Wed, 22 Apr 2026 10:08:08 GMT]]>@slyecho feel free to evaluate yourself using whatever tools you prefer

]]>https://board.circlewithadot.net/post/https://infosec.exchange/users/codinghorror/statuses/116447804697185176https://board.circlewithadot.net/post/https://infosec.exchange/users/codinghorror/statuses/116447804697185176Wed, 22 Apr 2026 10:08:08 GMT<![CDATA[Reply to "A recent 2026 empirical study titled "Beyond Code Snippets: Benchmarking LLMs on Repository-Level Question Answering" (published on arXiv/ResearchGate) explicitly tested LLMs on codebase comprehension. on Wed, 22 Apr 2026 08:39:06 GMT]]>@codinghorror 0 surprise there.

]]>https://board.circlewithadot.net/post/https://mastodon.sdf.org/users/doragasu/statuses/116447454574817162https://board.circlewithadot.net/post/https://mastodon.sdf.org/users/doragasu/statuses/116447454574817162Wed, 22 Apr 2026 08:39:06 GMT<![CDATA[Reply to "A recent 2026 empirical study titled "Beyond Code Snippets: Benchmarking LLMs on Repository-Level Question Answering" (published on arXiv/ResearchGate) explicitly tested LLMs on codebase comprehension. on Wed, 22 Apr 2026 02:49:14 GMT]]>@dalias @brianowen @codinghorror The number of billion dollar valuation security industry products that amount to a shiny web UI over a few FOSS tools ...

]]>https://board.circlewithadot.net/post/https://infosec.exchange/users/JessTheUnstill/statuses/116446078868655823https://board.circlewithadot.net/post/https://infosec.exchange/users/JessTheUnstill/statuses/116446078868655823Wed, 22 Apr 2026 02:49:14 GMT<![CDATA[Reply to "A recent 2026 empirical study titled "Beyond Code Snippets: Benchmarking LLMs on Repository-Level Question Answering" (published on arXiv/ResearchGate) explicitly tested LLMs on codebase comprehension. on Wed, 22 Apr 2026 01:58:35 GMT]]>@rjohnston I've never had that happen to me, personally, but I have pretty good resting bitch face to be fair.

]]>https://board.circlewithadot.net/post/https://infosec.exchange/users/codinghorror/statuses/116445879716276003https://board.circlewithadot.net/post/https://infosec.exchange/users/codinghorror/statuses/116445879716276003Wed, 22 Apr 2026 01:58:35 GMT<![CDATA[Reply to "A recent 2026 empirical study titled "Beyond Code Snippets: Benchmarking LLMs on Repository-Level Question Answering" (published on arXiv/ResearchGate) explicitly tested LLMs on codebase comprehension. on Tue, 21 Apr 2026 22:44:29 GMT]]>@codinghorror @bms48 change incentives to be for long term not quarterly. Give people doing work more autonomy to set their own standards. Possibly UBI will enable this shift in perspective from eeking out a paycheck to professional/citizen/human responsibility/opportunity.

]]>https://board.circlewithadot.net/post/https://social.lane-jayasinha.com/users/chris/statuses/01KPS3ETBAMD2JWNC86G1E1SYJhttps://board.circlewithadot.net/post/https://social.lane-jayasinha.com/users/chris/statuses/01KPS3ETBAMD2JWNC86G1E1SYJTue, 21 Apr 2026 22:44:29 GMT<![CDATA[Reply to "A recent 2026 empirical study titled "Beyond Code Snippets: Benchmarking LLMs on Repository-Level Question Answering" (published on arXiv/ResearchGate) explicitly tested LLMs on codebase comprehension. on Tue, 21 Apr 2026 21:47:54 GMT]]>@codinghorror I have yet to have an LLM tell me to RTFM and then end the conversation.

]]>https://board.circlewithadot.net/post/https://techhub.social/ap/users/115726408341208805/statuses/116444893956395105https://board.circlewithadot.net/post/https://techhub.social/ap/users/115726408341208805/statuses/116444893956395105Tue, 21 Apr 2026 21:47:54 GMT<![CDATA[Reply to "A recent 2026 empirical study titled "Beyond Code Snippets: Benchmarking LLMs on Repository-Level Question Answering" (published on arXiv/ResearchGate) explicitly tested LLMs on codebase comprehension. on Tue, 21 Apr 2026 21:09:44 GMT]]>@joe Agreed! I’m genuinely always in favor of repeating research like this given how fast the models are moving. Even the non-reasoning models are dramatically better today so I’d love to run an experiment on them too, it’s just concerning to me when 1-2 year old outdated material becomes considered a source of truth.

]]>https://board.circlewithadot.net/post/https://macaw.social/users/mergesort/statuses/116444743911361644https://board.circlewithadot.net/post/https://macaw.social/users/mergesort/statuses/116444743911361644Tue, 21 Apr 2026 21:09:44 GMT<![CDATA[Reply to "A recent 2026 empirical study titled "Beyond Code Snippets: Benchmarking LLMs on Repository-Level Question Answering" (published on arXiv/ResearchGate) explicitly tested LLMs on codebase comprehension. on Tue, 21 Apr 2026 20:56:36 GMT]]>@bms48 turns out far too many humans are pretty goddamned lazy and will ship the prototype. How do we change this?

]]>https://board.circlewithadot.net/post/https://infosec.exchange/users/codinghorror/statuses/116444692263468512https://board.circlewithadot.net/post/https://infosec.exchange/users/codinghorror/statuses/116444692263468512Tue, 21 Apr 2026 20:56:36 GMT<![CDATA[Reply to "A recent 2026 empirical study titled "Beyond Code Snippets: Benchmarking LLMs on Repository-Level Question Answering" (published on arXiv/ResearchGate) explicitly tested LLMs on codebase comprehension. on Tue, 21 Apr 2026 18:28:28 GMT]]>@mergesort sounds like a good opportunity for a one-up paper to try it again with the newer models. would be interesting to see what difference the "reasoning" really makes

]]>https://board.circlewithadot.net/post/https://f.duriansoftware.com/users/joe/statuses/116444109779942454https://board.circlewithadot.net/post/https://f.duriansoftware.com/users/joe/statuses/116444109779942454Tue, 21 Apr 2026 18:28:28 GMT<![CDATA[Reply to "A recent 2026 empirical study titled "Beyond Code Snippets: Benchmarking LLMs on Repository-Level Question Answering" (published on arXiv/ResearchGate) explicitly tested LLMs on codebase comprehension. on Tue, 21 Apr 2026 18:15:03 GMT]]>@joe More of an FYI for this repost in case you’re curious. (It’s mentioned in the abstract.) https://macaw.social/@mergesort/116444049426350678

]]>https://board.circlewithadot.net/post/https://macaw.social/users/mergesort/statuses/116444056999514889https://board.circlewithadot.net/post/https://macaw.social/users/mergesort/statuses/116444056999514889Tue, 21 Apr 2026 18:15:03 GMT<![CDATA[Reply to "A recent 2026 empirical study titled "Beyond Code Snippets: Benchmarking LLMs on Repository-Level Question Answering" (published on arXiv/ResearchGate) explicitly tested LLMs on codebase comprehension. on Tue, 21 Apr 2026 17:40:45 GMT]]>@codinghorror theft en masse as a business model

]]>https://board.circlewithadot.net/post/https://infosec.exchange/users/OvertonDoors/statuses/116443922146805758https://board.circlewithadot.net/post/https://infosec.exchange/users/OvertonDoors/statuses/116443922146805758Tue, 21 Apr 2026 17:40:45 GMT<![CDATA[Reply to "A recent 2026 empirical study titled "Beyond Code Snippets: Benchmarking LLMs on Repository-Level Question Answering" (published on arXiv/ResearchGate) explicitly tested LLMs on codebase comprehension. on Tue, 21 Apr 2026 16:48:45 GMT]]>@codinghorror@infosec.exchange

Link Preview Image
I'M Shocked! - Futurama GIF - Shocker Shocked Futurama - Discover & Share GIFs

The perfect Shocker Shocked Futurama Animated GIF for your conversation. Discover and Share the best GIFs on Tenor.

favicon

Tenor (tenor.com)

]]>https://board.circlewithadot.net/post/https://app.wafrn.net/fediverse/post/5c53ab46-850d-4deb-bb07-271b5f7c93b0https://board.circlewithadot.net/post/https://app.wafrn.net/fediverse/post/5c53ab46-850d-4deb-bb07-271b5f7c93b0Tue, 21 Apr 2026 16:48:45 GMT<![CDATA[Reply to "A recent 2026 empirical study titled "Beyond Code Snippets: Benchmarking LLMs on Repository-Level Question Answering" (published on arXiv/ResearchGate) explicitly tested LLMs on codebase comprehension. on Tue, 21 Apr 2026 16:46:32 GMT]]>@dalias @brianowen @codinghorror (and i want to be very clear this is not my code, it's really codex/openai, i barely glanced at some of it, let alone write anything other than prompts, i'm a dev with years of experience, not the best in the world for sure, but i know how to code, here i've been doing a PM's job, and not a very competent one, the tool had to cater to my whims and half ideas, and did quite well at that)

]]>https://board.circlewithadot.net/post/https://mas.to/users/tshirtman/statuses/116443708947693422https://board.circlewithadot.net/post/https://mas.to/users/tshirtman/statuses/116443708947693422Tue, 21 Apr 2026 16:46:32 GMT<![CDATA[Reply to "A recent 2026 empirical study titled "Beyond Code Snippets: Benchmarking LLMs on Repository-Level Question Answering" (published on arXiv/ResearchGate) explicitly tested LLMs on codebase comprehension. on Tue, 21 Apr 2026 16:41:57 GMT]]>@codinghorror
So, they're all like the AI on LinkedIn that will do a "smart" search for me that takes 100 times longer to give me the exact same list as the normal search option does? Because it probably just runs the search and fails a thousand process calls before just giving me the search?

]]>https://board.circlewithadot.net/post/https://infosec.exchange/users/RnDanger/statuses/116443690894044622https://board.circlewithadot.net/post/https://infosec.exchange/users/RnDanger/statuses/116443690894044622Tue, 21 Apr 2026 16:41:57 GMT<![CDATA[Reply to "A recent 2026 empirical study titled "Beyond Code Snippets: Benchmarking LLMs on Repository-Level Question Answering" (published on arXiv/ResearchGate) explicitly tested LLMs on codebase comprehension. on Tue, 21 Apr 2026 16:36:14 GMT]]>@dalias @brianowen @codinghorror there are certainly many doing just that, but i'm probably not alone in doing something completely different, my vibe coded app for my own personal use is an openXR remote display for my quest3, in rust, with a desktop (linux/macos) agent capturing, encoding and streaming to it, using rust on linux, swift on macos, and python to wrap things.

it was done in weeks what would have taken me months/years, assuming i would have found the time/motivation to even try.

]]>https://board.circlewithadot.net/post/https://mas.to/users/tshirtman/statuses/116443668425955497https://board.circlewithadot.net/post/https://mas.to/users/tshirtman/statuses/116443668425955497Tue, 21 Apr 2026 16:36:14 GMT<![CDATA[Reply to "A recent 2026 empirical study titled "Beyond Code Snippets: Benchmarking LLMs on Repository-Level Question Answering" (published on arXiv/ResearchGate) explicitly tested LLMs on codebase comprehension. on Tue, 21 Apr 2026 16:12:37 GMT]]>@brianowen @codinghorror This is exactly what it is. This is exactly what the web dev industry has been for decades. Millions of LoC of garbage to justify prices for what should be an easy in-house job using an existing CMS with minimal or no code and should be as easy as using Excel.

]]>https://board.circlewithadot.net/post/https://hachyderm.io/users/dalias/statuses/116443575607421807https://board.circlewithadot.net/post/https://hachyderm.io/users/dalias/statuses/116443575607421807Tue, 21 Apr 2026 16:12:37 GMT<![CDATA[Reply to "A recent 2026 empirical study titled "Beyond Code Snippets: Benchmarking LLMs on Repository-Level Question Answering" (published on arXiv/ResearchGate) explicitly tested LLMs on codebase comprehension. on Tue, 21 Apr 2026 16:07:07 GMT]]>@codinghorror I made a code going against "the recommended way of doing it" in several ways for certain task. It was a very conscious decision, these violations are needed to make something new I want. Recently I handed the code to my student and asked her to read and understand it. She handed me what she said was a written summary of her notes. It was a summary of the average recommended solutions on stack overflow, absolutely no mention of my very specific anti standard choices or code.

]]>https://board.circlewithadot.net/post/https://mastodon.gal/users/elrohir/statuses/116443553933190076https://board.circlewithadot.net/post/https://mastodon.gal/users/elrohir/statuses/116443553933190076Tue, 21 Apr 2026 16:07:07 GMT<![CDATA[Reply to "A recent 2026 empirical study titled "Beyond Code Snippets: Benchmarking LLMs on Repository-Level Question Answering" (published on arXiv/ResearchGate) explicitly tested LLMs on codebase comprehension. on Tue, 21 Apr 2026 16:02:19 GMT]]>@henryk @codinghorror I used to think the "just copying from Stack Overflow" jokes developers made were, you know, jokes. Then I watched all my coworkers embrace LLMs, and I was forced to conclude that they weren't joking.

So, depressing agreement.

]]>https://board.circlewithadot.net/post/https://cyberpunk.lol/users/Azuaron/statuses/116443535059699643https://board.circlewithadot.net/post/https://cyberpunk.lol/users/Azuaron/statuses/116443535059699643Tue, 21 Apr 2026 16:02:19 GMT<![CDATA[Reply to "A recent 2026 empirical study titled "Beyond Code Snippets: Benchmarking LLMs on Repository-Level Question Answering" (published on arXiv/ResearchGate) explicitly tested LLMs on codebase comprehension. on Tue, 21 Apr 2026 13:53:52 GMT]]>@codinghorror I'm genuinely curious how much of the hype is an army of mid level engineers all building the same five web apps as the other guy.

]]>https://board.circlewithadot.net/post/https://fosstodon.org/users/brianowen/statuses/116443029976265871https://board.circlewithadot.net/post/https://fosstodon.org/users/brianowen/statuses/116443029976265871Tue, 21 Apr 2026 13:53:52 GMT<![CDATA[Reply to "A recent 2026 empirical study titled "Beyond Code Snippets: Benchmarking LLMs on Repository-Level Question Answering" (published on arXiv/ResearchGate) explicitly tested LLMs on codebase comprehension. on Tue, 21 Apr 2026 13:16:40 GMT]]>@codinghorror I gots no problem with da one-shotting da boilerplate! But the actual useful application is a far cry from what Jensen, who pretends to be everyone's friend, wants you to do the "tokenmaxxing" for.

]]>https://board.circlewithadot.net/post/https://mastodon.social/ap/users/116175731239673526/statuses/116442883709546749https://board.circlewithadot.net/post/https://mastodon.social/ap/users/116175731239673526/statuses/116442883709546749Tue, 21 Apr 2026 13:16:40 GMT<![CDATA[Reply to "A recent 2026 empirical study titled "Beyond Code Snippets: Benchmarking LLMs on Repository-Level Question Answering" (published on arXiv/ResearchGate) explicitly tested LLMs on codebase comprehension. on Tue, 21 Apr 2026 13:15:37 GMT]]>@bms48 it's better for blank page ideation, and mashups / galactic brain fuzz testing in my opinion, and should always be double-checked by a human

]]>https://board.circlewithadot.net/post/https://infosec.exchange/users/codinghorror/statuses/116442879592117289https://board.circlewithadot.net/post/https://infosec.exchange/users/codinghorror/statuses/116442879592117289Tue, 21 Apr 2026 13:15:37 GMT<![CDATA[Reply to "A recent 2026 empirical study titled "Beyond Code Snippets: Benchmarking LLMs on Repository-Level Question Answering" (published on arXiv/ResearchGate) explicitly tested LLMs on codebase comprehension. on Tue, 21 Apr 2026 13:12:45 GMT]]>@codinghorror The really sad part of this was that the LLMs were caught directly in the act of fabricating erroneous output. I had a Git sparse-checkout of xnu directly in front of me. The technical matter related to TCP interactive behaviour.

]]>https://board.circlewithadot.net/post/https://mastodon.social/ap/users/116175731239673526/statuses/116442868311900184https://board.circlewithadot.net/post/https://mastodon.social/ap/users/116175731239673526/statuses/116442868311900184Tue, 21 Apr 2026 13:12:45 GMT<![CDATA[Reply to "A recent 2026 empirical study titled "Beyond Code Snippets: Benchmarking LLMs on Repository-Level Question Answering" (published on arXiv/ResearchGate) explicitly tested LLMs on codebase comprehension. on Tue, 21 Apr 2026 13:09:56 GMT]]>@codinghorror I have witnessed cloud based LLMs giving me authoritative-sound answers, when asked questions about macOS xnu kernel code, which came from Linux constructions, and were completely irrelevant. Generative AI is often useless and not fit for its advertised purpose. The limitations are structural and well known to ML researchers, and Noam Chomsky called it already years ago. I wish they'd just jog on and stop bothering real people with what is essentially a shell game.

]]>https://board.circlewithadot.net/post/https://mastodon.social/ap/users/116175731239673526/statuses/116442857251320680https://board.circlewithadot.net/post/https://mastodon.social/ap/users/116175731239673526/statuses/116442857251320680Tue, 21 Apr 2026 13:09:56 GMT<![CDATA[Reply to "A recent 2026 empirical study titled "Beyond Code Snippets: Benchmarking LLMs on Repository-Level Question Answering" (published on arXiv/ResearchGate) explicitly tested LLMs on codebase comprehension. on Tue, 21 Apr 2026 13:09:54 GMT]]>@codinghorror So ... roughly on par with normal developers then? 😉

]]>https://board.circlewithadot.net/post/https://chaos.social/users/henryk/statuses/116442857085014670https://board.circlewithadot.net/post/https://chaos.social/users/henryk/statuses/116442857085014670Tue, 21 Apr 2026 13:09:54 GMT