The hidden beauty of vibe coding
-
The hidden beauty of vibe coding
"It passed all the unit tests, the shape of the code looks right," he said. It's 3.7x more lines of code that performs 2,000 times worse than the actual SQLite. Two thousand times worse for a database is a non-viable product. It's a dumpster fire. Throw it away. All that money you spent on it is worthless."
AI still doesn't work very well in business, reckoning soon
interview: Codestrap founders say we need to dial down the hype and sort through the mess
(www.theregister.com)
@gerrymcgovern This is an unusual article. It mixes truth and misconceptions in awkward ways.
For example:
Smiley pointed to a recent attempt to rewrite SQLite in Rust using AI
This isn't what happened. It was a C Compiler that was rewritten. A different tester then rebuilt SQLite using both the AI and the official one. The AI one did worse.
But it did worse for very specific reasons. The AI version was only tested for correctness. It was only given unit tests as a parameter for success. It failed on real world performance tests, because it was never actually given that as a requirement.
Lines of code, number of [pull requests], these are liabilities. These are not measures of engineering excellence."... Measures of engineering excellence, said Smiley, include metrics like deployment frequency, lead time to production, change failure rate, mean time to restore, and incident severity.
So these are famously known as the DORA metrics. And they don't measure engineering excellence, ... /1
-
@gerrymcgovern This is an unusual article. It mixes truth and misconceptions in awkward ways.
For example:
Smiley pointed to a recent attempt to rewrite SQLite in Rust using AI
This isn't what happened. It was a C Compiler that was rewritten. A different tester then rebuilt SQLite using both the AI and the official one. The AI one did worse.
But it did worse for very specific reasons. The AI version was only tested for correctness. It was only given unit tests as a parameter for success. It failed on real world performance tests, because it was never actually given that as a requirement.
Lines of code, number of [pull requests], these are liabilities. These are not measures of engineering excellence."... Measures of engineering excellence, said Smiley, include metrics like deployment frequency, lead time to production, change failure rate, mean time to restore, and incident severity.
So these are famously known as the DORA metrics. And they don't measure engineering excellence, ... /1
@gerrymcgovern ... they measure the capabilities of the engineering platform along with the expertise of the people using that platform.
There are lots of companies with excellent engineers and crummy DORA scores because they don't have the institutional support to improve those metrics. Nor does the score mean the business is successful. You can have great DORA metrics and still lack for paying customers.
"The other challenge here is that the incentives are misaligned,"
But then he proceeds to list a bunch of examples for competing incentives. His examples of "misaligned" are really examples of "I would like to deliver less and get paid more"... /2
-
@gerrymcgovern ... they measure the capabilities of the engineering platform along with the expertise of the people using that platform.
There are lots of companies with excellent engineers and crummy DORA scores because they don't have the institutional support to improve those metrics. Nor does the score mean the business is successful. You can have great DORA metrics and still lack for paying customers.
"The other challenge here is that the incentives are misaligned,"
But then he proceeds to list a bunch of examples for competing incentives. His examples of "misaligned" are really examples of "I would like to deliver less and get paid more"... /2
@gerrymcgovern ...
If there's an incentives problem here, it's that companies have been paying for a lot of BS rituals and they're discovering that the BS generating machine is undermining part of the ritual. Companies have also been getting away with under-specifying success in order to pad results as "good". But Gen AIs will "fill in" the under-specificity with made up data. Or they will fail to deliver anything into the gap that some human was hoping would be filled.
But none of this is "misaligned". It's intentional ambiguity designed to protect business units. The AI is just exposing the BS for what it is.
OP is kind of talking about that BS problem. But he's taking weird micro angles to view subsets of the problem without calling out the greater problem. He's not wrong, but he's also not really right either.
️ // -
The hidden beauty of vibe coding
"It passed all the unit tests, the shape of the code looks right," he said. It's 3.7x more lines of code that performs 2,000 times worse than the actual SQLite. Two thousand times worse for a database is a non-viable product. It's a dumpster fire. Throw it away. All that money you spent on it is worthless."
AI still doesn't work very well in business, reckoning soon
interview: Codestrap founders say we need to dial down the hype and sort through the mess
(www.theregister.com)
@gerrymcgovern You love to see it:
"Insurers, he said, are already lobbying state-level insurance regulators to win a carve-out in business insurance liability policies so they are not obligated to cover AI-related workflows. "That kills the whole system," Deeks said."Someone will be left holding the bag when slop kills someone, bankrupts a business, or causes serious damage. Insurers will make sure it isn't them.
-
The hidden beauty of vibe coding
"It passed all the unit tests, the shape of the code looks right," he said. It's 3.7x more lines of code that performs 2,000 times worse than the actual SQLite. Two thousand times worse for a database is a non-viable product. It's a dumpster fire. Throw it away. All that money you spent on it is worthless."
AI still doesn't work very well in business, reckoning soon
interview: Codestrap founders say we need to dial down the hype and sort through the mess
(www.theregister.com)
@gerrymcgovern message from management: I didn’t read your test report, but our AI assistant told us we should fire all but one software engineer and push to prod, whatever that means. Let us know how it goes, we’ll be at the money burning party our AI assistant scheduled for us.
-
@gerrymcgovern The stats for the vibe-coded SQLite rewrite in Rust being literally thousands of times slower than SQLite are simply wild. Such as needing almost 2 seconds to do 100 single-ID lookups. I'm pretty sure I could improve on that operation just by slurping a CSV with unique row IDs into memory and doing a binary search on said row IDs. Would I want to? No, but I'm also not *trying* to build an actual RDB engine either.
Do you have a link you would like to share about this 1000 times slower implementation of a SQLite instance?
-
@PatrickTingen @gerrymcgovern "vibe coding" is a euphemism for plagiarism. even for personal projects or proofs of concept, plagiarism is unethical and impermissible. not to mention there's substantial cost and harm for every single prompt.
@poprox @PatrickTingen @gerrymcgovern
Given that, when a model is trained with GPL code, it has to be GPL'ed itself, would be fair.
No fun to say, but I think copyright/IP issues have a long way to go until the whole thing is settled and needless to say, is has to be fair -
@PatrickTingen @gerrymcgovern "vibe coding" is a euphemism for plagiarism. even for personal projects or proofs of concept, plagiarism is unethical and impermissible. not to mention there's substantial cost and harm for every single prompt.
@poprox @PatrickTingen @gerrymcgovern The problem is that all coders take from other people, copy code, and vibe coding does this better, in theory.
There is a lot of harm, and there are - IME - better ways of ding this. Or at least, there were, until they were all enshittified.
-
Do you have a link you would like to share about this 1000 times slower implementation of a SQLite instance?
@curious_carrot @dpnash @gerrymcgovern It's linked from the original Register article: https://medium.com/write-a-catalyst/an-ai-wrote-576-000-lines-to-replace-sqlite-7ea538826d72
-
Do you have a link you would like to share about this 1000 times slower implementation of a SQLite instance?
@curious_carrot @dpnash @gerrymcgovern yes. (and you have it too.)
oh, and it's 20000 times slower, actually.
-
The hidden beauty of vibe coding
"It passed all the unit tests, the shape of the code looks right," he said. It's 3.7x more lines of code that performs 2,000 times worse than the actual SQLite. Two thousand times worse for a database is a non-viable product. It's a dumpster fire. Throw it away. All that money you spent on it is worthless."
AI still doesn't work very well in business, reckoning soon
interview: Codestrap founders say we need to dial down the hype and sort through the mess
(www.theregister.com)
@gerrymcgovern The original article has it at 20000× worse x)
Your LLM Doesn't Write Correct Code. It Writes Plausible Code.
One of the simplest tests you can run on a database:
(blog.katanaquant.com)
-
The hidden beauty of vibe coding
"It passed all the unit tests, the shape of the code looks right," he said. It's 3.7x more lines of code that performs 2,000 times worse than the actual SQLite. Two thousand times worse for a database is a non-viable product. It's a dumpster fire. Throw it away. All that money you spent on it is worthless."
AI still doesn't work very well in business, reckoning soon
interview: Codestrap founders say we need to dial down the hype and sort through the mess
(www.theregister.com)
@gerrymcgovern KLOC as a measurement of productivity has come back into fashion yet again, for the *umpteenth* time. We never learn. Welcome to the 1970s.
-
M mttaggart@infosec.exchange shared this topic
