Welp, for the first semester ever, SOTA LLMs can do *every single assignment, from scratch (readmes, etc.), and get 100%*.
-
@krismicinski @shriramk @jfdm @csgordon @lindsey @jeremysiek goodness, I hope that prompting an LLM will not be a *huge* part of software engineering going forward. It's an incredibly inefficient way to go about the task. Frankly, I'm amazed at just how shoddy the current set of tools are.
@cross @krismicinski @shriramk @jfdm @csgordon @lindsey @jeremysiek It’s amazingly depressing to me, not because I worry about AI tools making the work obsolete, but because the remarkable effectiveness of AI tools in the face of their shoddiness drives home to me that the vast majority of programmers are redoing a thing that’s already been done most of the time. What a waste of human capital.
-
@GeorgWeissenbacher @krismicinski @jfdm @csgordon @lindsey @jeremysiek
Yes to most of that. I think it's not that hard to assess if that is what people were always assessing that.I actually disagree w/ your opening comment. Most intro CS educators will say (and have said), "I don't teach programming, I teach *problem solving*" (whatever the fuck that is). My response is, "great, this should be your liberation! Programming got easy, what are your «problem solving» ideas?"
@shriramk @GeorgWeissenbacher @krismicinski @jfdm @csgordon @lindsey @jeremysiek ... did programming get easy? Can one be said to be programming if one asks someone else (or an LLM) to write a program for you? Or is some other kind of (not- or not-quite-programming) interaction going on?
-
@cross @krismicinski @shriramk @jfdm @csgordon @lindsey @jeremysiek It’s amazingly depressing to me, not because I worry about AI tools making the work obsolete, but because the remarkable effectiveness of AI tools in the face of their shoddiness drives home to me that the vast majority of programmers are redoing a thing that’s already been done most of the time. What a waste of human capital.
@steve @cross @krismicinski @shriramk @jfdm @csgordon @lindsey @jeremysiek As a professional programmer now almost 20 years out from defending a thesis that was about code generation, I can't upvote this enough.
LLMs are an incredibly inefficient way to do code reuse.
-
@steve @cross @krismicinski @shriramk @jfdm @csgordon @lindsey @jeremysiek As a professional programmer now almost 20 years out from defending a thesis that was about code generation, I can't upvote this enough.
LLMs are an incredibly inefficient way to do code reuse.
@gwozniak @steve @krismicinski @shriramk @jfdm @csgordon @lindsey @jeremysiek yeah, I was going to comment that you had said something similar a few days or a week ago.
It's wild to me that we keep writing the same program over and over again and calling it "progress."
-
@lindsey @cross @krismicinski @jfdm @csgordon @jeremysiek
It is absolutely an open research question as to what will be the new "source" and "intermediate" languages. I think we'll have a much better shot at the latter (richly-typed, semantic specifications as part of code, etc.); for the former, I think we'll build good ones but the trick will be getting people to use them.@shriramk @lindsey @cross @krismicinski @jfdm @csgordon @jeremysiek Do you think that the "no one will look at the generated code anymore" future is inevitable? Given how often the industry has tried to generate programs directly from English-like specs before and failed, I'm quite skeptical, even if we have notably different tech this time around.
Internally at Google many folks (including high-level ones) are making this claim without evidence, as if it's obvious from its face, and I'm surprised how few people push back on it or at least ask for more proof.
-
@shriramk @lindsey @cross @krismicinski @jfdm @csgordon @jeremysiek Do you think that the "no one will look at the generated code anymore" future is inevitable? Given how often the industry has tried to generate programs directly from English-like specs before and failed, I'm quite skeptical, even if we have notably different tech this time around.
Internally at Google many folks (including high-level ones) are making this claim without evidence, as if it's obvious from its face, and I'm surprised how few people push back on it or at least ask for more proof.
@shriramk @lindsey @cross @krismicinski @jfdm @csgordon @jeremysiek LLM-based code generation reminds me of some of Bret Victor's talks: there are some cool ideas and convincing demos, but also a lot more work to do before one can say "we've solved all of the problems; everyone should be doing this all the time now".
-
@shriramk @lindsey @cross @krismicinski @jfdm @csgordon @jeremysiek LLM-based code generation reminds me of some of Bret Victor's talks: there are some cool ideas and convincing demos, but also a lot more work to do before one can say "we've solved all of the problems; everyone should be doing this all the time now".
@shriramk @lindsey @cross @krismicinski @jfdm @csgordon @jeremysiek To be fair: The LLM tooling is certainly more capable overall than the Bret Victor stuff. But I'm not yet convinced coding is 100% solved.
-
@shriramk @GeorgWeissenbacher @krismicinski @jfdm @csgordon @lindsey @jeremysiek ... did programming get easy? Can one be said to be programming if one asks someone else (or an LLM) to write a program for you? Or is some other kind of (not- or not-quite-programming) interaction going on?
@tonyg @GeorgWeissenbacher @krismicinski @jfdm @csgordon @lindsey @jeremysiek
I very much think of what I'm doing with Claude Code as a kind of programming — indeed, the kind of programming I always wished I could do! But if it makes you happier to use a different term for it (not "vibecoding", that has too many specific connotations and is definitely not how *I'm* doing things), and it's *useful* to have that other term…that's fine by me. I guess my slogan is: "Philosophy…but not too much". -
@shriramk @GeorgWeissenbacher @krismicinski @jfdm @csgordon @lindsey @jeremysiek ... did programming get easy? Can one be said to be programming if one asks someone else (or an LLM) to write a program for you? Or is some other kind of (not- or not-quite-programming) interaction going on?
@tonyg Unrelatedly, I was in the UK last month, and when I gave a talk in London (and later in Cambridge), was pleasantly surprised that Noel Walsh stopped in. I was reminiscing about how I'd attended a Racket meetup he organized in Islington back in 2003 or so, where I believe we met!
-
@gwozniak @steve @krismicinski @shriramk @jfdm @csgordon @lindsey @jeremysiek yeah, I was going to comment that you had said something similar a few days or a week ago.
It's wild to me that we keep writing the same program over and over again and calling it "progress."
@cross @gwozniak @steve @krismicinski @jfdm @csgordon @lindsey @jeremysiek
We were already doing this. The entire low-code/no-code movement was all about making sure we stopped doing that in some domains, though we built 20+ systems that each tried to do that. (-: -
@shriramk @lindsey @cross @krismicinski @jfdm @csgordon @jeremysiek To be fair: The LLM tooling is certainly more capable overall than the Bret Victor stuff. But I'm not yet convinced coding is 100% solved.
@jschuster @lindsey @cross @krismicinski @jfdm @csgordon @jeremysiek
Of course I don't think we'll never need to ever look at generated code again; that would be a foolish position. The interesting question is how much will people need to, and relative to what? If you have an amazing test suite or rich verified properties, for instance, how much do you need to review code? Most people aren't writing Dan Cross-level code. (The Bret Victor analogy is good.) -
@jfdm @csgordon @lindsey @jeremysiek @krismicinski
"we are doomed" is an incredibly disappointing take. You should have come to my "GenAI and CS Ed" talk (-:.If our only value-add was "my course was gated behind a needlessly difficult thing", that doesn't say much for the value of our courses.
@shriramk @jfdm @csgordon @lindsey @jeremysiek @krismicinski A challenge -- which I think we're seeing from many people in this thread! -- is that in recent years we (speaking broadly) have magnified Certification as an outcome relative Education _and_ conflated the two together in ours and, often, students' minds. New technology might have undermined how we do Certification right now; perhaps that will also encourage us to change how we do Education?
-
@cross @krismicinski @shriramk @jfdm @csgordon @lindsey @jeremysiek It’s amazingly depressing to me, not because I worry about AI tools making the work obsolete, but because the remarkable effectiveness of AI tools in the face of their shoddiness drives home to me that the vast majority of programmers are redoing a thing that’s already been done most of the time. What a waste of human capital.
@cross @krismicinski @shriramk @jfdm @csgordon @lindsey @jeremysiek (with AI tools, they are still redoing a thing that they shouldn’t need to, but faster and sloppier.)
