stray thought:
-
@zkat Wait - is the actual plan to prompt "Make me a therac-25"?@silverwizard @zkat No, it's more "I have no expertise in X-Ray picture analysis, so surely ChatGPT does!"
-
@silverwizard @zkat No, it's more "I have no expertise in X-Ray picture analysis, so surely ChatGPT does!"
-
@silverwizard @zkat Also even the current human-developed software for medical imaging analysis is rife with false positives. My partner did a scan recently and the automated analysis was added before the imaging technician comment, and they had to put in capitals that there was no cause for a specific concern raised by the automated system.
-
@silverwizard @zkat Also even the current human-developed software for medical imaging analysis is rife with false positives. My partner did a scan recently and the automated analysis was added before the imaging technician comment, and they had to put in capitals that there was no cause for a specific concern raised by the automated system.
-
stray thought:
I've heard of the term "Software Engineer" referred to as being "stolen glory", because engineering as a practice has long had MUCH higher standards than software development, when it came to reliability/quality/etc, because the stakes were usually so much higher, and the study required for qualification were correspondingly high.
LLMs seem to be leaning on this further: Software developers have embraced the idea that we can spit out any low-quality, unreliable, even straight-up dangerous garbage that, as long as it seems somewhat productive, is "valuable enough". They have dropped the floor on what we're able to achieve while deskilling many of us who at least were developing the skills required for quality production. Most importantly, they're making it so leadership actively discourages taking the actual time to make sure something won't fucking kill someone.
I was looking at a certain open source project's page and there was a link to its associated consultancy. On the front page was a gif of a prompt for an app generator LLM utility, and the first fucking prompt that showed up was "Write me an application that will act as an X-ray analyzer, looking for anomalies and reporting on them", and that was just obviously the most fucking irresponsible thing I'd seen all day. Imagine having some non-expert just write something that will put people's actual lives on the line like that, and sell it as if it has any qualification to make reliable diagnostics. This is the level of brazen irresponsibility we're dealing with.
@zkat thats surrendering the idea of what software should be to the worst actors though, large corporations want software to be irresponsible so its normal for things to fail and not directly their fault, shit like spaceflight computers and most earlier computer engineering id actually say qualifies as such without being 'stolen valor' or what have you, calling it that is succumbing to the lowered standard companies are trying to get away with rather than pointing out correctly that theyre dragging our expectations down to their profit margins so nobody ever even gets the idea of holding them accountable for shit software because 'software is just like this' -
@zkat my I agree with all of this.
My tl;dr has always been "Engineers are licensed. Software developers are not engineers."
-
@zkat you said it perfectly. Imagine if software developers had to take the PE. Or sign off on their work etc.
They hijacked the Engineer title and many CS grads I have met dont even enjoy the field. Its all just a cash grab of a position. Sad, really. -
@KormaChameleon @zkat I guess my point is: its a very human problem. I don't mean you or techies specifically.
-
@whitequark @zkat the ones with their PE licenses are. The others have a misleading job title, like "software engineer".
-
@zkat Wait - is the actual plan to prompt "Make me a therac-25"?
-
@zkat my I agree with all of this.
My tl;dr has always been "Engineers are licensed. Software developers are not engineers."
@dave@alvarado.social @zkat@toot.cat I don't know how it is where you are, but here in Québec to be employed as an "engineer" you need to be a member of the Order of Engineers. "Engineer" is regulated, "developer" and "programmer" are not.
But... Software engineers and software developers sit side-by-side in the office and do the same work to the same standards for the same compensation. I'm not sure there's any difference in practice.
When my employer was informed that it was illegally hiring software engineers, I had to sign a new contract with a different title, everything else stayed exactly the same.
-
stray thought:
I've heard of the term "Software Engineer" referred to as being "stolen glory", because engineering as a practice has long had MUCH higher standards than software development, when it came to reliability/quality/etc, because the stakes were usually so much higher, and the study required for qualification were correspondingly high.
LLMs seem to be leaning on this further: Software developers have embraced the idea that we can spit out any low-quality, unreliable, even straight-up dangerous garbage that, as long as it seems somewhat productive, is "valuable enough". They have dropped the floor on what we're able to achieve while deskilling many of us who at least were developing the skills required for quality production. Most importantly, they're making it so leadership actively discourages taking the actual time to make sure something won't fucking kill someone.
I was looking at a certain open source project's page and there was a link to its associated consultancy. On the front page was a gif of a prompt for an app generator LLM utility, and the first fucking prompt that showed up was "Write me an application that will act as an X-ray analyzer, looking for anomalies and reporting on them", and that was just obviously the most fucking irresponsible thing I'd seen all day. Imagine having some non-expert just write something that will put people's actual lives on the line like that, and sell it as if it has any qualification to make reliable diagnostics. This is the level of brazen irresponsibility we're dealing with.
@zkat Management would rather document some human process that triple checks your build and release process hasn't produced a blob that is too big for the partition and depend on that alone to avoid overwriting the next partition containing the backup in a device that's surgically embedded in someone than implement any checks in the tools that perform that writing because that would then have to be designed, documented, risk analyzed, tested, requirements traced...
Fuck me the overdesigner.
-
@dave@alvarado.social @zkat@toot.cat I don't know how it is where you are, but here in Québec to be employed as an "engineer" you need to be a member of the Order of Engineers. "Engineer" is regulated, "developer" and "programmer" are not.
But... Software engineers and software developers sit side-by-side in the office and do the same work to the same standards for the same compensation. I'm not sure there's any difference in practice.
When my employer was informed that it was illegally hiring software engineers, I had to sign a new contract with a different title, everything else stayed exactly the same.
@danso @zkat we have something similar in the US, but without the legal restriction on using the word "engineer" in all job titles.
What is a PE?
To become licensed, engineers must complete a four-year college degree, work under a Professional Engineer for at least four years, pass two intensive competency exams and earn a license from their state's licensure board. Then, to retain their licenses, PEs must continually maintain and improve their skills throughout their careers.
(www.nspe.org)
-
@zkat Management would rather document some human process that triple checks your build and release process hasn't produced a blob that is too big for the partition and depend on that alone to avoid overwriting the next partition containing the backup in a device that's surgically embedded in someone than implement any checks in the tools that perform that writing because that would then have to be designed, documented, risk analyzed, tested, requirements traced...
Fuck me the overdesigner.
@zkat You'd think regulatory would look at that and say, "Fuck you." Any text you read on the kinds of QMS you need for regulatory say you do not rely on human processes like that. You do shit like design your plugs so they can't fit wrong and you prove that shit by finding as many morons as you can and showing that none of them managed to fuck up.
These are direct quotes from ISO 62304
vOv
-
R relay@relay.an.exchange shared this topic
