i have thoughts on google’s modern-web-guidance thing…
-
RE: https://front-end.social/@SaraSoueidan/116610307042155106
3) lack of manual testing
afaict, only the use-case specific handcoded demos were manually tested. i can see an "eval" flow with e2e tests that regularly validate outputs from different models, which is good but *not* a replacement for real testing (esp. from disabled users).
there is no data to back up the claim that this skill actually helps these tools produce more accessible output. LLMs are black boxes and often ignore/misinterpret instructions, especially for nuanced a11y details.
…
4) marketing
it is too early to claim that this set of markdown files helps agents “build modern web experiences that are accessible”. i don’t want to call it misleading, but it’s highly aspirational at best.
ultimately, i see this as a net positive *harm reduction* measure. it may well increase baseline a11y of LLM output in the long term. we all want less slop on the web. but that’s such a low bar, and far from anything “game-changing”.
…
-
4) marketing
it is too early to claim that this set of markdown files helps agents “build modern web experiences that are accessible”. i don’t want to call it misleading, but it’s highly aspirational at best.
ultimately, i see this as a net positive *harm reduction* measure. it may well increase baseline a11y of LLM output in the long term. we all want less slop on the web. but that’s such a low bar, and far from anything “game-changing”.
…
5) anti-human
i actually really like the idea of creating high-quality guidance around general web dev stuff and specific modern web features. it would make for a great resource for humans to learn!
instead, this whole thing is optimized only for machines and will likely never be read by humans (unless they're sleuthing around like me), which is a shame.
if humans are not reading it, they're also less likely to find issues and contribute back.
…
-
5) anti-human
i actually really like the idea of creating high-quality guidance around general web dev stuff and specific modern web features. it would make for a great resource for humans to learn!
instead, this whole thing is optimized only for machines and will likely never be read by humans (unless they're sleuthing around like me), which is a shame.
if humans are not reading it, they're also less likely to find issues and contribute back.
…
i’ve already spent more time on this than i would like. i hate talking about these tools (which normalizes them).
there is nothing in it for me. i don’t want to talk about any of this shit. i’m not getting paid, and none of it is useful in my own work in any meaningful way.
but i've taken the time to properly understand these systems because i think it's important to stay informed. hopefully this thread was helpful for anyone who’s curious about this whole thing.
/fin.
-
i’ve already spent more time on this than i would like. i hate talking about these tools (which normalizes them).
there is nothing in it for me. i don’t want to talk about any of this shit. i’m not getting paid, and none of it is useful in my own work in any meaningful way.
but i've taken the time to properly understand these systems because i think it's important to stay informed. hopefully this thread was helpful for anyone who’s curious about this whole thing.
/fin.
@mayank the thing that keeps getting me is it is being marketed as "assurance" that the llm will produce modern web code, when it cannot by definition assure anything because of its non-determinism.
Meanwhile another part of Google basically just said "all websites are nothing more than training fodder for our own LLMs" so what exactly is the point here?
-
@mayank the thing that keeps getting me is it is being marketed as "assurance" that the llm will produce modern web code, when it cannot by definition assure anything because of its non-determinism.
Meanwhile another part of Google basically just said "all websites are nothing more than training fodder for our own LLMs" so what exactly is the point here?
@cthos indeed, i mentioned this in points 3 and 4.
i don't think the two parts of google even talk to each other. this particular effort is from the chrome/devrel team, who have historically been known to genuinely care about the web in their own way. but they also believe in LLMs and desperately want it to be less bad. so yeah, aspirational.
-
@cthos indeed, i mentioned this in points 3 and 4.
i don't think the two parts of google even talk to each other. this particular effort is from the chrome/devrel team, who have historically been known to genuinely care about the web in their own way. but they also believe in LLMs and desperately want it to be less bad. so yeah, aspirational.
@mayank I missed your middle two points, sorry for the duplication
I am also happy you see it too
-
i’ve already spent more time on this than i would like. i hate talking about these tools (which normalizes them).
there is nothing in it for me. i don’t want to talk about any of this shit. i’m not getting paid, and none of it is useful in my own work in any meaningful way.
but i've taken the time to properly understand these systems because i think it's important to stay informed. hopefully this thread was helpful for anyone who’s curious about this whole thing.
/fin.
@mayank the fact they're actively killing websites with their search changes and producing this feels less like harm reduction and more like blowing your head off with a shot gun and handing you a bandage
-
@mayank the fact they're actively killing websites with their search changes and producing this feels less like harm reduction and more like blowing your head off with a shot gun and handing you a bandage
@NicMakesStuff why did i picture that vividly lmao
-
@NicMakesStuff why did i picture that vividly lmao
@mayank haha whoops should put a content warning on there!
-
i have thoughts on google’s modern-web-guidance thing…
i’m looking at it solely from an accessibility perspective, for which there is a high-level guide, as well as small details sprinkled into use-case specific guides.
(the whole thing is open source so i was able to look pretty deep)
@mayank Digging into those big tech specs always makes me grateful for native iOS APIs. Trying to retroactively guide developers on web accessibility feels so much more convoluted than just having accessibility baked into the core UIKit or SwiftUI components from day one.
-
RE: https://front-end.social/@SaraSoueidan/116610307042155106
3) lack of manual testing
afaict, only the use-case specific handcoded demos were manually tested. i can see an "eval" flow with e2e tests that regularly validate outputs from different models, which is good but *not* a replacement for real testing (esp. from disabled users).
there is no data to back up the claim that this skill actually helps these tools produce more accessible output. LLMs are black boxes and often ignore/misinterpret instructions, especially for nuanced a11y details.
…
@mayank We have seen in this research by @kc that expert-level guidance only improves accessibility output of the slop machines by so much versus saying “make it accessible”. It’s ultimately trying to keep up the mirage that these things can produce acceptable output, while they are telling it repeatedly to not punch itself in the face. Because it otherwise would.
Read Casey’s full paper here: https://conesible.de/wab/
-
@mayank We have seen in this research by @kc that expert-level guidance only improves accessibility output of the slop machines by so much versus saying “make it accessible”. It’s ultimately trying to keep up the mirage that these things can produce acceptable output, while they are telling it repeatedly to not punch itself in the face. Because it otherwise would.
Read Casey’s full paper here: https://conesible.de/wab/
-
i have thoughts on google’s modern-web-guidance thing…
i’m looking at it solely from an accessibility perspective, for which there is a high-level guide, as well as small details sprinkled into use-case specific guides.
(the whole thing is open source so i was able to look pretty deep)
@mayank Thank you for taking the time to dig in and more so to share your findings, Mayank!
-
5) anti-human
i actually really like the idea of creating high-quality guidance around general web dev stuff and specific modern web features. it would make for a great resource for humans to learn!
instead, this whole thing is optimized only for machines and will likely never be read by humans (unless they're sleuthing around like me), which is a shame.
if humans are not reading it, they're also less likely to find issues and contribute back.
…
@mayank this is why i never understood the value of AGENT.md files. it's clear these companies WANT people to stop thinking of the details.
-
@mayank Thank you for taking the time to dig in and more so to share your findings, Mayank!
@SaraSoueidan and thank you for initiating the pushback! i wasn't originally planning to share any of this.
-
R relay@relay.mycrowd.ca shared this topic
