i have thoughts on google’s modern-web-guidance thing…
-
1) slop as the basis
the initial drafts (for accessibility and other high-level guides) were LLM-generated. human expertise was brought in later, which is good, but it makes the work largely remedial and leaves unresolved issues (both subtle details and larger gaps).
i get that there may have been tight deadlines due to i/o 26 and whatnot, but that’s part of the problem.
accessibility is too important to start from this questionable foundation …
Add discipline skills and researcher by rviscomi · Pull Request #430 · GoogleChrome/modern-web-guidance-src
Contribute to GoogleChrome/modern-web-guidance-src development by creating an account on GitHub.
GitHub (github.com)
RE: https://toot.cafe/@aardrian/116608683977339586
2) a11y needs manual invocation
the search flow makes sense for one-off use cases/features, but the a11y guide is high level and not guaranteed to be retrieved unless explicitly prompted. (the metadata says it’s to be used “when auditing or implementing accessibility features, keyboard navigation, or ARIA”)
a11y should be deeply ingrained into all UI work and not require opting in. anyone who’s not familiar/proficient will not know how to prompt and generate inaccessible output by default.
…
-
RE: https://toot.cafe/@aardrian/116608683977339586
2) a11y needs manual invocation
the search flow makes sense for one-off use cases/features, but the a11y guide is high level and not guaranteed to be retrieved unless explicitly prompted. (the metadata says it’s to be used “when auditing or implementing accessibility features, keyboard navigation, or ARIA”)
a11y should be deeply ingrained into all UI work and not require opting in. anyone who’s not familiar/proficient will not know how to prompt and generate inaccessible output by default.
…
RE: https://front-end.social/@SaraSoueidan/116610307042155106
3) lack of manual testing
afaict, only the use-case specific handcoded demos were manually tested. i can see an "eval" flow with e2e tests that regularly validate outputs from different models, which is good but *not* a replacement for real testing (esp. from disabled users).
there is no data to back up the claim that this skill actually helps these tools produce more accessible output. LLMs are black boxes and often ignore/misinterpret instructions, especially for nuanced a11y details.
…
-
RE: https://front-end.social/@SaraSoueidan/116610307042155106
3) lack of manual testing
afaict, only the use-case specific handcoded demos were manually tested. i can see an "eval" flow with e2e tests that regularly validate outputs from different models, which is good but *not* a replacement for real testing (esp. from disabled users).
there is no data to back up the claim that this skill actually helps these tools produce more accessible output. LLMs are black boxes and often ignore/misinterpret instructions, especially for nuanced a11y details.
…
4) marketing
it is too early to claim that this set of markdown files helps agents “build modern web experiences that are accessible”. i don’t want to call it misleading, but it’s highly aspirational at best.
ultimately, i see this as a net positive *harm reduction* measure. it may well increase baseline a11y of LLM output in the long term. we all want less slop on the web. but that’s such a low bar, and far from anything “game-changing”.
…
-
4) marketing
it is too early to claim that this set of markdown files helps agents “build modern web experiences that are accessible”. i don’t want to call it misleading, but it’s highly aspirational at best.
ultimately, i see this as a net positive *harm reduction* measure. it may well increase baseline a11y of LLM output in the long term. we all want less slop on the web. but that’s such a low bar, and far from anything “game-changing”.
…
5) anti-human
i actually really like the idea of creating high-quality guidance around general web dev stuff and specific modern web features. it would make for a great resource for humans to learn!
instead, this whole thing is optimized only for machines and will likely never be read by humans (unless they're sleuthing around like me), which is a shame.
if humans are not reading it, they're also less likely to find issues and contribute back.
…
-
5) anti-human
i actually really like the idea of creating high-quality guidance around general web dev stuff and specific modern web features. it would make for a great resource for humans to learn!
instead, this whole thing is optimized only for machines and will likely never be read by humans (unless they're sleuthing around like me), which is a shame.
if humans are not reading it, they're also less likely to find issues and contribute back.
…
i’ve already spent more time on this than i would like. i hate talking about these tools (which normalizes them).
there is nothing in it for me. i don’t want to talk about any of this shit. i’m not getting paid, and none of it is useful in my own work in any meaningful way.
but i've taken the time to properly understand these systems because i think it's important to stay informed. hopefully this thread was helpful for anyone who’s curious about this whole thing.
/fin.
-
i’ve already spent more time on this than i would like. i hate talking about these tools (which normalizes them).
there is nothing in it for me. i don’t want to talk about any of this shit. i’m not getting paid, and none of it is useful in my own work in any meaningful way.
but i've taken the time to properly understand these systems because i think it's important to stay informed. hopefully this thread was helpful for anyone who’s curious about this whole thing.
/fin.
@mayank the thing that keeps getting me is it is being marketed as "assurance" that the llm will produce modern web code, when it cannot by definition assure anything because of its non-determinism.
Meanwhile another part of Google basically just said "all websites are nothing more than training fodder for our own LLMs" so what exactly is the point here?
-
@mayank the thing that keeps getting me is it is being marketed as "assurance" that the llm will produce modern web code, when it cannot by definition assure anything because of its non-determinism.
Meanwhile another part of Google basically just said "all websites are nothing more than training fodder for our own LLMs" so what exactly is the point here?
@cthos indeed, i mentioned this in points 3 and 4.
i don't think the two parts of google even talk to each other. this particular effort is from the chrome/devrel team, who have historically been known to genuinely care about the web in their own way. but they also believe in LLMs and desperately want it to be less bad. so yeah, aspirational.
-
@cthos indeed, i mentioned this in points 3 and 4.
i don't think the two parts of google even talk to each other. this particular effort is from the chrome/devrel team, who have historically been known to genuinely care about the web in their own way. but they also believe in LLMs and desperately want it to be less bad. so yeah, aspirational.
@mayank I missed your middle two points, sorry for the duplication
I am also happy you see it too
-
i’ve already spent more time on this than i would like. i hate talking about these tools (which normalizes them).
there is nothing in it for me. i don’t want to talk about any of this shit. i’m not getting paid, and none of it is useful in my own work in any meaningful way.
but i've taken the time to properly understand these systems because i think it's important to stay informed. hopefully this thread was helpful for anyone who’s curious about this whole thing.
/fin.
@mayank the fact they're actively killing websites with their search changes and producing this feels less like harm reduction and more like blowing your head off with a shot gun and handing you a bandage
-
@mayank the fact they're actively killing websites with their search changes and producing this feels less like harm reduction and more like blowing your head off with a shot gun and handing you a bandage
@NicMakesStuff why did i picture that vividly lmao
-
@NicMakesStuff why did i picture that vividly lmao
@mayank haha whoops should put a content warning on there!
-
i have thoughts on google’s modern-web-guidance thing…
i’m looking at it solely from an accessibility perspective, for which there is a high-level guide, as well as small details sprinkled into use-case specific guides.
modern-web-guidance-src/guides/accessibility/accessibility/guide.md at main · GoogleChrome/modern-web-guidance-src
Contribute to GoogleChrome/modern-web-guidance-src development by creating an account on GitHub.
GitHub (github.com)
(the whole thing is open source so i was able to look pretty deep)
@mayank Digging into those big tech specs always makes me grateful for native iOS APIs. Trying to retroactively guide developers on web accessibility feels so much more convoluted than just having accessibility baked into the core UIKit or SwiftUI components from day one.
-
RE: https://front-end.social/@SaraSoueidan/116610307042155106
3) lack of manual testing
afaict, only the use-case specific handcoded demos were manually tested. i can see an "eval" flow with e2e tests that regularly validate outputs from different models, which is good but *not* a replacement for real testing (esp. from disabled users).
there is no data to back up the claim that this skill actually helps these tools produce more accessible output. LLMs are black boxes and often ignore/misinterpret instructions, especially for nuanced a11y details.
…
@mayank We have seen in this research by @kc that expert-level guidance only improves accessibility output of the slop machines by so much versus saying “make it accessible”. It’s ultimately trying to keep up the mirage that these things can produce acceptable output, while they are telling it repeatedly to not punch itself in the face. Because it otherwise would.
Read Casey’s full paper here: https://conesible.de/wab/
-
@mayank We have seen in this research by @kc that expert-level guidance only improves accessibility output of the slop machines by so much versus saying “make it accessible”. It’s ultimately trying to keep up the mirage that these things can produce acceptable output, while they are telling it repeatedly to not punch itself in the face. Because it otherwise would.
Read Casey’s full paper here: https://conesible.de/wab/
-
i have thoughts on google’s modern-web-guidance thing…
i’m looking at it solely from an accessibility perspective, for which there is a high-level guide, as well as small details sprinkled into use-case specific guides.
modern-web-guidance-src/guides/accessibility/accessibility/guide.md at main · GoogleChrome/modern-web-guidance-src
Contribute to GoogleChrome/modern-web-guidance-src development by creating an account on GitHub.
GitHub (github.com)
(the whole thing is open source so i was able to look pretty deep)
@mayank Thank you for taking the time to dig in and more so to share your findings, Mayank!
-
5) anti-human
i actually really like the idea of creating high-quality guidance around general web dev stuff and specific modern web features. it would make for a great resource for humans to learn!
instead, this whole thing is optimized only for machines and will likely never be read by humans (unless they're sleuthing around like me), which is a shame.
if humans are not reading it, they're also less likely to find issues and contribute back.
…
@mayank this is why i never understood the value of AGENT.md files. it's clear these companies WANT people to stop thinking of the details.
-
@mayank Thank you for taking the time to dig in and more so to share your findings, Mayank!
@SaraSoueidan and thank you for initiating the pushback! i wasn't originally planning to share any of this.
-
R relay@relay.mycrowd.ca shared this topic
