CIRCLE WITH A DOT

@SaraSoueidan and thank you for initiating the pushback! i wasn't originally planning to share any of this.

@yatil @kc ooh, instant bookmark! thanks for sharing

@NicMakesStuff why did i picture that vividly lmao

@cthos indeed, i mentioned this in points 3 and 4.

i don't think the two parts of google even talk to each other. this particular effort is from the chrome/devrel team, who have historically been known to genuinely care about the web in their own way. but they also believe in LLMs and desperately want it to be less bad. so yeah, aspirational.

i’ve already spent more time on this than i would like. i hate talking about these tools (which normalizes them).

there is nothing in it for me. i don’t want to talk about any of this shit. i’m not getting paid, and none of it is useful in my own work in any meaningful way.

but i've taken the time to properly understand these systems because i think it's important to stay informed. hopefully this thread was helpful for anyone who’s curious about this whole thing.

/fin.

5) anti-human

i actually really like the idea of creating high-quality guidance around general web dev stuff and specific modern web features. it would make for a great resource for humans to learn!

instead, this whole thing is optimized only for machines and will likely never be read by humans (unless they're sleuthing around like me), which is a shame.

if humans are not reading it, they're also less likely to find issues and contribute back.

…

4) marketing

it is too early to claim that this set of markdown files helps agents “build modern web experiences that are accessible”. i don’t want to call it misleading, but it’s highly aspirational at best.

ultimately, i see this as a net positive *harm reduction* measure. it may well increase baseline a11y of LLM output in the long term. we all want less slop on the web. but that’s such a low bar, and far from anything “game-changing”.

…

RE: https://front-end.social/@SaraSoueidan/116610307042155106

3) lack of manual testing

afaict, only the use-case specific handcoded demos were manually tested. i can see an "eval" flow with e2e tests that regularly validate outputs from different models, which is good but *not* a replacement for real testing (esp. from disabled users).

there is no data to back up the claim that this skill actually helps these tools produce more accessible output. LLMs are black boxes and often ignore/misinterpret instructions, especially for nuanced a11y details.

…

RE: https://toot.cafe/@aardrian/116608683977339586

2) a11y needs manual invocation

the search flow makes sense for one-off use cases/features, but the a11y guide is high level and not guaranteed to be retrieved unless explicitly prompted. (the metadata says it’s to be used “when auditing or implementing accessibility features, keyboard navigation, or ARIA”)

a11y should be deeply ingrained into all UI work and not require opting in. anyone who’s not familiar/proficient will not know how to prompt and generate inaccessible output by default.

…

1) slop as the basis

the initial drafts (for accessibility and other high-level guides) were LLM-generated. human expertise was brought in later, which is good, but it makes the work largely remedial and leaves unresolved issues (both subtle details and larger gaps).

i get that there may have been tight deadlines due to i/o 26 and whatnot, but that’s part of the problem.

accessibility is too important to start from this questionable foundation …

https://github.com/GoogleChrome/modern-web-guidance-src/pull/430/files

i have thoughts on google’s modern-web-guidance thing…

i’m looking at it solely from an accessibility perspective, for which there is a high-level guide, as well as small details sprinkled into use-case specific guides.

https://github.com/GoogleChrome/modern-web-guidance-src/blob/main/guides/accessibility/accessibility/guide.md

(the whole thing is open source so i was able to look pretty deep)