We observe increased error rates and are investigating #mastoadmin

[ordnung@chaos.social]

We observe increased error rates and are investigating #mastoadmin

[luc@chaos.social]

@ordnung Came here to search for the osm_tech account and see if they posted anything about an openstreetmap api outage

Found a different outage instead xD
What is it today

Good luck finding out what's up! Don't stress!

[ordnung@chaos.social]

Looks like the problem is solved. Systemd-resolved was in a strange stage where it still resolved requests via dig as usual but for example a curl or mtr timed out on the DNS stage. Restarting systemd-resolved solved the problem. #mastoadmin

[zornem@chaos.social]

@ordnung thank you!

[jay16k@chaos.social]

@ordnung sorry, not sorry

But thanks for keeping it running ️

[manawyrm@chaos.social]

@ordnung I've seen this exact issue in prod sooo, sooo many times and I still can't figure out what's wrong with the thing.

Ended up replacing it entirely for prod machines, but I'd still really like to know what's happening there.

There's some issue that looks the same, which sometimes (but not reproducibly) occurs when IPv6 addresses change while resolved is running, but I'm not sure if it's the same issue as the sudden/random hangs...
*sigh*

*hugops*, thanks for your time and effort!

[sandzwerg@chaos.social]

@ordnung thank you ️

[jay9@chaos.social]

@ordnung thank you!

[equinox@chaos.social]

@Jay16K uh, neither the DNS protocol nor general practices or shortcomings around DNS seem to be at fault here. Pretty much any protocol, when implemented by people with insufficient understanding, will suffer bugs like this.

I mean, I'm a L3 person, I'll happily shit on DNS all day long… but not without reason?

[nameless@chaos.social]

@manawyrm @ordnung Same with me. Messed around a bit with resolvectl but no luck so far. Happens since a few months/weeks. Don't know exact version though.