Why is checking the LOGS is always the 4th or 5th step in the troubleshooting flow?

[pejacoby@infosec.exchange]

@kajer amen brother!

Not once, not twice, but at least three times today on one 90 minute call…

Hey, you, tail /var/log/foo - see that? Fix that error.

Ok now tail it again - error changed? Progress! Fix that one now…

And one more time…no errors? holy shit it works now?

[ams@infosec.exchange]

@pejacoby @kajer Some of these folks never had their /var/log/messages tailed to their desktop background.

[kajer@infosec.exchange]

@AMS @pejacoby

i'm going to poke the bear as per usual...

journalctl

[rootwyrm@weird.autos]

@pejacoby @kajer and then you find out that like 90% of "modern" shitware that everyone has decreed is the New Hotness Which Must Be Used For All Microservices produces logs like this:

<50 lines of logo>
SERVER STARTED!
An unexpected error occurred
<EOF>

Or just:
<EOF>

[ams@infosec.exchange]

@kajer @pejacoby There's a reason any system I care about is running openrc.

[pejacoby@infosec.exchange]

@rootwyrm @kajer oh yes, so much this. One team didn’t realize Docker console logs flowed into LogAnalytics automatically until all that useless crap showed up as multi-thousand dollar bill one week. That got turned off…

[kajer@infosec.exchange]

@pejacoby @rootwyrm

One F100 org I was at had multi million dollar splunk instance... NEVER AGAIN

we got free tshirts tho, so that was neat.

[nuintari@mastodon.bsd.cafe]

@kajer @pejacoby Unmanaged switches pay for my groceries. I have so many stories that start with, "That one time nuintari got paid $500 to unplug a cable...."

[nuintari@mastodon.bsd.cafe]

@AMS @kajer @pejacoby journalctl is the first and foremost reason I despise systemd.

[rootwyrm@weird.autos]

@kajer @pejacoby oh, it's super great when you're trying to debug a stupid application in a Docker container and the ONLY way to get the information MAYBE is to set debug.
You got 10 minutes of *idle* runtime before 'docker logs shitheap' crashed from oversize.

[pejacoby@infosec.exchange]

@rootwyrm @kajer did I mention I’m having a second beer this glorious log evening?