Skip to content
  • Categories
  • Recent
  • Tags
  • Popular
  • World
  • Users
  • Groups
Skins
  • Light
  • Brite
  • Cerulean
  • Cosmo
  • Flatly
  • Journal
  • Litera
  • Lumen
  • Lux
  • Materia
  • Minty
  • Morph
  • Pulse
  • Sandstone
  • Simplex
  • Sketchy
  • Spacelab
  • United
  • Yeti
  • Zephyr
  • Dark
  • Cyborg
  • Darkly
  • Quartz
  • Slate
  • Solar
  • Superhero
  • Vapor

  • Default (Cyborg)
  • No Skin
Collapse
Brand Logo

CIRCLE WITH A DOT

  1. Home
  2. Uncategorized
  3. Why is checking the LOGS is always the 4th or 5th step in the troubleshooting flow?

Why is checking the LOGS is always the 4th or 5th step in the troubleshooting flow?

Scheduled Pinned Locked Moved Uncategorized
troubleshootingskillslogs
13 Posts 5 Posters 31 Views
  • Oldest to Newest
  • Newest to Oldest
  • Most Votes
Reply
  • Reply as topic
Log in to reply
This topic has been deleted. Only users with topic management privileges can see it.
  • pejacoby@infosec.exchangeP This user is from outside of this forum
    pejacoby@infosec.exchangeP This user is from outside of this forum
    pejacoby@infosec.exchange
    wrote last edited by
    #1

    Why is checking the LOGS is always the 4th or 5th step in the troubleshooting flow?

    The logs, they have iNfORMatiON that the programmer figured you’d like to know when shit goes sideways.

    Start there! Read that wisdom, even if it’s half misspelled and riddled with weird numbers. It SPEakS to you!

    #troubleshooting #skills #logs

    kajer@infosec.exchangeK rootwyrm@weird.autosR 2 Replies Last reply
    0
    • pejacoby@infosec.exchangeP pejacoby@infosec.exchange

      Why is checking the LOGS is always the 4th or 5th step in the troubleshooting flow?

      The logs, they have iNfORMatiON that the programmer figured you’d like to know when shit goes sideways.

      Start there! Read that wisdom, even if it’s half misspelled and riddled with weird numbers. It SPEakS to you!

      #troubleshooting #skills #logs

      kajer@infosec.exchangeK This user is from outside of this forum
      kajer@infosec.exchangeK This user is from outside of this forum
      kajer@infosec.exchange
      wrote last edited by
      #2

      @pejacoby I usually start with logs

      I have a syslog server collecting logs, like a psychopath. No graphana , no docker containers, just syslog-ng listening on UDP514 with allow-lists for known devices.

      that one device having shit connectivity? guess what? syslog told me that that port flapped and had ~32000 BPDU messages in ~3 seconds.

      What do I find? unmanaged switch inline and some loose ethernet cables.... Someone created a loop.

      Why BPDU guard wasn't enabled is a different story, but here we are.

      READ YOUR LOGS PEOPLE

      pejacoby@infosec.exchangeP nuintari@mastodon.bsd.cafeN 2 Replies Last reply
      0
      • kajer@infosec.exchangeK kajer@infosec.exchange

        @pejacoby I usually start with logs

        I have a syslog server collecting logs, like a psychopath. No graphana , no docker containers, just syslog-ng listening on UDP514 with allow-lists for known devices.

        that one device having shit connectivity? guess what? syslog told me that that port flapped and had ~32000 BPDU messages in ~3 seconds.

        What do I find? unmanaged switch inline and some loose ethernet cables.... Someone created a loop.

        Why BPDU guard wasn't enabled is a different story, but here we are.

        READ YOUR LOGS PEOPLE

        pejacoby@infosec.exchangeP This user is from outside of this forum
        pejacoby@infosec.exchangeP This user is from outside of this forum
        pejacoby@infosec.exchange
        wrote last edited by
        #3

        @kajer amen brother!

        Not once, not twice, but at least three times today on one 90 minute call…

        Hey, you, tail /var/log/foo - see that? Fix that error.

        Ok now tail it again - error changed? Progress! Fix that one now…

        And one more time…no errors? holy shit it works now?

        ams@infosec.exchangeA 1 Reply Last reply
        0
        • pejacoby@infosec.exchangeP pejacoby@infosec.exchange

          @kajer amen brother!

          Not once, not twice, but at least three times today on one 90 minute call…

          Hey, you, tail /var/log/foo - see that? Fix that error.

          Ok now tail it again - error changed? Progress! Fix that one now…

          And one more time…no errors? holy shit it works now?

          ams@infosec.exchangeA This user is from outside of this forum
          ams@infosec.exchangeA This user is from outside of this forum
          ams@infosec.exchange
          wrote last edited by
          #4

          @pejacoby @kajer Some of these folks never had their /var/log/messages tailed to their desktop background.

          kajer@infosec.exchangeK 1 Reply Last reply
          0
          • ams@infosec.exchangeA ams@infosec.exchange

            @pejacoby @kajer Some of these folks never had their /var/log/messages tailed to their desktop background.

            kajer@infosec.exchangeK This user is from outside of this forum
            kajer@infosec.exchangeK This user is from outside of this forum
            kajer@infosec.exchange
            wrote last edited by
            #5

            @AMS @pejacoby

            i'm going to poke the bear as per usual...

            journalctl

            ams@infosec.exchangeA 1 Reply Last reply
            0
            • pejacoby@infosec.exchangeP pejacoby@infosec.exchange

              Why is checking the LOGS is always the 4th or 5th step in the troubleshooting flow?

              The logs, they have iNfORMatiON that the programmer figured you’d like to know when shit goes sideways.

              Start there! Read that wisdom, even if it’s half misspelled and riddled with weird numbers. It SPEakS to you!

              #troubleshooting #skills #logs

              rootwyrm@weird.autosR This user is from outside of this forum
              rootwyrm@weird.autosR This user is from outside of this forum
              rootwyrm@weird.autos
              wrote last edited by
              #6

              @pejacoby @kajer and then you find out that like 90% of "modern" shitware that everyone has decreed is the New Hotness Which Must Be Used For All Microservices produces logs like this:

              <50 lines of logo>
              SERVER STARTED!
              An unexpected error occurred
              <EOF>

              Or just:
              <EOF>

              pejacoby@infosec.exchangeP 1 Reply Last reply
              0
              • kajer@infosec.exchangeK kajer@infosec.exchange

                @AMS @pejacoby

                i'm going to poke the bear as per usual...

                journalctl

                ams@infosec.exchangeA This user is from outside of this forum
                ams@infosec.exchangeA This user is from outside of this forum
                ams@infosec.exchange
                wrote last edited by
                #7

                @kajer @pejacoby There's a reason any system I care about is running openrc.

                nuintari@mastodon.bsd.cafeN 1 Reply Last reply
                0
                • rootwyrm@weird.autosR rootwyrm@weird.autos

                  @pejacoby @kajer and then you find out that like 90% of "modern" shitware that everyone has decreed is the New Hotness Which Must Be Used For All Microservices produces logs like this:

                  <50 lines of logo>
                  SERVER STARTED!
                  An unexpected error occurred
                  <EOF>

                  Or just:
                  <EOF>

                  pejacoby@infosec.exchangeP This user is from outside of this forum
                  pejacoby@infosec.exchangeP This user is from outside of this forum
                  pejacoby@infosec.exchange
                  wrote last edited by
                  #8

                  @rootwyrm @kajer oh yes, so much this. One team didn’t realize Docker console logs flowed into LogAnalytics automatically until all that useless crap showed up as multi-thousand dollar bill one week. That got turned off…

                  kajer@infosec.exchangeK 1 Reply Last reply
                  0
                  • pejacoby@infosec.exchangeP pejacoby@infosec.exchange

                    @rootwyrm @kajer oh yes, so much this. One team didn’t realize Docker console logs flowed into LogAnalytics automatically until all that useless crap showed up as multi-thousand dollar bill one week. That got turned off…

                    kajer@infosec.exchangeK This user is from outside of this forum
                    kajer@infosec.exchangeK This user is from outside of this forum
                    kajer@infosec.exchange
                    wrote last edited by
                    #9

                    @pejacoby @rootwyrm

                    One F100 org I was at had multi million dollar splunk instance... NEVER AGAIN

                    we got free tshirts tho, so that was neat.

                    rootwyrm@weird.autosR 1 Reply Last reply
                    0
                    • kajer@infosec.exchangeK kajer@infosec.exchange

                      @pejacoby I usually start with logs

                      I have a syslog server collecting logs, like a psychopath. No graphana , no docker containers, just syslog-ng listening on UDP514 with allow-lists for known devices.

                      that one device having shit connectivity? guess what? syslog told me that that port flapped and had ~32000 BPDU messages in ~3 seconds.

                      What do I find? unmanaged switch inline and some loose ethernet cables.... Someone created a loop.

                      Why BPDU guard wasn't enabled is a different story, but here we are.

                      READ YOUR LOGS PEOPLE

                      nuintari@mastodon.bsd.cafeN This user is from outside of this forum
                      nuintari@mastodon.bsd.cafeN This user is from outside of this forum
                      nuintari@mastodon.bsd.cafe
                      wrote last edited by
                      #10

                      @kajer @pejacoby Unmanaged switches pay for my groceries. I have so many stories that start with, "That one time nuintari got paid $500 to unplug a cable...."

                      1 Reply Last reply
                      0
                      • ams@infosec.exchangeA ams@infosec.exchange

                        @kajer @pejacoby There's a reason any system I care about is running openrc.

                        nuintari@mastodon.bsd.cafeN This user is from outside of this forum
                        nuintari@mastodon.bsd.cafeN This user is from outside of this forum
                        nuintari@mastodon.bsd.cafe
                        wrote last edited by
                        #11

                        @AMS @kajer @pejacoby journalctl is the first and foremost reason I despise systemd.

                        1 Reply Last reply
                        0
                        • kajer@infosec.exchangeK kajer@infosec.exchange

                          @pejacoby @rootwyrm

                          One F100 org I was at had multi million dollar splunk instance... NEVER AGAIN

                          we got free tshirts tho, so that was neat.

                          rootwyrm@weird.autosR This user is from outside of this forum
                          rootwyrm@weird.autosR This user is from outside of this forum
                          rootwyrm@weird.autos
                          wrote last edited by
                          #12

                          @kajer @pejacoby oh, it's super great when you're trying to debug a stupid application in a Docker container and the ONLY way to get the information MAYBE is to set debug.
                          You got 10 minutes of *idle* runtime before 'docker logs shitheap' crashed from oversize.

                          pejacoby@infosec.exchangeP 1 Reply Last reply
                          0
                          • rootwyrm@weird.autosR rootwyrm@weird.autos

                            @kajer @pejacoby oh, it's super great when you're trying to debug a stupid application in a Docker container and the ONLY way to get the information MAYBE is to set debug.
                            You got 10 minutes of *idle* runtime before 'docker logs shitheap' crashed from oversize.

                            pejacoby@infosec.exchangeP This user is from outside of this forum
                            pejacoby@infosec.exchangeP This user is from outside of this forum
                            pejacoby@infosec.exchange
                            wrote last edited by
                            #13

                            @rootwyrm @kajer did I mention I’m having a second beer this glorious log evening?

                            1 Reply Last reply
                            1
                            0
                            • R relay@relay.infosec.exchange shared this topic
                            Reply
                            • Reply as topic
                            Log in to reply
                            • Oldest to Newest
                            • Newest to Oldest
                            • Most Votes


                            • Login

                            • Login or register to search.
                            • First post
                              Last post
                            0
                            • Categories
                            • Recent
                            • Tags
                            • Popular
                            • World
                            • Users
                            • Groups