Mastodon Incident Report / Root cause analysis:
Uncategorized
1
Posts
1
Posters
3
Views
-
Mastodon Incident Report / Root cause analysis:
Earlier today, users experienced timeouts with Search, Hashtags, and Autocomplete.
Root Cause: Our setup separates the Mastodon frontend VPS (Hetzner) from backend services (for example Elasticsearch) via an OPNSense firewall. Suricata (our IPS) triggered a false-positive on internal traffic and aggressively blocked the VPS IP, severing the connection to the search database.
Resolution: We identified the false-positive, added the frontend IP to the whitelist, and traffic immediately normalized. Everything is back to green!
#mastodon #mastoadmin #burningboard #elasticsearch #firewall #opnsense #suricata #oopsie
