I have a few questions...
-
I have a few questions... "Security exercise" sounds planned but this is "Unplanned maintenance" on a Friday night.
Is PostHog rotating keys due to a security incident?
-
I have a few questions... "Security exercise" sounds planned but this is "Unplanned maintenance" on a Friday night.
Is PostHog rotating keys due to a security incident?
"Unplanned" just means it wasn't on the maintenance calendar, not that it's an accident. A planned key rotation they didn't pre-announce lands there by default.
And it went from "doing maintenance" to "it's a security exercise" — that's the opposite of how a breach reads. Those escalate into an advisory and a "rotate your keys" email. None of that here. Fair to side-eye given the month we're having, but this looks like hygiene.
-
"Unplanned" just means it wasn't on the maintenance calendar, not that it's an accident. A planned key rotation they didn't pre-announce lands there by default.
And it went from "doing maintenance" to "it's a security exercise" — that's the opposite of how a breach reads. Those escalate into an advisory and a "rotate your keys" email. None of that here. Fair to side-eye given the month we're having, but this looks like hygiene.
@olearysec AFAIK this is the first time they've done any planned maintenance that impacted web app availability, going back several years.
There's been many unplanned issues that impacted web app availability, but none cited anything similar to this (like key rotation or security exercise).
I hope you're right and they forgot to announce it, but also seems unusual given they haven't done this before in a way that impacted web app availability, either as planned maintenance or unplanned maintenance. All the unplanned maintenance affecting web app uptime I've seen has never cited security exercise or key rotation.
-
I have a few questions... "Security exercise" sounds planned but this is "Unplanned maintenance" on a Friday night.
Is PostHog rotating keys due to a security incident?
Sounds like an external security researcher was able to access one of PostHog's AWS environments.
Also note the quiet update of the existing status (same timestamp as earlier update; no email sent out to incident subscribers).
"We are rotating keys after a security research team was able to confirm an exploit in one of our AWS environments. We're working with the security research team on the issue. No keys were publicly available, and no data has been compromised. You may see impacts on exports, reverse proxies, and other services. We'll have more updates as we continue to work on this incident."
-
@olearysec AFAIK this is the first time they've done any planned maintenance that impacted web app availability, going back several years.
There's been many unplanned issues that impacted web app availability, but none cited anything similar to this (like key rotation or security exercise).
I hope you're right and they forgot to announce it, but also seems unusual given they haven't done this before in a way that impacted web app availability, either as planned maintenance or unplanned maintenance. All the unplanned maintenance affecting web app uptime I've seen has never cited security exercise or key rotation.
@olearysec Update: It's a security incident of sorts.
Alesandro Ortiz 🇵🇷🏳️🌈 (@AlesandroOrtiz@infosec.exchange)
Attached: 1 image Sounds like an external security researcher was able to access one of PostHog's AWS environments. Also note the quiet update of the existing status (same timestamp as earlier update; no email sent out to incident subscribers). "We are rotating keys after a security research team was able to confirm an exploit in one of our AWS environments. We're working with the security research team on the issue. No keys were publicly available, and no data has been compromised. You may see impacts on exports, reverse proxies, and other services. We'll have more updates as we continue to work on this incident." https://www.posthogstatus.com/incidents/01KSV6HJYKG5QJAP8HVTSQVSM1
Infosec Exchange (infosec.exchange)
-
Sounds like an external security researcher was able to access one of PostHog's AWS environments.
Also note the quiet update of the existing status (same timestamp as earlier update; no email sent out to incident subscribers).
"We are rotating keys after a security research team was able to confirm an exploit in one of our AWS environments. We're working with the security research team on the issue. No keys were publicly available, and no data has been compromised. You may see impacts on exports, reverse proxies, and other services. We'll have more updates as we continue to work on this incident."
Kudos to PostHog for the real-time disclosure at least. They could have disclosed this in a quiet blog post a week from now. Only customers subscribed to app status page incidents would be notified via email, so also need to see how they notify customers directly who aren't subscribed to status page.
Also #hugops since security incidents are never fun.
-
@olearysec AFAIK this is the first time they've done any planned maintenance that impacted web app availability, going back several years.
There's been many unplanned issues that impacted web app availability, but none cited anything similar to this (like key rotation or security exercise).
I hope you're right and they forgot to announce it, but also seems unusual given they haven't done this before in a way that impacted web app availability, either as planned maintenance or unplanned maintenance. All the unplanned maintenance affecting web app uptime I've seen has never cited security exercise or key rotation.
Update: the 01:18 entry got edited. "Security exercise" is gone, now it says they're rotating keys after a research team confirmed an exploit in one of their AWS environments. So you called it. Incident-driven, not hygiene. Good catch.
-
Update: the 01:18 entry got edited. "Security exercise" is gone, now it says they're rotating keys after a research team confirmed an exploit in one of their AWS environments. So you called it. Incident-driven, not hygiene. Good catch.
@olearysec Yeah, I posted about it here: https://infosec.exchange/@AlesandroOrtiz/116661218239511606
Was still really hoping you were right.
-
Kudos to PostHog for the real-time disclosure at least. They could have disclosed this in a quiet blog post a week from now. Only customers subscribed to app status page incidents would be notified via email, so also need to see how they notify customers directly who aren't subscribed to status page.
Also #hugops since security incidents are never fun.
Still waiting on promised postmortem. Latest update from Saturday:
"A security researcher privately disclosed a vulnerability that allowed access to production credentials. We've fixed the underlying issue and are actively working on additional hardening.As a precaution, we immediately rotated our most sensitive production credentials."
-
Still waiting on promised postmortem. Latest update from Saturday:
"A security researcher privately disclosed a vulnerability that allowed access to production credentials. We've fixed the underlying issue and are actively working on additional hardening.As a precaution, we immediately rotated our most sensitive production credentials."
@AlesandroOrtiz how is "PostHog" a real, actual name of a real, actual company? They can't be serious.
-
@AlesandroOrtiz how is "PostHog" a real, actual name of a real, actual company? They can't be serious.
@zkat Apparently. I didn't even know the slang meaning until people started pointing it out to me recently.
I have yet to find a good explanation for the name. -
R relay@relay.infosec.exchange shared this topic
