I felt a great disturbance in the Force, as if millions of domains suddenly cried out in terror and were suddenly silenced.
-
I felt a great disturbance in the Force, as if millions of domains suddenly cried out in terror and were suddenly silenced.
The .de TLD is the third largest in the world with ~18 million domains. All DNSSEC-aware resolvers didn’t serve any of those.
We just cancelled the Major Incident and sent everyone to bed. The remaining impact is outside our sphere of influence.
-
I felt a great disturbance in the Force, as if millions of domains suddenly cried out in terror and were suddenly silenced.
The .de TLD is the third largest in the world with ~18 million domains. All DNSSEC-aware resolvers didn’t serve any of those.
We just cancelled the Major Incident and sent everyone to bed. The remaining impact is outside our sphere of influence.
If only DNS was decentraiised
-
I felt a great disturbance in the Force, as if millions of domains suddenly cried out in terror and were suddenly silenced.
The .de TLD is the third largest in the world with ~18 million domains. All DNSSEC-aware resolvers didn’t serve any of those.
We just cancelled the Major Incident and sent everyone to bed. The remaining impact is outside our sphere of influence.
@masek Details?
Were these DNSSEC aware resolvers returning no-such domain responses?
Was there an error in the signing?
What tools are proving useful in understanding and diagnosing the issue?
-
I felt a great disturbance in the Force, as if millions of domains suddenly cried out in terror and were suddenly silenced.
The .de TLD is the third largest in the world with ~18 million domains. All DNSSEC-aware resolvers didn’t serve any of those.
We just cancelled the Major Incident and sent everyone to bed. The remaining impact is outside our sphere of influence.
I am still speechless about the incident. That is something that should have been impossible to happen at that level.
Personally, I really felt a disturbance in the force.
-
@masek Details?
Were these DNSSEC aware resolvers returning no-such domain responses?
Was there an error in the signing?
What tools are proving useful in understanding and diagnosing the issue?
@karlauerbach I will send an update later today. There is no word about the root cause yet. I am deeply disturbed.
-
@masek Details?
Were these DNSSEC aware resolvers returning no-such domain responses?
Was there an error in the signing?
What tools are proving useful in understanding and diagnosing the issue?
@karlauerbach Technically it was quite simple: they suddenly used a new ZSK (zone signing key) nobody else (especially the root NS) knew nothing about and killed the trust chain.
Every DNSSEC aware resolver refused to resolve any .de domain.
Workaround was that a hell lot of ISPs disabled DNSSEC on their resolvers.
They needed 150min to fix that problem (revert to the old ZSK), they didn't say a word about the cause yet.
An utter and complete shitshow.
-
@karlauerbach Technically it was quite simple: they suddenly used a new ZSK (zone signing key) nobody else (especially the root NS) knew nothing about and killed the trust chain.
Every DNSSEC aware resolver refused to resolve any .de domain.
Workaround was that a hell lot of ISPs disabled DNSSEC on their resolvers.
They needed 150min to fix that problem (revert to the old ZSK), they didn't say a word about the cause yet.
An utter and complete shitshow.
@masek That sounds crazy. DNSSEC is a sequence of keys in a hierarchy that starts at the root (or a trust anchor) and works down through the zones. One can't validate a record without that full sequence of keys.
As long as 20+ years ago when DNSSEC was a baby and I was arguing for competing roots one of the questions was whether DNSSEC tied the name hierarchy to exactly one root zone file - the answer is "yes, sort of" - and whether competing roots could use that with a different set of root NS records in that file - the answer is "yes".
I was always concerned about the key management - often the most complicated part of any crypto system - and I had faith in people like Patrick F. to figure it out (they did.)
I thought that ICANN and the ORSC had gone over and carefully rehearsed key updates and roll over procedures.
I was just, this week, discussing with ICANN folk about my 20 year old idea of establishing a worldwide DNS early warning system.
-
@masek That sounds crazy. DNSSEC is a sequence of keys in a hierarchy that starts at the root (or a trust anchor) and works down through the zones. One can't validate a record without that full sequence of keys.
As long as 20+ years ago when DNSSEC was a baby and I was arguing for competing roots one of the questions was whether DNSSEC tied the name hierarchy to exactly one root zone file - the answer is "yes, sort of" - and whether competing roots could use that with a different set of root NS records in that file - the answer is "yes".
I was always concerned about the key management - often the most complicated part of any crypto system - and I had faith in people like Patrick F. to figure it out (they did.)
I thought that ICANN and the ORSC had gone over and carefully rehearsed key updates and roll over procedures.
I was just, this week, discussing with ICANN folk about my 20 year old idea of establishing a worldwide DNS early warning system.
@karlauerbach There are some very German points to this. The operational instance belongs de facto to domain traders.
I will push for more information.
-
R relay@relay.infosec.exchange shared this topic
