{"id":1941,"date":"2020-09-09T13:12:43","date_gmt":"2020-09-09T11:12:43","guid":{"rendered":"https:\/\/signal.eu.org\/blog\/?p=1941"},"modified":"2020-09-09T13:37:22","modified_gmt":"2020-09-09T11:37:22","slug":"post-mortem-of-a-dnssec-incident-at-eu-org","status":"publish","type":"post","link":"https:\/\/signal.eu.org\/blog\/2020\/09\/09\/post-mortem-of-a-dnssec-incident-at-eu-org\/","title":{"rendered":"Post-mortem of a DNSSEC incident at eu.org"},"content":{"rendered":"\n
Tweet<\/a><\/div>\n\n

(or: the good, the bad and the ugly)<\/h4>\n\n\n\n

Abstract<\/h4>\n\n\n\n

Due a bug in zone generation, all updates for the EU.ORG zone were stuck from 2020-08-29 02:19 UTC to 2020-09-04 14:40 UTC. Then an incorrect fix was made, resulting in the publication of incorrect DNSSEC signatures for the zone from 2020-09-04 14:40 UTC to 2020-09-04 19:37:00 UTC. Then the final, correct fix was implemented.<\/p>\n\n\n\n

This episode, unoriginal albeit humbling, nevertheless yielded interesting returns of experience.<\/p>\n\n\n\n

All times in the rest of this document are UTC times.<\/p>\n\n\n\n

The software setup at eu.org<\/h4>\n\n\n\n

The primary DNS server for EU.ORG runs ISC<\/a>‘s BIND<\/a>. The zone is currently generated by Python and shell scripts from a Postgresql<\/a> database. This does not include DNSSEC records for the zone (except DS<\/code> records for delegations). DNSSEC records are generated and refreshed by dnssec-signzone<\/code>, one of the tools provided with bind<\/code>. Once the zone file has been updated, it is reloaded using rndc reload<\/code>, another command-line tool provided with bind<\/code>.<\/p>\n\n\n\n

Zone key rotation is handled by custom scripts which periodically check for key age and schedule key generation, pre-publication, activation and de-activation as needed, calling dnssec-keygen<\/code> to manage the key files.<\/p>\n\n\n\n

Setup for the failure: blocked updates<\/h4>\n\n\n\n

2020-08-29 02:19<\/strong>: due to a race condition in the zone generation process (issue #1<\/em>), the EU.ORG zone file disappeared.<\/p>\n\n\n\n

The last good and published version of the EU.ORG zone file, still loaded in the primary server, had serial number 2020082907, generated at 2020-08-29 01:12. In the case of a missing file, the reload obviously fails but bind<\/code> behaves nicely and keeps serving its older in-memory version of the file.<\/p>\n\n\n\n

However, the disappearance of the zone file caused all subsequent zone file generation processes to fail (issue #2<\/em>), as they were accessing the current version of the file to fetch the currently published serial number.<\/p>\n\n\n\n

The problem remained unnoticed (issue #3: incomplete monitoring<\/em>) until 4 September 2020, when a user notified us that his new domain was still undelegated.<\/p>\n\n\n\n

The ugly<\/h4>\n\n\n\n

Around 2020-09-04 14:40, a first fix was attempted: a known good version of the zone file was reinstalled to allow the zone generation process to succeed, then a new zone was generated, freshly DNSSEC-signed, and loaded.<\/p>\n\n\n\n

However, the above timeline conflicted with a scheduled key rotation of the zone-signing keys. The theoretical key rotation schedule was as follows:<\/p>\n\n\n\n

\"\"
Theoretical key rotation schedule<\/figcaption><\/figure>\n\n\n\n

<\/p>\n\n\n\n

The new key (14716) was due to be published from 2020-08-29 05:37, a few hours after the zone update process failed. It should have been present in concerned resolver caches about 24 hours later, alongside the previous key (22810), ready to be used to check signatures (RRSIG<\/code> records) of the zone which were supposed to be published from 2020-09-03 05:37.<\/p>\n\n\n\n

However, due the zone update suspension, this happened instead. The skipped steps are shown in gray. <\/p>\n\n\n\n

\"\"
Actual key rotation schedule (before fix)<\/figcaption><\/figure>\n\n\n\n

The zone was directly updated from the 2020-08-14\/2020-08-29 key configuration to 2020-09-04 14:40.<\/p>\n\n\n\n

A few minutes after 2020-09-04 14:40, it was apparent that something was amiss: the resolution of EU.ORG domains failed for people using resolvers with DNSSEC validation.<\/p>\n\n\n\n

The cause was quickly identified: since pre-publication for DNSKEY<\/code> 14716 was missed, most resolvers only had the unexpired DNSKEY<\/code> 22810 in their cache, while the only RRSIG<\/code> records available in the zone servers required key 14716.<\/p>\n\n\n\n

The bad<\/h4>\n\n\n\n

The obvious fix was to reactivate key 22810 and regenerate the zone signatures (RRSIG<\/code> records) with dnssec-signzone<\/code>. This also leaves in place the signatures with key 14716 (keeping the latter was needed for resolvers which had begun to cache key 14176).<\/p>\n\n\n\n

As a side note, it helped that the EU.ORG switched a few months ago to NSEC3<\/code> “opt-out” mode. This saves a lot of space (especially in nameserver memory) for zones with many delegations, which is especially useful if you temporarily need double signatures such as in this episode.<\/p>\n\n\n\n

A first implementation attempt was made at 2020-09-04 14:52 by updating the dates in the public key file (.key<\/code>) for key 22810, pushing the inactivation date to 2020-09-07 05:37:00 and the deletion date to 2020-09-09 05:37:00.<\/p>\n\n\n\n

Before update:<\/p>\n\n\n\n

; Created: 20200808100738 (Sat Aug  8 12:07:38 202)\n; Publish: 20200809053700 (Sun Aug  9 07:37:00 202)\n; Activate: 20200814053700 (Fri Aug 14 07:37:00 202)\n; Inactive: 20200903053700 (Thu Sep  3 07:37:00 202)\n; Delete: 20200905053700 (Sat Sep  5 07:37:00 202)\nEU.ORG. 172800 IN DNSKEY 256 3 8 AwEAAcHAqfeFzQqo9vFq8ZziaQs2...<\/pre>\n\n\n\n
Side remarks:<\/p>\n\n\n\n