<div dir="ltr"><div>Thank you, Job for your excellent and detailed and constructive analysis!</div><div><br></div><div>Now get some rest. :-) <br></div><div><br></div><div>Definitely explains why we saw what we saw (Routinator affected, RIPE Validator not).</div><div>At this point we are moving to pivot to FORT and perhaps rpki-client as well given the recent experiences.</div><div><br></div><div>Thanks again!</div><div><br></div><div><br></div><div>Tony<br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, Dec 2, 2020 at 4:33 PM Job Snijders <<a href="mailto:job@ntt.net">job@ntt.net</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Hi all,<br>
<br>
First of all to be very clear: there was no 'APNIC outage', APNIC did<br>
nothing wrong. This was a 'validator outage', and locally outages like<br>
these can continue to be experienced at any future moment until fixed<br>
versions are released and deployed. Note: network operators who run FORT<br>
or OpenBSD rpki-client side-by-side with routinator/octorpki will have<br>
seen a stable VRP merged set item count on their EBGP routers. In this<br>
situation RPKI validator software diversity helped the Internet remain<br>
more stable.<br>
<br>
APNIC staff are commendable for having seen an opportunity to implement<br>
a workaround for this routinator 0.8.1 quirk, but APNIC is just one of<br>
the tens of thousands of Certificate Authorities in the RPKI ecosystem.<br>
In short: the observed state of December 1st, 2020 00:00 UTC is an<br>
expected and normal state in the RPKI ecosystem.<br>
<br>
I appreciate George for reaching out to the community to draw more<br>
attention to the situation, as it seems we can learn from exploring this<br>
situation in great detail. For many in the community RPKI is a new<br>
technology. Also it appears a similar issue exists in Cloudflare's<br>
OctoRPKI, so I notified their developers too about the problem &<br>
solution. Since there are implementations with a bug in the same<br>
equivalence class, this case is best handed over to the IETF.<br>
<br>
While keeping in mind our human perception of the concept of time<br>
generally is somewhat incompatible with how time works in the X.509 /<br>
RPKI crypto world... here are my lengthy debug notes. :-)<br>
<br>
TL;DR: the VRP drop is an implementation issue in some RPKI validators, can happen again<br>
solution: wait for fixed version, or run multiple different RPKI validator implementations side by side<br>
there a bit of time pressure: this bug potentially interacts negatively with Juniper PR1483097.<br>
<br>
Every 20 minutes I copy all RPKI data from the Internet, run rpki-client<br>
[1], and store the original RPKI data files, the program's execution<br>
log, and the resulting VRP list as individual ZFS snapshots for<br>
post-mortem analysis. A copy of my data can be downloaded: it is an<br>
exact snapshot of all input data from that moment, to replay the event<br>
in various implementations.<br>
<a href="http://sobornost.net/~job/rpki-20201201-0001-adrian.sobornost.net.tar.gz" rel="noreferrer" target="_blank">http://sobornost.net/~job/rpki-20201201-0001-adrian.sobornost.net.tar.gz</a><br>
<br>
Looking at the process' log of December 1st, 2020 run starting at<br>
midnight for the string 'apnic':<br>
<br>
root@adrian:/tank/rpkirepositories/.zfs/snapshot/20201201-0001# fgrep apnic output/log<br>
Dec 01 00:00:01 rpki-client: <a href="https://tal.apnic.net/apnic.cer" rel="noreferrer" target="_blank">https://tal.apnic.net/apnic.cer</a>: https schema ignored<br>
Dec 01 00:00:01 <a href="http://rpki.apnic.net/repository" rel="noreferrer" target="_blank">rpki.apnic.net/repository</a>: pulling from network<br>
Dec 01 00:00:03 rpki-client: <a href="http://rpki.apnic.net/repository" rel="noreferrer" target="_blank">rpki.apnic.net/repository</a>: loaded from cache<br>
Dec 01 00:00:03 rpki-client: <a href="http://rpki.apnic.net/member_repository" rel="noreferrer" target="_blank">rpki.apnic.net/member_repository</a>: pulling from network<br>
Dec 01 00:00:03 rpki-client: <a href="http://rpki.sub.apnic.net/repository" rel="noreferrer" target="_blank">rpki.sub.apnic.net/repository</a>: pulling from network<br>
Dec 01 00:00:03 rpki-client: <a href="http://rpki.apnic.net/repository/B527EF581D6611E2BB468F7C72FD1FF2/ZwTFeTEC0uxi4JpTfGQbsyoqqhM.cer" rel="noreferrer" target="_blank">rpki.apnic.net/repository/B527EF581D6611E2BB468F7C72FD1FF2/ZwTFeTEC0uxi4JpTfGQbsyoqqhM.cer</a>: certificate has expired<br>
Dec 01 00:00:03 rpki-client: <a href="http://rpki.apnic.net/repository/B527EF581D6611E2BB468F7C72FD1FF2/9lv88f3YSSS6iXQmzBvPX6hvnQM.cer" rel="noreferrer" target="_blank">rpki.apnic.net/repository/B527EF581D6611E2BB468F7C72FD1FF2/9lv88f3YSSS6iXQmzBvPX6hvnQM.cer</a>: certificate has expired<br>
Dec 01 00:00:03 rpki-client: <a href="http://rpki.rand.apnic.net/repo" rel="noreferrer" target="_blank">rpki.rand.apnic.net/repo</a>: pulling from network<br>
Dec 01 00:00:04 rpki-client: <a href="http://rpki.apnic.net/repository/B527EF581D6611E2BB468F7C72FD1FF2/pBp2e-TKxusbiXQjNgwrQ1OsH_s.cer" rel="noreferrer" target="_blank">rpki.apnic.net/repository/B527EF581D6611E2BB468F7C72FD1FF2/pBp2e-TKxusbiXQjNgwrQ1OsH_s.cer</a>: certificate has expired<br>
Dec 01 00:00:04 rpki-client: <a href="http://rpki.apnic.net/repository/B527EF581D6611E2BB468F7C72FD1FF2/ZnMLuaQLNc_lmxGF9iLb0JAMbZA.cer" rel="noreferrer" target="_blank">rpki.apnic.net/repository/B527EF581D6611E2BB468F7C72FD1FF2/ZnMLuaQLNc_lmxGF9iLb0JAMbZA.cer</a>: certificate has expired<br>
Dec 01 00:00:04 rpki-client: <a href="http://rpki.apnic.net/repository/B527EF581D6611E2BB468F7C72FD1FF2/yZYCtJIcaINWT0smUVwdY-TPNkQ.cer" rel="noreferrer" target="_blank">rpki.apnic.net/repository/B527EF581D6611E2BB468F7C72FD1FF2/yZYCtJIcaINWT0smUVwdY-TPNkQ.cer</a>: certificate has expired<br>
Dec 01 00:00:04 rpki-client: <a href="http://rpki.apnic.net/repository/B527EF581D6611E2BB468F7C72FD1FF2/WFBPIARWFTaBikTQvkFutQVej0g.cer" rel="noreferrer" target="_blank">rpki.apnic.net/repository/B527EF581D6611E2BB468F7C72FD1FF2/WFBPIARWFTaBikTQvkFutQVej0g.cer</a>: certificate has expired<br>
Dec 01 00:00:05 rpki-client: <a href="http://rpki.apnic.net/repository/B527EF581D6611E2BB468F7C72FD1FF2/QmfPXQMASo_v3yE5XQ_oJFSLE8E.cer" rel="noreferrer" target="_blank">rpki.apnic.net/repository/B527EF581D6611E2BB468F7C72FD1FF2/QmfPXQMASo_v3yE5XQ_oJFSLE8E.cer</a>: certificate has expired<br>
Dec 01 00:00:05 rpki-client: <a href="http://rpki.sub.apnic.net/repository" rel="noreferrer" target="_blank">rpki.sub.apnic.net/repository</a>: loaded from cache<br>
Dec 01 00:00:05 rpki-client: <a href="http://rpki.apnic.net/repository/B527EF581D6611E2BB468F7C72FD1FF2/maB2Nu64AHCDMDGWpYxBvsxoj4A.cer" rel="noreferrer" target="_blank">rpki.apnic.net/repository/B527EF581D6611E2BB468F7C72FD1FF2/maB2Nu64AHCDMDGWpYxBvsxoj4A.cer</a>: certificate has expired<br>
Dec 01 00:00:05 rpki-client: <a href="http://rpki.apnic.net/repository/B527EF581D6611E2BB468F7C72FD1FF2/d0JlIBzwsNjMdvAm-Ir2i1XpkO4.cer" rel="noreferrer" target="_blank">rpki.apnic.net/repository/B527EF581D6611E2BB468F7C72FD1FF2/d0JlIBzwsNjMdvAm-Ir2i1XpkO4.cer</a>: certificate has expired<br>
Dec 01 00:00:05 rpki-client: <a href="http://rpki.apnic.net/repository/B3A24F201D6611E28AC8837C72FD1FF2/0I2GgcK-TUfCopBV9m5olVhGF_c.cer" rel="noreferrer" target="_blank">rpki.apnic.net/repository/B3A24F201D6611E28AC8837C72FD1FF2/0I2GgcK-TUfCopBV9m5olVhGF_c.cer</a>: certificate has expired<br>
Dec 01 00:00:06 rpki-client: <a href="http://rpki.rand.apnic.net/repo" rel="noreferrer" target="_blank">rpki.rand.apnic.net/repo</a>: loaded from cache<br>
Dec 01 00:00:12 rpki-client: <a href="http://rpki.apnic.net/member_repository" rel="noreferrer" target="_blank">rpki.apnic.net/member_repository</a>: loaded from cache<br>
<br>
(At the end of the process's run it had observed 62,154 VRPs under the<br>
APNIC TAL. A CSV & JSON file of the validation process output with all<br>
VRPs from that moment is also included in the tar.gz file.)<br>
<br>
In the above log we see that a number of certificates are expired,<br>
according to Tom's message [2] these certificates represents APNIC<br>
members whose membership has been closed. (for example: companies going<br>
out of business, or merger & acquisition) It is expected for<br>
organizations issuing cryptographic products to tie business events to<br>
validity periods in certificates.<br>
<br>
For the purpose of these notes I'll focus only on following the<br>
validation process towards 'ZwTFeTEC0uxi4JpTfGQbsyoqqhM.cer' in a manual<br>
fashion using command line utilities.<br>
<br>
After having pulled RPKI from the web (which operationally speaking<br>
end-to-end is a multi-hour process to get the data from signer to<br>
validator), a number of process steps have to be performed in order to<br>
produce a list of Validated ROA Payloads (VRPs). None of these steps can<br>
be skipped, and the order is important too.<br>
<br>
A single manifest file (<a href="https://tools.ietf.org/html/rfc6486" rel="noreferrer" target="_blank">https://tools.ietf.org/html/rfc6486</a>) actually is<br>
a bundle of a few things: a start & end date of the file listing, a list<br>
of filenames and sha256 hashes, and a EE certificate (which also has its<br>
own embedded start & end date!), a serial number, and references to<br>
other things such as which entity signed it.<br>
<br>
The first step is to figure out whether a given manifest file is 'valid'<br>
(are the signatures right) and 'current' (the timestamp on the<br>
validator's wall clock is between both the manifest's embedded start &<br>
end date AND the EE certificate validity dates), and the 'latest'<br>
(should the validator have to choose between two versions of the file,<br>
both valid and current, pick the one with the highest number).<br>
<br>
So at December 1st 00:00:03 UTC, the manifest's start & end date, and<br>
the EE certificate's start and end date were:<br>
<br>
$ tar fxz rpki-20201201-0001-adrian.sobornost.net.tar.gz<br>
$ cd 20201201-0001/data/<a href="http://rpki.apnic.net/repository/B527EF581D6611E2BB468F7C72FD1FF2" rel="noreferrer" target="_blank">rpki.apnic.net/repository/B527EF581D6611E2BB468F7C72FD1FF2</a><br>
<br>
$ ls -lahtr DmWk9f02tb1o6zySNAiXjJB6p58.mft<br>
-rw-r--r-- 1 job wheel 214K Nov 30 23:01 DmWk9f02tb1o6zySNAiXjJB6p58.mft<br>
<br>
This file's ctime appears to be November 30th, 23:01<br>
<br>
# check manifest's econtent start & end date<br>
$ strings DmWk9f02tb1o6zySNAiXjJB6p58.mft | head -2<br>
20201130230107Z<br>
20201202230107Z<br>
<br>
December 1st 00:00:03 is between November 30th 23:01:07 and December 2nd<br>
23:01:07: check!<br>
<br>
# check the manifest's embedded EE certificate start & end date:<br>
$ test-mft -vp DmWk9f02tb1o6zySNAiXjJB6p58.mft | openssl x509 -text | grep -A2 Validity<br>
Validity<br>
Not Before: Nov 30 23:01:07 2020 GMT<br>
Not After : Dec 2 23:01:07 2020 GMT<br>
<br>
December 1st 00:00:03 is between November 30th 23:01:07 and December 2nd<br>
23:01:07: check!<br>
<br>
With the dates and signatures of the manifest file check out to be 'all<br>
lights green', the next step is to process the manifest's file listing.<br>
A manifest 'file listing' is checked through two steps:<br>
<br>
- is the listed file present?<br>
- is the sha256 hash (in base64 format) listed on the manifest the<br>
same as the sha256 hash computed by the validator using a copy of<br>
the listed file?<br>
<br>
# looking at manifest file listing:<br>
$ test-mft -v DmWk9f02tb1o6zySNAiXjJB6p58.mft | grep -A1 ZwTFeTEC0uxi4JpTfGQbsyoqqhM.cer<br>
95: ZwTFeTEC0uxi4JpTfGQbsyoqqhM.cer<br>
hash YjVYYAHzd5UFgeKVJGa+2zLy6uQHH+j4EmiH43ypgZc=<br>
<br>
# checking whether file is present:<br>
$ ls -alhtr ZwTFeTEC0uxi4JpTfGQbsyoqqhM.cer<br>
-rw-r--r-- 1 job wheel 1.5K Nov 30 23:01 ZwTFeTEC0uxi4JpTfGQbsyoqqhM.cer<br>
<br>
# compute sha256 hash of the file<br>
$ sha256 -b ZwTFeTEC0uxi4JpTfGQbsyoqqhM.cer<br>
SHA256 (ZwTFeTEC0uxi4JpTfGQbsyoqqhM.cer) = YjVYYAHzd5UFgeKVJGa+2zLy6uQHH+j4EmiH43ypgZc=<br>
<br>
Indeed, the 'YjVYYAHzd5UFgeKVJGa+2zLy6uQHH+j4EmiH43ypgZc=' hash computed<br>
from the referenced certificate file is the same one as listed in the<br>
manifest file (which we inspected with test-mft)! Note that at this<br>
stage of the validation process the 'ZwTFeTEC0uxi4JpTfGQbsyoqqhM.cer'<br>
file has not been processed in any other way other than the equivalent<br>
of that 'sha256' OpenBSD utility.<br>
<br>
These 'jumps' from certificate to manifest to certificate using hashes &<br>
signatures serve multiple purposes: by first confirming a hash matches,<br>
the validator does not (yet) need to attempt any file content parsing<br>
(which would potentially be sensitive computing operations on an at that<br>
point in time a unknown and potentially dangerous file), and secondly:<br>
by checking the presence and hash of each file, the publication point's<br>
completeness and integrity is confirmed. Missing .roa files can result<br>
in network outages [3].<br>
<br>
At this point the manifest file has been completely processed, the next<br>
step in the validation process can commence. Each and every referenced<br>
file is opened by the validator, embedded certificates and sigantures<br>
are verified, and then again file contents processed (could be<br>
manifests, certificates, CRLs, or ROA files).<br>
<br>
Let's inspect ZwTFeTEC0uxi4JpTfGQbsyoqqhM.cer:<br>
<br>
$ openssl x509 -in ZwTFeTEC0uxi4JpTfGQbsyoqqhM.cer -inform DER -text | grep -A2 Validity<br>
Validity<br>
Not Before: Oct 23 10:14:32 2019 GMT<br>
Not After : Dec 1 00:00:00 2020 GMT<br>
<br>
As the validator's wall clock was December 1st 00:00:03, we can see that<br>
ZwTFeTEC0uxi4JpTfGQbsyoqqhM.cer expired '3 seconds ago'. Note that<br>
before we observed that creation time on the manifest file which<br>
referenced this .cer file was November 30th, at that time this<br>
ZwTFeTEC0uxi4JpTfGQbsyoqqhM.cer certificate was valid, present, and<br>
current!<br>
<br>
One could say that ZwTFeTEC0uxi4JpTfGQbsyoqqhM.cer is a child of<br>
DmWk9f02tb1o6zySNAiXjJB6p58.mft. The ZwTFeTEC0uxi4JpTfGQbsyoqqhM.cer<br>
file might not even be under control of the entity which generated<br>
DmWk9f02tb1o6zySNAiXjJB6p58.mft. A child's expiry does not result in the<br>
death of the parent. If a validator considers all referenced files on a<br>
manifest to be invalid, solely because *upon further inspection* a file<br>
contained contained an expired EE certificate, I'd say it is an<br>
'overreaction', a simple software defect. After all, there was a valid<br>
current manifest which listed a hash and that hash matched the file, so<br>
the file became eligible for X.509 certificate validation in the first place!<br>
<br>
It appears that Routinator conflates two distinct steps in the<br>
validation process:<br>
<br>
step 1) checking the validity of a RPKI manifest<br>
step 2) checking the validity of a file referenced from the in step 1 validated manifest<br>
<br>
A valid manifest referencing a (now expired) certificate is a legitimate<br>
state of being. What is not valid is for the manifest listing itself to<br>
be expired, or the manifest's EE certificate to be expired, or its CRL<br>
to be expired, or its parent certificate to be expired, or for any files<br>
listed on the manifest to be missing, or for any sha256 hashes to be<br>
different than listed on the manifest. Phew.... that's a mouthful of<br>
conditions! We're gonna have to work in IETF to capture this in simpler<br>
english.<br>
<br>
Conclusion<br>
==========<br>
<br>
I'm not saying validators should accept expired data, they shouldn't!<br>
But it is *expected* that Certificate Authorities (like LIRs, NIRs, or<br>
even RIRs) set the expiration dates on cryptographic objects to be<br>
aligned with the reality of business contracts. This is a *critical*<br>
feature of the RPKI and makes RPKI superior to IRR data: finally there<br>
/are/ expiration dates on the equivalent of 'route:' objects.<br>
<br>
A repeat of the 'december 1st' VRP drop situation can come into<br>
existence at any future moment under any Trust Anchor, under any<br>
Certificate Authority. Simply put: network solely relying on current<br>
versions of octorpki or routinator are somewhat at risk when billing<br>
cycles end. Also, I do not recommend downgrading to older versions<br>
because of <a href="https://www.nlnetlabs.nl/projects/rpki/security-advisories/" rel="noreferrer" target="_blank">https://www.nlnetlabs.nl/projects/rpki/security-advisories/</a><br>
(which perversely is a bug that *is not* resolved with rpki software<br>
diversity).<br>
<br>
I suspect it is OK for network operators to choose to sit this one out<br>
and just wait for a fixed version, provided it can be released in a<br>
manner of weeks. Because of Juniper PR1483097 (which probably still<br>
affects many currently deployed internet routers) the complete<br>
disappearance of VRPs can negatively impact internet traffic forwarding<br>
in the default-free zone, but as mentioned before impact is avoided both<br>
through multi-instance validator deployment combined with validator<br>
software diversity.<br>
<br>
There is a silver lining in all this: the most likely next occurance<br>
of this type of situation is January 1st, 2021, as then all kinds of<br>
LIR, NIR, or RIR business contracts are likely to start or stop. This<br>
gives nlnetlabs and cloudflare almost a full month to figure out a fix,<br>
release it, and for operators to deploy it in their networks during the<br>
holidays. The perfect excuse to escape any unwanted christmas dinner. ;-)<br>
<br>
I propose some of us continue discussion at <a href="mailto:sidrops@ietf.org" target="_blank">sidrops@ietf.org</a> where<br>
through wordsmithing in the draft-ietf-sidrops-6486bis effort so we help<br>
any future RPKI implementers from walking into the same problem.<br>
<br>
Kind regards,<br>
<br>
Job<br>
<br>
[1]: <a href="https://pkgs.org/search/?q=rpki-client" rel="noreferrer" target="_blank">https://pkgs.org/search/?q=rpki-client</a><br>
[2]: <a href="https://lists.nlnetlabs.nl/pipermail/rpki/2020-December/000238.html" rel="noreferrer" target="_blank">https://lists.nlnetlabs.nl/pipermail/rpki/2020-December/000238.html</a><br>
[3]: <a href="https://blog.apnic.net/2020/11/10/rpki-manifests-securely-declare-contents/" rel="noreferrer" target="_blank">https://blog.apnic.net/2020/11/10/rpki-manifests-securely-declare-contents/</a><br>
</blockquote></div>