[RPKI] APNIC had an unexpected drop in VRP 00:00 - 02:00

Chris Caputo ccaputo at alt.net
Wed Dec 2 03:32:50 UTC 2020


On Wed, 2 Dec 2020, George Michaelson via RPKI wrote:
> On Wed, Dec 2, 2020 at 3:45 AM Job Snijders <job at ntt.net> wrote:
> >
> > On Tue, Dec 01, 2020 at 01:29:58PM +1000, George Michaelson wrote:
> > > We have received reports that our RPKI repository was producing zero
> > > VRP from 00:00 to 02:00  today, Tuesday 01 December. This was visible
> > > in Seattle and may have been seen elsewhere.
> > >
> > > We are looking into what happened and will report back as soon as possible.
> >
> > Some preliminary analysis on my side suggests this event might have been
> > RRDP-specific.
> >
> > On (multiple) rsync-only RPKI collectors I did not observe a drop in
> > VRPs in the 00:00-02:00 UTC time frame. Hope this helps debugging.
> >
> > Kind regards,
> >
> > Job
> 
> We continue to investigate.
> 
> Not all RPs saw this, and it appears that the problem was due to
> recent updates to some relying party software.

With Routinator 0.8.1 I observed:

rsync://rpki.apnic.net/repository/B527EF581D6611E2BB468F7C72FD1FF2/ZwTFeTEC0uxi4JpTfGQbsyoqqhM.cer: CA certificate failed to validate.
CA for rsync://rpki.apnic.net/repository/B527EF581D6611E2BB468F7C72FD1FF2/ rejected, resources marked as unsafe:
   1.0.0.0/8
   14.0.0.0/8
   27.0.0.0/8
   36.0.0.0/8
   39.0.0.0/8
   42.0.0.0/7
   45.64.0.0-45.65.63.255
   45.112.0.0/12
   45.248.0.0/13
[...]

and:

rsync://rpki.apnic.net/repository/B3A24F201D6611E28AC8837C72FD1FF2/0I2GgcK-TUfCopBV9m5olVhGF_c.cer: CA certificate failed to validate.
CA for rsync://rpki.apnic.net/repository/B3A24F201D6611E28AC8837C72FD1FF2/ rejected, resources marked as unsafe:
   8.128.0.0/10
   8.208.0.0/12
   23.106.120.0/21
   23.106.248.0/21
   23.108.96.0/21
   23.111.12.0/22
   23.226.0.0/20
   23.232.128.0/17
   24.41.112.0/20
[...]

Decoded versions of those certificates are below.  Both expired at Dec 1 
00:00:00 2020 GMT and then the problem ensued.  Also at:

  http://console.rpki-client.org/rpki.apnic.net/repository/B527EF581D6611E2BB468F7C72FD1FF2/ZwTFeTEC0uxi4JpTfGQbsyoqqhM.cer.html
  http://console.rpki-client.org/rpki.apnic.net/repository/B3A24F201D6611E28AC8837C72FD1FF2/0I2GgcK-TUfCopBV9m5olVhGF_c.cer.html

Is there an explanation for how these expired certs contributed to a wide 
impact?

RIPE Validator reported "Not valid after time is in the past: 
2020-12-01T00:00:00.000Z" in regards to the APNIC trust anchor, at the 
time.  I'm guessing that is not Routinator based, or is it?

Thanks,
Chris

---------------------------------------------------------------

Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number: 47483 (0xb97b)
        Signature Algorithm: sha256WithRSAEncryption
        Issuer: CN = A90DC5BE, serialNumber = 0E65A4F5FD36B5BD68EB3C923408978C907AA79F
        Validity
            Not Before: Oct 23 10:14:32 2019 GMT
            Not After : Dec  1 00:00:00 2020 GMT
        Subject: CN = A91CFAC8, serialNumber = 6704C5793102D2EC62E09A537C641BB32A2AAA13
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                RSA Public-Key: (2048 bit)
                Modulus:
                    00:b9:42:87:00:97:d1:22:10:d4:0a:04:6d:17:d0:
                    a0:4c:85:a7:c1:8b:2f:94:cd:ab:fa:2d:d2:ce:4c:
                    82:65:98:03:b9:66:6e:a5:f2:35:d6:a3:df:89:42:
                    63:e4:9a:8a:92:64:fd:06:b1:fb:48:5d:72:11:03:
                    98:71:f3:30:1b:87:5d:b1:fa:00:a9:d9:b2:45:f3:
                    05:8e:4b:45:7c:42:cf:9b:cf:38:2c:d1:5e:fa:df:
                    b1:bb:15:30:57:37:9d:f2:b5:21:1f:bf:97:d8:3e:
                    ad:ba:86:62:88:8f:7a:54:b4:10:f4:d2:db:46:76:
                    79:34:93:ee:c4:88:da:2d:68:18:55:b7:f7:06:6c:
                    3f:63:87:7c:9b:76:ff:77:99:2f:39:59:b5:77:c5:
                    cb:07:d1:7e:45:f4:ed:e1:0a:d3:a0:76:90:ee:6d:
                    98:d3:20:d9:d1:67:79:12:25:09:bc:e4:2f:15:06:
                    38:54:79:84:77:a4:83:56:28:14:7f:b6:21:62:c4:
                    92:e7:ad:67:90:1b:da:94:17:b1:2f:20:f1:a4:9f:
                    9d:38:72:6c:4e:f8:9b:b7:b6:48:43:5b:38:16:89:
                    a0:1b:27:6a:02:3a:78:bd:3d:0b:8c:75:15:6e:41:
                    23:7d:b9:a4:c3:ea:08:92:a6:ce:c8:76:07:30:19:
                    41:c1
                Exponent: 65537 (0x10001)
        X509v3 extensions:
            X509v3 Subject Key Identifier: 
                67:04:C5:79:31:02:D2:EC:62:E0:9A:53:7C:64:1B:B3:2A:2A:AA:13
            X509v3 Authority Key Identifier: 
                keyid:0E:65:A4:F5:FD:36:B5:BD:68:EB:3C:92:34:08:97:8C:90:7A:A7:9F

            X509v3 Key Usage: critical
                Certificate Sign, CRL Sign
            X509v3 Basic Constraints: critical
                CA:TRUE
            X509v3 CRL Distribution Points: 

                Full Name:
                  URI:rsync://rpki.apnic.net/repository/B527EF581D6611E2BB468F7C72FD1FF2/DmWk9f02tb1o6zySNAiXjJB6p58.crl

            Authority Information Access: 
                CA Issuers - URI:rsync://rpki.apnic.net/repository/980652E0B77E11E7A96A39521A4F4FB4/DmWk9f02tb1o6zySNAiXjJB6p58.cer

            X509v3 Certificate Policies: critical
                Policy: 1.3.6.1.5.5.7.14.2
                  CPS: https://www.apnic.net/RPKI/CPS.pdf

            Subject Information Access: 
                CA Repository - URI:rsync://rpki.apnic.net/member_repository/A91CFAC8/3E167A58292711E692DAA117C4F9AE02/
                1.3.6.1.5.5.7.48.10 - URI:rsync://rpki.apnic.net/member_repository/A91CFAC8/3E167A58292711E692DAA117C4F9AE02/ZwTFeTEC0uxi4JpTfGQbsyoqqhM.mft
                1.3.6.1.5.5.7.48.13 - URI:https://rrdp.apnic.net/notification.xml

            sbgp-autonomousSysNum: critical
                0...0......
            sbgp-ipAddrBlock: critical
                0.0.....0......p0....0....$...
    Signature Algorithm: sha256WithRSAEncryption
         a7:93:36:5e:6f:51:35:71:09:52:a1:d7:58:5b:09:fd:41:bb:
         39:ee:a9:8f:77:93:94:cf:6e:0c:8d:f5:75:c7:6c:d3:70:95:
         ea:72:af:13:94:f5:d7:41:62:24:26:dd:1e:08:8d:d1:e3:cb:
         fe:e4:be:12:29:4a:ca:7f:f9:8f:98:f1:b4:0c:49:c9:12:8f:
         f7:18:f6:90:61:9e:da:fd:75:35:bf:5b:55:a6:39:24:8d:82:
         d4:cd:72:39:4d:03:c4:8f:e2:8f:bc:dd:48:c4:09:6e:61:6e:
         13:28:7b:58:bf:43:0b:58:b3:b7:fc:4d:93:90:05:15:10:fe:
         e9:7d:3c:17:7f:41:f4:5d:8b:62:27:77:f8:5f:d4:9e:e4:e7:
         8c:e0:96:d0:42:4a:e5:73:6f:dd:3d:47:77:be:0e:69:96:c1:
         ef:74:ef:e4:cb:df:63:81:35:b1:cb:73:c6:8f:ad:b2:c8:cb:
         c1:a0:f6:c4:ed:9f:a5:9f:f8:2b:3a:06:5b:cb:1e:5f:93:38:
         b5:e5:57:e0:05:f0:ee:e3:14:d2:7d:73:c4:29:f4:5d:87:71:
         1a:87:8a:e4:57:18:f3:79:02:50:0f:be:66:e5:f9:5c:c8:42:
         e4:6d:3c:37:33:47:a6:26:f8:68:37:a0:fa:3b:0d:dc:63:b3:
         f7:56:6b:0d

---------------------------------------------------------------

Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number: 11713 (0x2dc1)
        Signature Algorithm: sha256WithRSAEncryption
        Issuer: CN = A90DC5BE, serialNumber = 740165A80D1071970ABC09C02B71C1AC7C1D6E0E
        Validity
            Not Before: Sep  9 04:12:15 2019 GMT
            Not After : Dec  1 00:00:00 2020 GMT
        Subject: CN = A91FCEB1, serialNumber = D08D8681C2BE4D47C2A29055F66E6895584617F7
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                RSA Public-Key: (2048 bit)
                Modulus:
                    00:a8:b4:0e:5f:0e:c6:db:84:62:b1:5b:a3:23:36:
                    a4:7a:c2:91:ae:e1:35:8d:39:77:0c:83:46:c2:7b:
                    f4:aa:b3:7d:d2:60:d6:de:88:06:19:76:79:98:20:
                    ce:52:56:52:ee:88:09:9b:0c:54:33:79:84:09:ce:
                    4a:62:14:08:f5:ca:8d:ee:b6:64:4a:1d:fa:76:48:
                    22:29:36:7c:1e:5d:79:86:e7:df:da:70:cc:fd:72:
                    75:76:43:ad:c0:17:69:fa:b3:db:32:77:81:70:8b:
                    1f:b7:a2:0a:8b:61:96:f1:1e:88:e7:4a:8a:44:e6:
                    20:1a:25:63:ac:5d:7b:b8:4a:8e:bc:3c:ff:66:49:
                    20:8b:49:bd:5e:4e:6d:dc:f5:79:55:e4:f3:79:ee:
                    b6:c6:c3:e7:79:18:c8:46:da:ae:b5:c1:ad:71:8c:
                    57:c7:4a:e9:70:88:8d:f1:ab:92:cb:75:f0:51:d1:
                    df:f3:81:2e:84:f6:7a:0e:93:46:c5:84:99:29:cc:
                    0b:86:b6:4f:00:0a:24:4a:7d:b3:45:bd:2d:72:f6:
                    57:31:22:2c:a7:8e:30:9e:2c:0c:6f:cc:da:ca:c4:
                    72:48:42:55:92:54:f2:eb:e0:9b:33:33:c1:b0:82:
                    8a:76:3d:54:06:a2:24:99:72:61:99:de:2e:e5:7f:
                    4f:4f
                Exponent: 65537 (0x10001)
        X509v3 extensions:
            X509v3 Subject Key Identifier: 
                D0:8D:86:81:C2:BE:4D:47:C2:A2:90:55:F6:6E:68:95:58:46:17:F7
            X509v3 Authority Key Identifier: 
                keyid:74:01:65:A8:0D:10:71:97:0A:BC:09:C0:2B:71:C1:AC:7C:1D:6E:0E

            X509v3 Key Usage: critical
                Certificate Sign, CRL Sign
            X509v3 Basic Constraints: critical
                CA:TRUE
            X509v3 CRL Distribution Points: 

                Full Name:
                  URI:rsync://rpki.apnic.net/repository/B3A24F201D6611E28AC8837C72FD1FF2/dAFlqA0QcZcKvAnAK3HBrHwdbg4.crl

            Authority Information Access: 
                CA Issuers - URI:rsync://rpki.apnic.net/repository/980652E0B77E11E7A96A39521A4F4FB4/dAFlqA0QcZcKvAnAK3HBrHwdbg4.cer

            X509v3 Certificate Policies: critical
                Policy: 1.3.6.1.5.5.7.14.2
                  CPS: https://www.apnic.net/RPKI/CPS.pdf

            Subject Information Access: 
                CA Repository - URI:rsync://rpki.apnic.net/member_repository/A91FCEB1/9C14B4063A2111E79AA0D51CC4F9AE02/
                1.3.6.1.5.5.7.48.10 - URI:rsync://rpki.apnic.net/member_repository/A91FCEB1/9C14B4063A2111E79AA0D51CC4F9AE02/0I2GgcK-TUfCopBV9m5olVhGF_c.mft
                1.3.6.1.5.5.7.48.13 - URI:https://rrdp.apnic.net/notification.xml

            sbgp-ipAddrBlock: critical
                0.0.....0......*
    Signature Algorithm: sha256WithRSAEncryption
         b0:af:f5:85:0f:4c:0d:75:08:b1:1a:56:62:3b:74:85:a0:ea:
         06:6f:d6:de:1d:f5:04:b0:59:bd:80:e0:9d:ae:78:2b:23:c3:
         78:6d:30:f2:f4:af:96:f8:dc:eb:3e:4a:b4:a1:4f:46:33:ca:
         25:06:ba:31:3c:82:bc:09:59:15:a1:47:51:98:a5:57:17:82:
         1b:de:16:b4:58:41:d5:32:80:e2:55:78:21:66:5f:8f:b6:fb:
         0c:96:3d:d9:3b:58:bf:57:ad:cc:c1:af:e7:3c:71:9d:81:e1:
         98:23:24:77:a9:c6:59:1e:8f:1e:fe:a9:d2:0c:84:64:6d:a4:
         87:b0:65:bf:03:b2:18:be:9a:d1:48:2b:b1:1f:97:98:a4:ec:
         06:0a:ab:c2:e3:3f:5a:84:bd:01:00:29:b9:e5:fe:3f:cc:e1:
         be:5c:dc:76:a9:0f:21:13:45:7a:e2:06:5f:eb:98:c9:55:16:
         6c:31:19:64:78:2d:b6:df:c7:e3:3d:30:9b:ec:8c:8f:fe:39:
         04:29:48:c2:d8:b2:07:e6:41:e6:f8:15:f1:d8:8d:46:7c:95:
         d9:b8:51:53:67:0a:f9:88:8e:87:56:66:a9:df:fd:95:2f:01:
         17:c5:84:e2:ee:af:5c:36:c8:b2:4f:89:48:ec:50:7a:ae:17:
         a3:d6:dd:a1


More information about the RPKI mailing list