[RPKI] [RPKI-Deployers] APNIC had an unexpected drop in VRP 00:00 - 02:00

[Tony Tauber]

Hi George,

Could you possibly say a little more?

For our part, we run RIPE Validator.version=3.1-2020.07.06.14.28 (not the
most recent version, I believe) which didn't seem to be affected and also
Routinator 0.8.0 which did see it.
And, FWIW, the number of ROAs validated underneath the RIPE TA we saw drop
from over 62000 to 241 (where it stayed constant for the duration of the
incident.

The Routinator logs are quite verbose but I was able to see that during the
time of the problem the number of log messages per 12-minute refresh
interval grew by about 10x beyond what it had been (around 50 or 110 to
about 500 or 1000).

I could do more looking but if you had an idea of what you were looking for
it may save time or perhaps you've already found the problem.

Let me know if you want more info on what's in there.  Perhaps you're
already running a copy of each RP package and can check your own logs.

Tony



On Tue, Dec 1, 2020 at 9:55 PM George Michaelson <ggm at algebras.org> wrote:

> On Wed, Dec 2, 2020 at 3:45 AM Job Snijders <job at ntt.net> wrote:
> >
> > On Tue, Dec 01, 2020 at 01:29:58PM +1000, George Michaelson wrote:
> > > We have received reports that our RPKI repository was producing zero
> > > VRP from 00:00 to 02:00  today, Tuesday 01 December. This was visible
> > > in Seattle and may have been seen elsewhere.
> > >
> > > We are looking into what happened and will report back as soon as
> possible.
> >
> > Some preliminary analysis on my side suggests this event might have been
> > RRDP-specific.
> >
> > On (multiple) rsync-only RPKI collectors I did not observe a drop in
> > VRPs in the 00:00-02:00 UTC time frame. Hope this helps debugging.
> >
> > Kind regards,
> >
> > Job
>
> We continue to investigate.
>
> Not all RPs saw this, and it appears that the problem was due to
> recent updates to some relying party software.
>
> cheers
>
> -George
> --
> RPKI-Deployers mailing list
> RPKI-Deployers at puck.nether.net
> https://puck.nether.net/mailman/listinfo/rpki-deployers
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.nlnetlabs.nl/pipermail/rpki/attachments/20201201/6fa24301/attachment-0001.htm>