[RPKI] APNIC had an unexpected drop in VRP 00:00 - 02:00

[Lukas Tribus]

On Wed, 2 Dec 2020 at 03:54, George Michaelson via RPKI
<rpki at lists.nlnetlabs.nl> wrote:
>
> On Wed, Dec 2, 2020 at 3:45 AM Job Snijders <job at ntt.net> wrote:
> >
> > On Tue, Dec 01, 2020 at 01:29:58PM +1000, George Michaelson wrote:
> > > We have received reports that our RPKI repository was producing zero
> > > VRP from 00:00 to 02:00  today, Tuesday 01 December. This was visible
> > > in Seattle and may have been seen elsewhere.
> > >
> > > We are looking into what happened and will report back as soon as possible.
> >
> > Some preliminary analysis on my side suggests this event might have been
> > RRDP-specific.
> >
> > On (multiple) rsync-only RPKI collectors I did not observe a drop in
> > VRPs in the 00:00-02:00 UTC time frame. Hope this helps debugging.
> >
> > Kind regards,
> >
> > Job
>
> We continue to investigate.
>
> Not all RPs saw this, and it appears that the problem was due to
> recent updates to some relying party software.

If it helps, this is from fort 1.4.1 validation logs:

($ journalctl -u fort --facility local0 --utc --since="2020-12-01
00:00:00 UTC" --until "2020-12-01 02:30:00 UTC" -o short-iso
--no-hostname)

https://pastebin.com/raw/PaV1m4mG


With rsync, we have a lot of:
"Certificate validation failed: certificate has expired"

With RRDP we have some:
"does not match its expected hash."


Lukas