[RPKI] transcient differences between rpki-client and routinator

[Job Snijders]

Dear group,

(kristaps, claudio in CC)

I am running rpki-client and routinator in tandem every 15 minutes: the
output of both tools (based on the same input fetched with rsync) is
compared, and if the same the pipeline proceeds to publication a
'export.json' file - if not, an error is produced and a human (me)
alerted.

The rsync data was fetched around Thu 05 Dec 2019 13:30:12 UTC
rpki-client (first) and routinator (second) were done at Thu 05 Dec 2019 13:34:54 UTC

The flow is as following:

- rpki-client fetches all rsync data
- rpki-client runs its validation, spits it out as json, this is
  converted to cvs for easier comparing
- routinator runs with '-n' and .rpki-cache/repository/rsync is
  symlinked to the rpki-client directory (/var/cache/rpki-client/),
  also spits out json which is converted to csv

a snapshot of the data of that run is available here http://instituut.net/~job/rpki-repository.6E17IG9pm.tar.gz

the script that runs tools one after the other and compares the output
is available here: https://gist.github.com/job/ea11fc59b2411e042eaad1c1b0213c74

Now, what is very curious to me is that based on the same data input,
rpki-client and routinator don't /always/ produce the same output. I'd
say that it seems that 80% of the time they have the same output, and
20% of the time there are minute differences such as below:

    hanna:~ job$ diff rpki-repository.6E17IG9pm/export-routinator.cvs rpki-repository.6E17IG9pm/export-rpki-client.cvs
    34792d34791
    < AS207036,200.1.154.0/24,24,lacnic

Does any one have an idea what can explain these differences? Is there
perhaps some timestamp difference in an intermediate certificate where
routinator decides that the ROA for 200.1.154.0/24 is not valid, or is
there some check that rpki-client is maybe skipping over? What made
'200.1.154.0/24,24,AS207036' valid in the eyes of rpki-client, but not
in the eyes of routinator?

Kind regards,

Job