[RPKI] routinator server not updating?

[Martin Hoffmann]

Heisann!

Havard Eidnes via RPKI wrote:
> 
> The documentation for routinator states:
> 
>        The server will periodically update the local repository,
> every ten minutes by default, notify any clients of changes, and let
> them fetch validated data.  It will not, however, reread the trust
> anchor locators. Thus, if you update them, you will have to restart
>        Routinator.
> 
> First off, doing rsync with all the remote repositories every 10
> minutes sounds quite expensive in terms on the load on those
> servers(?), if that was indeed what would happen.

That certainly is becoming more relevant as the number of relying
parties grows. However, most servers have by now moved to RRDP and the
remaining ones probably should, too, for that very reason.

It has been suggested to use different refresh times for rsync and RRDP
(once we have Etag support, we could go even lower for RRDP), but that
would require rethinking the validation logic, so it hasn’t happened
yet.

> Secondly, the config file documentation says:
> 
>        refresh
>               An integer value specifying the number of seconds
> Routinator should wait between consecutive validation runs in server
> mode. The next validation run will happen earlier, if objects expire
>               earlier. The default is 600 seconds.
> 
> Does it only do re-validation and no actual refresh of the data
> from the upstream repositories in this period?  The corresponding
> program option has slightly different wording:
> 
>        --refresh=seconds
>               The amount of seconds the server should wait after
> having finished updating and validating the local repository before
>               starting to update again. The next update will earlier
> if objects in the repository expire earlier. The default value is
>               600 seconds.
> 
> I am guessing the difference in wording is not intentional, and
> that it is the intention that the local copy of the remote
> repositories are kept in sync by periodically refreshing them,
> and that no additional manual intervention should be required to
> keep the local copy up to date?

That is correct. A "validation run" intertwines both a repository
update and validation (we only update repository publication points
that are actually in use). The only exception is when the no-update
switch is used.

> However, I have configured the built-in http server to make it
> possible to do some monitoring, and it now says (among other
> things):
> 
> version: routinator/0.7.1
> serial: 10
> last-update-start-at:  2020-06-16 13:00:06.611424671 UTC
> last-update-start-ago: P22DT77524.959437S
> last-update-done-at:   2020-06-16 12:50:06.337468507 UTC
> last-update-done-ago:  P22DT78125.233393164S
> last-update-duration:  PT129.500793946S
> valid-roas: 38027
> 
> So ... last updated ... 22 days ago?!?  That timing mark
> coincides with when the routinator server was last re-started.

Yikes. The last-update-start-at is after the last-update-done-at, so it
appears Routinator got stuck somewhere during the update. Can you have
a quick look in the log whether there is anything suspicious? Probably
not as you are using the default log level, but maybe we are lucky.

Additionally, can you check whether Routinator has any rsync child
processes left?

Finally, what might help to determine where it got stuck might be a
list of the last modification times of all the files
under /var/db/rpki-cache/repository.

We have seen RRDP updates for certain servers to take an awfully long
time, even though there is a timeout set. I always attributed this to
bytes come trickling in slowly and the connection not be dead but maybe
I am wrong. I’ll definitely double check these timeouts.

> ...and while nitpicking (these might be relevant to the above
> main question, "why isn't routinator updating?"):
> 
>        disable-rsync
>               A boolean value that, if present and true, turns off
> the use of rsync.
> 
> Default value?

Not to go all philosophical, but can an optional value have a default
value?
 
>        rsync-command
>               A string specifying the command to use for running
> rsync. The default is simply rsync.
> 
> Searched for in $PATH?

Yes. Apparently there is some weirdness on Windows, but I guess that
doesn’t matter too much.

Kind regards,
Martin