[RPKI] Routinator memory usage

[Darren O'Connor]

Heya.

Hope, both are VMs yes, but they are provisioned identically. Looking a bit
deeper each server is using a slightly different 'version' of the CPU:

cat /proc/cpuinfo | grep mode
model           : 45
model name      : Intel(R) Xeon(R) CPU E5-2667 v3 @ 3.20GHz
model           : 45
model name      : Intel(R) Xeon(R) CPU E5-2667 v3 @ 3.20GHz

 cat /proc/cpuinfo  | grep mode
model           : 63
model name      : Intel(R) Xeon(R) CPU E5-2667 v3 @ 3.20GHz
model           : 63
model name      : Intel(R) Xeon(R) CPU E5-2667 v3 @ 3.20GHz

One thing I notice is that on the server using more memory right now, it
has more logs about rsync timeouts:

● routinator.service - Routinator 3000
   Loaded: loaded (/etc/systemd/system/routinator.service; enabled; vendor
preset: enabled)
   Active: active (running) since Sat 2019-10-19 21:59:07 EDT; 1 day 11h ago
     Docs: man:routinator(1)
 Main PID: 27905 (routinator)
    Tasks: 8 (limit: 4626)
   Memory: 747.3M
   CGroup: /system.slice/routinator.service
           └─27905 /root/.cargo/bin/routinator server --rtr 127.0.0.1:3323

Oct 21 09:10:35 bgp2 routinator[27905]: rsync://ca.rg.net/rpki: failed with
status exit code: 35
Oct 21 09:10:35 bgp2 routinator[27905]: rsync://ca.rg.net/rpki: rsync
error: timeout waiting for daemon connectio
Oct 21 09:20:53 bgp2 routinator[27905]: rsync://ca.rg.net/rpki: failed with
status exit code: 35
Oct 21 09:20:53 bgp2 routinator[27905]: rsync://ca.rg.net/rpki: rsync
error: timeout waiting for daemon connectio
Oct 21 09:21:10 bgp2 routinator[27905]: rsync://rpki-repository.nic.ad.jp/ap:
failed with status exit code: 35
Oct 21 09:21:10 bgp2 routinator[27905]: rsync://rpki-repository.nic.ad.jp/ap:
rsync error: timeout waiting for da
Oct 21 09:31:35 bgp2 routinator[27905]: rsync://rpki-repository.nic.ad.jp/ap:
failed with status exit code: 35
Oct 21 09:31:35 bgp2 routinator[27905]: rsync://rpki-repository.nic.ad.jp/ap:
rsync error: timeout waiting for da
Oct 21 09:31:48 bgp2 routinator[27905]: rsync://ca.rg.net/rpki: failed with
status exit code: 35
Oct 21 09:31:48 bgp2 routinator[27905]: rsync://ca.rg.net/rpki: rsync
error: timeout waiting for daemon connection


● routinator.service - Routinator 3000
   Loaded: loaded (/etc/systemd/system/routinator.service; enabled; vendor
preset: enabled)
   Active: active (running) since Sun 2019-10-20 17:47:29 EDT; 15h ago
 Main PID: 23658 (routinator)
    Tasks: 8 (limit: 4626)
   Memory: 645.9M
   CGroup: /system.slice/routinator.service
           └─23658 /root/.cargo/bin/routinator server --rtr 127.0.0.1:3323

Oct 20 20:57:14 bgp1 rpki-routinator[23658]: rsync://
rpki.afrinic.net/repository: failed with status exit code: 3
Oct 20 20:57:14 bgp1 rpki-routinator[23658]: rsync://
rpki.afrinic.net/repository: rsync error: timeout waiting fo
Oct 20 21:07:43 bgp1 rpki-routinator[23658]: rsync://
rpki.afrinic.net/repository: failed with status exit code: 3
Oct 20 21:07:43 bgp1 rpki-routinator[23658]: rsync://
rpki.afrinic.net/repository: rsync error: timeout waiting fo
Oct 20 21:18:12 bgp1 rpki-routinator[23658]: rsync://
rpki.afrinic.net/repository: failed with status exit code: 3
Oct 20 21:18:12 bgp1 rpki-routinator[23658]: rsync://
rpki.afrinic.net/repository: rsync error: timeout waiting fo
Oct 20 21:28:55 bgp1 rpki-routinator[23658]: rsync://
rpki.afrinic.net/repository: failed with status exit code: 3
Oct 20 21:28:55 bgp1 rpki-routinator[23658]: rsync://
rpki.afrinic.net/repository: rsync error: timeout waiting fo
Oct 20 21:39:24 bgp1 rpki-routinator[23658]: rsync://
rpki.afrinic.net/repository: failed with status exit code: 3
Oct 20 21:39:24 bgp1 rpki-routinator[23658]: rsync://
rpki.afrinic.net/repository: rsync error: timeout waiting fo


I might be completely wrong here, but maybe the memory is increasing each
time there is a timeout?

Thanks
Darren

On Mon, 21 Oct 2019 at 05:40, Martin Hoffmann <martin at nlnetlabs.nl> wrote:

> Darren O'Connor wrote:
> > On Sun, 20 Oct 2019, 13:08 Paul Hoogsteder, <paul at meanie.nl> wrote:
> >
> > > That seems excessive. My routinator uses 116/139M at the moment.
> > > That's on Raspbian so not completely comparable but I guess most
> > > datastructures will be comparable in size.
> >
> > Even stranger is that I have two instances on two identical servers.
> > One is 550MB and the second is using 657MB. I'm not sure why.
>
> It looks like memory consumption is tied to the number of validation
> threads which defaults to the number of CPUs (or cores, rather). Paul’s
> Pi probably has just one -- if I run Routinator with
> --validation-threads 1, I get to a similar number.
>
> Are you perhaps using VMs or somesuch with different number of cores
> allocated?
>
> Kind regards,
> Martin
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.nlnetlabs.nl/pipermail/rpki/attachments/20191021/cdcfdc8f/attachment.htm>