[RPKI] routinator-0.13.1 core dump

Mon Mar 18 10:52:20 UTC 2024

Hi Robert!

Robert Blayzor via RPKI wrote:
> I believe I have reproduced this crash and it's related to syslog. I 
> have run into this before, probably with the amplified noise the
> crashes were happening more often.
> 
> Reproducing it I can cause the crash if I stop/restart the syslog 
> service on the host at the right time.
> 
> Workaround: Disabled syslog and just set the log to file for now.

I think with Håvard’s help we narrowed this down to rsync writing too
much error output and Routinator collecting all of that and writing it
at once which causes syslog to be ... unhappy.

We have fix for that in the upcoming 0.14.0. I will also look into
trying to re-connect to syslog on failure rather than erroring out.

Cheers,
Martin

> 
> 
> On 3/16/24 09:00, Robert Blayzor via RPKI wrote:
> > I have been running into stability issues since the release of 0.13
> > and it's been getting worse. I'm having routinator servers crash
> > multiple times a day now and I just had two servers crash basically
> > at the same time, so it's getting some outside information that it
> > doesn't like.
> > 
> > The servers are running FreeBSD 14.0 and Rust 1.76 is installed on
> > the system. This has been happening since 0.13 and is now still
> > happening in 0.13.2, but now even more frequently it appears.
> > 
> > Unfortunately when routinator core dumps, it's not leaving a core
> > file behind, I'm just seeing:
> > 
> > pid 2982 (routinator), jid 0, uid 245: exited on signal 6 (no core
> > dump 
> > - bad address)
> > pid 93548 (routinator), jid 0, uid 245: exited on signal 6 (no core
> > dump 
> > - bad address)
> > pid 75744 (routinator), jid 0, uid 245: exited on signal 6 (no core
> > dump 
> > - bad address)
> > 
> > 
> > Literally the last noise in the logs on each server. This level of 
> > "noise" and failure looks like something is broken....
> > 
> > 
> > Mar 16 12:43:43 rs0 routinator[75742]: 
> > rsync://rpki.afrinic.net/repository/member_repository/F368F2D0/92F86E1C6E0511E8A1
> > B5854BF8AEA228/EA021982D2C311EE98D88865775412E6.roa: certificate
> > has expired.
> > Mar 16 12:43:43 rs0 routinator[75742]: 
> > rsync://rpki.afrinic.net/repository/member_repository/F368F2D0/92F86E1C6E0511E8A1
> > B5854BF8AEA228/6667703ED47D11EEAA6502BE775412E6.roa: certificate
> > has expired.
> > Mar 16 12:43:43 rs0 routinator[75742]: 
> > rsync://rpki.afrinic.net/repository/member_repository/F368F2D0/92F86E1C6E0511E8A1
> > B5854BF8AEA228/86D788C8D48111EE80528CC4775412E6.roa: certificate
> > has expired.
> > Mar 16 12:43:43 rs0 routinator[75742]: 
> > rsync://rpki.afrinic.net/repository/member_repository/F368F2D0/92F86E1C6E0511E8A1
> > B5854BF8AEA228/EFBE9CB6D16911EEA51004A8775412E6.roa: certificate
> > has expired.
> > Mar 16 12:43:43 rs0 routinator[75742]: 
> > rsync://rpki.afrinic.net/repository/member_repository/F368F2D0/92F86E1C6E0511E8A1
> > B5854BF8AEA228/8668158AD17311EEB69D94B8775412E6.roa: certificate
> > has expired.
> > Mar 16 12:43:43 rs0 routinator[75742]: 
> > rsync://rpki.afrinic.net/repository/member_repository/F368F2D0/92F86E1C6E0511E8A1
> > B5854BF8AEA228/B7C19E14D2CF11EEBB636E78775412E6.roa: certificate
> > has expired.
> > Mar 16 12:43:43 rs0 routinator[75742]: 
> > rsync://rpki.afrinic.net/repository/member_repository/F368F2D0/92F86E1C6E0511E8A1
> > B5854BF8AEA228/B8A5AEA0D48311EE85B40BC8775412E6.roa: certificate
> > has expired.
> > Mar 16 12:43:43 rs0 routinator[75742]: 
> > rsync://rpki.afrinic.net/repository/member_repository/F368F2D0/92F86E1C6E0511E8A1
> > B5854BF8AEA228/0B3122E8D17911EE951057C1775412E6.roa: certificate
> > has expired.
> > Mar 16 12:43:43 rs0 routinator[75742]: 
> > rsync://rpki.afrinic.net/repository/member_repository/F368F2D0/92F86E1C6E0511E8A1
> > B5854BF8AEA228/385F1D8AD2D011EE9BBD5C79775412E6.roa: certificate
> > has expired.
> > Mar 16 12:43:43 rs0 routinator[75742]: 
> > rsync://rpki.afrinic.net/repository/member_repository/F368F2D0/92F86E1C6E0511E8A1
> > B5854BF8AEA228/ED2CF494D2C611EEAA8E776A775412E6.roa: certificate
> > has expired.
> > Mar 16 12:43:43 rs0 kernel: pid 75744 (routinator), jid 0, uid 245: 
> > exited on signal 6 (no core dump - bad address)
> > 
> > 
> > 
> > 
> > Mar 16 08:48:48 rs0 routinator[95204]: 
> > rsync://rpki.afrinic.net/repository/member_repository/F368F2D0/92F86E1C6E0511E8A1
> > B5854BF8AEA228/11A8344ED2C411EEB833F865775412E6.roa: certificate
> > has expired.
> > Mar 16 08:48:48 rs0 routinator[95204]: 
> > rsync://rpki.afrinic.net/repository/member_repository/F368F2D0/92F86E1C6E0511E8A1
> > B5854BF8AEA228/3E62BCE0D2C711EEAEF6EA6A775412E6.roa: certificate
> > has expired.
> > Mar 16 08:48:48 rs0 routinator[95204]: 
> > rsync://rpki.afrinic.net/repository/member_repository/F368F2D0/92F86E1C6E0511E8A1
> > B5854BF8AEA228/C5EC9C26D2CC11EE8E67C973775412E6.roa: certificate
> > has expired.
> > Mar 16 08:48:48 rs0 routinator[95204]: 
> > rsync://rpki.afrinic.net/repository/member_repository/F368F2D0/92F86E1C6E0511E8A1
> > B5854BF8AEA228/8C32E506D2CE11EEA31CD576775412E6.roa: certificate
> > has expired.
> > Mar 16 08:48:48 rs0 routinator[95204]: 
> > rsync://rpki.afrinic.net/repository/member_repository/F368F2D0/92F86E1C6E0511E8A1
> > B5854BF8AEA228/F8491F0ED2C811EE8D92D66D775412E6.roa: certificate
> > has expired.
> > Mar 16 08:48:48 rs0 routinator[95204]: 
> > rsync://rpki.afrinic.net/repository/member_repository/F368F2D0/92F86E1C6E0511E8A1
> > B5854BF8AEA228/1543656CD2C711EE9822A16A775412E6.roa: certificate
> > has expired.
> > Mar 16 08:48:48 rs0 routinator[95204]: 
> > rsync://rpki.afrinic.net/repository/member_repository/F368F2D0/92F86E1C6E0511E8A1
> > B5854BF8AEA228/2040974A0F1911EE83AEF94F4AD9E6FC.roa: certificate
> > has expired.
> > Mar 16 08:48:48 rs0 routinator[95204]: 
> > rsync://rpki.afrinic.net/repository/member_repository/F368F2D0/92F86E1C6E0511E8A1
> > B5854BF8AEA228/A940268AD15D11EE8290C094775412E6.roa: certificate
> > has expired.
> > Mar 16 08:48:48 rs0 routinator[95204]: 
> > rsync://rpki.afrinic.net/repository/member_repository/F368F2D0/92F86E1C6E0511E8A1
> > B5854BF8AEA228/5812BEAED20111EE9D9842A6775412E6.roa: certificate
> > has expired.
> > Mar 16 08:48:48 rs0 kernel: pid 95206 (routinator), jid 0, uid 245: 
> > exited on signal 6 (no core dump - bad address)
> >   
> 