[RPKI] routinator-0.13.1 core dump

Sun Mar 17 12:49:51 UTC 2024

I believe I have reproduced this crash and it's related to syslog. I 
have run into this before, probably with the amplified noise the crashes 
were happening more often.

Reproducing it I can cause the crash if I stop/restart the syslog 
service on the host at the right time.

Workaround: Disabled syslog and just set the log to file for now.

On 3/16/24 09:00, Robert Blayzor via RPKI wrote:
> I have been running into stability issues since the release of 0.13 and 
> it's been getting worse. I'm having routinator servers crash multiple 
> times a day now and I just had two servers crash basically at the same 
> time, so it's getting some outside information that it doesn't like.
> 
> The servers are running FreeBSD 14.0 and Rust 1.76 is installed on the 
> system. This has been happening since 0.13 and is now still happening in 
> 0.13.2, but now even more frequently it appears.
> 
> Unfortunately when routinator core dumps, it's not leaving a core file 
> behind, I'm just seeing:
> 
> pid 2982 (routinator), jid 0, uid 245: exited on signal 6 (no core dump 
> - bad address)
> pid 93548 (routinator), jid 0, uid 245: exited on signal 6 (no core dump 
> - bad address)
> pid 75744 (routinator), jid 0, uid 245: exited on signal 6 (no core dump 
> - bad address)
> 
> 
> Literally the last noise in the logs on each server. This level of 
> "noise" and failure looks like something is broken....
> 
> 
> Mar 16 12:43:43 rs0 routinator[75742]: 
> rsync://rpki.afrinic.net/repository/member_repository/F368F2D0/92F86E1C6E0511E8A1
> B5854BF8AEA228/EA021982D2C311EE98D88865775412E6.roa: certificate has 
> expired.
> Mar 16 12:43:43 rs0 routinator[75742]: 
> rsync://rpki.afrinic.net/repository/member_repository/F368F2D0/92F86E1C6E0511E8A1
> B5854BF8AEA228/6667703ED47D11EEAA6502BE775412E6.roa: certificate has 
> expired.
> Mar 16 12:43:43 rs0 routinator[75742]: 
> rsync://rpki.afrinic.net/repository/member_repository/F368F2D0/92F86E1C6E0511E8A1
> B5854BF8AEA228/86D788C8D48111EE80528CC4775412E6.roa: certificate has 
> expired.
> Mar 16 12:43:43 rs0 routinator[75742]: 
> rsync://rpki.afrinic.net/repository/member_repository/F368F2D0/92F86E1C6E0511E8A1
> B5854BF8AEA228/EFBE9CB6D16911EEA51004A8775412E6.roa: certificate has 
> expired.
> Mar 16 12:43:43 rs0 routinator[75742]: 
> rsync://rpki.afrinic.net/repository/member_repository/F368F2D0/92F86E1C6E0511E8A1
> B5854BF8AEA228/8668158AD17311EEB69D94B8775412E6.roa: certificate has 
> expired.
> Mar 16 12:43:43 rs0 routinator[75742]: 
> rsync://rpki.afrinic.net/repository/member_repository/F368F2D0/92F86E1C6E0511E8A1
> B5854BF8AEA228/B7C19E14D2CF11EEBB636E78775412E6.roa: certificate has 
> expired.
> Mar 16 12:43:43 rs0 routinator[75742]: 
> rsync://rpki.afrinic.net/repository/member_repository/F368F2D0/92F86E1C6E0511E8A1
> B5854BF8AEA228/B8A5AEA0D48311EE85B40BC8775412E6.roa: certificate has 
> expired.
> Mar 16 12:43:43 rs0 routinator[75742]: 
> rsync://rpki.afrinic.net/repository/member_repository/F368F2D0/92F86E1C6E0511E8A1
> B5854BF8AEA228/0B3122E8D17911EE951057C1775412E6.roa: certificate has 
> expired.
> Mar 16 12:43:43 rs0 routinator[75742]: 
> rsync://rpki.afrinic.net/repository/member_repository/F368F2D0/92F86E1C6E0511E8A1
> B5854BF8AEA228/385F1D8AD2D011EE9BBD5C79775412E6.roa: certificate has 
> expired.
> Mar 16 12:43:43 rs0 routinator[75742]: 
> rsync://rpki.afrinic.net/repository/member_repository/F368F2D0/92F86E1C6E0511E8A1
> B5854BF8AEA228/ED2CF494D2C611EEAA8E776A775412E6.roa: certificate has 
> expired.
> Mar 16 12:43:43 rs0 kernel: pid 75744 (routinator), jid 0, uid 245: 
> exited on signal 6 (no core dump - bad address)
> 
> 
> 
> 
> Mar 16 08:48:48 rs0 routinator[95204]: 
> rsync://rpki.afrinic.net/repository/member_repository/F368F2D0/92F86E1C6E0511E8A1
> B5854BF8AEA228/11A8344ED2C411EEB833F865775412E6.roa: certificate has 
> expired.
> Mar 16 08:48:48 rs0 routinator[95204]: 
> rsync://rpki.afrinic.net/repository/member_repository/F368F2D0/92F86E1C6E0511E8A1
> B5854BF8AEA228/3E62BCE0D2C711EEAEF6EA6A775412E6.roa: certificate has 
> expired.
> Mar 16 08:48:48 rs0 routinator[95204]: 
> rsync://rpki.afrinic.net/repository/member_repository/F368F2D0/92F86E1C6E0511E8A1
> B5854BF8AEA228/C5EC9C26D2CC11EE8E67C973775412E6.roa: certificate has 
> expired.
> Mar 16 08:48:48 rs0 routinator[95204]: 
> rsync://rpki.afrinic.net/repository/member_repository/F368F2D0/92F86E1C6E0511E8A1
> B5854BF8AEA228/8C32E506D2CE11EEA31CD576775412E6.roa: certificate has 
> expired.
> Mar 16 08:48:48 rs0 routinator[95204]: 
> rsync://rpki.afrinic.net/repository/member_repository/F368F2D0/92F86E1C6E0511E8A1
> B5854BF8AEA228/F8491F0ED2C811EE8D92D66D775412E6.roa: certificate has 
> expired.
> Mar 16 08:48:48 rs0 routinator[95204]: 
> rsync://rpki.afrinic.net/repository/member_repository/F368F2D0/92F86E1C6E0511E8A1
> B5854BF8AEA228/1543656CD2C711EE9822A16A775412E6.roa: certificate has 
> expired.
> Mar 16 08:48:48 rs0 routinator[95204]: 
> rsync://rpki.afrinic.net/repository/member_repository/F368F2D0/92F86E1C6E0511E8A1
> B5854BF8AEA228/2040974A0F1911EE83AEF94F4AD9E6FC.roa: certificate has 
> expired.
> Mar 16 08:48:48 rs0 routinator[95204]: 
> rsync://rpki.afrinic.net/repository/member_repository/F368F2D0/92F86E1C6E0511E8A1
> B5854BF8AEA228/A940268AD15D11EE8290C094775412E6.roa: certificate has 
> expired.
> Mar 16 08:48:48 rs0 routinator[95204]: 
> rsync://rpki.afrinic.net/repository/member_repository/F368F2D0/92F86E1C6E0511E8A1
> B5854BF8AEA228/5812BEAED20111EE9D9842A6775412E6.roa: certificate has 
> expired.
> Mar 16 08:48:48 rs0 kernel: pid 95206 (routinator), jid 0, uid 245: 
> exited on signal 6 (no core dump - bad address)
> 

-- 
inoc.net!rblayzor
XMPP: rblayzor.AT.inoc.net
PGP:  https://pgp.inoc.net/rblayzor/