[Unbound-users] Issues with Unbound 1.4.18

Tue Sep 4 10:29:34 UTC 2012

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hi Phil,

Thanks for the coredump!  This bug is related to your query volume and
openssl locking, it has been fixed in the development version at 1 August.

workarounds would be to run with num-threads: 1 (probably easiest), or
to have unbound compiled --without-pthreads (forks a new process
instead of a new thread for multicore machines).  Or disable
validation of DNSSEC.  Or use the patch.

There is also a patch, attached.

It is caused because, although unbound does not use openssl's locking
and threading, openssl has global tables in which it needs locking.
This is why it can run so long without a problem, these tables simply
do not change, so the absence of locking is no problem, and the race
condition window is very small, but something happens inside openssl
memory allocation (reference counting) that causes problems.  The
patch initializes locks for openssl.

Best regards,
   Wouter

On 09/04/2012 12:01 PM, Phil Davies wrote:
> Hi Wouter,
> 
> I managed to get unbound to generate a core dump. The back trace 
> suggests a possible issue with DNSSEC and OpenSSL related to when 
> verifying rsa keys any way here it is:
> 
> # gdb /usr/local/sbin/unbound /usr/local/etc/unbound/unbound.core 
> GNU gdb 6.1.1 [FreeBSD] Copyright 2004 Free Software Foundation,
> Inc. GDB is free software, covered by the GNU General Public
> License, and you are welcome to change it and/or distribute copies
> of it under certain conditions. Type "show copying" to see the
> conditions. There is absolutely no warranty for GDB.  Type "show
> warranty" for details. This GDB was configured as
> "amd64-marcel-freebsd"... Core was generated by `unbound'. Program
> terminated with signal 11, Segmentation fault. Reading symbols from
> /usr/local/lib/libssl.so.8...done. Loaded symbols for
> /usr/local/lib/libssl.so.8 Reading symbols from
> /usr/local/lib/libldns.so.1...done. Loaded symbols for
> /usr/local/lib/libldns.so.1 Reading symbols from
> /lib/libutil.so.8...done. Loaded symbols for /lib/libutil.so.8 
> Reading symbols from /usr/local/lib/libevent-1.4.so.4...done. 
> Loaded symbols for /usr/local/lib/libevent-1.4.so.4 Reading symbols
> from /usr/local/lib/libcrypto.so.8...done. Loaded symbols for
> /usr/local/lib/libcrypto.so.8 Reading symbols from
> /lib/libthr.so.3...done. Loaded symbols for /lib/libthr.so.3 
> Reading symbols from /lib/libc.so.7...done. Loaded symbols for
> /lib/libc.so.7 Reading symbols from /usr/lib/libz.so...done. Loaded
> symbols for /usr/lib/libz.so Reading symbols from
> /libexec/ld-elf.so.1...done. Loaded symbols for
> /libexec/ld-elf.so.1 #0  0x00000000005b10c0 in rbtree_null_node () 
> [New Thread 8013bc1c0 (LWP 100105/unbound)] [New Thread 8012041c0
> (LWP 100174/unbound)] (gdb) bt #0  0x00000000005b10c0 in
> rbtree_null_node () #1  0x0000000800ca001a in int_rsa_verify
> (dtype=672, m=0x7fffffffdab0 
> "?�\203�)�!\232��\202�\206zK\224�5\204\026�D\216m\233\r��\213��0@��\002\b",
>
> 
m_len=32, rm=0x0, prm_len=0x0, sigbuf=0x82967e77d "\\N���", siglen=128,
> rsa=0x82be59940) at rsa_sign.c:199 #2  0x0000000800ca032b in
> RSA_verify (dtype=Variable "dtype" is not available. ) at
> rsa_sign.c:317 #3  0x0000000800ca48d3 in pkey_rsa_verify
> (ctx=Variable "ctx" is not available. ) at rsa_pmeth.c:371 #4
> 0x0000000800ccd2c2 in EVP_VerifyFinal (ctx=0x7fffffffdbb0, 
> sigbuf=0x82967e77d "\\N���", siglen=Variable "siglen" is not
> available. ) at p_verify.c:91 #5  0x0000000000477e10 in
> verify_canonrrset (buf=0x80266cac0, algo=8, sigblock=0x82967e77d
> "\\N���", sigblock_len=128, key=0x82976b562 
> "\001\003��m�֮�B\034\t`W�$�ȽY�|\215\204\203l\027�D}\215�\177���2\237�+\036�hc�>�p_\211U\031\205�\177�y\227\211K\207�g\234�o\214<AYT\034\016��p|�\001��AvOpI;S$�\004�i\211ZA��\211ͳ�l\177W\030X\216\\�#�xP�lo[F�F\"��\006���G�\001\027",
>
>  keylen=130, reason=0x7fffffffe1c0) at validator/val_secalgo.c:534 
> #6  0x000000000047b05d in dnskey_verify_rrset_sig
> (region=0x802737000, buf=0x80266cac0, ve=0x801243150,
> now=1346696668, rrset=0x82967e3d0, dnskey=0x7fffffffdfa0,
> dnskey_idx=1, sig_idx=0, sortree=0x7fffffffdec8, 
> buf_canon=0x7fffffffdd38, reason=0x7fffffffe1c0) at 
> validator/val_sigcrypt.c:1349 #7  0x0000000000479639 in
> dnskeyset_verify_rrset_sig (env=0x8013fd3e8, ve=0x801243150,
> now=1346696668, rrset=0x82967e3d0, dnskey=0x7fffffffdfa0,
> sig_idx=0, sortree=0x7fffffffdec8, reason=0x7fffffffe1c0) at
> validator/val_sigcrypt.c:607 #8  0x0000000000479127 in
> dnskeyset_verify_rrset (env=0x8013fd3e8, ve=0x801243150,
> rrset=0x82967e3d0, dnskey=0x7fffffffdfa0, sigalg=0x82976b700 "\b",
> reason=0x7fffffffe1c0) at validator/val_sigcrypt.c:504 #9
> 0x000000000047bbc3 in val_verify_rrset (env=0x8013fd3e8, 
> ve=0x801243150, rrset=0x82967e3d0, keys=0x7fffffffdfa0, 
> sigalg=0x82976b700 "\b", reason=0x7fffffffe1c0) at 
> validator/val_utils.c:334 #10 0x000000000047be11 in
> val_verify_rrset_entry (env=0x8013fd3e8, ve=0x801243150,
> rrset=0x82967e3d0, kkey=0x82976b360, reason=0x7fffffffe1c0) at
> validator/val_utils.c:380 #11 0x0000000000475c3b in list_is_secure
> (env=0x8013fd3e8, ve=0x801243150, list=0x82967e308, num=3,
> kkey=0x82976b360, reason=0x7fffffffe1c0) at
> validator/val_nsec3.c:1351 #12 0x0000000000475dab in
> nsec3_prove_nods (env=0x8013fd3e8, ve=0x801243150,
> list=0x82967e308, num=3, qinfo=0x82967c080, kkey=0x82976b360,
> reason=0x7fffffffe1c0) at validator/val_nsec3.c:1378 #13
> 0x000000000046cc9d in ds_response_to_ke (qstate=0x829768080, 
> vq=0x82976b250, id=0, rcode=0, msg=0x82967e298, qinfo=0x82967c080, 
> ke=0x7fffffffe250) at validator/validator.c:2438 #14
> 0x000000000046d0d1 in process_ds_response (qstate=0x829768080, 
> vq=0x82976b250, id=0, rcode=0, msg=0x82967e298, qinfo=0x82967c080, 
> origin=0x82967eb40) at validator/validator.c:2555 #15
> 0x000000000046dda6 in val_inform_super (qstate=0x82967c080, id=0, 
> super=0x829768080) at validator/validator.c:2895 #16
> 0x00000000004425c8 in mesh_walk_supers (mesh=0x802a64480, 
> mstate=0x82967c030) at services/mesh.c:924 #17 0x0000000000442b0b
> in mesh_continue (mesh=0x802a64480, mstate=0x82967c030,
> s=module_finished, ev=0x7fffffffe32c) at services/mesh.c:1041 #18
> 0x0000000000442cc1 in mesh_run (mesh=0x802a64480, 
> mstate=0x82967c030, ev=module_event_moddone, e=0x0) at
> services/mesh.c:1072 #19 0x0000000000440ae7 in mesh_report_reply
> (mesh=0x802a64480, e=0x82967e278, reply=0x7fffffffe680, what=0) at
> services/mesh.c:488 #20 0x00000000004167b5 in
> worker_handle_service_reply (c=0x80265f140, arg=0x82967e278,
> error=0, reply_info=0x7fffffffe680) at daemon/worker.c:287 #21
> 0x00000000004899f1 in serviced_callbacks (sq=0x8044db1c0, error=0, 
> c=0x80265f140, rep=0x7fffffffe680) at
> services/outside_network.c:1511 #22 0x000000000048ab18 in
> serviced_udp_callback (c=0x80265f140, arg=0x8044db1c0, error=0,
> rep=0x7fffffffe680) at services/outside_network.c:1782 #23
> 0x00000000004866d3 in outnet_udp_cb (c=0x80265f140,
> arg=0x801bf1500, error=0, reply_info=0x7fffffffe680) at
> services/outside_network.c:462 #24 0x000000000047f4dc in
> comm_point_udp_callback (fd=179, event=2, arg=0x80265f140) at
> util/netevent.c:656 #25 0x0000000800ac2302 in event_process_active
> (base=0x8013cf900) at event.c:400 #26 0x0000000800ac269c in
> event_base_loop (base=0x8013cf900, flags=0) at event.c:552 #27
> 0x0000000800ac236a in event_base_dispatch (event_base=0x8013cf900) 
> at event.c:420 #28 0x000000000047e16b in comm_base_dispatch
> (b=0x8013c3360) at util/netevent.c:265 #29 0x0000000000419d99 in
> worker_work (worker=0x8013fc000) at daemon/worker.c:1208 #30
> 0x000000000040cee6 in daemon_fork (daemon=0x80120b140) at 
> daemon/daemon.c:487 #31 0x0000000000416059 in run_daemon
> (cfgfile=0x48dafe "/unbound.conf", cmdline_verbose=0, debug_mode=0)
> at daemon/unbound.c:662 #32 0x0000000000416261 in main (argc=0,
> argv=0x7fffffffe970) at daemon/unbound.c:756 (gdb)
> 
> For reference i'm using openssl version : openssl-1.0.1_4
> (installed from the ports) . Let me know if you need any further
> information.
> 
> Thanks
> 
> Phil Davies

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.12 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iQIcBAEBAgAGBQJQRdgOAAoJEJ9vHC1+BF+NFgUQAKcCMVJpV31U72bwMImMb+uK
9jnQxXEWK5ZITNrUFaE1um5tMi7rkMdBxb9VyTBxhndB25DL2SdAK9mR0bp+W/ip
fiB6vu4rd2nxBlmhiKttT+0QNG0fbWj4oviUp0DFhmDO/u5HunnhcqQaH1DywWZ/
DE5HQHdLUkJE3vfq4sQ/cgMQcnFfSbh6yNA9C15jwavwDBpwGBMNPTOVOOH79jPg
3VUjBaXkoeprkZ396Y1DnGfSlu5DIJB8RUecUZ2vqddyA66vG4igAixiThXnej33
enV+0sS2wq9SxvR0yAzXtOQxWpZByN6ldpHeRHp0eFy/KcPV0RUdlJ7g1JZ5Xwzc
VoypP3qh0PRA7wSmagJ7xRgHHKcUITNGRssIMDRA4nAky9xt2BCHXDEh2lefuQ6Q
Jx8LdWAP5HojlUV4EWYNfa2IA2FXr+xv31XQo8wcEfWeEZ0JVUMWmiPQiL5DLgVH
iwb86HIQmT7x92BpPZxaWkW3ueLwJyw0vGMN6TWh5amGtF9XcBVdNnBsSGTt5rus
VWGVRzR3jnmPmxnXh/RwEAp+TyQnm+o1ni/fUk3rY6qKtkCajHEN4riLBsjLRmfd
83RLhB8olsChLpcyKR8VogP3gXryl/tJuSOsUsJxS3lfr9YbI07Kq4VIs8A38kWO
0wxb+9PJrkl6Os7PL6JO
=XMfg
-----END PGP SIGNATURE-----
-------------- next part --------------
A non-text attachment was scrubbed...
Name: patch_openssl_threads.diff
Type: text/x-patch
Size: 2637 bytes
Desc: not available
URL: <http://lists.nlnetlabs.nl/pipermail/unbound-users/attachments/20120904/d26aa8dc/attachment.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: patch_openssl_threads.diff.sig
Type: application/pgp-signature
Size: 543 bytes
Desc: not available
URL: <http://lists.nlnetlabs.nl/pipermail/unbound-users/attachments/20120904/d26aa8dc/attachment-0001.bin>