[Unbound-users] unbound occasionally does not answer
Attila Nagy
bra at fsn.hu
Fri Sep 18 11:12:41 UTC 2009
Hello,
Better late, than never. :)
No, I still have my problems, and I have reached the point where I would
like to do something against it.
The setup is similar as Felix's (even in terms of the qps values), but
the problem seems to be different.
The environment is: FreeBSD 7/amd64 STABLE, quad core CPUs, but only one
used by unbound (it runs unthreaded). It's compiled with libev, statically.
Configuration:
server:
verbosity: 0
statistics-interval: 0
extended-statistics: yes
#num-threads: 2
outgoing-range: 8192
msg-cache-size: 1700M
num-queries-per-thread: 8192
rrset-cache-size: 4000M
do-daemonize: no
access-control: 0.0.0.0/0 allow
access-control: ::/0 allow
chroot: ""
username: ""
directory: "/etc"
As I can't turn on query logging (it effectively kills the server), I've
traced it with ktrace and checked the missing responses there.
What I can see is the investigated packets (queries) for which there was
no response, doesn't appear in the trace (but tcpdump sees it), so it
seems that unbound doesn't even get them.
top says:
PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU COMMAND
1522 bind 1 4 -15 7305M 7168M kqread 1 864:59 33.54% unbound
So unbound doesn't max out the CPU (core) it runs on.
I have started a sequential lookup of several hostnames, in the form of
$i.zone, where $i is an integer ranging 0 to 20000, the query fires
every milliseconds. All of the records are in the cache.
The test shows success ratios of:
2009-09-14 17:18:52+0200 [-] XXX sequential: 98.219090 out of 47953.000000
2009-09-14 17:28:52+0200 [-] XXX sequential: 97.950930 out of 48217.000000
2009-09-14 17:38:52+0200 [-] XXX sequential: 98.321538 out of 47901.000000
2009-09-14 17:48:52+0200 [-] XXX sequential: 98.096286 out of 48169.000000
2009-09-14 17:58:52+0200 [-] XXX sequential: 98.095040 out of 48190.000000
2009-09-14 18:08:52+0200 [-] XXX sequential: 98.133461 out of 48164.000000
2009-09-14 18:18:52+0200 [-] XXX sequential: 98.222950 out of 48226.000000
2009-09-14 18:28:52+0200 [-] XXX sequential: 98.217362 out of 48187.000000
2009-09-14 18:38:52+0200 [-] XXX sequential: 97.964281 out of 48042.000000
2009-09-14 18:48:52+0200 [-] XXX sequential: 97.921717 out of 48261.000000
2009-09-14 18:58:52+0200 [-] XXX sequential: 97.506948 out of 48214.000000
2009-09-14 19:08:52+0200 [-] XXX sequential: 97.060414 out of 48068.000000
2009-09-14 19:18:52+0200 [-] XXX sequential: 97.214335 out of 48247.000000
2009-09-14 19:28:52+0200 [-] XXX sequential: 96.783492 out of 48220.000000
2009-09-14 19:38:52+0200 [-] XXX sequential: 96.252864 out of 48010.000000
2009-09-14 19:48:52+0200 [-] XXX sequential: 96.240118 out of 48193.000000
2009-09-14 19:58:52+0200 [-] XXX sequential: 95.753322 out of 48014.000000
2009-09-14 20:08:52+0200 [-] XXX sequential: 95.321613 out of 47837.000000
2009-09-14 20:18:52+0200 [-] XXX sequential: 94.027737 out of 48022.000000
2009-09-14 20:28:52+0200 [-] XXX sequential: 93.054829 out of 47803.000000
2009-09-14 20:38:52+0200 [-] XXX sequential: 93.724317 out of 47453.000000
2009-09-14 20:48:52+0200 [-] XXX sequential: 94.639940 out of 47742.000000
2009-09-14 20:58:52+0200 [-] XXX sequential: 95.425759 out of 48161.000000
2009-09-14 21:08:52+0200 [-] XXX sequential: 94.936893 out of 48172.000000
2009-09-14 21:18:52+0200 [-] XXX sequential: 96.082117 out of 47883.000000
2009-09-14 21:28:52+0200 [-] XXX sequential: 97.559304 out of 48142.000000
2009-09-14 21:38:52+0200 [-] XXX sequential: 97.683090 out of 47736.000000
2009-09-14 21:48:52+0200 [-] XXX sequential: 98.036436 out of 47974.000000
2009-09-14 21:58:52+0200 [-] XXX sequential: 97.378246 out of 47945.000000
2009-09-14 22:08:52+0200 [-] XXX sequential: 98.166017 out of 47983.000000
2009-09-14 22:18:52+0200 [-] XXX sequential: 98.314385 out of 48113.000000
2009-09-14 22:28:52+0200 [-] XXX sequential: 98.599084 out of 48040.000000
2009-09-14 22:38:52+0200 [-] XXX sequential: 98.412136 out of 47989.000000
2009-09-14 22:48:52+0200 [-] XXX sequential: 98.695048 out of 48201.000000
2009-09-14 22:58:52+0200 [-] XXX sequential: 98.513611 out of 48305.000000
2009-09-14 23:08:52+0200 [-] XXX sequential: 98.920180 out of 48434.000000
2009-09-14 23:18:52+0200 [-] XXX sequential: 99.060181 out of 48520.000000
2009-09-14 23:28:52+0200 [-] XXX sequential: 98.936499 out of 48519.000000
2009-09-14 23:38:52+0200 [-] XXX sequential: 98.935555 out of 48382.000000
2009-09-14 23:48:52+0200 [-] XXX sequential: 99.001955 out of 48595.000000
2009-09-14 23:58:52+0200 [-] XXX sequential: 99.253300 out of 48480.000000
2009-09-15 00:08:52+0200 [-] XXX sequential: 99.149614 out of 48331.000000
2009-09-15 00:18:52+0200 [-] XXX sequential: 99.285111 out of 48539.000000
2009-09-15 00:28:52+0200 [-] XXX sequential: 99.225860 out of 48570.000000
2009-09-15 00:38:52+0200 [-] XXX sequential: 99.267158 out of 48578.000000
2009-09-15 00:48:52+0200 [-] XXX sequential: 99.262403 out of 48536.000000
2009-09-15 00:58:52+0200 [-] XXX sequential: 99.395223 out of 48613.000000
2009-09-15 01:08:52+0200 [-] XXX sequential: 99.512507 out of 48411.000000
2009-09-15 01:18:52+0200 [-] XXX sequential: 99.639863 out of 48315.000000
2009-09-15 01:28:52+0200 [-] XXX sequential: 99.446795 out of 48445.000000
2009-09-15 01:38:52+0200 [-] XXX sequential: 99.641119 out of 48484.000000
2009-09-15 01:48:52+0200 [-] XXX sequential: 99.656541 out of 48623.000000
2009-09-15 01:58:52+0200 [-] XXX sequential: 99.645806 out of 48561.000000
2009-09-15 02:08:52+0200 [-] XXX sequential: 99.614957 out of 48566.000000
2009-09-15 02:18:52+0200 [-] XXX sequential: 99.499526 out of 48554.000000
2009-09-15 02:28:52+0200 [-] XXX sequential: 99.494784 out of 48692.000000
2009-09-15 02:38:52+0200 [-] XXX sequential: 99.554204 out of 48677.000000
2009-09-15 02:48:52+0200 [-] XXX sequential: 99.603517 out of 48678.000000
2009-09-15 02:58:52+0200 [-] XXX sequential: 99.746867 out of 48591.000000
2009-09-15 03:08:52+0200 [-] XXX sequential: 99.570842 out of 48467.000000
2009-09-15 03:18:52+0200 [-] XXX sequential: 99.661702 out of 48478.000000
2009-09-15 03:28:52+0200 [-] XXX sequential: 99.728875 out of 48686.000000
2009-09-15 03:38:52+0200 [-] XXX sequential: 99.691504 out of 48623.000000
2009-09-15 03:48:52+0200 [-] XXX sequential: 99.714585 out of 48701.000000
2009-09-15 03:58:52+0200 [-] XXX sequential: 99.591510 out of 48716.000000
2009-09-15 04:08:52+0200 [-] XXX sequential: 99.583351 out of 48002.000000
2009-09-15 04:18:52+0200 [-] XXX sequential: 99.878633 out of 48613.000000
2009-09-15 04:28:52+0200 [-] XXX sequential: 99.640406 out of 48666.000000
2009-09-15 04:38:52+0200 [-] XXX sequential: 99.585105 out of 48687.000000
2009-09-15 04:48:52+0200 [-] XXX sequential: 99.768124 out of 48733.000000
2009-09-15 04:58:52+0200 [-] XXX sequential: 99.609704 out of 48681.000000
2009-09-15 05:08:52+0200 [-] XXX sequential: 99.454856 out of 48611.000000
2009-09-15 05:18:52+0200 [-] XXX sequential: 99.595258 out of 48673.000000
2009-09-15 05:28:52+0200 [-] XXX sequential: 99.790477 out of 48682.000000
2009-09-15 05:38:52+0200 [-] XXX sequential: 99.607077 out of 48610.000000
2009-09-15 05:48:52+0200 [-] XXX sequential: 99.728512 out of 48621.000000
2009-09-15 05:58:52+0200 [-] XXX sequential: 99.584687 out of 48638.000000
2009-09-15 06:08:52+0200 [-] XXX sequential: 99.467724 out of 48659.000000
2009-09-15 06:18:52+0200 [-] XXX sequential: 99.644516 out of 48666.000000
2009-09-15 06:28:52+0200 [-] XXX sequential: 99.434750 out of 48651.000000
2009-09-15 06:38:52+0200 [-] XXX sequential: 99.528710 out of 48590.000000
2009-09-15 06:48:52+0200 [-] XXX sequential: 99.324658 out of 48568.000000
2009-09-15 06:58:52+0200 [-] XXX sequential: 99.274856 out of 48680.000000
2009-09-15 07:08:52+0200 [-] XXX sequential: 99.056467 out of 48435.000000
2009-09-15 07:18:52+0200 [-] XXX sequential: 98.914725 out of 48467.000000
2009-09-15 07:28:52+0200 [-] XXX sequential: 99.149702 out of 48336.000000
2009-09-15 07:38:52+0200 [-] XXX sequential: 99.084450 out of 48277.000000
2009-09-15 07:48:52+0200 [-] XXX sequential: 99.111203 out of 48380.000000
2009-09-15 07:58:52+0200 [-] XXX sequential: 99.013334 out of 48446.000000
2009-09-15 08:08:52+0200 [-] XXX sequential: 99.009881 out of 48378.000000
2009-09-15 08:18:52+0200 [-] XXX sequential: 98.528652 out of 48391.000000
2009-09-15 08:28:52+0200 [-] XXX sequential: 98.908414 out of 48370.000000
2009-09-15 08:38:52+0200 [-] XXX sequential: 98.827193 out of 48175.000000
2009-09-15 08:48:52+0200 [-] XXX sequential: 98.987568 out of 48102.000000
2009-09-15 08:58:52+0200 [-] XXX sequential: 98.569997 out of 47902.000000
2009-09-15 09:08:52+0200 [-] XXX sequential: 98.492483 out of 48026.000000
2009-09-15 09:18:52+0200 [-] XXX sequential: 98.269443 out of 48077.000000
2009-09-15 09:28:52+0200 [-] XXX sequential: 98.546045 out of 48007.000000
2009-09-15 09:38:52+0200 [-] XXX sequential: 98.399799 out of 47869.000000
2009-09-15 09:48:52+0200 [-] XXX sequential: 98.709502 out of 47811.000000
2009-09-15 09:58:52+0200 [-] XXX sequential: 98.537404 out of 47655.000000
2009-09-15 10:08:52+0200 [-] XXX sequential: 98.451378 out of 47849.000000
2009-09-15 10:18:52+0200 [-] XXX sequential: 98.264597 out of 47597.000000
2009-09-15 10:28:52+0200 [-] XXX sequential: 98.417808 out of 47529.000000
2009-09-15 10:38:52+0200 [-] XXX sequential: 98.366144 out of 47495.000000
2009-09-15 10:48:52+0200 [-] XXX sequential: 98.117186 out of 47429.000000
2009-09-15 10:58:52+0200 [-] XXX sequential: 98.697592 out of 47297.000000
2009-09-15 11:08:52+0200 [-] XXX sequential: 98.296661 out of 47084.000000
2009-09-15 11:18:52+0200 [-] XXX sequential: 98.474591 out of 47266.000000
2009-09-15 11:28:52+0200 [-] XXX sequential: 98.322687 out of 47278.000000
2009-09-15 11:38:52+0200 [-] XXX sequential: 98.527923 out of 47416.000000
2009-09-15 11:48:52+0200 [-] XXX sequential: 98.084607 out of 47301.000000
(the first number is the success ratio in percent, the second number is
the number of sent queries in that time interval (so this is a ten
minutes average))
As you can see, during off-hours the success rate is better, but still
not 100%.
First I thought of an OS issue, but currently I don't think the problem
is there.
Investigating the issue further, I realized that the timeouts come in
batches. They group together.
It seems that the query success rate is OK most of the cases, then about
1.2 seconds of massive timeouts come and then everything goes back to
normal (well, more normal than during those 1.2 seconds, but not
perfect, ie 100% response ratio)...
I have tried to increase the UDP buffer size of the OS, but because of
the large number of outgoing ports configured, they took too much space,
so I've modified unbound, so it sets its SO_RCVBUF only on its listen
port (BTW, I think such a configuration option would be nice).
The default (net.inet.udp.recvspace) is 42080, I've gradually increased
that to 256MB (I would say, it's insanely large) in the hope that it
helps unbound get the packets, which arrive during it does *something*.
I could get some improvement with this, there are even some periods,
where the success rate was 100%, but it' still not perfect:
2009-09-15 14:38:52+0200 [-] XXX sequential: 99.051233 out of 47430.000000
2009-09-15 14:48:52+0200 [-] XXX sequential: 99.710347 out of 47298.000000
2009-09-15 14:58:52+0200 [-] XXX sequential: 99.888882 out of 47697.000000
2009-09-15 15:08:52+0200 [-] XXX sequential: 99.762475 out of 47995.000000
2009-09-15 15:18:52+0200 [-] XXX sequential: 99.808570 out of 47537.000000
2009-09-15 15:28:52+0200 [-] XXX sequential: 99.862677 out of 48062.000000
2009-09-15 15:38:52+0200 [-] XXX sequential: 99.698107 out of 47699.000000
2009-09-15 15:48:52+0200 [-] XXX sequential: 99.848422 out of 48160.000000
2009-09-15 15:58:52+0200 [-] XXX sequential: 99.875514 out of 48198.000000
2009-09-15 16:08:52+0200 [-] XXX sequential: 99.699083 out of 48186.000000
2009-09-15 16:18:52+0200 [-] XXX sequential: 99.864521 out of 47978.000000
2009-09-15 16:28:52+0200 [-] XXX sequential: 99.712572 out of 48012.000000
2009-09-15 16:38:52+0200 [-] XXX sequential: 99.874543 out of 47825.000000
2009-09-15 16:48:52+0200 [-] XXX sequential: 99.868906 out of 48057.000000
2009-09-15 16:58:52+0200 [-] XXX sequential: 99.904021 out of 47927.000000
2009-09-15 17:08:52+0200 [-] XXX sequential: 99.910614 out of 48106.000000
2009-09-15 17:18:52+0200 [-] XXX sequential: 99.908695 out of 48190.000000
2009-09-15 17:28:52+0200 [-] XXX sequential: 99.902315 out of 48114.000000
2009-09-15 17:38:52+0200 [-] XXX sequential: 99.916715 out of 48028.000000
2009-09-15 17:48:52+0200 [-] XXX sequential: 99.912966 out of 48257.000000
2009-09-15 17:58:52+0200 [-] XXX sequential: 99.892526 out of 48384.000000
2009-09-15 18:08:52+0200 [-] XXX sequential: 99.882346 out of 48447.000000
2009-09-15 18:18:52+0200 [-] XXX sequential: 99.909243 out of 48481.000000
2009-09-15 18:28:52+0200 [-] XXX sequential: 99.882491 out of 48507.000000
2009-09-15 18:38:52+0200 [-] XXX sequential: 99.869665 out of 48337.000000
2009-09-15 18:48:52+0200 [-] XXX sequential: 99.886764 out of 48571.000000
2009-09-15 18:58:52+0200 [-] XXX sequential: 99.882566 out of 48538.000000
2009-09-15 19:08:52+0200 [-] XXX sequential: 99.466975 out of 48403.000000
2009-09-15 19:18:52+0200 [-] XXX sequential: 99.826961 out of 48544.000000
2009-09-15 19:28:52+0200 [-] XXX sequential: 99.814739 out of 48580.000000
2009-09-15 19:38:52+0200 [-] XXX sequential: 99.807819 out of 48392.000000
2009-09-15 19:48:52+0200 [-] XXX sequential: 99.816842 out of 48592.000000
2009-09-15 19:58:52+0200 [-] XXX sequential: 99.439580 out of 48535.000000
2009-09-15 20:08:52+0200 [-] XXX sequential: 99.779327 out of 48488.000000
2009-09-15 20:18:52+0200 [-] XXX sequential: 99.776809 out of 48389.000000
2009-09-15 20:28:52+0200 [-] XXX sequential: 99.789782 out of 48521.000000
2009-09-15 20:38:52+0200 [-] XXX sequential: 99.746872 out of 48197.000000
2009-09-15 20:48:52+0200 [-] XXX sequential: 99.767087 out of 48516.000000
2009-09-15 20:58:52+0200 [-] XXX sequential: 99.802270 out of 48551.000000
2009-09-15 21:08:52+0200 [-] XXX sequential: 99.806146 out of 48490.000000
2009-09-15 21:18:52+0200 [-] XXX sequential: 99.849512 out of 48509.000000
2009-09-15 21:28:52+0200 [-] XXX sequential: 99.880587 out of 48571.000000
2009-09-15 21:38:52+0200 [-] XXX sequential: 99.826069 out of 48295.000000
2009-09-15 21:48:52+0200 [-] XXX sequential: 99.888726 out of 48529.000000
2009-09-15 21:58:52+0200 [-] XXX sequential: 99.936082 out of 48500.000000
2009-09-15 22:08:52+0200 [-] XXX sequential: 99.948361 out of 48413.000000
2009-09-15 22:18:52+0200 [-] XXX sequential: 99.962945 out of 48576.000000
2009-09-15 22:28:52+0200 [-] XXX sequential: 99.979399 out of 48541.000000
2009-09-15 22:38:52+0200 [-] XXX sequential: 99.985530 out of 48377.000000
2009-09-15 22:48:52+0200 [-] XXX sequential: 99.991753 out of 48505.000000
2009-09-15 22:58:52+0200 [-] XXX sequential: 99.989729 out of 48680.000000
2009-09-15 23:08:52+0200 [-] XXX sequential: 99.989719 out of 48634.000000
2009-09-15 23:18:52+0200 [-] XXX sequential: 99.991774 out of 48628.000000
2009-09-15 23:28:52+0200 [-] XXX sequential: 99.981514 out of 48685.000000
2009-09-15 23:38:52+0200 [-] XXX sequential: 99.993813 out of 48492.000000
2009-09-15 23:48:52+0200 [-] XXX sequential: 99.989739 out of 48726.000000
2009-09-15 23:58:52+0200 [-] XXX sequential: 99.995883 out of 48579.000000
2009-09-16 00:08:52+0200 [-] XXX sequential: 99.983505 out of 48500.000000
2009-09-16 00:18:52+0200 [-] XXX sequential: 99.993835 out of 48662.000000
2009-09-16 00:28:52+0200 [-] XXX sequential: 99.991785 out of 48691.000000
2009-09-16 00:38:52+0200 [-] XXX sequential: 99.993838 out of 48682.000000
2009-09-16 00:48:52+0200 [-] XXX sequential: 99.989746 out of 48762.000000
2009-09-16 00:58:52+0200 [-] XXX sequential: 99.993848 out of 48768.000000
2009-09-16 01:08:52+0200 [-] XXX sequential: 99.987645 out of 48563.000000
2009-09-16 01:18:52+0200 [-] XXX sequential: 99.993791 out of 48315.000000
2009-09-16 01:28:52+0200 [-] XXX sequential: 99.993836 out of 48667.000000
2009-09-16 01:38:52+0200 [-] XXX sequential: 99.997943 out of 48622.000000
2009-09-16 01:48:52+0200 [-] XXX sequential: 100.000000 out of 48700.000000
2009-09-16 01:58:52+0200 [-] XXX sequential: 99.991787 out of 48706.000000
2009-09-16 02:08:52+0200 [-] XXX sequential: 99.993803 out of 48412.000000
2009-09-16 02:18:52+0200 [-] XXX sequential: 99.991786 out of 48700.000000
2009-09-16 02:28:52+0200 [-] XXX sequential: 99.985670 out of 48849.000000
2009-09-16 02:38:52+0200 [-] XXX sequential: 99.993847 out of 48760.000000
2009-09-16 02:48:52+0200 [-] XXX sequential: 99.989764 out of 48845.000000
2009-09-16 02:58:52+0200 [-] XXX sequential: 99.987708 out of 48811.000000
2009-09-16 03:08:52+0200 [-] XXX sequential: 99.993821 out of 48552.000000
2009-09-16 03:18:52+0200 [-] XXX sequential: 99.991775 out of 48634.000000
2009-09-16 03:28:52+0200 [-] XXX sequential: 99.989757 out of 48813.000000
2009-09-16 03:38:52+0200 [-] XXX sequential: 99.985646 out of 48767.000000
2009-09-16 03:48:52+0200 [-] XXX sequential: 99.993854 out of 48816.000000
2009-09-16 03:58:52+0200 [-] XXX sequential: 99.989767 out of 48863.000000
2009-09-16 04:08:52+0200 [-] XXX sequential: 99.995847 out of 48157.000000
2009-09-16 04:18:52+0200 [-] XXX sequential: 100.000000 out of 47829.000000
2009-09-16 04:28:52+0200 [-] XXX sequential: 99.991659 out of 47956.000000
2009-09-16 04:38:52+0200 [-] XXX sequential: 99.997918 out of 48025.000000
2009-09-16 04:48:52+0200 [-] XXX sequential: 99.995839 out of 48065.000000
2009-09-16 04:58:52+0200 [-] XXX sequential: 99.991676 out of 48052.000000
2009-09-16 05:08:52+0200 [-] XXX sequential: 99.989592 out of 48042.000000
2009-09-16 05:18:52+0200 [-] XXX sequential: 99.983612 out of 48816.000000
2009-09-16 05:28:52+0200 [-] XXX sequential: 99.987711 out of 48826.000000
2009-09-16 05:38:52+0200 [-] XXX sequential: 99.987700 out of 48781.000000
2009-09-16 05:48:52+0200 [-] XXX sequential: 99.991803 out of 48796.000000
2009-09-16 05:58:52+0200 [-] XXX sequential: 99.991793 out of 48740.000000
2009-09-16 06:08:52+0200 [-] XXX sequential: 99.995901 out of 48796.000000
2009-09-16 06:18:52+0200 [-] XXX sequential: 99.991815 out of 48871.000000
2009-09-16 06:28:52+0200 [-] XXX sequential: 99.993864 out of 48892.000000
2009-09-16 06:38:52+0200 [-] XXX sequential: 99.987697 out of 48767.000000
2009-09-16 06:48:52+0200 [-] XXX sequential: 99.987702 out of 48790.000000
2009-09-16 06:58:52+0200 [-] XXX sequential: 99.991809 out of 48834.000000
2009-09-16 07:08:52+0200 [-] XXX sequential: 99.997945 out of 48650.000000
2009-09-16 07:18:52+0200 [-] XXX sequential: 99.991786 out of 48698.000000
2009-09-16 07:28:52+0200 [-] XXX sequential: 99.989733 out of 48701.000000
2009-09-16 07:38:52+0200 [-] XXX sequential: 99.993821 out of 48551.000000
2009-09-16 07:48:52+0200 [-] XXX sequential: 99.985591 out of 48582.000000
2009-09-16 07:58:52+0200 [-] XXX sequential: 99.995873 out of 48461.000000
2009-09-16 08:08:52+0200 [-] XXX sequential: 99.997943 out of 48608.000000
2009-09-16 08:18:52+0200 [-] XXX sequential: 99.993842 out of 48718.000000
2009-09-16 08:28:52+0200 [-] XXX sequential: 99.989735 out of 48710.000000
2009-09-16 08:38:52+0200 [-] XXX sequential: 99.995881 out of 48551.000000
2009-09-16 08:48:52+0200 [-] XXX sequential: 99.991781 out of 48667.000000
2009-09-16 08:58:52+0200 [-] XXX sequential: 100.000000 out of 48660.000000
2009-09-16 09:08:52+0200 [-] XXX sequential: 99.995883 out of 48578.000000
2009-09-16 09:18:52+0200 [-] XXX sequential: 99.991760 out of 48544.000000
2009-09-16 09:28:52+0200 [-] XXX sequential: 99.991715 out of 48279.000000
2009-09-16 09:38:52+0200 [-] XXX sequential: 99.987506 out of 48023.000000
2009-09-16 09:48:52+0200 [-] XXX sequential: 99.966567 out of 47857.000000
2009-09-16 09:58:52+0200 [-] XXX sequential: 99.985362 out of 47820.000000
2009-09-16 10:08:52+0200 [-] XXX sequential: 99.993761 out of 48088.000000
2009-09-16 10:18:52+0200 [-] XXX sequential: 99.981277 out of 48069.000000
2009-09-16 10:28:52+0200 [-] XXX sequential: 99.981191 out of 47850.000000
2009-09-16 10:38:52+0200 [-] XXX sequential: 99.991636 out of 47824.000000
2009-09-16 10:48:52+0200 [-] XXX sequential: 99.133225 out of 47994.000000
2009-09-16 10:58:52+0200 [-] XXX sequential: 99.355925 out of 47510.000000
2009-09-16 11:08:52+0200 [-] XXX sequential: 99.970744 out of 47853.000000
2009-09-16 11:18:52+0200 [-] XXX sequential: 99.974811 out of 47640.000000
2009-09-16 11:28:52+0200 [-] XXX sequential: 99.970220 out of 47012.000000
2009-09-16 11:38:52+0200 [-] XXX sequential: 99.966165 out of 47288.000000
2009-09-16 11:48:52+0200 [-] XXX sequential: 99.520686 out of 47568.000000
2009-09-16 11:58:52+0200 [-] XXX sequential: 99.575127 out of 47779.000000
2009-09-16 12:08:52+0200 [-] XXX sequential: 99.896959 out of 47554.000000
2009-09-16 12:18:52+0200 [-] XXX sequential: 99.323040 out of 47861.000000
2009-09-16 12:28:52+0200 [-] XXX sequential: 99.396808 out of 47746.000000
2009-09-16 12:38:52+0200 [-] XXX sequential: 98.885220 out of 47543.000000
2009-09-16 12:48:52+0200 [-] XXX sequential: 99.117734 out of 47378.000000
2009-09-16 12:58:52+0200 [-] XXX sequential: 99.300862 out of 47201.000000
2009-09-16 13:08:52+0200 [-] XXX sequential: 99.126166 out of 47263.000000
2009-09-16 13:18:52+0200 [-] XXX sequential: 99.153883 out of 47393.000000
2009-09-16 13:28:52+0200 [-] XXX sequential: 98.976261 out of 47180.000000
2009-09-16 13:38:52+0200 [-] XXX sequential: 99.243430 out of 47451.000000
2009-09-16 13:48:52+0200 [-] XXX sequential: 98.913250 out of 47297.000000
2009-09-16 13:58:52+0200 [-] XXX sequential: 99.069846 out of 47304.000000
2009-09-16 14:08:52+0200 [-] XXX sequential: 98.965437 out of 47363.000000
2009-09-16 14:18:52+0200 [-] XXX sequential: 98.784528 out of 47389.000000
2009-09-16 14:28:52+0200 [-] XXX sequential: 98.970310 out of 47490.000000
2009-09-16 14:38:52+0200 [-] XXX sequential: 98.614124 out of 47479.000000
2009-09-16 14:48:52+0200 [-] XXX sequential: 98.742204 out of 47623.000000
2009-09-16 14:58:52+0200 [-] XXX sequential: 98.644524 out of 47511.000000
2009-09-16 15:08:52+0200 [-] XXX sequential: 98.776360 out of 47563.000000
2009-09-16 15:18:52+0200 [-] XXX sequential: 98.802520 out of 47934.000000
2009-09-16 15:28:52+0200 [-] XXX sequential: 98.791915 out of 47596.000000
2009-09-16 15:38:52+0200 [-] XXX sequential: 98.591431 out of 47708.000000
2009-09-16 15:48:52+0200 [-] XXX sequential: 98.838642 out of 47875.000000
2009-09-16 15:58:52+0200 [-] XXX sequential: 98.554277 out of 47727.000000
2009-09-16 16:08:52+0200 [-] XXX sequential: 98.820814 out of 47660.000000
2009-09-16 16:18:52+0200 [-] XXX sequential: 98.758178 out of 47994.000000
2009-09-16 16:28:52+0200 [-] XXX sequential: 98.821362 out of 47767.000000
2009-09-16 16:38:52+0200 [-] XXX sequential: 98.901352 out of 47786.000000
2009-09-16 16:48:52+0200 [-] XXX sequential: 98.701408 out of 47744.000000
2009-09-16 16:58:52+0200 [-] XXX sequential: 98.630137 out of 47742.000000
2009-09-16 17:08:52+0200 [-] XXX sequential: 98.525500 out of 47745.000000
2009-09-16 17:18:52+0200 [-] XXX sequential: 98.724129 out of 48124.000000
2009-09-16 17:28:52+0200 [-] XXX sequential: 98.679048 out of 47920.000000
2009-09-16 17:38:52+0200 [-] XXX sequential: 98.808527 out of 47756.000000
2009-09-16 17:48:52+0200 [-] XXX sequential: 98.703562 out of 48209.000000
2009-09-16 17:58:52+0200 [-] XXX sequential: 98.599278 out of 48475.000000
2009-09-16 18:08:52+0200 [-] XXX sequential: 98.465473 out of 48484.000000
2009-09-16 18:18:52+0200 [-] XXX sequential: 98.427003 out of 48379.000000
2009-09-16 18:28:52+0200 [-] XXX sequential: 98.434754 out of 48363.000000
2009-09-16 18:38:52+0200 [-] XXX sequential: 98.201021 out of 48194.000000
2009-09-16 18:48:52+0200 [-] XXX sequential: 98.503921 out of 48460.000000
2009-09-16 18:58:52+0200 [-] XXX sequential: 98.612860 out of 48445.000000
2009-09-16 19:08:52+0200 [-] XXX sequential: 98.349281 out of 48282.000000
2009-09-16 19:18:52+0200 [-] XXX sequential: 98.203753 out of 48490.000000
2009-09-16 19:28:52+0200 [-] XXX sequential: 98.276146 out of 48496.000000
2009-09-16 19:38:52+0200 [-] XXX sequential: 98.096699 out of 48232.000000
2009-09-16 19:48:52+0200 [-] XXX sequential: 98.128469 out of 48463.000000
2009-09-16 19:58:52+0200 [-] XXX sequential: 98.435858 out of 48525.000000
2009-09-16 20:08:52+0200 [-] XXX sequential: 98.236776 out of 48434.000000
2009-09-16 20:18:52+0200 [-] XXX sequential: 98.078709 out of 48457.000000
2009-09-16 20:28:52+0200 [-] XXX sequential: 98.240040 out of 48467.000000
2009-09-16 20:38:52+0200 [-] XXX sequential: 98.357834 out of 48229.000000
2009-09-16 20:48:52+0200 [-] XXX sequential: 98.198997 out of 48473.000000
2009-09-16 20:58:52+0200 [-] XXX sequential: 98.288891 out of 48565.000000
2009-09-16 21:08:52+0200 [-] XXX sequential: 98.235064 out of 48557.000000
2009-09-16 21:18:52+0200 [-] XXX sequential: 98.209793 out of 48486.000000
2009-09-16 21:28:52+0200 [-] XXX sequential: 98.274264 out of 48559.000000
2009-09-16 21:38:52+0200 [-] XXX sequential: 98.264584 out of 48461.000000
2009-09-16 21:48:52+0200 [-] XXX sequential: 98.167474 out of 48294.000000
2009-09-16 21:58:52+0200 [-] XXX sequential: 98.548094 out of 48488.000000
2009-09-16 22:08:52+0200 [-] XXX sequential: 98.524054 out of 48308.000000
2009-09-16 22:18:52+0200 [-] XXX sequential: 98.486842 out of 48640.000000
2009-09-16 22:28:52+0200 [-] XXX sequential: 98.758212 out of 48559.000000
2009-09-16 22:38:52+0200 [-] XXX sequential: 98.849142 out of 48138.000000
2009-09-16 22:48:52+0200 [-] XXX sequential: 98.645868 out of 48592.000000
2009-09-16 22:58:52+0200 [-] XXX sequential: 98.734775 out of 48687.000000
2009-09-16 23:08:52+0200 [-] XXX sequential: 98.990086 out of 48717.000000
2009-09-16 23:18:52+0200 [-] XXX sequential: 99.279054 out of 48686.000000
2009-09-16 23:28:52+0200 [-] XXX sequential: 99.287591 out of 48708.000000
2009-09-16 23:38:52+0200 [-] XXX sequential: 99.123892 out of 48510.000000
2009-09-16 23:48:52+0200 [-] XXX sequential: 99.099524 out of 48752.000000
2009-09-16 23:58:52+0200 [-] XXX sequential: 99.306156 out of 48570.000000
2009-09-17 00:08:52+0200 [-] XXX sequential: 99.397281 out of 48613.000000
2009-09-17 00:18:52+0200 [-] XXX sequential: 99.306040 out of 48706.000000
2009-09-17 00:28:52+0200 [-] XXX sequential: 99.403820 out of 48643.000000
2009-09-17 00:38:52+0200 [-] XXX sequential: 99.586998 out of 48668.000000
2009-09-17 00:48:52+0200 [-] XXX sequential: 99.526144 out of 48749.000000
2009-09-17 00:58:52+0200 [-] XXX sequential: 99.297131 out of 48800.000000
2009-09-17 01:08:52+0200 [-] XXX sequential: 99.678953 out of 48591.000000
2009-09-17 01:18:52+0200 [-] XXX sequential: 99.604605 out of 48559.000000
2009-09-17 01:28:52+0200 [-] XXX sequential: 99.608828 out of 48572.000000
2009-09-17 01:38:52+0200 [-] XXX sequential: 99.535810 out of 48687.000000
2009-09-17 01:48:52+0200 [-] XXX sequential: 99.719072 out of 48767.000000
2009-09-17 01:58:52+0200 [-] XXX sequential: 99.624492 out of 48734.000000
2009-09-17 02:08:52+0200 [-] XXX sequential: 99.719686 out of 48517.000000
2009-09-17 02:18:52+0200 [-] XXX sequential: 99.589735 out of 48749.000000
2009-09-17 02:28:52+0200 [-] XXX sequential: 99.510787 out of 48854.000000
2009-09-17 02:38:52+0200 [-] XXX sequential: 99.725201 out of 48763.000000
2009-09-17 02:48:52+0200 [-] XXX sequential: 99.569363 out of 48765.000000
2009-09-17 02:58:52+0200 [-] XXX sequential: 99.782751 out of 48792.000000
2009-09-17 03:08:52+0200 [-] XXX sequential: 99.862114 out of 48591.000000
2009-09-17 03:18:52+0200 [-] XXX sequential: 99.761346 out of 48606.000000
2009-09-17 03:28:52+0200 [-] XXX sequential: 99.700427 out of 48736.000000
2009-09-17 03:38:52+0200 [-] XXX sequential: 99.608165 out of 48745.000000
2009-09-17 03:48:52+0200 [-] XXX sequential: 99.813746 out of 48858.000000
2009-09-17 03:58:52+0200 [-] XXX sequential: 99.838188 out of 48822.000000
2009-09-17 04:08:52+0200 [-] XXX sequential: 99.638736 out of 48441.000000
2009-09-17 04:18:52+0200 [-] XXX sequential: 99.807551 out of 48844.000000
2009-09-17 04:28:52+0200 [-] XXX sequential: 99.840288 out of 48838.000000
2009-09-17 04:38:52+0200 [-] XXX sequential: 99.571792 out of 48808.000000
2009-09-17 04:48:52+0200 [-] XXX sequential: 99.743716 out of 48774.000000
2009-09-17 04:58:52+0200 [-] XXX sequential: 99.647707 out of 48823.000000
2009-09-17 05:08:52+0200 [-] XXX sequential: 99.737748 out of 48808.000000
2009-09-17 05:18:52+0200 [-] XXX sequential: 99.635433 out of 48825.000000
2009-09-17 05:28:52+0200 [-] XXX sequential: 99.891465 out of 48832.000000
2009-09-17 05:38:52+0200 [-] XXX sequential: 99.813372 out of 48760.000000
2009-09-17 05:48:52+0200 [-] XXX sequential: 99.524610 out of 48802.000000
2009-09-17 05:58:52+0200 [-] XXX sequential: 99.784990 out of 48835.000000
2009-09-17 06:08:52+0200 [-] XXX sequential: 99.895453 out of 48782.000000
2009-09-17 06:18:52+0200 [-] XXX sequential: 99.556887 out of 48746.000000
2009-09-17 06:28:52+0200 [-] XXX sequential: 99.711083 out of 48803.000000
2009-09-17 06:38:52+0200 [-] XXX sequential: 99.477052 out of 48762.000000
2009-09-17 06:48:52+0200 [-] XXX sequential: 99.505925 out of 48778.000000
2009-09-17 06:58:52+0200 [-] XXX sequential: 99.476311 out of 48693.000000
2009-09-17 07:08:52+0200 [-] XXX sequential: 99.448843 out of 48625.000000
2009-09-17 07:18:52+0200 [-] XXX sequential: 99.117605 out of 48731.000000
2009-09-17 07:28:52+0200 [-] XXX sequential: 99.275496 out of 48723.000000
2009-09-17 07:38:52+0200 [-] XXX sequential: 99.189400 out of 48606.000000
2009-09-17 07:48:52+0200 [-] XXX sequential: 99.452787 out of 48610.000000
2009-09-17 07:58:52+0200 [-] XXX sequential: 98.805588 out of 48392.000000
2009-09-17 08:08:52+0200 [-] XXX sequential: 98.743133 out of 48056.000000
2009-09-17 08:18:52+0200 [-] XXX sequential: 98.583287 out of 48069.000000
2009-09-17 08:28:52+0200 [-] XXX sequential: 98.713679 out of 48044.000000
2009-09-17 08:38:52+0200 [-] XXX sequential: 99.015238 out of 48235.000000
2009-09-17 08:48:52+0200 [-] XXX sequential: 98.766275 out of 48309.000000
2009-09-17 08:58:52+0200 [-] XXX sequential: 98.907511 out of 48330.000000
2009-09-17 09:08:52+0200 [-] XXX sequential: 98.878644 out of 48156.000000
2009-09-17 09:18:52+0200 [-] XXX sequential: 98.846425 out of 48198.000000
2009-09-17 09:28:52+0200 [-] XXX sequential: 98.556521 out of 48009.000000
2009-09-17 09:38:52+0200 [-] XXX sequential: 98.719674 out of 47566.000000
2009-09-17 09:48:52+0200 [-] XXX sequential: 98.632922 out of 48205.000000
2009-09-17 09:58:52+0200 [-] XXX sequential: 98.540040 out of 48289.000000
2009-09-17 10:08:52+0200 [-] XXX sequential: 98.419498 out of 48086.000000
2009-09-17 10:18:52+0200 [-] XXX sequential: 98.651765 out of 48137.000000
2009-09-17 10:28:52+0200 [-] XXX sequential: 98.481359 out of 48201.000000
2009-09-17 10:38:52+0200 [-] XXX sequential: 98.181360 out of 47618.000000
2009-09-17 10:48:52+0200 [-] XXX sequential: 98.563857 out of 48115.000000
2009-09-17 10:58:52+0200 [-] XXX sequential: 98.635701 out of 48010.000000
2009-09-17 11:08:52+0200 [-] XXX sequential: 98.518706 out of 47391.000000
2009-09-17 11:18:52+0200 [-] XXX sequential: 98.671398 out of 47343.000000
2009-09-17 11:28:52+0200 [-] XXX sequential: 98.473282 out of 47422.000000
2009-09-17 11:38:52+0200 [-] XXX sequential: 98.508940 out of 47483.000000
2009-09-17 11:48:52+0200 [-] XXX sequential: 98.503558 out of 47780.000000
2009-09-17 11:58:52+0200 [-] XXX sequential: 98.334840 out of 47563.000000
2009-09-17 12:08:52+0200 [-] XXX sequential: 98.269186 out of 47261.000000
2009-09-17 12:18:52+0200 [-] XXX sequential: 97.525567 out of 48011.000000
2009-09-17 12:28:52+0200 [-] XXX sequential: 98.173315 out of 47682.000000
2009-09-17 12:38:52+0200 [-] XXX sequential: 98.655575 out of 47604.000000
2009-09-17 12:48:52+0200 [-] XXX sequential: 98.342261 out of 47233.000000
2009-09-17 12:58:52+0200 [-] XXX sequential: 98.327035 out of 47102.000000
2009-09-17 13:08:53+0200 [-] XXX sequential: 98.568464 out of 47222.000000
2009-09-17 13:18:53+0200 [-] XXX sequential: 98.509520 out of 47166.000000
2009-09-17 13:28:53+0200 [-] XXX sequential: 98.442592 out of 47258.000000
2009-09-17 13:38:53+0200 [-] XXX sequential: 98.241431 out of 46970.000000
2009-09-17 13:48:53+0200 [-] XXX sequential: 98.395143 out of 47107.000000
2009-09-17 13:58:53+0200 [-] XXX sequential: 98.342319 out of 47295.000000
2009-09-17 14:08:53+0200 [-] XXX sequential: 98.218475 out of 47319.000000
2009-09-17 14:18:53+0200 [-] XXX sequential: 97.218942 out of 47428.000000
2009-09-17 14:28:53+0200 [-] XXX sequential: 97.923539 out of 47292.000000
2009-09-17 14:38:53+0200 [-] XXX sequential: 98.139081 out of 47181.000000
2009-09-17 14:48:53+0200 [-] XXX sequential: 97.903838 out of 47420.000000
2009-09-17 14:58:53+0200 [-] XXX sequential: 98.127106 out of 47787.000000
2009-09-17 15:08:53+0200 [-] XXX sequential: 97.960124 out of 47748.000000
2009-09-17 15:18:53+0200 [-] XXX sequential: 97.968374 out of 47745.000000
2009-09-17 15:28:53+0200 [-] XXX sequential: 97.993451 out of 47644.000000
2009-09-17 15:38:53+0200 [-] XXX sequential: 97.882164 out of 47218.000000
2009-09-17 15:48:53+0200 [-] XXX sequential: 97.816511 out of 47218.000000
2009-09-17 15:58:53+0200 [-] XXX sequential: 97.877503 out of 47397.000000
2009-09-17 16:08:53+0200 [-] XXX sequential: 97.919224 out of 47242.000000
2009-09-17 16:18:53+0200 [-] XXX sequential: 98.372407 out of 47432.000000
2009-09-17 16:28:53+0200 [-] XXX sequential: 97.971343 out of 46829.000000
2009-09-17 16:38:53+0200 [-] XXX sequential: 97.862177 out of 47104.000000
The netstat output may be important:
netstat -s -p udp
udp:
2952534541 datagrams received
0 with incomplete header
0 with bad data length field
2154 with bad checksum
3918458 with no checksum
7911265 dropped due to no socket
19 broadcast/multicast datagrams undelivered
7015112 dropped due to full socket buffers
0 not for hashed pcb
2937605991 delivered
3090340606 datagrams output
0 times multicast source filter matched
Here, "dropped due to no socket" grows constantly (5-20 packets per
sec), but setting net.inet.udp.log_in_vain to 1 (which logs which
packets were dropped) tells that these packets are the ones, which come
back from the authoritative DNS servers, not the one, coming for port 53
(obviously unbound closed the socket, due to a timeout).
The other (and more interesting stuff) is "dropped due to full socket
buffers".
That counter has grown 0 to 130 drops per second (the exact value of
course moves with the actual query rate, so at nights, it is low, while
at daytime, it's at its peak) originally. With increasing SO_RCVBUF I
could push it down to 0 to 2-3 drops per second.
The counters are queried about every five minutes, so this is about a
five minutes average, as I've said, the drops come in batches, like this
(there is one second sleep between every printouts):
fullsockbufs.value 11939010
fullsockbufs.value 11939028
fullsockbufs.value 11939028
fullsockbufs.value 11939028
fullsockbufs.value 11939028
fullsockbufs.value 11939028
fullsockbufs.value 11939028
fullsockbufs.value 11939028
fullsockbufs.value 11939028
fullsockbufs.value 11939028
fullsockbufs.value 11939028
fullsockbufs.value 11939028
fullsockbufs.value 11939028
fullsockbufs.value 11939028
fullsockbufs.value 11939028
fullsockbufs.value 11939036
fullsockbufs.value 11939036
fullsockbufs.value 11939036
fullsockbufs.value 11939036
fullsockbufs.value 11939036
fullsockbufs.value 11939036
fullsockbufs.value 11939036
fullsockbufs.value 11939036
fullsockbufs.value 11939036
fullsockbufs.value 11939036
fullsockbufs.value 11939036
fullsockbufs.value 11939036
fullsockbufs.value 11939036
fullsockbufs.value 11939065
fullsockbufs.value 11939065
fullsockbufs.value 11939065
fullsockbufs.value 11939065
fullsockbufs.value 11939065
fullsockbufs.value 11939065
fullsockbufs.value 11939065
fullsockbufs.value 11939065
fullsockbufs.value 11939065
fullsockbufs.value 11939065
fullsockbufs.value 11939065
fullsockbufs.value 11939065
fullsockbufs.value 11939092
fullsockbufs.value 11939093
fullsockbufs.value 11939093
fullsockbufs.value 11939093
fullsockbufs.value 11939093
fullsockbufs.value 11939093
fullsockbufs.value 11939093
fullsockbufs.value 11939093
fullsockbufs.value 11939093
fullsockbufs.value 11939093
fullsockbufs.value 11939093
fullsockbufs.value 11939093
fullsockbufs.value 11939093
fullsockbufs.value 11939093
fullsockbufs.value 11939093
fullsockbufs.value 11939093
fullsockbufs.value 11939093
fullsockbufs.value 11939093
fullsockbufs.value 11939093
^C
Should multi threading help this (so the cache management could run in
its own thread, so maybe it doesn't affect other threads' performance -I
don't know about the details)?
BTW, with threading I have two problems:
- it slows down normal operation (at least on FreeBSD, so it means lower
qps)
- it increases the cache memory usage, which of course makes the hit
ratio slightly worse
So to summarize the above:
- I think it would be good to have a receive socket size configuration
option, which would make possible to change SO_RCVBUF size only on the
listening socket(s). The patch is trivial, but I'm sure you can do it
better/faster :)
- what could be done to make unbound not block for about a second and
drop a lot of queries (is it possible with some cache tuning, or is
changing code needed, or is is just an impossible thing with an
unthreaded application?)
ps: using memcached as unbound's cache store is a long time request on
my part, I would be curious whether it's affected by the same problems,
or not (it also have an asynchronous mono threaded run-mode)...
Thanks and sorry for the long message.
On 08/14/09 08:57, W.C.A. Wijngaards wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> Hi Felix,
>
> Can you tell me the names of the queries (off-list ? ).
>
> Perhaps they are misconfigured domains of some sort.
> I believe I solved Attila's problems from June.
>
> If you dig for them yourself, do you get an answer?
> And with dig +cdflag (to see if they are 'bogus').
>
> Best regards,
> Wouter
>
> On 08/13/2009 07:28 PM, Felix Schueren wrote:
>
>> Hello,
>>
>> and sorry for starting a new thread when the last one is from June - I
>> subscribed only recently and the mail archive does not show mail ids.
>> Anyway, I'm referring to the thread from Attila Nagy: "No answers from
>> unbound occasionally" - we're seeing much the same symptoms (using
>> 1.3.2), i.e. unbound sometimes not answering queries, and debugging is a
>> pain - we're running a load-balanced setup with currently 3 unbound
>> nodes, around 22k q/s, each unbound node doing ~6-8k qps. It usually
>> works fast & fine, but some queries appear to get eaten - we're getting
>> occasional dns resolution errors when using unbound as cache that we
>> never got with our dnscache (djb) setup. I've debugged as far as seeing
>> that the queries reach unbound (they get logged), but I don't know
>> whether unbound answered and the packets got lost, or if unbound simply
>> did not answer at all.
>>
>> Any ideas on debugging this? "Occasionally" means a couple 100 queries
>> per day (out of roughly 1.1G total queries). My "exceeded" (jostle)
>> counter is 0, average number of waiting requests is <10 per hosts.
>>
>> Kind regards,
>>
>> Felix
>>
>>
>
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.9 (GNU/Linux)
> Comment: Using GnuPG with Fedora - http://enigmail.mozdev.org/
>
> iEYEARECAAYFAkqFCsQACgkQkDLqNwOhpPjhGQCcDmsqJIQGbLO8NTQPLhd8g9JI
> CYQAn0BXNC3gIr+h88ktUPrA4LFIL3YG
> =bx1R
> -----END PGP SIGNATURE-----
> _______________________________________________
> Unbound-users mailing list
> Unbound-users at unbound.net
> http://unbound.nlnetlabs.nl/mailman/listinfo/unbound-users
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.nlnetlabs.nl/pipermail/unbound-users/attachments/20090918/28a2a7b7/attachment.htm>
More information about the Unbound-users
mailing list