[Unbound-users] unbound performance tuning

[Wouter Wijngaards]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hi Beastie,

Beastie wrote:
> Hi.
> 
> I'm using unbound for a couple of months as a default recursor in one
> middle level installation and I like it so far :)

Thanks

> Now I have a paln to replace BIND with unbound in one big ISP network
> (average recursion load is 50K requests/sec).
> I have a question about how to tune unbound to allow him handle that
> volumes. That parameters should I adjust to get maximum performance?
> We have two CPU socket quard core Xeon server with 12GB of RAM running
> FreeBSD 7 STABLE. Unbound will be compilled without libevent and
> without threading, and we will start 8 unbound processes (num-threads:
> 8) using 1024MB cache size for each.
> 
> I have a couple of ideas about that should be tuned, but I would like
> to ask - am I right, or I'm missing somthing? This post is request for
> comments.

I'll try to tell you what I think I know.

> AFAIK unbound handle incoming requests this way:
> 
> .............................................|.rrset-cache-size:.|...................................
> (Client).==>.|.num-queries-per-thread:.|.==>.|.msg-cache-size:.|.==>.|.outgoing-range:.|.==>.(Resolving)

No, client queries are not stored in the msg-cache.   The msg-cache
contains server replies.

> So, I should pay attention for that potential bottlenecks:
> 1) num-queries-per-thread: - "The number of queries that every thread
> will service simultaneously". If one process will have more than this
> nuber of simultaneousl requests then it will drop exceeded requests?

It will still try to answer from cache, but if that fails the message
can be dropped.

A high number of num-queries-per-thread is useful to suck-up 'peak
load'. On the average, useful resolving of 'outgoing-range' number of
messages at the same time gives good performance I think.

> 2) msg-cache-size: - After queue has arrived into unbound it will be
> saved in msg-cache and will wait there until resolving process will be
> free to handle it? If msg-cache has no free space then new arrived
> queues will be dropped?

No. It does not work like that.  The cache is used to store DNS
information - both resource records and message formats.

The msg-cache (and rrset-cache) stores DNS info, using MRU algo, if a
new entry arrives, the LRU elements are deleted until there is
sufficient space for it.

Thus, the caches are not part of your flow line.

> 3) outgoing-range: - the nuber of random ports that resolving process
> can simultaneously use for sending requests out of unbound. If this
> nuber is low, then msg-cache will grow in size and that can produce
> queue dropping?

Right and wrong.  Right about that it is the number of ports that the
resolving process can simultaneously use for sending requests out of
unbound. And if low, then it produces problems with queing.

(Wrong about msg-cache growing in size).  What grows in size is probably
the requestlist.  The queries that wait for resolving (i.e. they were
not answered from cache) are in the requestlist.  This is an abstraction
- - it is not a simple list - but it is easy to understand.   The
requestlist will grow in size if the outgoing range is too small.

The num-queries-per-thread is the maximum value for the requestlist.

The statistics printout includes the requestlist size; the average and
the maximum.  It also includes the number of times the requestlist was
exceeded.   This can give you a good indication what the size of your
requestlist is in practice, and if its too small.

For good performance, I would think outgoing-range and
num-queries-per-thread should be about equal and large enough ...

> So, if that is right, I should set num-queries-per-thread: to 10240,
> msg-cache-size: to 512MB and outgoing-range: to somthing close to 1024
> (1024 - the limit of unbound's builtinmini-event event handler because
> I don't use libevent). I will tune FreeBSD sysctl limits for this
> (default FreeBSD 7 ulimit -n is 12328 and ulimit -d is 512M).

It would be good to compile with libevent - as it removes the 1024
restriction on the outgoing-range - and you need lots.   AFAIK there are
no problems on FreeBSD 7 with libevent(-kqueue backend).

Then I would put 6400 as outgoing-range.  For peak performance (8*6
about 48000 queries at the same time (that miss the cache) for your 50k
qps server).

> 
> I assume this unbound.conf will be ok?

Small problem, you allocate 512m msg and 1024m rrset cache per thread,
but that adds up to 12G.  Exactly your max.  I would leave some spare
room - for other processes, other data - so,  256m and 512m.
(note for the reader: because he compiled without threading, unbound is
forking and uses the cache memory per-processor, instead of shared mem).

even though you put verbosity: 0, it will still print (fatal) errors.
But since you set logfile: "" (stderr), and use-syslog: no; the stderr
stream goes nowhere.  Set use-syslog: yes to get errors printed in your
var/log/messages (depending on your log config).

If verbosity: 0 is printing too many log lines, let me know.

Best regards,
   Wouter

> 
> server:
>         verbosity: 0
>         num-threads: 8
>         interface: 0.0.0.0
>         port: 53
>         outgoing-range: 980
>         msg-cache-size: 512m
>         msg-cache-slabs: 8
>         num-queries-per-thread: 10240
>         rrset-cache-size: 1024m
>         rrset-cache-slabs: 8
>         cache-max-ttl: 86400
>         infra-host-ttl: 900
>         infra-lame-ttl: 900
>         infra-cache-slabs: 4
>         infra-cache-numhosts: 10000
>         infra-cache-lame-size: 10k
>         do-ip4: yes
>         do-ip6: no
>         do-udp: yes
>         do-tcp: yes
>         do-daemonize: yes
> 
>         access-control: 0.0.0.0/0 allow


>         access-control: ::0/0 allow

This line does nothing since do-ip6: is "no".

> 
>         chroot: "/usr/local/etc/unbound"
>         username: "unbound"
>         directory: "/usr/local/etc/unbound"
>         logfile: ""
>         use-syslog: no
>         pidfile: "/usr/local/etc/unbound/unbound.pid"
>         root-hints: "/usr/local/etc/unbound/named.cache"
> 
>         hide-identity: yes
>         hide-version: yes
>         harden-glue: yes
>         module-config: "iterator"
> 
> 
> Thanks in advance for any comment.
> _______________________________________________
> Unbound-users mailing list
> Unbound-users at unbound.net
> http://unbound.nlnetlabs.nl/mailman/listinfo/unbound-users

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (GNU/Linux)

iEYEARECAAYFAkj8P5sACgkQkDLqNwOhpPju6ACfR7oksAeXVIqh4elU2jiFq2nk
xbgAn1pIwjS9+YE2GVdU0G9h+vV4gza8
=rwtQ
-----END PGP SIGNATURE-----