[Unbound-users] Unbound vs MS Resolver

[Carsten Strotmann]

Hi Olaf,
Hi Unbound-Users,

I've did some benchmarking yesterday. I will continue later this month
(with different hardware, see below), but here are the first
results. I'm posting it here because you might have some ideas on how to
improve the benchmark-setup.

DISCLAIMER 1: the hardware used is not "Enterprise" or "Telco" level, I'm
aware of that. It is what I had available. 

DISCLAIMER 2: this is a benchmark in a synthetic lab environment. It is
not how DNS resolvers work in the real Internet. This results can give
indications, but do not replace your own measurements in your own
environment using your own hardware. 

Lab setup:

* one "load generating" machine running Unbound's own "perf" tool
  (Thinkpad E330 4 Core, 16GB RAM, running Fedora Linux 20) , sending
  1.000.000 queries to the DNS resolver (out of a zone of 10.000.000
  IPv4-A-records). I've tested using more than one "load generating"
  machine, but with the limited hardware of the resolver, more
  load-machines did not change the measured results

* one "authoritative" DNS server, hosting a root zone containing
  10.000.000 A-Records. BIND 9.9.5, Solaris 11, 8 GB. No delegation,
  everything is inside one large root zone. TTL is 86400.

* one DNS resolver for testing Unbound vs. WinDNS. AMD Athlon 64 X2 Dual
  Core 2.31 Ghz 1 Socket 2 Cores 256 L1 Cache 1 MB L2 Cache, 2GB RAM 533
  Mhz DIMM, Ethernet Intel PRO/100S Desktop NIC

All machines were connected via a HP ProCurve 1410 16G Switch

I did 3 tests: Unbound on FreeBSD 10-RELEASE, Unbound on Windows 2012
ServerCore and Windows DNS Server on Windows 2012 ServerCore

Memory was measured by "top" on FreeBSD and by Windows 2012 taskmgr.

Each setup was tested in 2 runs. Run 1 with cold (empty cache) and Run 2
with filled cache (no DNS resolution towards the authoritative).

1 FreeBSD 10 unbound 1.4.20
===========================

  1.000.000 Records

   Run                 Memory  average qps  CPU
   Run 1 (cold cache)                 9000  90 %
   Run 2 (warm cache)  736M          43000  79 %

2 Windows 2012 Core unbound 1.4.22
==================================

  1.000.000 Records

   Run                 Memory  average qps  CPU
   Run 1 (cold cache)                 4113  53%
   Run 2 (warm cache)  398M          30900  50% (63% System)

3 Windows 2012 Core dns.exe 23D99296 (6.2 build 9200)
=====================================================

  1.000.000 Records

   Run                 Memory  average qps  CPU
   Run 1 (cold cache)                 3160  15-60%
   Run 2 (warm cache)  747M          37443  78% (98% System)

Observations:

* According to Sysinternals "process explorer"
  (http://technet.microsoft.com/en-us/sysinternals/bb896653.aspx),
  Unbound on Windows is a single threaded application. It does not make
  use of multiple cores.

* According to the same "process explorer", dns.exe is
  multi-threaded. 

* dns.exe qps throughput and CPU usage is fluctuating while resolving to the
  authoritative servers. QPS was between 1000-6000 with an average on
  3160 during this time. Unbound is more steady while doing resolving.

* dns.exe uses more memory during resolution (up to 1.2 GB seen), but
  frees memory after getting into stable state (all queries cached)

* dns.exe is faster when only answering from cache (probably due to
  multi-threading), Unbound is faster resolving from the authoritative
  servers.

* memory allocation on Windows seems to be more resourceful compared to
  FreeBSD.

I will re-run these tests on a 4 Core Dell R210 Server later this month
and report the numbers here.

Please let me know if you see flaws in my measurement setup (there are
probably some) and how to fix them.

Best regards

Carsten

Olaf Kolkman writes:

> Folk,
>
> We recently got a query from somebody in the industry asking how Unbound compares to Windows Resolver performance in terms of qps and CPU (memory is under user control).
>
> All our benchmarking is Unix based and we never really did a comparison ourselves. Is there somebody on this list that has performed such measurements, or knows of a reference?


-- 
Carsten Strotmann
Email: cas at strotmann.de
Blog: strotmann.de