Memory tuning with Unbound RPZ implementation

[dns at todoo.biz]

> Le 19 sept. 2022 à 18:15, dns--- via Unbound-users <unbound-users at lists.nlnetlabs.nl> a écrit :
> 
> Hello, 
> 
> We are preparing a special config of unbound which will allow one to use RPZ and benefit from a global GUI. 
> This will be embedded on firewall devices such as APU devices (4GB of RAM and 1GHz quad core AMD-G series ).
> 
> The zones we are dealing with could be of quite large size (largest is 468Mo) in size. 
> 
> We have observed that after downloading the rpz files from our main RPZ server, It loads the server and uses ±3Go of RAM.
> This seems quite strange because the total size of loaded RPZ files in the provided ex. below is 275Mo. 
> 
> 
> last pid: 64525;  load averages:  0.63,  0.57,  0.43   
> up 0+03:03:48  16:05:00   35 processes:  2 running, 33 sleeping
> CPU: 22.0% user,  0.0% nice,  9.1% system,  0.0% interrupt, 68.9% idle
> Mem: 2263M Active, 752M Inact, 2316K Laundry, 422M Wired, 245M Buf, 481M Free
> Swap: 764M Total, 60M Used, 703M Free, 7% Inuse
> 19073 unbound       4  52    0  2924M  2833M kqread   2   7:43   0.00% unbound
> 
> 
> 
> The questions are the following: 
> 
> 1. Is there any specific parameters that we could use in order to minimise the RPZ memory impact ? 
> 
> 2. By default RPZ will use an AXFR zone transfer, can you confirm that this zone transfer is compressed ? (by RFC It should be)
> 
> 3. What would you suggest as a general advise in order to tune the parameters (knowing that the zone content will change but not very often - like once a week)… 
> 
> 
> These are the default parameters we are using: 
> 
> ##
> # Server configuration
> ##
> server:
> chroot: /var/unbound
> username: unbound
> directory: /var/unbound
> pidfile: /var/run/unbound.pid
> root-hints: /var/unbound/root.hints
> use-syslog: yes
> port: 53
> verbosity: 1
> extended-statistics: no
> log-queries: yes
> hide-identity: no
> hide-version: no
> harden-referral-path: no
> do-ip4: yes
> do-ip6: yes
> do-udp: yes
> do-tcp: yes
> do-daemonize: yes
> so-reuseport: yes
> module-config: "iterator"
> cache-max-ttl: 86400
> cache-min-ttl: 0
> harden-dnssec-stripped: no
> serve-expired: no
> outgoing-num-tcp: 10
> incoming-num-tcp: 10
> num-queries-per-thread: 4096
> outgoing-range: 8192
> infra-host-ttl: 900
> infra-cache-numhosts: 10000
> unwanted-reply-threshold: 0
> jostle-timeout: 200
> msg-cache-size: 4m
> rrset-cache-size: 8m
> num-threads: 4
> msg-cache-slabs: 8
> rrset-cache-slabs: 8
> infra-cache-slabs: 8
> key-cache-slabs: 8
> 
> 
> 
> Thanks for your feedback. 

Not so many answers for the time being… 

A short follow-up to this thread. 

After couple of further tests:

Loading zones will "occupy" the processor and put it at 100% for as long as all zones are loaded
About 8'30" to load 276Mo of RPZ zones (4920269 total zones)
Once loaded the processor is at 2924Mo usage
and the load is at 80% globally
I have read this thread: https://github.com/NLnetLabs/unbound/issues/318 <https://github.com/NLnetLabs/unbound/issues/318>
There don't seem to be any memory leak, simply a very slow loading process and a huge usage of memory.

Most probably this is not so optimized in memory.



Comparison with bind

As a comparison, the process loaded on our bind9 (master RPZ server) serves zones for 18063096 total zones and uses 8245348 bytes

So about 8Go or RAM usage.

So the bind9 has a memory performance ratio of : 2190,8

While the unbound has a memory performance ratio of: 1682,7 also the loading process is way faster. 



In other word : I think we should advise using our filtering with firewall devices with a minimum of 8Go of RAM (and this is a real minimum).


If anyone can come up with a way to optimise this specially the loading process. 
I'd be happy to read the comments. 


Sincerely yours. 


—


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.nlnetlabs.nl/pipermail/unbound-users/attachments/20220920/742e831c/attachment-0001.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: LOGO_OCTOPUS_90.png
Type: image/png
Size: 4732 bytes
Desc: not available
URL: <http://lists.nlnetlabs.nl/pipermail/unbound-users/attachments/20220920/742e831c/attachment-0001.png>