increasing memory usage (using rpz zones)

[Hanspeter Kunz]

On Fri, 2020-10-16 at 15:56 +0200, Ralph Dolmans via Unbound-users
wrote:
> Hi Hanspeter,
> 
> On 14-10-2020 23:29, Hanspeter Kunz via Unbound-users wrote:
> > Hi all,
> > 
> > [replying to my own post]
> > 
> > Apparently it is normal that unbound uses *a lot of RAM* after the
> > initial load of the rpz zones (point 1 below).
> 
> Does your RPZ zone contain a lot of records with the local data RPZ
> action? Due to the way the memory allocation is done here this can
> result in a very memory hungry Unbound instance. We are working on a
> fix
> for this.

I am not entirely sure what "local data RPZ action" means. almost all
our records in the rpz zones are CNAMES.

> > The problem 2, however, is still unsolved. But I was able to track
> > down
> > the memory leak (point 2 below) somewhat: it only occurs if unbound
> > is
> > configured to listen on more than one IP address.
> > 
> > I filed a bug here:
> > https://github.com/NLnetLabs/unbound/issues/318
> 
> Thanks, appreciate the detailed report. My first hunch is that this
> is
> related to the zone transfer because of the involvement of the
> configured IP addresses. We'll continue working on this and
> communicate
> about this issue on github.

perfect, thanks,
hp

> -- Ralph
> 
> > Best,
> > Hp
> > 
> > On Tue, 2020-09-29 at 14:25 +0200, Hanspeter Kunz via Unbound-users
> > wrote:
> > > Hi all,
> > > 
> > > I recently upgraded to unbound 1-11.0 because I needed support
> > > for
> > > rpz
> > > zones. while the rpz zones work fine, I realized rather quickly,
> > > that
> > > the memory unbound uses is much higher than before and
> > > *increases*
> > > over
> > > time. when I use a rather large rpz zone it actually unbound's
> > > memory
> > > requirements increases rather dramatically. 
> > > 
> > > These are my observations:
> > > 
> > > ---
> > > 
> > > 1. memory consumption at startup (as reported by pmap)
> > > a) starting unbound without any rpz zones: 248M
> > > b) starting unbound with all but the problematic rpz zone: 526M
> > > c) starting unbound including the problematic rpz zone: 10G (!)
> > > 
> > > the total file size of my rpz zones ist 103M, the problematic rpz
> > > zone
> > > has a size of 95M. i.e. it is much larger than the other zones.
> > > it
> > > has
> > > ~1 million entries.
> > > 
> > > are such increases in memory consumption to be expected? 
> > > 
> > > restarting unbound for case c) also take a very long time (~20s),
> > > which, however, is reasonable.
> > > 
> > > ---
> > > 
> > > 2. increased memory consumption after rpz zone transfers/reloads
> > > as soon as an rpz zone is updated (and I assume unbound triggers
> > > a
> > > transfer and a reload of the zone), memory consumtion increases
> > > further, for example
> > > 
> > > startup: 10G
> > > 1st reload of the problematic rpz zone: 19G (!!)
> > > 
> > > when one if the smaller rpz zone is reloaded, memory consumption
> > > increases too, but only much slower.
> > > 
> > > ---
> > > 
> > > I am running debian stable (buster), unbound 1.11.0 (from debian
> > > backports).
> > > 
> > > unbound is configured for 4 threads and I used the suggestions
> > > here
> > > 
> > >   https://nlnetlabs.nl/documentation/unbound/howto-optimise/
> > > 
> > > I can send unbound.conf and rpz.conf if that helps, although
> > > essentially the same settings worked well for years (unbound
> > > 1.6.0-
> > > 3).
> > > 
> > > Best,
> > > Hp
-- 
Hanspeter Kunz                  University of Zurich
Systems Administrator           Department of Informatics
Email: hkunz at ifi.uzh.ch         Binzmühlestrasse 14
Tel: +41.(0)44.63-56714         Office 2.E.07
http://www.ifi.uzh.ch           CH-8050 Zurich, Switzerland

Spamtraps: hkunz.bogus at ailab.ch hkunz.bogus at ifi.uzh.ch
---
A wise man can see more from a mountain top than a fool can from the bottom
of a well.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 5395 bytes
Desc: not available
URL: <http://lists.nlnetlabs.nl/pipermail/unbound-users/attachments/20201020/b237c17e/attachment.bin>