increasing memory usage (using rpz zones)

[Ralph Dolmans]

Hi Hanspeter,

On 14-10-2020 23:29, Hanspeter Kunz via Unbound-users wrote:
> Hi all,
> 
> [replying to my own post]
> 
> Apparently it is normal that unbound uses *a lot of RAM* after the
> initial load of the rpz zones (point 1 below).

Does your RPZ zone contain a lot of records with the local data RPZ
action? Due to the way the memory allocation is done here this can
result in a very memory hungry Unbound instance. We are working on a fix
for this.

> 
> The problem 2, however, is still unsolved. But I was able to track down
> the memory leak (point 2 below) somewhat: it only occurs if unbound is
> configured to listen on more than one IP address.
> 
> I filed a bug here:
> https://github.com/NLnetLabs/unbound/issues/318

Thanks, appreciate the detailed report. My first hunch is that this is
related to the zone transfer because of the involvement of the
configured IP addresses. We'll continue working on this and communicate
about this issue on github.

-- Ralph

> 
> Best,
> Hp
> 
> On Tue, 2020-09-29 at 14:25 +0200, Hanspeter Kunz via Unbound-users
> wrote:
>> Hi all,
>>
>> I recently upgraded to unbound 1-11.0 because I needed support for
>> rpz
>> zones. while the rpz zones work fine, I realized rather quickly, that
>> the memory unbound uses is much higher than before and *increases*
>> over
>> time. when I use a rather large rpz zone it actually unbound's memory
>> requirements increases rather dramatically. 
>>
>> These are my observations:
>>
>> ---
>>
>> 1. memory consumption at startup (as reported by pmap)
>> a) starting unbound without any rpz zones: 248M
>> b) starting unbound with all but the problematic rpz zone: 526M
>> c) starting unbound including the problematic rpz zone: 10G (!)
>>
>> the total file size of my rpz zones ist 103M, the problematic rpz
>> zone
>> has a size of 95M. i.e. it is much larger than the other zones. it
>> has
>> ~1 million entries.
>>
>> are such increases in memory consumption to be expected? 
>>
>> restarting unbound for case c) also take a very long time (~20s),
>> which, however, is reasonable.
>>
>> ---
>>
>> 2. increased memory consumption after rpz zone transfers/reloads
>> as soon as an rpz zone is updated (and I assume unbound triggers a
>> transfer and a reload of the zone), memory consumtion increases
>> further, for example
>>
>> startup: 10G
>> 1st reload of the problematic rpz zone: 19G (!!)
>>
>> when one if the smaller rpz zone is reloaded, memory consumption
>> increases too, but only much slower.
>>
>> ---
>>
>> I am running debian stable (buster), unbound 1.11.0 (from debian
>> backports).
>>
>> unbound is configured for 4 threads and I used the suggestions here
>>
>>   https://nlnetlabs.nl/documentation/unbound/howto-optimise/
>>
>> I can send unbound.conf and rpz.conf if that helps, although
>> essentially the same settings worked well for years (unbound 1.6.0-
>> 3).
>>
>> Best,
>> Hp