Question about thread logging statistics

Wed May 21 19:20:08 UTC 2025

Thanks Yorgos,

I updated one of the server to 1.23 and now the requestlist stats are 
consistently 0 for exceeded:

    [1747854620] unbound[1:6] info: server stats for thread 6:
    requestlist max 73 avg 63.6041 exceeded 0 jostled 0
    [1747854620] unbound[1:2] info: server stats for thread 2:
    requestlist max 71 avg 59.7928 exceeded 0 jostled 0
    [1747854620] unbound[1:3] info: server stats for thread 3:
    requestlist max 72 avg 64.8742 exceeded 0 jostled 0

Increasing *max-sent-count* definitely resolved for our initial problem 
where the DNSSEC records were failing for some domains that are 
frequently used by DNS clients.

This unbound server forwards all requests for "domain.com" to a set of 
internal DNS servers, and I can see in the logs that there are a couple 
of valid hostnames within that domain, however subdomains like 
"dev.domain.com" are probably configured on those internal nameservers 
to forward to some IP addresses that were retired a couple of years ago. 
I'm working with the admin of those nameservers to verify which DNS 
records are valid and hopefully stop forwarding requests to dead IP 
addresses.

-Mike Durkin

On 5/21/25 8:00 AM, unbound-users-request at lists.nlnetlabs.nl wrote:
>
> ----------------------------------------------------------------------
>
> Message: 1
> Date: Tue, 20 May 2025 12:37:50 -0400
> From: Mike Durkin<mike at celtic-dreamer.com>
> To:unbound-users at lists.nlnetlabs.nl
> Subject: Question about thread logging statistics
> Message-ID:<c0dedfce-6683-4ca4-b969-0dd13b73773f at celtic-dreamer.com>
> Content-Type: text/plain; charset="utf-8"; Format="flowed"
>
> Hi
>
> We are using unbound docker containers (version 1.22) in our corporate
> environment and after fixing an issue with DNSSEC records, I wanted to
> ask about some of the logging statistics to see if there still might be
> an performance issue.
>
> Last week, we were getting reports of certain domains not resolving and
> I saw error messages like the following in the logs:
>
>   ? ? [1747490624] unbound[1:2] info: validation failure
> <wpad.domain.com. A IN>: SERVFAIL [exceeded the maximum number of sends]
> no DS for DS domain.com. while building chain of trust
>   ??? [1747493945] unbound[1:1] error: SERVFAIL <wpad.domain.com. A IN>:
> exceeded the maximum number of sends
>
> I ended up adding the following which seemed to resolve the issue:
>
>   ??? ??? max-sent-count: 200
>
> I had tried some lower values initially, but that didn't resolve the
> problem until I bumped it up to 200.
>
>
> So at the moment we are not getting any reports for DNS client failues,
> but I am seeing the following in the logs:
>
>      [1747757273] unbound[1:0] info: server stats for thread 0:
>      requestlist max 78 avg 68.4251 exceeded 84 jostled 0
>      [1747757333] unbound[1:0] info: server stats for thread 0:
>      requestlist max 72 avg 66.9528 exceeded 55 jostled 0
>      [1747757393] unbound[1:0] info: server stats for thread 0:
>      requestlist max 78 avg 66.9892 exceeded 62 jostled 0
>
>
> The thread server stats is always showing a significant number for
> exceeded. The host where the container is running is not overloaded. I
> do see in the logs that there are a significant number of requests for
> legacy subdomains that are no longer in use and cause error messages
> like the following:
>
>      [1747758108] unbound[1:0] error: SERVFAIL <db01-dev.dev.domain.com.
>      A IN>: all the configured stub or forward servers failed, at zone
>      domain.com. from 10.10.32.2 got SERVFAIL
>
> My main question is, would those requests that are being forwarded and
> timing out with a client error "no servers could be reached" be a source
> for the "exceeded" count in the thread server stats?
>
> Thanks,
>
> -Mike Durkin
>
>
>
>
>
> -------------- next part --------------
> An HTML attachment was scrubbed...
> URL:<http://lists.nlnetlabs.nl/pipermail/unbound-users/attachments/20250520/ea79f3c0/attachment-0001.htm>
>
> ------------------------------
>
> Message: 2
> Date: Wed, 21 May 2025 10:24:15 +0200
> From: Yorgos Thessalonikefs<yorgos at nlnetlabs.nl>
> To:unbound-users at lists.nlnetlabs.nl
> Subject: Re: Question about thread logging statistics
> Message-ID:<09d7a0de-cc2d-44fc-8b88-87fecf79959d at nlnetlabs.nl>
> Content-Type: text/plain; charset=UTF-8; format=flowed
>
> Hi Mike,
>
> The "exceeded" number are queries that were dropped because the request
> list (queries from clients) was full.
>
> However versions 1.21.0 up to and including 1.23.0 wrongfully use
> the statistic for queries that exceed the discard-timeout [1] and/or
> wait-limit [2] options.
>
> Version 1.23.0 fixes that by introducing an explicit counter for that
> accessible from the 'stats' command (total.num.queries_discard_timeout
> and total.num.queries_wait_limit [3]) and stop counting those drops in
> the "exceeded".
>
> What I believe happens in your case is because you increased
> max-sent-count to 200, those queries are now slow to respond back and
> Unbound drops the replies to those clients because discard-timeout is
> exceeded, or because they are slow those clients exceed their wait-limit.
> (And wrongfully counts those in the "exceeded" in the log output)
>
> Btw, did increasing max-sent-count actually help in your case?
>
> Is your Unbound configured specially for domain.com or it just uses a
> '.' forwarder?
> I mainly am asking about the last error log you shared.
>
> Best regards,
> -- Yorgos
>
> [1]
> https://unbound.docs.nlnetlabs.nl/en/latest/manpages/unbound.conf.html#unbound-conf-discard-timeout"
> [2]
> https://unbound.docs.nlnetlabs.nl/en/latest/manpages/unbound.conf.html#unbound-conf-wait-limit
> [3]
> https://unbound.docs.nlnetlabs.nl/en/latest/manpages/unbound-control.html#statistic-counters
>
> On 20/05/2025 18:37, Mike Durkin via Unbound-users wrote:
>> Hi
>>
>> We are using unbound docker containers (version 1.22) in our corporate
>> environment and after fixing an issue with DNSSEC records, I wanted to
>> ask about some of the logging statistics to see if there still might be
>> an performance issue.
>>
>> Last week, we were getting reports of certain domains not resolving and
>> I saw error messages like the following in the logs:
>>
>>   ? ? [1747490624] unbound[1:2] info: validation failure
>> <wpad.domain.com. A IN>: SERVFAIL [exceeded the maximum number of sends]
>> no DS for DS domain.com. while building chain of trust
>>   ??? [1747493945] unbound[1:1] error: SERVFAIL <wpad.domain.com. A IN>:
>> exceeded the maximum number of sends
>>
>> I ended up adding the following which seemed to resolve the issue:
>>
>>   ??? ??? max-sent-count: 200
>>
>> I had tried some lower values initially, but that didn't resolve the
>> problem until I bumped it up to 200.
>>
>>
>> So at the moment we are not getting any reports for DNS client failues,
>> but I am seeing the following in the logs:
>>
>>      [1747757273] unbound[1:0] info: server stats for thread 0:
>>      requestlist max 78 avg 68.4251 exceeded 84 jostled 0
>>      [1747757333] unbound[1:0] info: server stats for thread 0:
>>      requestlist max 72 avg 66.9528 exceeded 55 jostled 0
>>      [1747757393] unbound[1:0] info: server stats for thread 0:
>>      requestlist max 78 avg 66.9892 exceeded 62 jostled 0
>>
>>
>> The thread server stats is always showing a significant number for
>> exceeded. The host where the container is running is not overloaded. I
>> do see in the logs that there are a significant number of requests for
>> legacy subdomains that are no longer in use and cause error messages
>> like the following:
>>
>>      [1747758108] unbound[1:0] error: SERVFAIL <db01-dev.dev.domain.com.
>>      A IN>: all the configured stub or forward servers failed, at zone
>>      domain.com. from 10.10.32.2 got SERVFAIL
>>
>> My main question is, would those requests that are being forwarded and
>> timing out with a client error "no servers could be reached" be a source
>> for the "exceeded" count in the thread server stats?
>>
>> Thanks,
>>
>> -Mike Durkin
>>
>>
>>
>>
>>
>>
>
>
> ------------------------------
>
> Subject: Digest Footer
>
> _______________________________________________
> Unbound-users mailing list
> Unbound-users at lists.nlnetlabs.nl
> https://lists.nlnetlabs.nl/mailman/listinfo/unbound-users
>
>
> ------------------------------
>
> End of Unbound-users Digest, Vol 65, Issue 5
> ********************************************

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.nlnetlabs.nl/pipermail/unbound-users/attachments/20250521/3d599cc4/attachment.htm>