<div dir="ltr">Hi Ralph,<br><br>Thank you for response, do dump_cache with cron it's good idea, also probably I can merge it with command to get request_list and join all of it with my cron to collect tcpdump traffic<br>But now, I know next: <br>amazon doesn't like NXDOMAIN records, if query arrived to nonexistent domain, unbound forwards this query to aws vpc dns server and aws spend a lot of time to return answer.<br>Probably it can be our issue, but I am not 100% sure<br><br><br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">чт, 11 июл. 2019 г. в 12:36, Ralph Dolmans via Unbound-users <<a href="mailto:unbound-users@nlnetlabs.nl">unbound-users@nlnetlabs.nl</a>>:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Hi Eduard,<br>
<br>
Hard to say why this happens periodically to you. Do you see an increase<br>
in the incoming queries when this happens? Maybe running out of some<br>
buffer space? Or do you by any chance periodically perform an expensive<br>
operation on unbound, like doing a dump_cache from cron? Are there any<br>
errors written to the log?<br>
<br>
-- Ralph<br>
<br>
On 11-07-19 10:34, Eduard Ahmatgareev via Unbound-users wrote:<br>
> Hi everyone,<br>
> <br>
> I faced with intersting issue with unbound server and couldn't figure<br>
> out without your help<br>
> We used unbound as primary dns resolver in our aws infrastructure, but<br>
> from time to time unbound server is not responding to queries from our<br>
> clients<br>
> Also I found by tcpdump and wireshark a lot of retransmission DNS<br>
> requests from clients in the subnets.<br>
> But this issue present periodically, our clients get timeout issue<br>
> during the day.<br>
> from 100 queries, timeout can be get for 3-8 queries.<br>
> <br>
> For debug I used command:<br>
> perf trace -p $(pidof unbound) --duration=10<br>
> and got next:<br>
> 13.285 (599.741 ms): unbound/15943 epoll_pwait(epfd:<br>
> 54<anon_inode:[eventpoll]>, events: 0x564955c6ae10, maxevents: 128,<br>
> timeout: -1, sigsetsize: 8) = -1 EINTR Interrupted system call<br>
> 616.016 (94.403 ms): unbound/15943 epoll_pwait(epfd:<br>
> 54<anon_inode:[eventpoll]>, events: 0x564955c6ae10, maxevents: 128,<br>
> timeout: -1, sigsetsize: 8) = 1<br>
> 710.662 (130.206 ms): unbound/15943 epoll_pwait(epfd:<br>
> 54<anon_inode:[eventpoll]>, events: 0x564955c6ae10, maxevents: 128,<br>
> timeout: -1, sigsetsize: 8) = 1<br>
> 616.649 (224.502 ms): unbound/15952 epoll_pwait(epfd:<br>
> 42<anon_inode:[eventpoll]>, events: 0x7faea89ea7f0, maxevents: 128,<br>
> timeout: -1, sigsetsize: 8) = 1<br>
> 850.606 (112.947 ms): unbound/15952 epoll_pwait(epfd:<br>
> 42<anon_inode:[eventpoll]>, events: 0x7faea89ea7f0, maxevents: 128,<br>
> timeout: -1, sigsetsize: 8) = 1<br>
> 13.453 (1160.129 ms): unbound/15951 epoll_pwait(epfd:<br>
> 37<anon_inode:[eventpoll]>, events: 0x7faea47ca3e0, maxevents: 64,<br>
> timeout: -1, sigsetsize: 8) = 1<br>
> 840.904 (335.113 ms): unbound/15943 epoll_pwait(epfd:<br>
> 54<anon_inode:[eventpoll]>, events: 0x564955c6ae10, maxevents: 128,<br>
> timeout: -1, sigsetsize: 8) = 1<br>
> 710.891 (465.469 ms): unbound/15950 epoll_pwait(epfd:<br>
> 36<anon_inode:[eventpoll]>, events: 0x7faeac8b2680, maxevents: 128,<br>
> timeout: -1, sigsetsize: 8) = 1<br>
> 13.769 (1174.857 ms): unbound/15954 epoll_pwait(epfd:<br>
> 48<anon_inode:[eventpoll]>, events: 0x7fae98747c20, maxevents: 128,<br>
> timeout: -1, sigsetsize: 8) = 1<br>
> 1176.048 (17.121 ms): unbound/15943 epoll_pwait(epfd:<br>
> 54<anon_inode:[eventpoll]>, events: 0x564955c6ae10, maxevents: 128,<br>
> timeout: -1, sigsetsize: 8) = -1 EINTR Interrupted system call<br>
> 1175.740 (21.495 ms): unbound/15951 epoll_pwait(epfd:<br>
> 37<anon_inode:[eventpoll]>, events: 0x7faea47ca3e0, maxevents: 64,<br>
> timeout: -1, sigsetsize: 8) = 1<br>
> 1177.587 (19.955 ms): unbound/15950 epoll_pwait(epfd:<br>
> 36<anon_inode:[eventpoll]>, events: 0x7faeac8b2680, maxevents: 128,<br>
> timeout: 264, sigsetsize: 8) = 1<br>
> 1196.914 (11.097 ms): unbound/15954 epoll_pwait(epfd:<br>
> 48<anon_inode:[eventpoll]>, events: 0x7fae98747c20, maxevents: 128,<br>
> timeout: -1, sigsetsize: 8) = 1<br>
> <br>
> <br>
> <br>
> our infra:<br>
> ec2: c5.2xlarge (16gb mem, 8cores, 60gb gp2) <br>
> dist: amazon linux 2<br>
> <br>
> unbound-libs-1.6.6-1.amzn2.0.2.x86_64<br>
> unbound-python-1.6.6-1.amzn2.0.2.x86_64<br>
> unbound-1.6.6-1.amzn2.0.2.x86_64<br>
> <br>
> conf:<br>
> server:<br>
> verbosity: 1<br>
> num-threads: 8<br>
> statistics-interval: 0<br>
> extended-statistics: yes<br>
> statistics-cumulative: no<br>
> msg-cache-slabs: 4<br>
> rrset-cache-slabs: 4<br>
> infra-cache-slabs: 4<br>
> key-cache-slabs: 4<br>
> rrset-cache-size: 100m<br>
> msg-cache-size: 50m<br>
> so-rcvbuf: 4m<br>
> so-sndbuf: 4m<br>
> so-reuseport: yes<br>
> outgoing-range: 8192<br>
> num-queries-per-thread: 4096<br>
> do-daemonize: no<br>
> prefetch: yes<br>
> rrset-roundrobin: yes<br>
> logfile: ""<br>
> use-syslog: no<br>
> directory: "/etc/unbound"<br>
> chroot: ""<br>
> log-queries: no<br>
> access-control: <a href="http://0.0.0.0/0" rel="noreferrer" target="_blank">0.0.0.0/0</a> <<a href="http://0.0.0.0/0" rel="noreferrer" target="_blank">http://0.0.0.0/0</a>> allow<br>
> interface: 0.0.0.0<br>
> interface-automatic: yes<br>
> port: 53<br>
> do-ip4: yes<br>
> do-ip6: no<br>
> do-udp: yes<br>
> do-tcp: yes<br>
> username: "unbound"<br>
> pidfile: "/var/run/unbound/unbound.pid"<br>
> root-hints: /etc/unbound/root.hints<br>
> key-cache-size: 32m<br>
> local-zone: "10.in-addr.arpa." nodefault<br>
> <br>
> remote-control:<br>
> control-enable: yes<br>
> <br>
> any ideas?<br>
> <br>
</blockquote></div>