<html><head><meta http-equiv="Content-Type" content="text/html charset=us-ascii"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;">Hi Wouter, <div class=""><br class=""></div><div class=""><br class=""><blockquote type="cite" class="">On Apr 6, 2016, at 2:49 PM, W.C.A. Wijngaards <<a href="mailto:wouter@nlnetlabs.nl" class="">wouter@nlnetlabs.nl</a>> wrote:<br class=""><br class="">Signed PGP part<br class="">Hi Kabindra,<br class=""><br class="">I have not heard of this before, how is TCP affecting NSD?</blockquote>After couple thousand of TCP queries, NSD goes unresponsive for both TCP and UDP. </div><div class="">[kabindra@1 ~]$ dig @`hostname` -p 5350 ch txt hostname.bind<br class=""><br class="">; <<>> DiG 9.8.1 <<>> @<replaced> -p 5350 ch txt hostname.bind<br class="">; (2 servers found)<br class="">;; global options: +cmd<br class="">;; connection timed out; no servers could be reached<br class="">[kabindra@1 ~]$ dig @`hostname` -p 5350 ch txt hostname.bind +tcp<br class=""><br class="">; <<>> DiG 9.8.1 <<>> @ <replaced> -p 5350 ch txt hostname.bind +tcp<br class="">; (2 servers found)<br class="">;; global options: +cmd<br class="">;; connection timed out; no servers could be reached</div><div class=""><br class=""></div><div class="">One thing we noticed, we have set the server-count to 4, so it should have 4 child process forked, right? when NSD goes unresponsive, we see couple of <defunct> process and more than 4 child processes. </div><div class="">also, these NSD processes are using lots of CPU. I have left this box out of service for almost 2 days now after going unresponsive but you can see the cpu usage on the below image, it's not coming down.</div><div class=""><br class=""></div><div class=""><img apple-inline="yes" id="AAF272C8-B23C-4AA6-8880-3A2C0B34A70E" height="261" width="569" apple-width="yes" apple-height="yes" src="cid:577B3730-0AF8-490E-98EB-13C22B1538FF@ktm-office.woodynet.net" class=""></div><div class=""><br class=""></div><div class=""><img apple-inline="yes" id="641AA326-A9D8-4A59-AEAC-403166AF5794" height="266" width="570" apple-width="yes" apple-height="yes" src="cid:36FE66B7-AAE8-4A01-A75F-941F08298B32@ktm-office.woodynet.net" class=""></div><div class=""><br class=""></div><div class=""><br class=""><blockquote type="cite" class=""> </blockquote><br class=""><blockquote type="cite" class=""> NSD has a<br class="">fixed number of tcp connections, configured in tcp-count: 100 from the<br class="">nsd.conf file. That should be what is services. You should increase<br class="">that count to increase responsiveness to TCP.<br class=""></blockquote><div class="">Yes, that's what we changed earlier to increase responsiveness to TCP. </div><br class=""><blockquote type="cite" class=""><br class="">UDP should be unaffected.<br class=""></blockquote><div class="">That is not the case we are seeing. </div><br class=""><blockquote type="cite" class=""><br class="">The backlog is for tcp connections waiting to be accepted. 256 is<br class="">reasonably portable, reasonably large. I don't see how that value is<br class="">your problem.</blockquote><div class="">It has been so far and should be true for most of the users but recently with the increase in TCP traffic, I doubt that's still the case. With the RRL implemented I believe it's going to increase some amount of TCP traffic than what it used to be, right?<br class=""></div><div class="">So say if I increase the number of tcp-counts to 1024 but my backlog is set to 256, will I still be able to get 1024 connections at a time or will I be limited to 256 connections concurrently?</div><br class=""><blockquote type="cite" class=""> Is your kernel and networking subsystem failing?<br class=""></blockquote><div class=""><br class=""></div><div class="">I don't think so, if it was the problem I would see problem for other services on that server as well, right?</div><div class=""><br class=""></div><br class=""><blockquote type="cite" class=""><br class="">The OS can return EMFILE or ENFILE to accept(), nsd starts to stop<br class="">accepting TCP connections to relieve buffer stress on the OS. But<br class="">again, UDP should not have been impacted?<br class=""></blockquote><div class="">Again, that's not the case we are seeing. </div><br class=""><blockquote type="cite" class=""><br class="">Are you using so-reuseport: yes?</blockquote><div class="">Nope.</div><div class=""><br class=""></div><div class=""><br class=""></div><blockquote type="cite" class=""> I have had reports that it disrupts<br class="">connectivity (depending on OS, particular version of the OS, and more<br class="">recent versions of NSD do not use reuseport on TCP anymore).<br class=""></blockquote><div class=""><br class=""></div><div class="">Sorry, forgot to mention earlier, we are on CentOS 6 and NSD 4.1.8. </div><div class=""><br class=""></div><div class="">Thanks. </div><br class=""><blockquote type="cite" class=""><br class="">Best regards, Wouter<br class=""><br class="">On 05/04/16 18:28, Kabindra Shrestha wrote:<br class="">> Hi,<br class="">><br class="">> We are seeing some large number of TCP connections to our DNS<br class="">> servers (in thousands) and NSD goes unresponsive after certain time<br class="">> and doesn't recover, it stops responding to UDP as well. We tried<br class="">> increasing the number of tcp-counts but it doesn't help. I noticed<br class="">> the TCP backlog is hardcoded to 256 in NSD config, so even with<br class="">> customised TCP backlogs on the system its still being throttled at<br class="">> around 256. Is there anyway we can change this value without<br class="">> recompiling the NSD.<br class="">><br class="">><br class="">> [kabindra@05 nsd-4.1.8]$ grep BACKLOG * config.h.in:#undef<br class="">> TCP_BACKLOG configure:#define TCP_BACKLOG 256<br class="">> configure.ac:AC_DEFINE_UNQUOTED([TCP_BACKLOG], [256], [Define to<br class="">> the backlog to be used with listen.])<br class="">><br class="">><br class="">> We are using NSD4.1.8.<br class="">><br class="">> ( From one of the servers that went unresponsive, we have seen that<br class="">> TCP number closing to 10k. )<br class="">><br class="">> #ss -s Total: 5591 (kernel 5640) TCP: 5067 (estab 4968, closed 4,<br class="">> orphaned 0, synrecv 0, timewait 3/0), ports 28<br class="">><br class="">> Transport Total IP IPv6 *<span class="Apple-tab-span" style="white-space:pre"> </span> 5640 - - RAW<br class="">> 0 0 0 UDP<span class="Apple-tab-span" style="white-space:pre"> </span> 122 63 59 TCP<span class="Apple-tab-span" style="white-space:pre"> </span> 5063<br class="">> 5017 46 INET<span class="Apple-tab-span" style="white-space:pre"> </span> 5185 5080 105 FRAG<span class="Apple-tab-span" style="white-space:pre"> </span> 0 0<br class="">> 0<br class="">><br class="">><br class="">> Thanks.<br class="">><br class="">> Regards, Kabindra Shrestha<br class="">><br class="">><br class="">><br class="">> _______________________________________________ nsd-users mailing<br class="">> list <a href="mailto:nsd-users@NLnetLabs.nl" class="">nsd-users@NLnetLabs.nl</a><br class="">> <a href="https://open.nlnetlabs.nl/mailman/listinfo/nsd-users" class="">https://open.nlnetlabs.nl/mailman/listinfo/nsd-users</a><br class="">><br class=""><br class="">_______________________________________________<br class="">nsd-users mailing list<br class=""><a href="mailto:nsd-users@NLnetLabs.nl" class="">nsd-users@NLnetLabs.nl</a><br class="">https://open.nlnetlabs.nl/mailman/listinfo/nsd-users<br class=""></blockquote><br class=""><div class="">Regards,<br class="">Kabindra Shrestha<br class=""></div><br class=""></div></body></html>