[Unbound-users] Is there any memory leak in unbound?
Tao Ma
matao_reg at yahoo.com.cn
Thu Sep 17 09:47:09 UTC 2009
Hi Wouter,I'm very glad to tell you,I patched the diff file you gave me and now unbound works well on the server,thanks very much!
I'm sorry that I didn't tell you clearly about the compile problem.When we found we couldn't compile the code with "--enable-alloc-checks",we added another option "--enable-static-exe" and then finished compiling smoothly,that's all.
Thanks again for your warmhearted help.
Best Regards,
Tao Ma
--- 09年9月16日，周三, W.C.A. Wijngaards <wouter at nlnetlabs.nl> 写道：
发件人: W.C.A. Wijngaards <wouter at nlnetlabs.nl>
主题: Re: [Unbound-users] Is there any memory leak in unbound?
收件人: "Tao Ma" <matao_reg at yahoo.com.cn>
日期: 2009年9月16日,周三,下午8:47
Hi Tao Ma,
Does this solve your memory leak problem?
Index: libunbound/libworker.c
===================================================================
--- libunbound/libworker.c (revision 1831)
+++ libunbound/libworker.c (working copy)
@@ -514,6 +514,7 @@
ldns_buffer_write_u16_at(w->back->udp_buff, 2, qflags);
if(local_zones_answer(ctx->local_zones, &qinfo, &edns,
w->back->udp_buff, w->env->scratch)) {
+ regional_free_all(w->env->scratch);
libworker_fillup_fg(q, LDNS_RCODE_NOERROR,
w->back->udp_buff, sec_status_insecure);
libworker_delete(w);
@@ -630,6 +631,7 @@
ldns_buffer_write_u16_at(w->back->udp_buff, 2, qflags);
if(local_zones_answer(w->ctx->local_zones, &qinfo, &edns,
w->back->udp_buff, w->env->scratch)) {
+ regional_free_all(w->env->scratch);
q->msg_security = sec_status_insecure;
add_bg_result(w, q, w->back->udp_buff, UB_NOERROR);
free(qinfo.qname);
Index: daemon/worker.c
===================================================================
--- daemon/worker.c (revision 1831)
+++ daemon/worker.c (working copy)
@@ -806,6 +806,7 @@
}
if(local_zones_answer(worker->daemon->local_zones, &qinfo, &edns,
c->buffer, worker->scratchpad)) {
+ regional_free_all(worker->scratchpad);
if(ldns_buffer_limit(c->buffer) == 0) {
comm_point_drop_reply(repinfo);
return 0;
Sorry about the compile problem with unbound-host and alloc-checks, I'll
look at it later.
Best regards,
Wouter
On 09/16/2009 10:05 AM, Tao Ma wrote:
> Hi Wouter,thanks for your reply.
>
> We use a 32-bit Gentoo and I don't think there's a special memory
> allocator in libc.
> I can always find the memory consumed by unbound keep growing from ouput
> of 'ps' and 'top',the result of sbrk estimate seems to be in accord with
> this.
>
> Following your suggestion,we configure with --enable-alloc-checks and
> compile unbound again,now we can find the "Memory conditions" value keep
> growing in the log.
>
> We use memstats to parse the log,but found nothing but the following ouput:
> Reading /var/unbound/log_test of size 38767531
> 1% 2% 3% 4% 5% 6% 7% 8% 9% 10% 11% 12% 13% 14% 15% 16% 17% 18% 19% 20%
> 21% 22% 23% 24% 25% 26% 27% 28% 29% 30% 31% 32% 33% 34% 35% 36% 37% 38%
> 39% 40% 41% 42% 43% 44% 45% 46% 47% 48% 49% 50% 51% 52% 53% 54% 55% 56%
> 57% 58% 59% 60% 61% 62% 63% 64% 65% 66% 67% 68% 69% 70% 71% 72% 73% 74%
> 75% 76% 77% 78% 79% 80% 81% 82% 83% 84% 85% 86% 87% 88% 89% 90% 91% 92%
> 93% 94% 95% 96% 97% 98% 99% 100% done
> ------------
> 0 / 0 total in 0 code lines
>
> We got a complete record in the log from the start of unbound to the
> exiting of unbound,and the log file has been attatched to the mail,maybe
> it will help you to analyse.
>
> Let me know if you find any mistakes in my operation.
>
> By the way,when we configure unbound-1.3.3 with "./configure
> --prefix=/usr/local/unbound --enable-alloc-checks" and compile,we got
> the following error:
> build/smallapp/.libs/unbound-host.o: In function `print_rd':
> /home/mt/tools/unbound-1.3.3/smallapp/unbound-host.c:233: undefined
> reference to `unbound_stat_malloc_log'
> /home/mt/tools/unbound-1.3.3/smallapp/unbound-host.c:255: undefined
> reference to `unbound_stat_free_log'
> build/smallapp/.libs/unbound-host.o: In function `pretty_rdata':
> /home/mt/tools/unbound-1.3.3/smallapp/unbound-host.c:247: undefined
> reference to `unbound_stat_free_log'
> build/smallapp/.libs/unbound-host.o: In function `lookup':
> /home/mt/tools/unbound-1.3.3/smallapp/unbound-host.c:384: undefined
> reference to `unbound_stat_free_log'
> collect2: ld returned 1 exit status
> make: *** [unbound-host] Error 1
>
> We did some modification to the code and finished compiling.Is there
> anything wrong in the code?
>
> Thanks & Best regards,
> Tao Ma
>
>
>
> --- *09年9月15日，周二, W.C.A. Wijngaards /<wouter at NLnetLabs.nl>/* 写道：
>
>
> 发件人: W..C.A. Wijngaards <wouter at NLnetLabs.nl>
> 主题: Re: [Unbound-users] Is there any memory leak in unbound?
> 收件人: "W.C.A. Wijngaards" <wouter at nlnetlabs.nl>
> 抄送: "Tao Ma" <matao_reg at yahoo.com.cn>, unbound-users at unbound.net
> 日期: 2009年9月15日,周二,下午5:29
>
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> Hi Tao Ma,
>
> Just realized something else, the sbrk estimate is a dirty hack.
> Gentoo may report a different sbrk() value, and the dirty hack
> does not work. The sbrk() value keeps growing but maybe it is
> not true that unbound's memory keeps growing. Does the 'top' or
> 'ps' memory number also grow?
>
> Best regards,
> Wouter
>
> On 09/15/2009 11:26 AM, W.C.A. Wijngaards wrote:
> > Hi Tao Ma,
> >
> >> debug: cache memory msg=16528 rrset=33040 infra=672 val=24888
> >
> > The values from your log file look very reasonable. No problems here.
> > The sbrk keeps growing?
> >
> > Ok, it is possible to enable even more memory debug, configure with
> > --enable-alloc-checks and compile again. Unbound prints:
> > info: Memory conditions: %u front=%u back=%u mesh=%u msg=%u rrset=%u
> > infra=%u iter=%u val=%u alloccache=%u globalalloccache=%u me=%u
> > But with values.
> >
> > It also prints a log line for *every* alloc call. So, you can see
> where
> > in the code it allocates all that memory. This will make the
> > server very slow. If you send me the log file (off-list to
> > wouter at nlnetlabs.nl </mc/compose?to=wouter at nlnetlabs.nl> , then I
> can parse it with testcode/memstats.c;
> > or you can do that yourself: make memstats ; ./memstats < logfile ).
> >
> > To answer your question, unbound keeps some memory around, up to a
> > maximum, because free-ing something that you need later again is
> slow.
> > This should top out at 40Mb for the standard config.
> >
> > Gentoo? Is it 64bit? Does it use a 'special' memory allocator in
> libc?
> >
> > Best regards,
> > Wouter
> >
> > On 09/11/2009 10:06 AM, Tao Ma wrote:
> >> What's more,I found that the memory occupied by unbound would not be
> >> freed when all requets finished,that means,unbound only consume
> memory
> >> but never free them.
> >> What does unbound hold the memory for?It quite like that a
> memory leak
> >> happens...
> >> Anyone else has been in the same trouble?
> >
> >
> >> Best Regards,
> >> Tao Ma
> >
> >
> >> --- *09t911氓h聰, Tao Ma /<matao_reg at yahoo.com.cn
> </mc/compose?to=matao_reg at yahoo.com.cn>>/* 聶S
> >
> >
> >> 脩枚潞: Tao Ma <matao_reg at yahoo.com.cn
> </mc/compose?to=matao_reg at yahoo.com.cn>>
> >> ;聵: Re: [Unbound-users] Is there any memory leak in unbound?
> >> 6枚潞: "W.C.A. Wijngaards" <wouter at nlnetlabs.nl
> </mc/compose?to=wouter at nlnetlabs.nl>>
> >> 聞: unbound-users at unbound.net
> </mc/compose?to=unbound-users at unbound.net>
> >> 氓: 2009t911氓,h聰,
> > H11:50
> >
> >> Hi Wouter,thanks very much for your reply.
> >
> >> I tried the method you mentioned,here is the info I got from the
> log:
> >> debug: cache memory msg=16528 rrset=33040 infra=672 val=24888
> >> This info keeps the same from the start of the unbound(I have only
> >> about 5 records in unbound).
> >
> >> I also found the value of mem..total.sbrk printed by
> "unbound-control
> >> stats" keeps growing.The item "mem.total.sbrk" is an estimate of the
> >> heap size of unbound in bytes,it is close to the memory used by
> unbound.
> >
> >> I have a script keeps recording mem.total.sbrk in a file,and found
> >> when mem.total.sbrk grew to 2017873920,unbound was killed by
> >> system(My server has a 2G memory).
> >
> >> Unbound runs in a Gentoo Linux with 2.6.23 kernel,does the problem
> >> relate to the OS?
> >
> >> Thanks & Best regards,
> >> Tao Ma
> >
> >
> >
> >> --- *09t910氓h脹, W.C.A. Wijngaards /<wouter at NLnetLabs.nl
> </mc/compose?to=wouter at NLnetLabs.nl>>/*
> >> 聶S
> >
> >
> >> 脩枚潞: W.C.A. Wijngaards <wouter at NLnetLabs.nl
> </mc/compose?to=wouter at NLnetLabs.nl>>
> >> ;聵: Re: [Unbound-users] 脼 Re: Is there any memory leak in
> >> unbound?
> >> 6枚潞: "Tao Ma" <matao_reg at yahoo.com.cn
> </mc/compose?to=matao_reg at yahoo.com.cn>>
> >> 聞: "Paul Wouters" <paul at xelerance.com
> </mc/compose?to=paul at xelerance.com>>, unbound-users at unbound.net
> </mc/compose?to=unbound-users at unbound.net>
> >> 氓: 2009t910氓,h脹,H10:25
> >
> >> -----BEGIN PGP SIGNED MESSAGE-----
> >> Hash: SHA1
> >
> >> Hi Tao Ma,
> >
> >> The memory statistics that unbound makes can help?
> >
> >> It prints them into the log file (depending on verbosity
> >> level and --enable-debug to configure) like this:
> >> debug: cache memory msg=1620127 rrset=2964258 infra=738306
> >> val=128832
> >
> >> Also the memory statistics are printed when you enable extended
> >> statistics and do unbound-control stats, like this:
> >> mem.cache.rrset=2980222
> >> mem.cache.message=1633031
> >> mem.mod.iterator=16460
> >> mem.mod.validator=130126
> >
> >> Best regards,
> >> Wouter
> >
> >
> >>
> ------------------------------------------------------------------------
> >> }漏:aI`脩庐卤:ah掳
> > 驴
> >>
> <http://cn.rd.yahoo.com/mail_cn/tagline/card/*http://card.mail.cn.yahoo.com/>
> >
> >> -----b:D枚聟鹿-----
> >
> >> _______________________________________________
> >> Unbound-users mailing list
> >> Unbound-users at unbound.net
> </mc/compose?to=Unbound-users at unbound.net>
> </mc/compose?to=Unbound-users at unbound.net
> </mc/compose?to=Unbound-users at unbound.net>>
> >> http://unbound.nlnetlabs.nl/mailman/listinfo/unbound-users
> >
> >
> >>
> ------------------------------------------------------------------------
> >> }漏:aI`脩庐卤:ah掳
> > 驴
> >>
> <http://cn.rd.yahoo.com/mail_cn/tagline/card/*http://card.mail.cn.yahoo.com/>
> >
> >
> _______________________________________________
> Unbound-users mailing list
> Unbound-users at unbound.net </mc/compose?to=Unbound-users at unbound.net>
> http://unbound.nlnetlabs.nl/mailman/listinfo/unbound-users
>
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.9 (GNU/Linux)
> Comment: Using GnuPG with Fedora - http://enigmail.mozdev.org/
>
> iEYEARECAAYFAkqvXmkACgkQkDLqNwOhpPisoACeKatx2Pl4/B7k5MtJ86KTPBaR
> 2WkAn0sd6bW9UDjCbdQ4pOLHW2WiFOg+
> =NrkI
> -----END PGP SIGNATURE-----
>
>
> ------------------------------------------------------------------------
> 好玩贺卡等你发，邮箱贺卡全新上线！
> <http://cn.rd.yahoo.com/mail_cn/tagline/card/*http://card.mail.cn.yahoo.com/>
>
___________________________________________________________
好玩贺卡等你发，邮箱贺卡全新上线！
http://card.mail.cn.yahoo.com/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.nlnetlabs.nl/pipermail/unbound-users/attachments/20090917/d4dd1c52/attachment.htm>
More information about the Unbound-users
mailing list