From paul at xelerance.com  Tue Apr 12 18:15:56 2011
From: paul at xelerance.com (Paul Wouters)
Date: Tue, 12 Apr 2011 14:15:56 -0400 (EDT)
Subject: [ldns-users] ldns-python bug in resolver.get_addr_by_name()
Message-ID: <alpine.LFD.1.10.1104121413380.23536@newtla.xelerance.com>


The following works:

>>> import ldns
>>> resolver = ldns.ldns_resolver.new_frm_file("/etc/resolv.conf")
>>> dnn = ldns.ldns_dname("www.google.com")
>>> addr = resolver.get_addr_by_name(dnn)
>>> for rr in addr.rrs():
...     print rr
... 
www.l.google.com.	300	IN	A	74.125.224.50

[..]

However, this does not work:

>>> addr = resolver.get_addr_by_name("www.google.com")
Traceback (most recent call last):
   File "<stdin>", line 1, in <module>
   File "/usr/lib/python2.7/site-packages/ldns.py", line 3898, in get_addr_by_name
     return _ldns.ldns_get_rr_list_addr_by_name(self, name, aclass, flags)
TypeError: in method 'ldns_get_rr_list_addr_by_name', argument 2 of type 'ldns_rdf *'
>>>

The docs claim that get_addr_by_name() can do the string->dname conversion....

Paul


From bedrich.kosata at nic.cz  Wed Apr 13 06:45:55 2011
From: bedrich.kosata at nic.cz (Bedrich Kosata)
Date: Wed, 13 Apr 2011 08:45:55 +0200
Subject: [ldns-users] pyldns - memory leaks and double freeing
In-Reply-To: <4D8C56E1.7040301@nic.cz>
References: <4D8C56E1.7040301@nic.cz>
Message-ID: <4DA546A3.5030600@nic.cz>

Hi again,

as nobody seemed to have anything against my proposal, I prepared the 
patches that should fix the problems I outlined before :)

When the patches are applied, the python bindings should always have 
rr_list and rr structs separated in memory. This is accomplished by 
always cloning rr data when either adding them to an rr_list or 
retrieving them from the rr_list.
Most of the changes should be transparent and should not require 
modifications to existing code. The only outside change I had to make 
was in the "ldns_verify_rrsig_keylist_notime" and 
"ldns_verify_rrsig_keylist" functions, because it copies pointers to the 
correct keys into a second rr_list. I solved this by replacing each of 
these functions by two new ones (thus creating four functions). All 
versions take only 3 arguments and differ in the way they treat the 
good_keys value. One versions name ends with "_status_only" (e.g. 
ldns_verify_rrsig_keylist_notime_status_only) and this does not report 
the good_keys at all, just the result of verification. The other ends 
with "_" (e.g. ldns_verify_rrsig_keylist_notime_) and returns a tuple 
(status, good_keys) where good_keys are indexes of the keys in the key 
list supplied to the function. Thus the whole business of two rr_lists 
sharing the same data is sidestepped.

The patches are made for each file separately because the fixes are 
mostly independent. I also included several test scripts which should 
show how the situation was fixed. They either show a situation where the 
old version would crash with a segmentation fault or display the amount 
of memory the app has used thus pointing to a memory leak. When used 
with patched sources, none of the scripts should crash and none should 
report more that ~30MB of memory used.

Best regards

Beda

p.s.- the patch for ldns_rdf.i is in fact not related to the rest and 
fixes and unrelated memory leak in ldns_rdf.new_frm_str

p.p.s.- I did not touch a related issue of rr and rdf relationship. In 
current version, the rdf data does not live beyond the lifetime of the 
corresponding rr. This is demonstrated in the attached 
test-rrlist-get-rdf.py. The problem could again be solved by cloning all 
rdfs on retrieval and addition from/into an rr.
However, I will wait how the current patch is received before I start 
thinking about fixing it.


On 03/25/2011 09:48 AM, Bedrich Kosata wrote:
> Hi everybody,
>
> while trying to find a cause of a memory leak in a simple script, I
> found a nest of memory related issues in the python bindings.
> The problems are all related to one common problem - who takes care of
> memory of composite objects, such as ldns_rr_lists or ldns_pkt.
> For example, in the current version, ldns_pkt bindings use ldns_pkt_free
> to free a packet structure, however, when a rr_list is taken from the
> packet and returned from a function, the packet gets out of scope, is
> freed and the rr_list refers to already freed memory.
> On the other hand, a rr_list only frees its own memory, not memory of
> the stored rrs. This can lead to memory leaks.
> I am attaching two scripts that demonstrate these problems (it might be
> necessary to have the sources patched with the "freeing None" patch I
> sent last week).
> I would be willing to have a stab at the problems (provided I get
> clearance from my boss :)), but the only solution I think would be clean
> enough, is to clone the necessary bits where needed. This might lead to
> some inefficiency and slowdowns (probably not big), so I would like to
> ask if this is ok.
> If there is anyone else willing to fix this, I would be happy to act as
> a tester.
>
> Cheers
>
> Beda
>
> p.s.- test-pkt-free.py crashes with a segmentation fault,
> test-rr-list.py ends up eating about 130 MB of memory and more than
> 400000 python objects which python cannot free.
>
>
>
> _______________________________________________
> ldns-users mailing list
> ldns-users at open.nlnetlabs.nl
> http://open.nlnetlabs.nl/mailman/listinfo/ldns-users


-- 
Bedrich Kosata
CZ.NIC Labs <http://labs.nic.cz>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: patches.tgz
Type: application/x-gtar
Size: 2712 bytes
Desc: not available
URL: <http://lists.nlnetlabs.nl/pipermail/ldns-users/attachments/20110413/eb8707f0/attachment.gtar>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: test-rrlist-get-rdf.py
Type: text/x-python
Size: 533 bytes
Desc: not available
URL: <http://lists.nlnetlabs.nl/pipermail/ldns-users/attachments/20110413/eb8707f0/attachment.py>

From willem at NLnetLabs.nl  Wed Apr 13 11:39:30 2011
From: willem at NLnetLabs.nl (Willem Toorop)
Date: Wed, 13 Apr 2011 13:39:30 +0200
Subject: [ldns-users] pyldns - wire2pkt
In-Reply-To: <4D8C58B9.90708@nic.cz>
References: <4D8C58B9.90708@nic.cz>
Message-ID: <4DA58B72.5040906@nlnetlabs.nl>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hi Bedrich,

I've applied this patch to trunk.
I'll have a look at the memory patches now...

Cheers, Willem

On 03/25/11 09:56, Bedrich Kosata wrote:
> Hi again,
> 
> I attach a patch which makes wire2pkt usable from python. The previous
> version always complained about wrong type of arguments.
> A test code is attached as well.
> 
> Cheers
> 
> Beda
> 
> 
> 
> _______________________________________________
> ldns-users mailing list
> ldns-users at open.nlnetlabs.nl
> http://open.nlnetlabs.nl/mailman/listinfo/ldns-users

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.17 (FreeBSD)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iQIcBAEBAgAGBQJNpYtyAAoJEOX4+CEvd6SYvUMP+wevHXclKursoIcZ/UBQhY1v
1emdmh+bTF1aRkbq1+TaBwfgHBRarNlnrUnMVeYf0tDdwdog/GkdEOR+kiG35KXz
SD1xHP53HwyIGvWU61qZwIFL9IS1AZcJffglwMkrZ8avZf/wZ1Kk6DJzdFblXir7
lh8hcMop5VI6vjWaYzq7wLhPHLhsPRwYhe58OZU1xlUuFs4MIROHfnCJqTEAyIfo
iepqia/Ky8ZJAFPKdp9Re9mLYgyU4Vs+ALbBxUdqJlYsCVF6dwOT+5nJ8YF4nf62
KOXIfTHLPNuOSbcaTmG6PTiKWM8yO6Iv1jpA7ofk5V77+SlXLhYsXeVDzrf/0d5n
SZhyeNRmdTncXO/g5PHwflMvRhFzbLXa1pg4DJvMAf8DPRmLc6UWViwFxn7V/eXD
qwkcfdtv5FpZpUZNTy0r4PWQtW2tIMCphSo52OksWaILpl/WDGYIR/V94nTHmRbQ
AX43T5scmPXrikdEYRhcze8UfOSM/WHErJguMVlsCV+B2BoDzqa5oXHyfsSbuYXY
j9w9B4/y/KSQr5TpAjy1iweNOT+6RjjwjaXxDXQfvN3scfEs+dWHJyxqNek0E4hf
2M3/MA7bbeC0xJgkwU5qy+T0d26VWqdS6lDYm6JVWjEv+rssFJH6GZwiA4Jel8Jk
WzZwscErveBOAjghx1HG
=wvw0
-----END PGP SIGNATURE-----


From willem at NLnetLabs.nl  Wed Apr 13 14:54:46 2011
From: willem at NLnetLabs.nl (Willem Toorop)
Date: Wed, 13 Apr 2011 16:54:46 +0200
Subject: [ldns-users] pyldns - memory leaks and double freeing
In-Reply-To: <4DA546A3.5030600@nic.cz>
References: <4D8C56E1.7040301@nic.cz> <4DA546A3.5030600@nic.cz>
Message-ID: <4DA5B936.30606@nlnetlabs.nl>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hi Bedrich,

I've applied your patches in trunk.
The tests run all fine with me.
Thanks for fixing this.

Best regards, Willem


On 04/13/11 08:45, Bedrich Kosata wrote:
> Hi again,
> 
> as nobody seemed to have anything against my proposal, I prepared the
> patches that should fix the problems I outlined before :)
> 
> When the patches are applied, the python bindings should always have
> rr_list and rr structs separated in memory. This is accomplished by
> always cloning rr data when either adding them to an rr_list or
> retrieving them from the rr_list.
> Most of the changes should be transparent and should not require
> modifications to existing code. The only outside change I had to make
> was in the "ldns_verify_rrsig_keylist_notime" and
> "ldns_verify_rrsig_keylist" functions, because it copies pointers to the
> correct keys into a second rr_list. I solved this by replacing each of
> these functions by two new ones (thus creating four functions). All
> versions take only 3 arguments and differ in the way they treat the
> good_keys value. One versions name ends with "_status_only" (e.g.
> ldns_verify_rrsig_keylist_notime_status_only) and this does not report
> the good_keys at all, just the result of verification. The other ends
> with "_" (e.g. ldns_verify_rrsig_keylist_notime_) and returns a tuple
> (status, good_keys) where good_keys are indexes of the keys in the key
> list supplied to the function. Thus the whole business of two rr_lists
> sharing the same data is sidestepped.
> 
> The patches are made for each file separately because the fixes are
> mostly independent. I also included several test scripts which should
> show how the situation was fixed. They either show a situation where the
> old version would crash with a segmentation fault or display the amount
> of memory the app has used thus pointing to a memory leak. When used
> with patched sources, none of the scripts should crash and none should
> report more that ~30MB of memory used.
> 
> Best regards
> 
> Beda
> 
> p.s.- the patch for ldns_rdf.i is in fact not related to the rest and
> fixes and unrelated memory leak in ldns_rdf.new_frm_str
> 
> p.p.s.- I did not touch a related issue of rr and rdf relationship. In
> current version, the rdf data does not live beyond the lifetime of the
> corresponding rr. This is demonstrated in the attached
> test-rrlist-get-rdf.py. The problem could again be solved by cloning all
> rdfs on retrieval and addition from/into an rr.
> However, I will wait how the current patch is received before I start
> thinking about fixing it.
> 
> 
> On 03/25/2011 09:48 AM, Bedrich Kosata wrote:
>> Hi everybody,
>>
>> while trying to find a cause of a memory leak in a simple script, I
>> found a nest of memory related issues in the python bindings.
>> The problems are all related to one common problem - who takes care of
>> memory of composite objects, such as ldns_rr_lists or ldns_pkt.
>> For example, in the current version, ldns_pkt bindings use ldns_pkt_free
>> to free a packet structure, however, when a rr_list is taken from the
>> packet and returned from a function, the packet gets out of scope, is
>> freed and the rr_list refers to already freed memory.
>> On the other hand, a rr_list only frees its own memory, not memory of
>> the stored rrs. This can lead to memory leaks.
>> I am attaching two scripts that demonstrate these problems (it might be
>> necessary to have the sources patched with the "freeing None" patch I
>> sent last week).
>> I would be willing to have a stab at the problems (provided I get
>> clearance from my boss :)), but the only solution I think would be clean
>> enough, is to clone the necessary bits where needed. This might lead to
>> some inefficiency and slowdowns (probably not big), so I would like to
>> ask if this is ok.
>> If there is anyone else willing to fix this, I would be happy to act as
>> a tester.
>>
>> Cheers
>>
>> Beda
>>
>> p.s.- test-pkt-free.py crashes with a segmentation fault,
>> test-rr-list.py ends up eating about 130 MB of memory and more than
>> 400000 python objects which python cannot free.
>>
>>
>>
>> _______________________________________________
>> ldns-users mailing list
>> ldns-users at open.nlnetlabs.nl
>> http://open.nlnetlabs.nl/mailman/listinfo/ldns-users
> 
> 
> 
> 
> _______________________________________________
> ldns-users mailing list
> ldns-users at open.nlnetlabs.nl
> http://open.nlnetlabs.nl/mailman/listinfo/ldns-users

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.17 (FreeBSD)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iQIcBAEBAgAGBQJNpbk2AAoJEOX4+CEvd6SYMx0P/i4X059sn9fXRk7WzEdOdsz4
AtDbi1B/rf2D26WN2rjrQwJRhmaC6s20YKeo9h4cFBqcZrROaOWseeMqN24iOYsI
Rt1ltWPbe+2th+G1o9r6F7keZU3i1PUxPGEgzpTe8o8xicq5tP9pJlpI/43ISj2T
KHKLi1Ps6B/TPOM519q50FCapTFU/IlaybONGJ12Ts3QhtwUB/Q0uUL5DmJWM8B6
iz2Jqy5/ABJYmyiVDup2rtVW/nZI0t032PCP8qHBEVlfZvRyX7Cm0TMLpXLH80cH
w6AXkdj+FUOD7uDi8cxGxMeQnYg/OtgjerTaVqc0+1v3XgwxflReIxpZ1rudUz81
YGf04XFuzk8+2/X3wukfXLoNAMQCSaqsQLtR5N1u/JMDt5KlBRcpzEPQJjLbxTBc
ABsCuCuhrVgVclMTkLXZ/aIVAcx1Rjwjgppxr3Rc0Ixp+gPEWZb8SZc5ZAvucAI1
lVHVi7gohiFdQ9tILByxkFVvbzTYvbkojUJ2zgobJuGdfJ/KnIQ/mt9lZ3PF3IEu
jkPpNRKGqY9diTuMQt9hQvKbXQF0KedP1kjUYFDhHql2li0xpdVzvHq+Xpgc/jGj
owj4faYKHo9QSxbMhshuypV3+QF3Ck9l2IwGCTWw2qM9ffXKH+eB8HA5aXsuz70p
dSfoo4W83DQRqjdS3Vk/
=LBN9
-----END PGP SIGNATURE-----


From willem at NLnetLabs.nl  Wed Apr 13 14:55:07 2011
From: willem at NLnetLabs.nl (Willem Toorop)
Date: Wed, 13 Apr 2011 16:55:07 +0200
Subject: [ldns-users] ldns-python bug in resolver.get_addr_by_name()
In-Reply-To: <alpine.LFD.1.10.1104121413380.23536@newtla.xelerance.com>
References: <alpine.LFD.1.10.1104121413380.23536@newtla.xelerance.com>
Message-ID: <4DA5B94B.2030809@nlnetlabs.nl>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hi Paul,

Indeed, the documentation showed such usage in the example.
The version in trunk now actually supports this behavior too.
Thanks for the report!

Willem


On 04/12/11 20:15, Paul Wouters wrote:
> 
> The following works:
> 
>>>> import ldns
>>>> resolver = ldns.ldns_resolver.new_frm_file("/etc/resolv.conf")
>>>> dnn = ldns.ldns_dname("www.google.com")
>>>> addr = resolver.get_addr_by_name(dnn)
>>>> for rr in addr.rrs():
> ...     print rr
> ... www.l.google.com.    300    IN    A    74.125.224.50
> 
> [..]
> 
> However, this does not work:
> 
>>>> addr = resolver.get_addr_by_name("www.google.com")
> Traceback (most recent call last):
>   File "<stdin>", line 1, in <module>
>   File "/usr/lib/python2.7/site-packages/ldns.py", line 3898, in
> get_addr_by_name
>     return _ldns.ldns_get_rr_list_addr_by_name(self, name, aclass, flags)
> TypeError: in method 'ldns_get_rr_list_addr_by_name', argument 2 of type
> 'ldns_rdf *'
>>>>
> 
> The docs claim that get_addr_by_name() can do the string->dname
> conversion....
> 
> Paul
> _______________________________________________
> ldns-users mailing list
> ldns-users at open.nlnetlabs.nl
> http://open.nlnetlabs.nl/mailman/listinfo/ldns-users

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.17 (FreeBSD)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iQIcBAEBAgAGBQJNpblKAAoJEOX4+CEvd6SYhv0P/1iGlla5AfN6y3vXcf47+D44
NqxqppPhe+cGM62cdrwltlX/DZW0T3Q/olNlJwJMFxURsHZmBEKJousJq6CoMV6L
+badmltP4OltJRLZUYFezJzpJpR/zS9e3ovkVB4P9FisEwglv2JF0Xam8yPMpeTK
Xy0tWvsoDFecKVjlzgy1jaZgugxbOUCfWY/IcE8YTLq3u6miEO9+6VRJPchrfQF3
eb2DiBy+xYlb2xR8dd1yrg1jT31GR351oX8mdvG0vLjX2u3uuI6VuhzTWt9NodC+
E75oEFeT2g/rbsLCjFAaU0eVPGoD0dC0cUEDKpZwtmZQHHkaW2PrILmmF3OE/Sgw
nMaTLKF33CunVCB/iGEzT8W0z49qJnD4Xp2e9mI1FurZ2vDjzWNg5HnGv17Zew8h
++4mDjsEeVjPxwpYdluKK2jCQN+HFtkAttnHQ33fDqXJwWSEjp3+gBAvgeTMS6MG
k1CQmquGoEPYWJur/dxzEtOX5Q8KlgoPWRHn+hTrvlv3n95p9sagPyS3GgRd8voS
AAXYFOqfdRl1te19odlCQlJrakHU8NE8pyv8fFE4QN+wmgC9biqAXWUoRX+ZYEcj
BFK8ypoPm+ronIgbFS4BEYyPKQ4NbYke4GhHv3/r6DSUjUePodKgIBLnW5b0rggG
CEVvbYBB0ri0moUJkzZd
=nRJM
-----END PGP SIGNATURE-----


From paul at xelerance.com  Thu Apr 14 02:11:25 2011
From: paul at xelerance.com (Paul Wouters)
Date: Wed, 13 Apr 2011 22:11:25 -0400 (EDT)
Subject: [ldns-users] ldns-python bug in resolver.get_addr_by_name()
In-Reply-To: <4DA5B94B.2030809@nlnetlabs.nl>
References: <alpine.LFD.1.10.1104121413380.23536@newtla.xelerance.com>
 <4DA5B94B.2030809@nlnetlabs.nl>
Message-ID: <alpine.LFD.1.10.1104132211080.18910@newtla.xelerance.com>

On Wed, 13 Apr 2011, Willem Toorop wrote:

> Indeed, the documentation showed such usage in the example.
> The version in trunk now actually supports this behavior too.
> Thanks for the report!

Thanks!

Paul


From klaus.mailinglists at pernau.at  Wed Apr 27 13:25:01 2011
From: klaus.mailinglists at pernau.at (Klaus Darilion)
Date: Wed, 27 Apr 2011 15:25:01 +0200
Subject: [ldns-users] drill uses local resolvers
Message-ID: <4DB8192D.7070100@pernau.at>

Hi!

I experienced strange problems with drill 1.6.6 and 1.6.9. Although I
specify root servers with -r, sometimes (50% chance) the local
configured resolver is asked for a root-server list. With drill 1.6.4
this random behavior was not seen yet.

Are there any known issues?

Thanks
Klaus