[ldns-users] pyldns - memory leaks and double freeing

Wed May 4 07:26:23 UTC 2011

Hi Willem,

by looking at the current SVN code, I found out, that I actually did not 
include part of the code into the patch I sent last time.
This is the part about RRSIG checking - marked in my last post below.
I am attaching the patch for ldns_rr.i now.

Best regards

Beda

p.s.- I am currently working on some more memory issues related to rr 
and rdf sharing the same data. I hope to be able to send something soon.

On 04/13/2011 04:54 PM, Willem Toorop wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> Hi Bedrich,
>
> I've applied your patches in trunk.
> The tests run all fine with me.
> Thanks for fixing this.
>
> Best regards, Willem
>
>
> On 04/13/11 08:45, Bedrich Kosata wrote:
>> Hi again,
>>
>> as nobody seemed to have anything against my proposal, I prepared the
>> patches that should fix the problems I outlined before :)
>>
>> When the patches are applied, the python bindings should always have
>> rr_list and rr structs separated in memory. This is accomplished by
>> always cloning rr data when either adding them to an rr_list or
>> retrieving them from the rr_list.

This was not in the last patch..

>> Most of the changes should be transparent and should not require
>> modifications to existing code. The only outside change I had to make
>> was in the "ldns_verify_rrsig_keylist_notime" and
>> "ldns_verify_rrsig_keylist" functions, because it copies pointers to the
>> correct keys into a second rr_list. I solved this by replacing each of
>> these functions by two new ones (thus creating four functions). All
>> versions take only 3 arguments and differ in the way they treat the
>> good_keys value. One versions name ends with "_status_only" (e.g.
>> ldns_verify_rrsig_keylist_notime_status_only) and this does not report
>> the good_keys at all, just the result of verification. The other ends
>> with "_" (e.g. ldns_verify_rrsig_keylist_notime_) and returns a tuple
>> (status, good_keys) where good_keys are indexes of the keys in the key
>> list supplied to the function. Thus the whole business of two rr_lists
>> sharing the same data is sidestepped.
>>

.. and here it ends

>> The patches are made for each file separately because the fixes are
>> mostly independent. I also included several test scripts which should
>> show how the situation was fixed. They either show a situation where the
>> old version would crash with a segmentation fault or display the amount
>> of memory the app has used thus pointing to a memory leak. When used
>> with patched sources, none of the scripts should crash and none should
>> report more that ~30MB of memory used.
>>
>> Best regards
>>
>> Beda
>>
>> p.s.- the patch for ldns_rdf.i is in fact not related to the rest and
>> fixes and unrelated memory leak in ldns_rdf.new_frm_str
>>
>> p.p.s.- I did not touch a related issue of rr and rdf relationship. In
>> current version, the rdf data does not live beyond the lifetime of the
>> corresponding rr. This is demonstrated in the attached
>> test-rrlist-get-rdf.py. The problem could again be solved by cloning all
>> rdfs on retrieval and addition from/into an rr.
>> However, I will wait how the current patch is received before I start
>> thinking about fixing it.
>>
>>
>> On 03/25/2011 09:48 AM, Bedrich Kosata wrote:
>>> Hi everybody,
>>>
>>> while trying to find a cause of a memory leak in a simple script, I
>>> found a nest of memory related issues in the python bindings.
>>> The problems are all related to one common problem - who takes care of
>>> memory of composite objects, such as ldns_rr_lists or ldns_pkt.
>>> For example, in the current version, ldns_pkt bindings use ldns_pkt_free
>>> to free a packet structure, however, when a rr_list is taken from the
>>> packet and returned from a function, the packet gets out of scope, is
>>> freed and the rr_list refers to already freed memory.
>>> On the other hand, a rr_list only frees its own memory, not memory of
>>> the stored rrs. This can lead to memory leaks.
>>> I am attaching two scripts that demonstrate these problems (it might be
>>> necessary to have the sources patched with the "freeing None" patch I
>>> sent last week).
>>> I would be willing to have a stab at the problems (provided I get
>>> clearance from my boss :)), but the only solution I think would be clean
>>> enough, is to clone the necessary bits where needed. This might lead to
>>> some inefficiency and slowdowns (probably not big), so I would like to
>>> ask if this is ok.
>>> If there is anyone else willing to fix this, I would be happy to act as
>>> a tester.
>>>
>>> Cheers
>>>
>>> Beda
>>>
>>> p.s.- test-pkt-free.py crashes with a segmentation fault,
>>> test-rr-list.py ends up eating about 130 MB of memory and more than
>>> 400000 python objects which python cannot free.
>>>
>>>
>>>
>>> _______________________________________________
>>> ldns-users mailing list
>>> ldns-users at open.nlnetlabs.nl
>>> http://open.nlnetlabs.nl/mailman/listinfo/ldns-users
>>
>>
>>
>>
>> _______________________________________________
>> ldns-users mailing list
>> ldns-users at open.nlnetlabs.nl
>> http://open.nlnetlabs.nl/mailman/listinfo/ldns-users
>
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v2.0.17 (FreeBSD)
> Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
>
> iQIcBAEBAgAGBQJNpbk2AAoJEOX4+CEvd6SYMx0P/i4X059sn9fXRk7WzEdOdsz4
> AtDbi1B/rf2D26WN2rjrQwJRhmaC6s20YKeo9h4cFBqcZrROaOWseeMqN24iOYsI
> Rt1ltWPbe+2th+G1o9r6F7keZU3i1PUxPGEgzpTe8o8xicq5tP9pJlpI/43ISj2T
> KHKLi1Ps6B/TPOM519q50FCapTFU/IlaybONGJ12Ts3QhtwUB/Q0uUL5DmJWM8B6
> iz2Jqy5/ABJYmyiVDup2rtVW/nZI0t032PCP8qHBEVlfZvRyX7Cm0TMLpXLH80cH
> w6AXkdj+FUOD7uDi8cxGxMeQnYg/OtgjerTaVqc0+1v3XgwxflReIxpZ1rudUz81
> YGf04XFuzk8+2/X3wukfXLoNAMQCSaqsQLtR5N1u/JMDt5KlBRcpzEPQJjLbxTBc
> ABsCuCuhrVgVclMTkLXZ/aIVAcx1Rjwjgppxr3Rc0Ixp+gPEWZb8SZc5ZAvucAI1
> lVHVi7gohiFdQ9tILByxkFVvbzTYvbkojUJ2zgobJuGdfJ/KnIQ/mt9lZ3PF3IEu
> jkPpNRKGqY9diTuMQt9hQvKbXQF0KedP1kjUYFDhHql2li0xpdVzvHq+Xpgc/jGj
> owj4faYKHo9QSxbMhshuypV3+QF3Ck9l2IwGCTWw2qM9ffXKH+eB8HA5aXsuz70p
> dSfoo4W83DQRqjdS3Vk/
> =LBN9
> -----END PGP SIGNATURE-----

-- 
Bedrich Kosata
CZ.NIC Labs <http://labs.nic.cz>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: ldns_rr.i-rrsig_wrappers.patch
Type: text/x-patch
Size: 3001 bytes
Desc: not available
URL: <http://lists.nlnetlabs.nl/pipermail/ldns-users/attachments/20110504/cfd8d55f/attachment.bin>