backend abstraction layer

[Paul G]

erik,

----- Original Message ----- 
From: "Erik Rozendaal" <erik at NLnetLabs.nl>
To: "Paul G" <paul at rusko.us>
Cc: <nsd-users at NLnetLabs.nl>
Sent: Thursday, September 02, 2004 8:25 AM
Subject: Re: backend abstraction layer


> Erik Rozendaal wrote:
>
> > But support for any kind of specific dynamic or SQL backend will most
> > likely have to remain a separate patch or plugin.
>
> Actually, to clarify a bit, dynamic backends will have to remain
> separate. See the requirements (#9) and especially the non-requirements
> (#4) in the REQUIREMENTS file.

yep, saw that, which was exactly why i was asking about inclusion chances.
the requirements and development policies make a boatload of sense and to be
honest, nsd is one of the best-run foss projects i have come across; i would
not presume to suggest any changes to that.

> So to reiterate, if some internal changes are needed to make it easier
> for you to implement whatever you need and those changes do not
> complicate NSD the changes can be included.  But anything beyond that
> will have to remain as a separate patch/plugin/project.

apropos a separate plugin, this is exactly what i'm playing with right now.
upon closer inspection/more tinkering, i think this would work fine as a
plugin with a few internal changes. this would work well with the general
philosophy of the project, since it could be dumped in /contrib for those
who want to use/play with it. i hope you don't mind if i run the following
implementation ideas past you and anyone else who might care to chime in to
see if i'm on the right track. please note that i haven't looked at a few
things yet (i decided to start hacking a bit first), so some of this might
be completely off-the-wall. you have been extremely courteous and helpful in
your past replies, but if you think this is going too far, please feel free
to tell me where to stick it (god knows it would have happened much earlier
with pretty much every other foss project) <g>

generic backend plugin

requirements:

1. must provide an implementation independent set of apis backends can
implement
2. must support static entries (load once, query many)
3. must support atomic, non service affecting updates of said entries with
per entry granularity
4. must support dynamic entries (load on each query, eg query a mysql
backend on every query)
5. must avoid blocking queries against static entries while processing a
query against a dynamic entry

* entry = first class citizen of my choice, a zone to start with

implementation thoughts:

1. pretty self-explanatory - i don't care how/where you get me the list of
domains and records for them, just give them to me when i ask.
2. loading should take place on plugin init. in my recent hacking, i already
added a second version of register_data() that does a domain_table_insert()
if the domain name is not present, which allows me to work with domains not
listed in nsd.zones. taken a step further, i could go on to do
domain_add_rrset() for those zones as well.
3. this is something i especially need input on. since this would be done
asynchronously, i'd either do a per namedb or per entry write lock and do
the update. there doesn't seem to be a way to modify an entry currently and
what i'd really like to do is insert a modified entry and switch to using
that atomically so that queries against it wouldn't have to be blocked or
dropped on the floor during the update. while this looks doable, it does
involve tinkering with the internals so i'm looking for both pointers and
guidance as to how much of a faux pas it would be.
4 and 5. i figure having each backend implementation run as a separate
process (don't want to touch threads because of portability issues) capable
of doing async i/o with its storage and handing off the query to it if it's
dynamic would work for this. however, i admit i didn't look at enough of the
code yet to see if this would pose a problem (ie, does nsd already deal with
queries asynchronously?).

a couple assorted thoughts:

* there needs to be a way to short circuit the query processing; ie, if the
plugin wants to satisfy the query in the query_received callback, such as
with queries against dynamic entries, there is no need for nsd to do
anything else. a possibly clever solution would be to actually do a namedb
update here and let nsd process the query as it does normally. this would
work well with the "least amount of change to nsd code" principle.
* the performance in the "standard" codepath as well as memory consumption
should not be affected (too much). this will take some thinking, because
ideally you'd want to bypass any processing by the plugin if the zone comes
from nsd.db and you don't want to add a per entry flag. a small hash table
should be an acceptable workaround unless there's a better idea.


that's it for me, long night, excuse the rambling ;] thanks again for your
help.

cheers,
paul