<div dir="ltr">Hi Yuri,<div><br></div><div>I was wondering if you'd made any head way on the ECS TTL bug or the bug in unbound-control that causes ECS queries to be counted as misses even when they are returned from cache.</div>
<div><br></div><div>Thanks,</div><div>Larry</div></div><div class="gmail_extra"><br clear="all"><div>-Larry</div>
<br><br><div class="gmail_quote">On Tue, May 6, 2014 at 9:51 AM, Larry Havemann <span dir="ltr"><<a href="mailto:larry@edgecast.com" target="_blank">larry@edgecast.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr">Here is an example showing the behavior described above:<div><div><br></div><div>Client 1 subnet <a href="http://110.232.0.0/24" target="_blank">110.232.0.0/24</a> Asia:</div><div>root@dnsr001:/usr/etc/unbound# /EdgeCast/ecdns/bin/dig_iana @localhost <a href="http://gp1.wpc.edgecastcdn.net" target="_blank">gp1.wpc.edgecastcdn.net</a> +client=<a href="http://110.232.0.0/24" target="_blank">110.232.0.0/24</a></div>
<div><br></div><div>; <<>> DiG 9.9.3-P1 <<>> @localhost <a href="http://gp1.wpc.edgecastcdn.net" target="_blank">gp1.wpc.edgecastcdn.net</a> +client=<a href="http://110.232.0.0/24" target="_blank">110.232.0.0/24</a></div>
<div class=""><div>; (2 servers found)</div>
<div>;; global options: +cmd</div><div>;; Got answer:</div></div><div>;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 12638</div><div class=""><div>;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1</div>
<div>
<br></div><div>;; OPT PSEUDOSECTION:</div><div>; EDNS: version: 0, flags:; udp: 4096</div><div>; CLIENT-SUBNET: <a href="http://110.232.0.0/24/19" target="_blank">110.232.0.0/24/19</a></div><div>;; QUESTION SECTION:</div>
<div>;<a href="http://gp1.wpc.edgecastcdn.net" target="_blank">gp1.wpc.edgecastcdn.net</a>.<span style="white-space:pre-wrap"> </span>IN<span style="white-space:pre-wrap"> </span>A</div>
<div><br></div><div>;; ANSWER SECTION:</div></div><div><b><a href="http://gp1.wpc.edgecastcdn.net" target="_blank">gp1.wpc.edgecastcdn.net</a>. 3600<span style="white-space:pre-wrap"> </span>IN<span style="white-space:pre-wrap"> </span>A<span style="white-space:pre-wrap"> </span>117.18.232.133 <=== Asia IP</b></div>
<div><br></div><div>;; Query time: 45 msec</div><div>;; SERVER: 127.0.0.1#53(127.0.0.1)</div><div>;; WHEN: Tue May 06 16:32:57 UTC 2014</div><div>;; MSG SIZE rcvd: 79</div><div><br></div><div>Client 2 no subnet:</div><div>
root@dnsr001:/usr/etc/unbound# /EdgeCast/ecdns/bin/dig_iana @localhost <a href="http://gp1.wpc.edgecastcdn.net" target="_blank">gp1.wpc.edgecastcdn.net</a> </div><div><br></div><div>; <<>> DiG 9.9.3-P1 <<>> @localhost <a href="http://gp1.wpc.edgecastcdn.net" target="_blank">gp1.wpc.edgecastcdn.net</a></div>
<div class="">
<div>; (2 servers found)</div><div>;; global options: +cmd</div><div>;; Got answer:</div></div><div>;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 44048</div><div class=""><div>;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1</div>
<div><br></div><div>;; OPT PSEUDOSECTION:</div><div>; EDNS: version: 0, flags:; udp: 4096</div><div>;; QUESTION SECTION:</div><div>;<a href="http://gp1.wpc.edgecastcdn.net" target="_blank">gp1.wpc.edgecastcdn.net</a>.<span style="white-space:pre-wrap"> </span>IN<span style="white-space:pre-wrap"> </span>A</div>
<div><br></div><div>;; ANSWER SECTION:</div></div><div><b><a href="http://gp1.wpc.edgecastcdn.net" target="_blank">gp1.wpc.edgecastcdn.net</a>. 3600<span style="white-space:pre-wrap"> </span>IN<span style="white-space:pre-wrap"> </span>A<span style="white-space:pre-wrap"> </span>72.21.81.253 <==== NA IP</b></div>
<div><br></div><div>;; Query time: 2 msec</div><div>;; SERVER: 127.0.0.1#53(127.0.0.1)</div><div>;; WHEN: Tue May 06 16:33:01 UTC 2014</div><div>;; MSG SIZE rcvd: 68</div><div><br></div><div>Client 3 <a href="http://46.22.74.0/24" target="_blank">46.22.74.0/24</a> Europe: </div>
<div>root@dnsr001:/usr/etc/unbound# /EdgeCast/ecdns/bin/dig_iana @localhost <a href="http://gp1.wpc.edgecastcdn.net" target="_blank">gp1.wpc.edgecastcdn.net</a> +client=<a href="http://46.22.74.0/24" target="_blank">46.22.74.0/24</a></div>
<div><br></div>
<div>; <<>> DiG 9.9.3-P1 <<>> @localhost <a href="http://gp1.wpc.edgecastcdn.net" target="_blank">gp1.wpc.edgecastcdn.net</a> +client=<a href="http://46.22.74.0/24" target="_blank">46.22.74.0/24</a></div>
<div class=""><div>; (2 servers found)</div>
<div>;; global options: +cmd</div><div>;; Got answer:</div></div><div>;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 33795</div><div class=""><div>;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1</div>
<div>
<br></div><div>;; OPT PSEUDOSECTION:</div><div>; EDNS: version: 0, flags:; udp: 4096</div></div><div>; CLIENT-SUBNET: <a href="http://46.22.74.0/24/0" target="_blank">46.22.74.0/24/0</a></div><div class=""><div>;; QUESTION SECTION:</div>
<div>;<a href="http://gp1.wpc.edgecastcdn.net" target="_blank">gp1.wpc.edgecastcdn.net</a>.<span style="white-space:pre-wrap"> </span>IN<span style="white-space:pre-wrap"> </span>A</div>
<div><br></div><div>;; ANSWER SECTION:</div></div><div><b><a href="http://gp1.wpc.edgecastcdn.net" target="_blank">gp1.wpc.edgecastcdn.net</a>. 3583<span style="white-space:pre-wrap"> </span>IN<span style="white-space:pre-wrap"> </span>A<span style="white-space:pre-wrap"> </span>72.21.81.253 <===== NA IP</b></div>
<div class="">
<div><br></div><div>;; Query time: 0 msec</div><div>;; SERVER: 127.0.0.1#53(127.0.0.1)</div></div><div>;; WHEN: Tue May 06 16:33:18 UTC 2014</div><div>;; MSG SIZE rcvd: 79</div></div><div><br></div><div><br></div><div>This is a show stopper because a client coming in from 46.22.74.0 is in Europe and with this result is being directed to servers located in North America. This is a major performance hit for the end user, the correct response should be:</div>
<div><div>root@dnsr001:/usr/etc/unbound# /EdgeCast/ecdns/bin/dig_iana @localhost <a href="http://gp1.wpc.edgecastcdn.net" target="_blank">gp1.wpc.edgecastcdn.net</a> +client=<a href="http://46.22.74.0/24" target="_blank">46.22.74.0/24</a></div>
<div><br>
</div><div>; <<>> DiG 9.9.3-P1 <<>> @localhost <a href="http://gp1.wpc.edgecastcdn.net" target="_blank">gp1.wpc.edgecastcdn.net</a> +client=<a href="http://46.22.74.0/24" target="_blank">46.22.74.0/24</a></div>
<div class=""><div>; (2 servers found)</div>
<div>;; global options: +cmd</div><div>;; Got answer:</div></div><div>;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 31443</div><div class=""><div>;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1</div>
<div>
<br></div><div>;; OPT PSEUDOSECTION:</div><div>; EDNS: version: 0, flags:; udp: 4096</div></div><div>; CLIENT-SUBNET: <a href="http://46.22.74.0/24/24" target="_blank">46.22.74.0/24/24</a></div><div class=""><div>;; QUESTION SECTION:</div>
<div>;<a href="http://gp1.wpc.edgecastcdn.net" target="_blank">gp1.wpc.edgecastcdn.net</a>.<span style="white-space:pre-wrap"> </span>IN<span style="white-space:pre-wrap"> </span>A</div>
<div><br></div><div>;; ANSWER SECTION:</div></div><div><b><a href="http://gp1.wpc.edgecastcdn.net" target="_blank">gp1.wpc.edgecastcdn.net</a>. 3600<span style="white-space:pre-wrap"> </span>IN<span style="white-space:pre-wrap"> </span>A<span style="white-space:pre-wrap"> </span>93.184.221.133 <==== Europe IP</b></div>
<div class="">
<div><br></div><div>;; Query time: 3 msec</div><div>;; SERVER: 127.0.0.1#53(127.0.0.1)</div></div><div>;; WHEN: Tue May 06 16:35:07 UTC 2014</div><div>;; MSG SIZE rcvd: 79</div></div><div><br></div><div><br></div><div>The use case where this becomes a show stopper is this. As a CDN we pull content from many different customers. If the customer has geoip rules setup so that content in the EU is pulled from an EU origin server and content for NA is pulled from an NA server this would cause the pull to happen from the wrong part of the world. Now if 1000's of servers in the EU and 1000's of servers in NA are all pulling from the same location instead of distributing the load correctly as described in the geoip rule the NA origin server could fail. This would lead to their content either never loading or becoming stale. So for me the result from regular cache is in fact wrong. I did find a way to force it to always do ecs lookup by white listing 1-255.0.0.0/8, but to me this seems like a very sloppy way of doing things.</div>
<div><br></div><div>I think the performance hit would be acceptable to anyone enabling ECS so long as it is documented that there is a performance hit. </div></div><div class="gmail_extra"><span class="HOEnZb"><font color="#888888"><br clear="all">
<div>-Larry</div></font></span><div><div class="h5">
<br><br><div class="gmail_quote">On Tue, May 6, 2014 at 2:36 AM, Yuri Schaeffer <span dir="ltr"><<a href="mailto:yuri@nlnetlabs.nl" target="_blank">yuri@nlnetlabs.nl</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div>-----BEGIN PGP SIGNED MESSAGE-----<br>
Hash: SHA1<br>
<br>
Hi Larry,<br>
<br>
</div><div>> This creates an issue where you have a user coming in with ecs<br>
> <a href="http://1.2.3.0/24" target="_blank">1.2.3.0/24</a> gets the correct answer, then user coming without ecs<br>
> get the default answer and finally user with ecs <a href="http://1.2.4.0/24" target="_blank">1.2.4.0/24</a> get<br>
> default instead of the correct answer.<br>
<br>
</div>Did you test this?<br>
<br>
I believe this is not what would happen. The final user asking for ecs<br>
<a href="http://1.2.4.0/24" target="_blank">1.2.4.0/24</a> would get an answer from the ecs cache. In case it was not in<br>
cache an upstream lookup would be done with ecs <a href="http://1.2.4.0/24" target="_blank">1.2.4.0/24</a>. Yielding the<br>
most accurate response.<br>
<br>
Also, I'd like to stress that an answer from the 'regular' cache is not<br>
wrong, merely suboptimal. It is what a non ECS aware resolver would<br>
answer.<br>
<br>
Please note there is no need for the clients to 'speak' ECS. When you<br>
whitelist a server to do ECS Unbound will ask it for the most specific<br>
answer for its client. Relaying ECS, specially to non-whitelisted<br>
servers is a courtesy to the client and not mandated by the draft.<br>
<div><br>
> To me and for my use this is a major show stopper.<br>
<br>
</div>I'm interested in your usecase and what functionality you are looking<br>
for. What do you expect from a recursor? Do you not intent to use the<br>
whitelist functionality?<br>
<div><br>
> Is there anyway the look up order could be reversed if ecs is<br>
> enabled? So that all queries first try ecs then goto normal<br>
> cache?<br>
<br>
</div>While possible it would affect every single incoming query. It is<br>
assumed ECS is only communicated with a fraction of authority servers.<br>
It would mean a significant performance hit, especially since the ECS<br>
cache is not a straight forward key:value lookup.<br>
<br>
Best regards,<br>
<div>Yuri<br>
-----BEGIN PGP SIGNATURE-----<br>
Version: GnuPG v1<br>
Comment: Using GnuPG with Icedove - <a href="http://www.enigmail.net/" target="_blank">http://www.enigmail.net/</a><br>
<br>
</div>iEYEARECAAYFAlNorSsACgkQI3PTR4mhavhdeACfUpkA6wiZMnzgTbgN/ZKcYL65<br>
JdkAniIjFO1w00N3rnPU+Yy55C16Mx/X<br>
=lk90<br>
-----END PGP SIGNATURE-----<br>
</blockquote></div><br></div></div></div>
</blockquote></div><br></div>