RPZ: is this config correct?

RayG rgsub1 at btinternet.com
Wed Nov 11 16:14:06 UTC 2020


Hi Eduardo,

Thanks for the suggestion, that is certainly an easier way to get the debugging output.

Looking through the logs and in greater detail I wonder if I have seen the issue.

See these two commands:

C:\Program Files\Unbound>I:\wget64.exe https://151.101.130.49/downloads/rpz
--2020-11-11 16:01:48--  https://151.101.130.49/downloads/rpz
Connecting to 151.101.130.49:443... connected.
    ERROR: certificate common name 'c.sni.fastly.net' doesn't match requested host name '151.101.130.49'.
To connect to 151.101.130.49 insecurely, use `--no-check-certificate'.

C:\Program Files\Unbound>I:\wget64.exe https://urlhaus.abuse.ch/downloads/rpz
--2020-11-11 16:02:54--  https://urlhaus.abuse.ch/downloads/rpz
Resolving urlhaus.abuse.ch (urlhaus.abuse.ch)... 151.101.66.49, 151.101.2.49, 151.101.194.49, ...
Connecting to urlhaus.abuse.ch (urlhaus.abuse.ch)|151.101.66.49|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 130762 (128K) [text/plain]
Saving to: 'rpz'

rpz                           100%[=================================================>] 127.70K  --.-KB/s    in 0.04s

2020-11-11 16:02:54 (3.15 MB/s) - 'rpz' saved [130762/130762]

And the data is there in the "rpz" file.

I see in the unbound log file:

10/11/2020 15:05:14 C:\Program Files\Unbound\unbound.exe[15932:0] debug: auth zone rpz.urlhaus.abuse.ch. transfer next HTTP fetch from 151.101.122.49 started
...
10/11/2020 15:05:24 C:\Program Files\Unbound\unbound.exe[15932:0] debug: xfr stopped, connection timeout to urlhaus.abuse.ch
...
10/11/2020 15:05:24 C:\Program Files\Unbound\unbound.exe[15932:0] debug: auth zone rpz.urlhaus.abuse.ch. transfer failed, wait

Which suggests the transfer is being done using the IP address rather than the DNS name and as we can see from above with wget we get a certificate error.

Is this what is causing things to go wrong?

Is unbound using the DNS name or the IP address?

RayG

-----Original Message-----
From: Eduardo Schoedler <listas at esds.com.br> 
Sent: 10 November 2020 17:44
To: Unbound-users <unbound-users at lists.nlnetlabs.nl>
Cc: RayG <rgsub1 at btinternet.com>
Subject: Re: RPZ: is this config correct?

RayG,

You can try stop unbound and run it in foreground:

unbound -d -vvvvv


And look for some errors.

Em ter., 10 de nov. de 2020 às 13:54, RayG via Unbound-users <unbound-users at lists.nlnetlabs.nl> escreveu:
>
> Hi George,
>
> OK thanks for that info.
>
> I have no issue getting the data using the URL in a browser but unbound flatly refuses to successfully retrieve it.
>
>      name: "URLHaus"
>      zonefile: "C:\ProgramData\Unbound\Logs\urlhaus.zone"
>      url: https://urlhaus.abuse.ch/downloads/rpz
>      rpz-log: yes
>      rpz-log-name: "URLHausRPZ"
>      rpz-action-override: nxdomain
>
> I have tried with an empty file, a populated file and no file but nothing works. No file is ever created.
>
> What else can I try?
>
> This is the configuration file I have removed my local network configuration:
>
> #
> # UnboundConfiguration @ 2020-11-10
> #
> server: # MyConfig.conf
> include: "MyConfigUpdates.conf" # Version 1.12.0
> include: "MyPerformance.conf"
> include: "MyUseMixedCase.conf"
> include: "MyBlocklist.conf"
> include: "MyLocalHostNetwork.conf"
> include: "MyLocalNetwork.conf"
> include: "MyForwardZonesTLS.conf" # Calls - MyDoTConfig.conf
> include: "MyRemoteControl.conf"
> include: "MyResponsePolicyZones.conf"
> include: "MyAddToBlockList.conf"
> server: # MyConfigUpdates.conf
>      verbosity: 1
>      statistics-interval: 3600
>      extended-statistics: no
>      num-threads: 4
>      do-ip4: yes
>      do-ip6: yes
>      do-udp: yes
>      do-tcp: yes
>      access-control: 0.0.0.0/0 refuse
>      access-control: 127.0.0.0/8 allow_snoop
>      access-control: ::0/0 refuse
>      access-control: ::1 allow_snoop
>      logfile: "C:\ProgramData\Unbound\logs\unbound.log"
>      use-syslog: no
>      stream-wait-size: 16m
>      msg-cache-size: 8m
>      msg-cache-slabs: 8
>      rrset-cache-size: 8m
>      rrset-cache-slabs: 8
>      infra-cache-slabs: 8
>      log-identity: ""
>      log-time-ascii: yes
>      log-queries: yes
>      log-replies: yes
>      log-tag-queryreply: yes
>      log-servfail: yes
>      root-hints: "RootHints.conf"
>      hide-identity: yes
>      hide-version: yes
>      harden-short-bufsize: yes
>      harden-large-queries: yes
>      harden-glue: yes
>      harden-dnssec-stripped: yes
>      harden-below-nxdomain: yes
>      harden-referral-path: yes
>      harden-algo-downgrade: yes
>      qname-minimisation: yes
>      aggressive-nsec: yes
>      private-address: 0.0.0.0/8       # Broadcast address
>      private-address: 10.0.0.0/8
>      private-address: 100.64.0.0/10
>      private-address: 127.0.0.0/8     # Loopback Localhost
>      private-address: 169.254.0.0/16
>      private-address: 172.16.0.0/12
>      private-address: 192.0.0.0/24    # IANA IPv4 special purpose net
>      private-address: 192.0.2.0/24    # Documentation network TEST-NET
>      private-address: 192.168.0.0/16
>      private-address: 198.18.0.0/15   # Used for testing inter-network communications
>      private-address: 198.51.100.0/24 # Documentation network TEST-NET-2
>      private-address: 203.0.113.0/24  # Documentation network TEST-NET-3
>      private-address: 233.252.0.0/24  # Documentation network MCAST-TEST-NET
>      private-address: ::/128            # Unspecified address IPV4 0.0.0.0 http://www.iana.org/go/rfc4291
>      private-address: ::1/128   # Loopback Localhost http://www.iana.org/go/rfc4291
>      private-address: 2001::/23 # IETF Protocol Assignments http://www.iana.org/go/rfc2928
>      private-address: 2001:db8::/32     # Documentation network IPv6 http://www.iana.org/go/rfc3849
>      private-address: 2001:2::/48       # is reserved for Benchmarking http://www.iana.org/go/rfc5180 http://www.rfc-editor.org/errata_search.php?eid=1752
>      private-address: fc00::/7  # Unique local address (ULA) part of "fc00::/7", not defined yet
>      private-address: fd00::/8  # Unique local address (ULA) part of "fc00::/7", "/48" prefix group
>      private-address: fe80::/10 # Link-local address (LLA) = 169.254.0.0/16
>      private-address: ::ffff:0:0/96     # IPv4-mapped Address http://www.iana.org/go/rfc4291 ::ffff:x.x.x.x
>      prefetch: yes
>      prefetch-key: yes
>      minimal-responses: no
>      module-config: "respip validator iterator"
>      auto-trust-anchor-file: "C:\Program Files\Unbound\root.key"
>      val-log-level: 2
> server: # MyPerformance.conf
>      outgoing-range: 4096
>      outgoing-num-tcp: 40
>      incoming-num-tcp: 40
>      so-reuseport: no
>      target-fetch-policy: "4 3 2 1 0 0"
>      stream-wait-size: 16m
> server: # MyUseMixedCase.conf
>      use-caps-for-id: no
> server: # MyLocalHostNetwork.conf
>      private-domain: "localhost"
>      local-zone: "localhost." static
>      local-data: "localhost. IN NS localhost."
>      local-data: "localhost. IN SOA localhost. nobody1.invalid. 1 3600 1200 604800 10800"
>      local-data: "localhost. IN A 127.0.0.1"
>      local-data: "localhost. IN AAAA ::1"
>      local-data-ptr: "127.0.0.1 localhost."
>      local-data-ptr: "::1 localhost."
>      local-zone: "127.in-addr.arpa." static
>      local-data: "127.in-addr.arpa. 10800 IN NS localhost."
>      local-data: "127.in-addr.arpa. 10800 IN SOA localhost. nobody2.invalid. 1 3600 1200 604800 10800"
>      local-data: "1.0.0.127.in-addr.arpa. 10800 IN PTR localhost."
>      local-zone: "1.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.ip6.arpa." static
>      local-data: "1.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.ip6.arpa. 10800 IN NS localhost."
>      local-data: "1.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.ip6.arpa. 10800 IN SOA localhost. nobody3.invalid. 1 3600 1200 604800 10800"
>      local-data: "1.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.ip6.arpa. 10800 IN PTR localhost."
> server: # MyLocalNetwork.conf
>      private-domain: "homelan"
>      local-zone: "homelan" static
>      local-data: "@ IN SOA localhost. nobody4.invalid. 1 3600 1200 604800 10800"
>      local-data: "IN NS localhost."
> #
> # I have removed my local network configuration from this section.
> #
> forward-zone: # MyForwardZones.conf
>      name: "."
>      forward-tls-upstream: yes
>      forward-first: no
>      forward-addr: 1.1.1.1 at 853#cloudflare-dns.com
>      forward-addr: 1.0.0.1 at 853#cloudflare-dns.com
>      forward-addr: 2606:4700:4700::1111 at 853#cloudflare-dns.com
>      forward-addr: 2606:4700:4700::1001 at 853#cloudflare-dns.com
>      forward-addr: 2620:fe::fe at 853#dns.quad9.net
>      forward-addr: 9.9.9.9 at 853#dns.quad9.net
>      forward-addr: 8.8.8.8 at 853#Dns.google
>      forward-addr: 8.8.4.4 at 853#Dns.google
>      forward-addr: 2001:4860:4860::8888 at 853#Dns.google
>      forward-addr: 2001:4860:4860::8844 at 853#Dns.google
>      forward-addr: 94.130.110.185 at 853#ns1.dnsprivacy.at
>      forward-addr: 94.130.110.178 at 853#ns2.dnsprivacy.at
> include: "MyDoTConfig.conf"
> server: # MyDoTConfig.conf
>      tls-port: 853
>      tls-ciphers: "DHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-SHA256:DHE-RSA-AES128-SHA256:ECDHE-RSA-AES256-SHA384:ECDHE-RSA-AES128-SHA256"
>      tls-ciphersuites: "TLS_AES_128_GCM_SHA256:TLS_AES_128_CCM_8_SHA256:TLS_AES_128_CCM_SHA256:TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256"
>      tls-upstream: yes
>      tls-win-cert: yes
> remote-control: # MyRemoteControl.conf
>      control-enable: yes
>      control-use-cert: yes
>      control-interface: x.x.x.x
>      control-port: xxxxx
>      server-key-file: "C:\ProgramData\Unbound\Info\unbound_server.key"
>      server-cert-file: "C:\ProgramData\Unbound\Info\unbound_server.pem"
>      control-key-file: "C:\ProgramData\Unbound\Info\unbound_control.key"
>      control-cert-file: "C:\ProgramData\Unbound\Info\unbound_control.pem"
>      rpz: # MyResponsePolicyZones.conf
>      name: "URLHaus"
>      zonefile: "C:\ProgramData\Unbound\Logs\urlhaus.zone"
>      url: https://urlhaus.abuse.ch/downloads/rpz
>      rpz-log: yes
>      rpz-log-name: "URLHausRPZ"
>      rpz-action-override: nxdomain
> server: # MyAddToBlockList.conf
>      local-zone: home always_nxdomain
> server: # MyBlockList.conf
> # Reset requested, all Blocklist entries removed.
>
>
> -----Original Message-----
> From: George Thessalonikefs <george at nlnetlabs.nl>
> Sent: 10 November 2020 15:47
> To: RayG <rgsub1 at btinternet.com>; unbound-users at lists.nlnetlabs.nl
> Subject: Re: RPZ: is this config correct?
>
> Hi RayG,
>
> You don't have to create the file before starting unbound. If the file is there unbound will try to parse it.
>
> You don't have to manually populate the file with anything unless the rpz source is only a file. Not for your case though.
>
> I see in your log:
> ...
> 10/11/2020 15:00:14 C:\Program Files\Unbound\unbound.exe[15932:0] debug:
> read zonefile C:\ProgramData\Unbound\Logs\rpz.urlhaus.abuse.ch for rpz.urlhaus.abuse.ch.
> ...
> 10/11/2020 15:05:24 C:\Program Files\Unbound\unbound.exe[15932:0] debug:
> auth zone rpz.urlhaus.abuse.ch. transfer failed, wait ...
>
> The first one shows unbound reading from the file it created from a previous run probably.
>
> The second one shows that unbound could not complete the transfer and will try later.
>
> Best regards,
> -- George
>
> On 10/11/2020 16:33, RayG wrote:
> > OK so with log level set at 4 I don’t see in the log file lines like 
> > you list below
> >
> > The log file is very large so I cannot attach it to this email even as pasted text.
> >
> > Here is a link:
> > https://1drv.ms/u/s!As73rPtzISrU4mTvPONvZCWVSCWD?e=MJ18Lx
> >
> > A couple of points.
> >
> > 1). Do I have to create the zone file before starting unbound?
> > 2). Do I need to populate the file with anything?
> >
> > I have tried all ways with no success.
> >
> > Thanks
> >
> > RayG
> >
> > -----Original Message-----
> > From: George Thessalonikefs <george at nlnetlabs.nl>
> > Sent: 09 November 2020 17:35
> > To: unbound-users at lists.nlnetlabs.nl
> > Cc: RayG <rgsub1 at btinternet.com>
> > Subject: Re: RPZ: is this config correct?
> >
> > Hi RayG,
> >
> > On verbosity >= 4 you could see the following entries that relate to rpz (from my own run where download and file creation succeed):
> >       debug: auth zone rpz.urlhaus.abuse.ch. transfer next HTTP fetch from
> >       debug: http download downloads/rpz of size
> >       info: auth zone http downloaded content preview:
> >       debug: auth zone rpz.urlhaus.abuse.ch. updated to serial
> >       debug: write zonefile file.name for rpz.urlhaus.abuse.ch.
> >
> > local-zone answers are before the rpz zones, so you will not see entries in the log file for those.
> >
> > Best regards,
> > -- George
> >
> > On 09/11/2020 17:55, RayG wrote:
> >> Hi George,
> >>
> >> Thanks for the reply.
> >>
> >> I agree the XFR may not work but the URL should get the zone file.
> >>
> >> Given I have pointed the zone file at unbound's log directory and 
> >> it can write the log OK I think it should be able to write the zone file.
> >> It writes the log OK.
> >>
> >> Again if I have understood correctly these two lines:
> >>
> >> rpz-log: yes
> >> rpz-log-name: "URLHausRPZ"
> >>
> >> Make unbound write entries in the log file using the label "URLHausRPZ"
> >>
> >> I see none of those or a zone file in the log directory.
> >>
> >> What should I be looking for in unbound's log file to show it's working?
> >>
> >> One other question does it matter that some of the same entries may 
> >> have been entered in a block list file
> >>
> >> Which has entries looking like this:
> >>
> >> local-zone: "0.nextyourcontent.com" refuse # Source:
> >> https://someonewhocares.org/hosts/
> >> local-zone: "0.r.msn.com" refuse # Source:
> >> http://winhelp2002.mvps.org/hosts.txt
> >> local-zone: "0.start.bz" refuse # Source:
> >> http://sysctl.org/cameleon/hosts.win
> >> local-zone: "180clubrealestate.com" refuse # See:
> >> https://urlhaus.abuse.ch/host/180clubrealestate.com
> >>
> >> RayG
> >>
> >> -----Original Message-----
> >> From: George Thessalonikefs <george at nlnetlabs.nl>
> >> Sent: 09 November 2020 11:07
> >> To: unbound-users at lists.nlnetlabs.nl
> >> Subject: Re: RPZ: is this config correct?
> >>
> >> Hi RayG,
> >>
> >> You are correct that the file should be written by unbound. Are you 
> >> sure that unbound has write permissions in that directory?
> >>
> >> You could also use IP addresses for XFRs and they will be probed 
> >> for the SOA value and also tried if the url does not work.
> >>
> >> However, I don't think that they offer the service over XFR. At 
> >> least they only advertise the url on their website.
> >>
> >> Best regards,
> >> -- George
> >>
> >> On 07/11/2020 16:17, RayG via Unbound-users wrote:
> >>> Hi,
> >>>
> >>> No response to this post as yet?
> >>>
> >>> Any help appreciated.
> >>>
> >>> RayG
> >>>
> >>> *From:*RayG <rgsub1 at btinternet.com>
> >>> *Sent:* 14 October 2020 15:59
> >>> *To:* 'unbound-users at lists.nlnetlabs.nl'
> >>> <unbound-users at lists.nlnetlabs.nl>
> >>> *Subject:* RPZ: is this config correct?
> >>>
> >>> I have created the following RPZ entry for unbound and added 
> >>> respip to the module configuration.
> >>>
> >>> rpz:
> >>>
> >>>         name: "rpz.urlhaus.abuse.ch."
> >>>
> >>>         zonefile: "c:\programdata\unbound\logs\URLHaus.rpz"
> >>>
> >>>         url: https://urlhaus.abuse.ch/downloads/rpz
> >>> <https://urlhaus.abuse.ch/downloads/rpz>
> >>>
> >>>         rpz-log: yes
> >>>
> >>>         rpz-log-name: "URLHausRPZ"
> >>>
> >>> If I understand things correctly unbound should fetch the zone 
> >>> file using the URL and store the data in the zonefile. I created 
> >>> an empty zone file but it is not being populated by unbound. I 
> >>> cannot see any relevant issues in the log file.  I also do not 
> >>> (have not yet) seen any entries in the log file with the appended log name item.
> >>>
> >>> Do I have the correct configuration and understanding?
> >>>
> >>> Following on would it be correct to add these masters to the
> >> configuration:
> >>>
> >>>         master: 151.101.130.49
> >>>
> >>>         master: 151.101.66.49
> >>>
> >>>         master: 151.101.194.49
> >>>
> >>>         master: 151.101.2.49
> >>>
> >>> C:\>dig urlhaus.abuse.ch.
> >>>
> >>> ; <<>> DiG 9.16.6 <<>> urlhaus.abuse.ch.
> >>>
> >>> ;; global options: +cmd
> >>>
> >>> ;; Got answer:
> >>>
> >>> ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 1870
> >>>
> >>> ;; flags: qr rd ra; QUERY: 1, ANSWER: 5, AUTHORITY: 0, ADDITIONAL: 
> >>> 1
> >>>
> >>> ;; OPT PSEUDOSECTION:
> >>>
> >>> ; EDNS: version: 0, flags:; udp: 4096
> >>>
> >>> ;; QUESTION SECTION:
> >>>
> >>> ;urlhaus.abuse.ch.              IN      A
> >>>
> >>> ;; ANSWER SECTION:
> >>>
> >>> urlhaus.abuse.ch.       3037    IN      CNAME
> >> p2.shared.global.fastly.net.
> >>>
> >>> p2.shared.global.fastly.net. 29 IN      A       151.101.130.49
> >>>
> >>> p2.shared.global.fastly.net. 29 IN      A       151.101.194.49
> >>>
> >>> p2.shared.global.fastly.net. 29 IN      A       151.101.2.49
> >>>
> >>> p2.shared.global.fastly.net. 29 IN      A       151.101.66.49
> >>>
> >>> The URL Returns data like this:
> >>>
> >>> $TTL 30
> >>>
> >>> @ SOA rpz.urlhaus.abuse.ch. hostmaster.urlhaus.abuse.ch. 
> >>> 2010141440
> >>> 300
> >>> 1800 604800 30
> >>>
> >>> NS localhost.
> >>>
> >>> ;
> >>>
> >>> ; abuse.ch URLhaus Response Policy Zones (RPZ)
> >>>
> >>> ; Last updated: 2020-10-14 14:40:12 (UTC)
> >>>
> >>> ;
> >>>
> >>> ; Terms Of Use: https://urlhaus.abuse.ch/api/ 
> >>> <https://urlhaus.abuse.ch/api/>
> >>>
> >>> ; For questions please contact urlhaus [at] abuse.ch
> >>>
> >>> ;
> >>>
> >>> testentry.rpz.urlhaus.abuse.ch CNAME . ; Test entry for testing 
> >>> URLhaus RPZ
> >>>
> >>> 1am.co.nz CNAME . ; Malware download (2020-08-17), see 
> >>> https://urlhaus.abuse.ch/host/1am.co.nz/
> >>> <https://urlhaus.abuse.ch/host/1am.co.nz/>
> >>>
> >>> 1ca.co.za CNAME . ; Malware download (2020-08-28), see 
> >>> https://urlhaus.abuse.ch/host/1ca.co.za/
> >>> <https://urlhaus.abuse.ch/host/1ca.co.za/>
> >>>
> >>> 1med.kiev.ua CNAME . ; Malware download (2020-10-14), see 
> >>> https://urlhaus.abuse.ch/host/1med.kiev.ua/
> >>> <https://urlhaus.abuse.ch/host/1med.kiev.ua/>
> >>>
> >>> 21robo.com CNAME . ; Malware download (2019-02-20), see 
> >>> https://urlhaus.abuse.ch/host/21robo.com/
> >>> <https://urlhaus.abuse.ch/host/21robo.com/>
> >>>
> >>
> >>
> >
> >
>


--
Eduardo Schoedler



More information about the Unbound-users mailing list