[RPKI] HA strategy

Wed Feb 19 16:06:16 UTC 2020

Hi Aistis,

Re-sending because my email client (or I..) somehow messed up and my reply was not plain text as it should have been..

> On 19 Feb 2020, at 13:04, Aistis Zenkevičius <aistis at heficed.com> wrote:
> 
> Hi Tim
> 
> Thanks a lot for an extensive reply. So in essense, I have 16 hours to bring my repo back up one way or another. 

You have 16 hours to fix your Krill CA and/or Repo instances.

It's good to keep your repositories available as much as possible - i.e. the data. Most RPs will work with cached data, but it's best to be safe.

> 
> Good idea regarding Krill instance dedicated solely for CA purposes and then configuring it to publish into Repo Krill. I assume there's no way to configure it right now so it publishes into multiple repos, right? This, at least on paper, would simplify data dir synchronization between redundant repo servers.
> 

The RPKI standards only allow a single publication point in RPKI objects. So, you cannot get redundancy through secondary repositories, using other public URIs.

Theoretically you could have multiple repository servers that share the same public URI - (DNS, anycast, or simply loadbalanced) - while they have separate back-end service ports - and then CAs could, theoretically, be configured to publish with each separately. However, this would be quite complicated of the CAs, and it could easily result in differences in the objects that are available at each repo instance, which would be confusing.

All in all I think that sharing the data between the front-ends to the repo (httpd/rsyncd) is the easiest for now. Long term we could look into repo servers that are aware of each other, and that can synchronise somehow - though I am not sure that the benefit is worth the complexity just yet. Secondary servers is something that would have to go through the IETF and I think there would be some pushback there.

> So in my case, setup should move from (single instance):
> 
> 1. Add CA <ca handle>
> 2. Grab CA identity XML, hand it over to RIR
> 3. Configure embedded repo for <ca handle>
> 4. Add parent for <ca handle> with RIR response XML
> 5. Configure HTTPS proxy pass for RRDP
> 6. Configure rsyncd to serve files
> 
> To:
> 
> 1. Create CA Krill instance somewhere safe
> 2. Create Repo Krill instance
> 3. Add CA <ca handle> on CA Krill
> 4. Grab publisher request
> 5. Add CA Krill as publisher on Repo Krill, grab response
> 6. repo update rfc8183 <repo response> on CA Krill
> 7. [???] Add parent for <ca handle> with RIR response XML on CA Krill [???]

yes

> 
> This can be HA'd to ones taste, either geo redundant or single site with tools that one likes:
> 8. Configure HTTPS proxy pass for RRDP
> 9. Configure rsyncd to serve files
> 
> Something feels off above. Did I get it right?
> 

I would leave the Repo and CA Krills as single instances. Hot standby can work, provided that you keep their data dirs on reliable storage.

I would then HA the https proxy for RRDP and rsyncd based on the datadir which is updated by a single instance Repo Krill.

The RRDP content needs to be available to the world, so even though Krill can serve this data you are better off including the files on your proxy host. E.g. something like:

root at krillprod:/etc/nginx/sites-enabled# cat rrdp.rpki.nlnetlabs.nl 
server {
    server_name rrdp.rpki.nlnetlabs.nl;

    location /rrdp/ {
        alias /var/lib/krill/repo/rrdp/;
    }

    listen 443 ssl; # managed by Certbot
    listen [::]:443 ssl;
    ssl_certificate /etc/letsencrypt/live/rrdp.rpki.nlnetlabs.nl/fullchain.pem; # managed by Certbot
    ssl_certificate_key /etc/letsencrypt/live/rrdp.rpki.nlnetlabs.nl/privkey.pem; # managed by Certbot
    include /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot
    ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbot
}
server {
    if ($host = rrdp.rpki.nlnetlabs.nl) {
        return 301 https://$host$request_uri;
    } # managed by Certbot

    server_name rrdp.rpki.nlnetlabs.nl;
    listen 80;
    listen [::]:80;
    return 404; # managed by Certbot
}

If you want to make the API and UI (next week in 0.5!) available you can proxy everything else to Krill - you could then of course also choose to limit access in that config to your own network.

Tim

> 
> Thanks,
> Aistis
> 
> From: Tim Bruijnzeels <tim at nlnetlabs.nl> 
> Sent: 2020 m. vasario 17 d., pirmadienis 15:07
> To: Aistis Zenkevičius <aistis at heficed.com>
> Cc: rpki at lists.nlnetlabs.nl
> Subject: Re: [RPKI] HA strategy
> 
> Hi Aistis,
> 
> First of all, thank you for using Krill and congratulations on being the first to run it in production under the RIPE NCC!
> 
> I will just stick to the generic story here. My colleague Ximon is on holidays at the moment, but he may be able to provide some docker specific insights at some point.
> 
> Krill does not yet support running in a cluster or multi-master setup. This is something we want to look at later this year. For now the best approach is that you have a stand-by node, and that you use some kind of redundant storage for your main krill instance's data directory. If your main krill node dies, you can then bring up a secondary node using the same data. Linux HA may be useful, and there are docker strategies, that I don't have hands-on experience with, but which can help here.
> 
> When Krill is unavailable you will not be able to make updates to your ROAs, and Krill will not (re-)publish objects at a repository.
> 
> However, this does not need to affect relying parties as long as the repository remains available, and Krill comes back before objects start to become stale (you have 16 hours with the current built-in defaults, we can think of stretching this). So, for example, users of Krill under http://nic.br have the option of running their CA and publishing with http://nic.br. So their uptime requirements for Krill are relaxed, because http://nic.br makes the repository available.
> 
> If you need to run your own repository, then it is advisable to use multiple rsync servers and multiple http servers. You can also use a CDN in front of your http servers. Sadly, our documentation is still somewhat lacking on how to do this part of the setup. This will be fixed soon. In the meantime let me try to give an overview here.
> 
> Krill can use an embedded repository if so configured. You will need to set the following directives in the config file:
> 
> repo_enabled = false
> rsync_base = "rsync://<YOURHOST>/repo/"
> rrdp_service_uri = "https://<YOURHOST>/rrdp/"
> 
> Krill will write the repository files under its data directory:
> 
> $DATA_DIR/repo/rsync/current/    Contains the files for rsync
> $DATA_DIR/repo/rrdp/             Contains the files for https (RRDP as the protocol is called in the RFC)
> 
> You can share the contents of these directories with your repository servers in various ways. You could have some kind of redundant shared filesystem where the krill CA can write, and your repository servers can read. But you could also synchronise the contents of these directories in another way, e.g. rsync them over every couple of minutes. If you are using a shared filesystem, then please note that the rsync 'current' directory cannot be the mountpoint. Krill tries to write the entire repo to a new folder under $DATA_DIR/repo/rsync and then renames it. This is done to minimise issues with files being updated while validators are fetching data.
> 
> You will then need to configure your rsync deamons to expose a 'module' for your files. Make sure that the rsync URI including the 'module' matches the 'rsync_base' in Krill's configuration file. Basic configuration can then be as simple as:
> 
> root at krillprod:~# cat /etc/rsyncd.conf
> uid = nobody
> gid = nogroup
> max connections = 50
> socket options = SO_KEEPALIVE
>  
> [repo]
> path = /var/lib/krill/data/repo/rsync/current/
> comment = RPKI repository
> read only = yes
> 
> For the http you will need to set up your favourite http server and make sure that it has a valid https certificate (eg use letsencrypt). Then you can make the files found under, or copied from, $DATA_DIR/repo/rrdp here. Make sure that the public URI to the rrdp base dir matches the value of 'rrdp_service_uri' in your krill.conf. If you want, you can also use a CDN in front of this server to further reduce your load and uptime requirements. If you do, be sure that the public URI matched the directive in the krill.conf because this will be used in certificates.
> 
> 
> Now, to make matters even more interesting.. it is very hard - almost impossible - to change any of this set up in Krill's embedded repository after it has been initialised. This is at least in part because we want to encourage users to use a repository provided by their NIR or RIR whenever possible. That said, we understand that you may not have that option today..
> 
> 
> Therefore we recommend that, if you run your own repository, you do so by running a separate Krill instance for this purpose only. Do not configure any CAs under it. Just let your CA publish at this repository as is described here: https://rpki.readthedocs.io/en/latest/krill/remote-publishing.html#krill-as-repository-server
> 
> 
> If you find that there is an issue with your repository, e.g. your URIs are wrong, then you can set up a new Krill instance with embedded repository. When you are satisfied that this one is correct, you can migrate your CA to it by adding them as a publisher under the new repository server, and then updating your CA to use the new repository. Krill will then make sure that objects are moved properly, and that a new certificate is requested from your parent(s) to match the new location.
> 
> 
> Note, if/when your RIR starts providing a repository service you can also update your CA to start publishing there.
> 
> 
> 
> 
> As I said, I understand that this part of the documentation still needs some love. I will look at this as soon as I am done with the upcoming 0.5 release. For now, I hope this helps.
> 
> 
> 
> Thanks
> Tim
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> On 17 Feb 2020, at 11:42, Aistis Zenkevičius via RPKI <mailto:rpki at lists.nlnetlabs.nl> wrote:
> 
> Hi list
> 
> Was wondering if you have tips on running Krill in redundant configuration. I've ran it for half a week out of a prebuilt docker instance with a couple of CAs with no issue and plan on moving forward with a lot of more. 
> 
> My preference is multi-DC redundance, in case one of the sites goes down. So for the frontend I'm thinking between the lines of DNS with minimal TTL (e.g. via Route53) or unicast, rsync served files can probably be sinced with some rudimentary script, but what about CA backend itself?
> 
> Thanks
> Aistis
> 
> 
> 
> -- 
> RPKI mailing list
> mailto:RPKI at lists.nlnetlabs.nl
> https://lists.nlnetlabs.nl/mailman/listinfo/rpki