[RPKI] HA strategy

Wed Feb 19 12:04:47 UTC 2020

Hi Tim

Thanks a lot for an extensive reply. So in essense, I have 16 hours to bring my repo back up one way or another. 

Good idea regarding Krill instance dedicated solely for CA purposes and then configuring it to publish into Repo Krill. I assume there's no way to configure it right now so it publishes into multiple repos, right? This, at least on paper, would simplify data dir synchronization between redundant repo servers.

So in my case, setup should move from (single instance):

1. Add CA <ca handle>
2. Grab CA identity XML, hand it over to RIR
3. Configure embedded repo for <ca handle>
4. Add parent for <ca handle> with RIR response XML
5. Configure HTTPS proxy pass for RRDP
6. Configure rsyncd to serve files

To:

1. Create CA Krill instance somewhere safe
2. Create Repo Krill instance
3. Add CA <ca handle> on CA Krill
4. Grab publisher request
5. Add CA Krill as publisher on Repo Krill, grab response
6. repo update rfc8183 <repo response> on CA Krill
7. [???] Add parent for <ca handle> with RIR response XML on CA Krill [???]

This can be HA'd to ones taste, either geo redundant or single site with tools that one likes:
8. Configure HTTPS proxy pass for RRDP
9. Configure rsyncd to serve files

Something feels off above. Did I get it right?

Thanks,
Aistis

From: Tim Bruijnzeels <tim at nlnetlabs.nl> 
Sent: 2020 m. vasario 17 d., pirmadienis 15:07
To: Aistis Zenkevičius <aistis at heficed.com>
Cc: rpki at lists.nlnetlabs.nl
Subject: Re: [RPKI] HA strategy

Hi Aistis,

First of all, thank you for using Krill and congratulations on being the first to run it in production under the RIPE NCC!

I will just stick to the generic story here. My colleague Ximon is on holidays at the moment, but he may be able to provide some docker specific insights at some point.

Krill does not yet support running in a cluster or multi-master setup. This is something we want to look at later this year. For now the best approach is that you have a stand-by node, and that you use some kind of redundant storage for your main krill instance's data directory. If your main krill node dies, you can then bring up a secondary node using the same data. Linux HA may be useful, and there are docker strategies, that I don't have hands-on experience with, but which can help here.

When Krill is unavailable you will not be able to make updates to your ROAs, and Krill will not (re-)publish objects at a repository.

However, this does not need to affect relying parties as long as the repository remains available, and Krill comes back before objects start to become stale (you have 16 hours with the current built-in defaults, we can think of stretching this). So, for example, users of Krill under http://nic.br have the option of running their CA and publishing with http://nic.br. So their uptime requirements for Krill are relaxed, because http://nic.br makes the repository available.

If you need to run your own repository, then it is advisable to use multiple rsync servers and multiple http servers. You can also use a CDN in front of your http servers. Sadly, our documentation is still somewhat lacking on how to do this part of the setup. This will be fixed soon. In the meantime let me try to give an overview here.

Krill can use an embedded repository if so configured. You will need to set the following directives in the config file:

repo_enabled = false
rsync_base = "rsync://<YOURHOST>/repo/"
rrdp_service_uri = "https://<YOURHOST>/rrdp/"

Krill will write the repository files under its data directory:

$DATA_DIR/repo/rsync/current/    Contains the files for rsync
$DATA_DIR/repo/rrdp/             Contains the files for https (RRDP as the protocol is called in the RFC)

You can share the contents of these directories with your repository servers in various ways. You could have some kind of redundant shared filesystem where the krill CA can write, and your repository servers can read. But you could also synchronise the contents of these directories in another way, e.g. rsync them over every couple of minutes. If you are using a shared filesystem, then please note that the rsync 'current' directory cannot be the mountpoint. Krill tries to write the entire repo to a new folder under $DATA_DIR/repo/rsync and then renames it. This is done to minimise issues with files being updated while validators are fetching data.

You will then need to configure your rsync deamons to expose a 'module' for your files. Make sure that the rsync URI including the 'module' matches the 'rsync_base' in Krill's configuration file. Basic configuration can then be as simple as:

root at krillprod:~# cat /etc/rsyncd.conf
uid = nobody
gid = nogroup
max connections = 50
socket options = SO_KEEPALIVE

[repo]
path = /var/lib/krill/data/repo/rsync/current/
comment = RPKI repository
read only = yes

For the http you will need to set up your favourite http server and make sure that it has a valid https certificate (eg use letsencrypt). Then you can make the files found under, or copied from, $DATA_DIR/repo/rrdp here. Make sure that the public URI to the rrdp base dir matches the value of 'rrdp_service_uri' in your krill.conf. If you want, you can also use a CDN in front of this server to further reduce your load and uptime requirements. If you do, be sure that the public URI matched the directive in the krill.conf because this will be used in certificates.

Now, to make matters even more interesting.. it is very hard - almost impossible - to change any of this set up in Krill's embedded repository after it has been initialised. This is at least in part because we want to encourage users to use a repository provided by their NIR or RIR whenever possible. That said, we understand that you may not have that option today..

Therefore we recommend that, if you run your own repository, you do so by running a separate Krill instance for this purpose only. Do not configure any CAs under it. Just let your CA publish at this repository as is described here: https://rpki.readthedocs.io/en/latest/krill/remote-publishing.html#krill-as-repository-server

If you find that there is an issue with your repository, e.g. your URIs are wrong, then you can set up a new Krill instance with embedded repository. When you are satisfied that this one is correct, you can migrate your CA to it by adding them as a publisher under the new repository server, and then updating your CA to use the new repository. Krill will then make sure that objects are moved properly, and that a new certificate is requested from your parent(s) to match the new location.

Note, if/when your RIR starts providing a repository service you can also update your CA to start publishing there.

As I said, I understand that this part of the documentation still needs some love. I will look at this as soon as I am done with the upcoming 0.5 release. For now, I hope this helps.

Thanks
Tim

On 17 Feb 2020, at 11:42, Aistis Zenkevičius via RPKI <mailto:rpki at lists.nlnetlabs.nl> wrote:

Hi list

Was wondering if you have tips on running Krill in redundant configuration. I've ran it for half a week out of a prebuilt docker instance with a couple of CAs with no issue and plan on moving forward with a lot of more. 

My preference is multi-DC redundance, in case one of the sites goes down. So for the frontend I'm thinking between the lines of DNS with minimal TTL (e.g. via Route53) or unicast, rsync served files can probably be sinced with some rudimentary script, but what about CA backend itself?

Thanks
Aistis

-- 
RPKI mailing list
mailto:RPKI at lists.nlnetlabs.nl
https://lists.nlnetlabs.nl/mailman/listinfo/rpki