Release notes for IRRd 4.1.0

IRRd 4.1.0 adds several major new features, including RPKI-aware mode, a new daemon architecture with full multiprocessing, synthetic NRTM, and a scope filter.

The changes between 4.0.x and 4.1.0 are major. Please read these release notes carefully before upgrading. Upgrading to IRRd 4.1.0 requires several changes to the deployment setup. Downgrading back to IRRd 4.0.x, if needed, also requires steps before downgrading the installed IRRd package.

New features

RPKI-aware mode

RPKI-aware mode is now available, where IRRd imports RPKI ROAs and can filter or reject RPSL objects that are in conflict with ROAs. Pseudo-IRR objects are generated for all ROAs. When RPKI-aware mode is enabled, RPKI becomes an invalid as a regular IRR source name, as it is reserved for pseudo-IRR objects from ROAs. RPKI-aware mode also affects mirroring.

IRRd 4.1.0 includes several database migrations to support RPKI-aware mode, whether enabled or disabled, and facilitate performance improvements needed for RPKI-aware mode. Running these migrations is required to run IRRd 4.1.0, even if RPKI-aware mode is never enabled.

Danger

The impact of running IRRd in RPKI-aware mode can be dramatic, and it is strongly recommended to read the RPKI integration documentation very carefully before enabling it, to understand all consequences. By default, RPKI-aware mode is enabled. RPKI-aware mode can be disabled entirely, or certain sources can be excluded from RPKI validation.

New daemon architecture

IRRd has a new daemon architecture, where all whois queries, HTTP requests, mirror processes and the preloader run in their own process. This improves performance significantly, especially where many processes are running on servers with many cores. Previously, the entire IRRd process was limited to one CPU core.

To communicate between processes, IRRd now requires a running Redis instance. The commands to start IRRd and several IRRd scripts have also changed. The --uid parameter is no longer supported.

Synthetic NRTM

The irrd_load_database command allowed loading RPSL data from local files, often used to load data generated by scripts or other systems. Data imported this way could not be mirrored over NRTM.

In IRRd 4.1, the irrd_update_database command has been added. This supports periodically updating a source to the state in a particular file, and automatically generates journal entries for any differences, allowing NRTM mirroring. See the mirroring documentation for further details.

Other changes

  • A scopefilter has been added. This allows you to filter RPSL objects matching certain prefixes and AS numbers from your IRRd instance. By default, the scope filter is disabled.

  • The default for server.whois.max_connections has been reduced from 50 to 10. In 4.1, IRRd whois workers use considerably more memory, about 200 MB each, and one worker is started for each permitted connection. Therefore, at the default 10 connections, the whois processes use about 2 GB of memory, at 50 connections, this is about 10 GB.

  • The last-modified attribute is set every time an object is created or updated in an authoritative source. You can apply this to all existing authoritative objects with the new irrd_set_last_modified_auth command.

  • Serial handling in IRRd has changed. If you mirror a source over NRTM, and keep a local journal, IRRd used to keep these serials identical. The NRTM ADD from the original source would be stored in the local journal under the same serial number, unless it was ignored by the object class filter. In 4.1, IRRd always keeps its own range of serials. That means serials for the same change may be different between different mirrors. If users change their NRTM source to a different one, they should reload the entire database from that source, not just resume NRTM streaming. Simply changing the NRTM host may lead to missing data.

  • The !j command has changed, and now is exclusively used to check mirroring status. It returns what the most recent serial processed from a mirror is. For more extensive status information, like the local serials in the journal, use the new !J command.

  • IRRd starts a maximum of three mirror processes at the same time, to reduce peak loads. A further three, if needed, are started 15 seconds later, regardless of whether the previous ones have finished.

  • HTTP(s) downloads are now supported for the sources.{name}.import_source and sources.{name}.import_serial_source settings.

  • A number of new configuration options were added, and some are required. See the configuration documentation for more information on these options.

  • RIPE style query responses now always end with two empty lines, consistent with the RIPE database.

  • A custom flexible logging config can now be set with the log.logging_config_path.

  • A timeout was added for FTP connections.

  • Memory usage during large RPSL imports has been reduced.

  • A bug was fixed where some invalid objects could cause parser exceptions.

Steps required to upgrade

The following steps are required to upgrade to IRRd 4.1.0, regardless of whether RPKI-aware mode is enabled or not.

  • Disable all cron and e-mail triggered tasks. There should be no calls to any IRRd scripts during the upgrade process.

  • Upgrade the IRRd package from within the virtualenv with pip install -U irrd

  • Install a Redis instance as documented in the deployment guide and configure the redis_url setting.

  • Note that unix sockets are strongly recommended over TCP sockets for both PostgreSQL and Redis, for improved performance. The effect of this is more significant with the new multi-process daemon architecture.

  • Set piddir to a directory where IRRd can write its PID file, irrd.pid.

  • Run the database migrations, using the same command used to create the tables initially in deployment. Important note: some of the migrations change large amounts of data, and may take up to 15-45 minutes to run in total. While the migrations are running, IRRd should be shut down and any cron / e-mail triggered tasks must be disabled. There must be no calls to irrd_submit_email or irrd_load_database.

  • Update any startup scripts or systemd for IRRd to call the new daemon process, with the new command line arguments, and use setcap to allow IRRd to bind to privileged ports: see the updated deployment guide.

  • Remove the --irrd_pidfile parameter from calls to irrd_submit_email and irrd_load_database.

  • Ensure that RPKI-aware mode is configured as desired. By default it is enabled.

  • Start IRRd and re-enable the cron / e-mail triggered tasks.

  • If you would like to set last-modified for existing authoritative objects, use the new irrd_set_last_modified_auth command.

Downgrading from 4.1 to 4.0.x

If you are running IRRd 4.1, and would like to downgrade back to 4.0.x, the database schema needs to be modified. You can either restore an older copy of your database, start with a fresh database, or use the database migrations.

If you want to use the database migrations, run this command before downgrading your local package installation to 4.0.x:

/home/irrd/irrd-venv/bin/irrd_database_downgrade --version 28dc1cd85bdc

If you would like to re-upgrade to 4.1 later on, you will need to run irrd_database_upgrade again, as noted in the steps above. The downgrade migration typically takes a few seconds.