Why do a puppetdb intervention?

The PostgreSQL database under PuppetDB is bloated, its performance is degraded. This means that commands sent to PuppetDB (replace facts, replace catalogs) are waiting in a queue until the data can be entered in the database. Due to the degraded performance the queue's length is continuously increasing (120 000+ at the moment of writing), which means several hours of delay. New exported resources, or facts with change values cannot be queried from PuppetDB for the amount of this delay.

The reason for bloat is partially because of a bug in PuppetDB. When a catalog is sent to PuppetDB a hash is calculated. If the catalog stored has the same hash as the new one, then only a timestamp is updated. If there is a difference, the new catalog is stored, and the old ones are wiped periodically, at the garbage collection (hourly). The problem is that in the current version of PuppetDB a different hash can generated for the very same catalog, causing a lot of unnecessary writes, increasing table bloat. The automatic maintenance jobs in the database cannot keep up with this bloat, and regular maintenance is required.

The rate at which catalogs are replaced is called catalog duplication rate. Our rate is currently 5%, which is a very low value, meaning that 95% of incoming catalogs contain changes. That obviously cannot be true.

The above figure shows the low catalog duplication rate in our node population, as well as the high queue.

This figure shows the growth of the queue in the last 3 days.

In order do this with minimal, or no downtime at all, PuppetDB is switched to a backup database, that is restored from a backup created on the master database. Due to the amount of data, this procedure takes 2 hours.The command queue processing is not halted, so any changes that were entering the databse during that two hours are essentially lost, until the next Puppet run. Keeping a consistent state would require taking PuppetDB offline for the duration of the backup and restore operations