Alert rules

This page contains a markdown version of the alert rules described in the postgresql-k8s-operator repository. The following file(s) are the source of truth:

• src/prometheus_alert_rules/postgresql_rules.yaml

• src/prometheus_alert_rules/pgbouncer_rules.yaml

• src/prometheus_alert_rules/patroni_rules.yaml

This documentation describes the latest alert rule expressions. See the YAML file(s) listed above if you require an older version.

PostgresqlExporterK8s

Alert	Severity	Notes
PostgresqlDown		PostgreSQL instance is down. If you are not upgrading or configuring cross-region async replication clusters, check for errors in the Loki logs.
PostgresqlRestarted		PostgreSQL instance has restarted. If you are not enabling/disabling TLS or upgrading or configuring cross-region async replication clusters, check for errors in the Loki logs.
PostgresqlExporterError		PostgreSQL instance is showing an exporter error. There may be a buggy query in query.yaml
PostgresqlTableNotAutoVacuumed		A PostgreSQL table in instance is not auto vacuumed. A table has not been auto vacuumed for 7 days.Double-check your VACUUM settings.
PostgresqlTableNotAutoAnalyzed		A PostgreSQL table in instance is not auto analyzed. A table has not been auto analyzed for 7 days.Double-check your AUTOVACUUM ANALYZE settings.
PostgresqlTooManyConnections		PostgreSQL instance is using > 80% of the maximum connections. Consider checking how many connections the client application is opening, or using PgBouncer in front of the database.
PostgresqlNotEnoughConnections		PostgreSQL instance does not have enough connections. PostgreSQL instance should have more connections (> 5). Consider double-checking how many connections the client application is opening and/or using PgBouncer in front of the database.
PostgresqlDeadLocks		PostgreSQL instance has dead locks. See more details with the pg_locks view.
PostgresqlHighRollbackRate		PostgreSQL instance has a high rollback rate instance. The ratio of transactions being aborted compared to committed is > 2 %. This is probably happening due to unoptimized configurations related to commit delay, connections, memory, and WAL files.
PostgresqlCommitRateLow		PostgreSQL instance has a low commit rate. PostgreSQL seems to be processing very few transactions. Check for long-running queries and configuration issues, like insufficient cache size.
PostgresqlLowXidConsumption		PostgreSQL instance shows low XID consumption. PostgreSQL seems to be consuming transaction IDs very slowly. Run ANALYZE to update the optimizer statistics, ensure that query plans are correct, and double-check your VACUUM settings.
PostgresqlHighRateStatementTimeout		PostgreSQL instance shows a high rate of statement timeout. Either tune statement_timeout when sending queries or use EXPLAIN ANALYZE to understand how the queries can be improved.
PostgresqlHighRateDeadlock		PostgreSQL instance shows a high deadlock rate. More details can be obtained through the pg_locks view.
PostgresqlUnusedReplicationSlot		PostgreSQL instance has unused replication slots. Check if a replica is not using any of them before deleting it.
PostgresqlTooManyDeadTuples		PostgreSQL instance has too many dead tuples. Double-check your VACUUM settings.
PostgresqlConfigurationChanged		PostgreSQL instance configuration has changed. PostgreSQL database configuration has changed.
PostgresqlSslCompressionActive		PostgreSQL instance SSL compression is active. Database connections with SSL compression are enabled. This may add significant jitter in replication delay.Replicas should turn off SSL compression via sslcompression=0 in recovery.conf.
PostgresqlTooManyLocksAcquired		PostgreSQL instance has acquired too many locks. If this alert happens frequently, you may need to increase the PostgreSQL setting max_locks_per_transaction.
PostgresqlBloatIndexHigh(>80%)		PostgreSQL instance has a high bloat index (> 80%). An index is bloated.Consider running REINDEX INDEX CONCURRENTLY <index name>;
PostgresqlBloatTableHigh(>80%)		PostgreSQL instance has a high bloat table (> 80%). A table is bloated.Consider running VACUUM {{ $labels.relname }};
PostgresqlInvalidIndex		PostgreSQL instance )= has an invalid index. A table has an invalid index. Consider running DROP INDEX <index name>;

PgbouncerExporterK8s

Alert	Severity	Notes
PgbouncerActiveConnections		PgBouncer instance has > 200 active connections Consider checking the client application responsible for generating those additional connections.
PgbouncerErrors		PgBouncer instance is logging errors. This may be due to a a server restart or an admin typing commands at the PgBouncer console.
PgbouncerMaxConnections		PgBouncer instance has reached max_client_conn. Consider checking how many connections the client application is opening.

PatroniExporterK8s

Alert	Severity	Notes
PatroniPostgresqlDown		Patroni PostgreSQL instance is down. Check for errors in the Loki logs.
PatroniHasNoLeader		Patroni instance has no leader node. A leader node (neither primary nor standby) cannot be found inside a cluster. Check for errors in the Loki logs.

PgbackrestExporterK8s

Alert	Severity	Notes
PgBackRestBackupError		Backup failed for a stanza. The last pgBackRest backup ended with error status > 0. Check the pgBackRest logs for the stanza.
PgBackRestBackupTooOld		No recent backup available. The last pgBackRest backup is older than 7 days. Consider checking your backup schedule, capacity, and logs.
PgBackRestStanzaError		A stanza has reported errors. Status > 0 indicates problems such as missing stanza path or no valid backups. Check pgBackRest logs for details.
PgBackRestRepoError		A repository has reported errors. Status > 0 indicates the repository may be inaccessible, out of space, or otherwise unhealthy. Check pgBackRest logs and storage system.
PgBackRestExporterError		The pgBackRest exporter failed to fetch data. Metric pgbackrest_exporter_status == 0 indicates exporter-side issues. This may be a misconfiguration or runtime error; check exporter logs.