document MISSING-BASE

diff --git a/input/howto/postgres-backup.creole b/input/howto/postgres-backup.creole
index 9c07596..3afeaa6 100644 (file)
--- a/input/howto/postgres-backup.creole
+++ b/input/howto/postgres-backup.creole
@@ -100,3 +100,51 @@ to see the port for the cluster, and run
   sudo -u debbackup /usr/local/bin/postgres-make-base-backups <host>:<port>
 }}}
 probably best to do that in a screen as it might take a while.
+
+== MISSING-BASE ==
+
+e.g.:
+{{{
+sudo -u debbackup /usr/lib/nagios/plugins/dsa-check-backuppg  | grep BASE
+[fasolo, dak] MISSING-BASE: dak.BASE.backuphost.debian.org-20180211-012002-fasolo.debian.org-dak-9.6-backup.tar.gz
+}}}
+
+This means that we started doing a base backup (as witnessed by a .backup file
+next to a WAL), but for some reason we don't have the corresponding base file.
+{{{
+root@backuphost:/srv/backups/pg/fasolo# ls -l *backup*
+-rw------- 1 debbackup debbackup 9201093916 Jan 14 06:18 dak.BASE.backuphost.debian.org-20180114-012001-fasolo.debian.org-dak-9.6-backup.tar.gz
+-rw------- 1 debbackup debbackup 9227651542 Jan 21 06:25 dak.BASE.backuphost.debian.org-20180121-012001-fasolo.debian.org-dak-9.6-backup.tar.gz
+-rw------- 1 debbackup debbackup 9266306750 Jan 28 07:59 dak.BASE.backuphost.debian.org-20180128-012001-fasolo.debian.org-dak-9.6-backup.tar.gz
+-rw------- 1 debbackup debbackup 9312602089 Feb  5 11:00 dak.BASE.backuphost.debian.org-20180204-012001-fasolo.debian.org-dak-9.6-backup.tar.gz
+-rw------- 1 debbackup debbackup 9346830509 Feb 12 10:25 dak.BASE.backuphost.debian.org-20180212-094930-fasolo.debian.org-dak-9.6-backup.tar.gz
+-rw------- 1 debbackup debbackup        353 Jan 14 06:18 dak.WAL.0000000100000033000000A6.00000028.backup
+-rw------- 1 debbackup debbackup        350 Jan 20 11:20 dak.WAL.00000001000000350000008C.00000028.backup
+-rw------- 1 debbackup debbackup        353 Jan 21 06:25 dak.WAL.000000010000003600000068.00000028.backup
+-rw------- 1 debbackup debbackup        353 Jan 28 07:59 dak.WAL.0000000100000038000000E3.00000028.backup
+-rw------- 1 debbackup debbackup        353 Feb  5 11:00 dak.WAL.000000010000003B00000090.00000028.backup
+-rw------- 1 debbackup debbackup        350 Feb  5 15:49 dak.WAL.000000010000003B0000009B.00000108.backup
+-rw------- 1 debbackup debbackup        353 Feb 11 10:09 dak.WAL.000000010000003D000000AC.00000028.backup
+-rw------- 1 debbackup debbackup        353 Feb 12 10:25 dak.WAL.000000010000003E00000027.00000178.backup
+}}}
+
+{{{.backup}}} files are created on the postgres server and shipped to the
+backup hosts whenever a base backup is initiated.  We do some labelling, so
+we know which backup host the corresponding tarball should end up with.
+
+e.g.:
+{{{
+root@backuphost:/srv/backups/pg/fasolo# cat dak.WAL.000000010000003B00000090.00000028.backup
+START WAL LOCATION: 3B/90000028 (file 000000010000003B00000090)
+STOP WAL LOCATION: 3B/97CF2138 (file 000000010000003B00000097)
+CHECKPOINT LOCATION: 3B/90000098
+BACKUP METHOD: streamed
+BACKUP FROM: master
+START TIME: 2018-02-05 10:25:28 UTC
+LABEL: backuphost.debian.org-20180204-012001-fasolo.debian.org-dak-9.6-backup
+STOP TIME: 2018-02-05 10:59:50 UTC
+}}}
+
+To fix this, verify we have a later base tarball, or that we are fine for some other reason,
+and remove the corresponding .backup file.  In the case above, we would remove
+{{{dak.WAL.000000010000003D000000AC.00000028.backup}}}.