Validation error with multi timeline chain #251
Description
Process
- Creating FULL backup with
--streamoption - Creating DELTA backup with
--streamoption - Doing manual switchover. Timelines are changed
- Creating another DELTA backup
- Run
pg_probackup-11 validation
Expected result
Backup validation process completes successfully:
INFO: Backup QFQ609 data files are valid
INFO: Backup QFQ609 WAL segments are valid
...
INFO: All backups are valid
Current result
Backup validation process ended with error:
INFO: Validating backup QFQ609
INFO: Backup QFQ609 data files are valid
WARNING: Thread [1]: Could not read WAL record at 61/CA000028
ERROR: Thread [1]: WAL segment "/var/backup/backups/db-mt/QFQ609/database/pg_wal/0000004A00000061000000CA" is absent
WARNING: There are not enough WAL records to consistenly restore backup QFQ609 from START LSN: 61/CA000028 to STOP LSN: 61/CA000198
WARNING: Backup QFQ609 WAL segments are corrupted
Backup status before validation:
===========================================================================================================================================
Instance Version ID Recovery Time Mode WAL Mode TLI Time Data WAL Zratio Start LSN Stop LSN Status
===========================================================================================================================================
db-mt 11 QFQ609 2020-08-27 13:41:17+00 DELTA STREAM 75/74 21s 356kB 16MB 47.21 61/CA000028 61/CA000198 OK
db-mt 11 QFQ5U3 2020-08-27 13:36:06+00 DELTA STREAM 74/74 1m:10s 201kB 16MB 81.79 61/C805C420 61/C8079110 OK
db-mt 11 QFQ5QG 2020-08-27 13:35:04+00 DELTA STREAM 74/74 1m:1s 388kB 16MB 43.37 61/C805C420 61/C80604F8 OK
db-mt 11 QFQ5AK 2020-08-27 13:25:04+00 FULL STREAM 74/0 1m:35s 1466MB 16MB 3.45 61/C801C1D0 61/C80361B0 OK
db-mt 11 QFQ1KZ 2020-08-27 12:05:43+00 DELTA STREAM 73/73 22s 317kB 16MB 52.40 61/C4000028 61/C4000198 OK
db-mt 11 QFQ1JD 2020-08-27 12:04:45+00 DELTA STREAM 73/73 21s 201kB 16MB 81.76 61/C2000028 61/C2000198 OK
db-mt 11 QFQ1HJ 2020-08-27 12:03:39+00 DELTA STREAM 73/73 21s 342kB 16MB 48.99 61/C0000028 61/C0000198 OK
db-mt 11 QFQ1CH 2020-08-27 12:01:51+00 FULL STREAM 73/0 1m:40s 1466MB 16MB 3.45 61/BE000028 61/BE01B5F0 OK
Some helpful stdout's:
ls -la /var/backup/backups/db-mt/QFQ609/database/pg_wal/
total 16396
drwx------ 2 postgres users 4096 Aug 27 13:40 .
drwx------ 19 postgres users 4096 Aug 27 13:41 ..
-rw------- 1 postgres users 16777216 Aug 27 13:41 0000004B00000061000000CA
-rw------- 1 postgres users 3269 Aug 27 13:40 0000004B.history
pg_probackup-11 show -B /var/backup --archive
ARCHIVE INSTANCE 'db-mt'
================================================================================================================================
TLI Parent TLI Switchpoint Min Segno Max Segno N segments Size Zratio N backups Status
================================================================================================================================
75 74 61/C9000098 0000004B00000061000000C9 0000004B00000061000000CA 2 35kB 930.10 1 OK
74 0 0/0 0000004A00000061000000C8 0000004A00000061000000C8 1 121kB 135.86 3 OK
73 0 0/0 0000004900000061000000BD 0000004900000061000000C8 12 764kB 257.21 4 OK
pg_probackup-11 show -B /var/backup --instance db-mt -i QFQ609
#Configuration
backup-mode = DELTA
stream = true
compress-alg = zlib
compress-level = 6
from-replica = false
#Compatibility
block-size = 8192
xlog-block-size = 8192
checksum-version = 1
program-version = 2.4.2
server-version = 11
#Result backup info
timelineid = 75
start-lsn = 61/CA000028
stop-lsn = 61/CA000198
start-time = '2020-08-27 13:40:57+00'
end-time = '2020-08-27 13:41:18+00'
recovery-xid = 6729008
recovery-time = '2020-08-27 13:41:17+00'
data-bytes = 364482
wal-bytes = 16777216
uncompressed-bytes = 17205767
pgdata-bytes = 5301082639
status = CORRUPT
parent-backup-id = 'QFQ5U3'
primary_conninfo = 'user=USER host=AA.BB.CC.DD port=PORT sslmode=prefer sslcompression=0 gssencmode=prefer krbsrvname=postgres target_session_attrs=any'
content-crc = 2463672895
Backup status after validation:
Leonid, [27 Aug 2020 at 18:46:03 (27 Aug 2020 at 18:46:30)]:
...ACKUP INSTANCE 'db-mt'
============================================================================================================================================
Instance Version ID Recovery Time Mode WAL Mode TLI Time Data WAL Zratio Start LSN Stop LSN Status
============================================================================================================================================
db-mt 11 QFQ609 2020-08-27 13:41:17+00 DELTA STREAM 75/74 21s 356kB 16MB 47.21 61/CA000028 61/CA000198 CORRUPT
db-mt 11 QFQ5U3 2020-08-27 13:36:06+00 DELTA STREAM 74/74 1m:10s 201kB 16MB 81.79 61/C805C420 61/C8079110 OK
db-mt 11 QFQ5QG 2020-08-27 13:35:04+00 DELTA STREAM 74/74 1m:1s 388kB 16MB 43.37 61/C805C420 61/C80604F8 OK
db-mt 11 QFQ5AK 2020-08-27 13:25:04+00 FULL STREAM 74/0 1m:35s 1466MB 16MB 3.45 61/C801C1D0 61/C80361B0 OK
db-mt 11 QFQ1KZ 2020-08-27 12:05:43+00 DELTA STREAM 73/73 22s 317kB 16MB 52.40 61/C4000028 61/C4000198 OK
db-mt 11 QFQ1JD 2020-08-27 12:04:45+00 DELTA STREAM 73/73 21s 201kB 16MB 81.76 61/C2000028 61/C2000198 OK
db-mt 11 QFQ1HJ 2020-08-27 12:03:39+00 DELTA STREAM 73/73 21s 342kB 16MB 48.99 61/C0000028 61/C0000198 OK
db-mt 11 QFQ1CH 2020-08-27 12:01:51+00 FULL STREAM 73/0 1m:40s 1466MB 16MB 3.45 61/BE000028 61/BE01B5F0 OK