Here’s a weird error message for my collection…

I’ve been working with a customer to setup a RMAN backup on his standby database (backup is going to SBT). After a while, we decided that we couldn’t avoid using the RMAN catalog so we created one.
Now, when using the catalog, we can backup the database from either the primary or standby instances and that will register to a shared catalog so we could restore from either database.

RMAN> LIST DB_UNIQUE_NAME OF DATABASE;


List of Databases
DB Key  DB Name  DB ID            Database Role    Db_unique_name
------- ------- ----------------- ---------------  ------------------
1       ORCL     1415951511       PRIMARY          ORCL
1       ORCL     1415951511       STANDBY          ORCLDG

The thing is, that if we change the DB_UNIQUE_NAME of the backup pieces (using RMAN’s CHANGE command), they will now be “owned” by the wrong database server and the files will not be available when running CROSSCHECK command.

RMAN> CHANGE BACKUP FOR DB_UNIQUE_NAME orcldg RESET DB_UNIQUE_NAME TO orcl;

change backup piece db_unique_name
backup piece handle=/data/fast_recovery_area/ORCLDG/backupset/2016_05_01/o1_mf_nnndf_TAG20160501T174121_cld5dkrj_.bkp RECID=15 STAMP=910719681
change backup piece db_unique_name
backup piece handle=/data/fast_recovery_area/ORCLDG/autobackup/2016_05_01/o1_mf_s_910715737_cld5dm03_.bkp RECID=16 STAMP=910719682
Changed 2 objects db_unique_name

When I want to remove the files (using the RMAN DELETE command), the files are unavailable since they are not in that server

RMAN> delete backup of datafile 5;

allocated channel: ORA_DISK_1
channel ORA_DISK_1: SID=48 device type=DISK

List of Backup Pieces
BP Key  BS Key  Pc# Cp# Status      Device Type Piece Name
------- ------- --- --- ----------- ----------- ----------
473     469     1   1   AVAILABLE   DISK        /data/fast_recovery_area/ORCLDG/backupset/2016_05_01/o1_mf_nnndf_TAG20160501T163703_cld1mz84_.bkp
510     506     1   1   AVAILABLE   DISK        /data/fast_recovery_area/ORCLDG/backupset/2016_05_01/o1_mf_nnndf_TAG20160501T173016_cld4qrcc_.bkp
545     541     1   1   AVAILABLE   DISK        /data/fast_recovery_area/ORCLDG/backupset/2016_05_01/o1_mf_nnndf_TAG20160501T173153_cld4tsrw_.bkp
583     579     1   1   AVAILABLE   DISK        /data/fast_recovery_area/ORCLDG/backupset/2016_05_01/o1_mf_nnndf_TAG20160501T173345_cld4yb3w_.bkp
612     608     1   1   AVAILABLE   DISK        /data/fast_recovery_area/ORCLDG/backupset/2016_05_01/o1_mf_nnndf_TAG20160501T173630_cld53gn8_.bkp
654     650     1   1   AVAILABLE   DISK        /data/fast_recovery_area/ORCLDG/backupset/2016_05_01/o1_mf_nnndf_TAG20160501T174121_cld5dkrj_.bkp
705     701     1   1   AVAILABLE   DISK        /data/fast_recovery_area/ORCLDG/backupset/2016_05_01/o1_mf_nnndf_TAG20160501T174429_cld5lfo6_.bkp

Do you really want to delete the above objects (enter YES or NO)? yes

RMAN-06207: WARNING: 7 objects could not be deleted for DISK channel(s) due
RMAN-06208:          to mismatched status.  Use CROSSCHECK command to fix status
RMAN-06210: List of Mismatched objects
RMAN-06211: ==========================
RMAN-06212:   Object Type   Filename/Handle
RMAN-06213: --------------- ---------------------------------------------------
RMAN-06214: Backup Piece    /data/fast_recovery_area/ORCLDG/backupset/2016_05_01/o1_mf_nnndf_TAG20160501T163703_cld1mz84_.bkp
RMAN-06214: Backup Piece    /data/fast_recovery_area/ORCLDG/backupset/2016_05_01/o1_mf_nnndf_TAG20160501T173016_cld4qrcc_.bkp
RMAN-06214: Backup Piece    /data/fast_recovery_area/ORCLDG/backupset/2016_05_01/o1_mf_nnndf_TAG20160501T173153_cld4tsrw_.bkp
RMAN-06214: Backup Piece    /data/fast_recovery_area/ORCLDG/backupset/2016_05_01/o1_mf_nnndf_TAG20160501T173345_cld4yb3w_.bkp
RMAN-06214: Backup Piece    /data/fast_recovery_area/ORCLDG/backupset/2016_05_01/o1_mf_nnndf_TAG20160501T173630_cld53gn8_.bkp
RMAN-06214: Backup Piece    /data/fast_recovery_area/ORCLDG/backupset/2016_05_01/o1_mf_nnndf_TAG20160501T174121_cld5dkrj_.bkp
RMAN-06214: Backup Piece    /data/fast_recovery_area/ORCLDG/backupset/2016_05_01/o1_mf_nnndf_TAG20160501T174429_cld5lfo6_.bkp

So I ran the CROSSCHECK on the primary database; it discovered that the files are missing and marked them as EXPIRED.

RMAN> crosscheck backup;

using channel ORA_DISK_1
crosschecked backup piece: found to be 'EXPIRED'
backup piece handle=/data/fast_recovery_area/ORCLDG/backupset/2016_05_01/o1_mf_nnndf_TAG20160501T163622_cld1lpkp_.bkp RECID=3 STAMP=910715782
crosschecked backup piece: found to be 'EXPIRED'
backup piece handle=/data/fast_recovery_area/ORCLDG/autobackup/2016_05_01/o1_mf_s_910715736_cld1lqv6_.bkp RECID=4 STAMP=910715783
crosschecked backup piece: found to be 'EXPIRED'
…
Crosschecked 16 objects

I then deleted the expired objects thinking I could re-add them on the standby server if needed.

RMAN> delete expired backup;

using channel ORA_DISK_1

List of Backup Pieces
BP Key  BS Key  Pc# Cp# Status      Device Type Piece Name
------- ------- --- --- ----------- ----------- ----------
471     467     1   1   EXPIRED     DISK        /data/fast_recovery_area/ORCLDG/backupset/2016_05_01/o1_mf_nnndf_TAG20160501T163622_cld1lpkp_.bkp
472     468     1   1   EXPIRED     DISK        /data/fast_recovery_area/ORCLDG/autobackup/2016_05_01/o1_mf_s_910715736_cld1lqv6_.bkpy
473     469     1   1   EXPIRED     DISK        /data/fast_recovery_area/ORCLDG/backupset/2016_05_01/o1_mf_nnndf_TAG20160501T163703_cld1mz84_.bkp
…
Do you really want to delete the above objects (enter YES or NO)? es
deleted backup piece
backup piece handle=/data/fast_recovery_area/ORCLDG/backupset/2016_05_01/o1_mf_nnndf_TAG20160501T163622_cld1lpkp_.bkp RECID=3 STAMP=910715782
deleted backup piece
backup piece handle=/data/fast_recovery_area/ORCLDG/autobackup/2016_05_01/o1_mf_s_910715736_cld1lqv6_.bkp RECID=4 STAMP=910715783
…
Deleted 16 EXPIRED objects

So now, my files are on disk, not in either control files or any catalog.
On Primary:

RMAN> list backup;

specification does not match any backup in the repository

On Standby:

RMAN> list backup;

specification does not match any backup in the repository

When I tried to re-add the files to the catalog I got this error:

RMAN> catalog start with '/data/fast_recovery_area/ORCLDG/backupset/2016_05_01/';

searching for all files that match the pattern /data/fast_recovery_area/ORCLDG/backupset/2016_05_01/
no files found to be unknown to the database

RMAN> catalog start with '/data/fast_recovery_area/ORCLDG/backupset/2016_05_01/*.bkp';

searching for all files that match the pattern /data/fast_recovery_area/ORCLDG/backupset/2016_05_01/*.bkp
no files found to be unknown to the database

RMAN> catalog start with '/data/fast_recovery_area/ORCLDG/backupset/2016_05_01/o1_mf_nnndf_TAG20160501T163622_cld1lpkp_.bkp' ;

searching for all files that match the pattern /data/fast_recovery_area/ORCLDG/backupset/2016_05_01/o1_mf_nnndf_TAG20160501T163622_cld1lpkp_.bkp
no files found to be unknown to the database

Kind of weird – it seems that there is some kind of record for that, but I couldn’t find any. The only valid workaround I could find was to move the files to a new directory and catalog that directory:

[oracle@lnx-oracle-2 backupset]$ mv 2016_05_01/ 2016_05_01b/

And then

RMAN> catalog start with '/data/fast_recovery_area/ORCLDG/backupset/2016_05_01b';

searching for all files that match the pattern /data/fast_recovery_area/ORCLDG/backupset/2016_05_01b

List of Files Unknown to the Database
=====================================
File Name: /data/fast_recovery_area/ORCLDG/backupset/2016_05_01b/o1_mf_nnndf_TAG20160501T173016_cld4qrcc_.bkp
File Name: /data/fast_recovery_area/ORCLDG/backupset/2016_05_01b/o1_mf_nnndf_TAG20160501T174429_cld5lfo6_.bkp
File Name: /data/fast_recovery_area/ORCLDG/backupset/2016_05_01b/o1_mf_nnndf_TAG20160501T173345_cld4yb3w_.bkp
File Name: /data/fast_recovery_area/ORCLDG/backupset/2016_05_01b/o1_mf_nnndf_TAG20160501T173153_cld4tsrw_.bkp
File Name: /data/fast_recovery_area/ORCLDG/backupset/2016_05_01b/o1_mf_nnndf_TAG20160501T163703_cld1mz84_.bkp
File Name: /data/fast_recovery_area/ORCLDG/backupset/2016_05_01b/o1_mf_nnndf_TAG20160501T174121_cld5dkrj_.bkp
File Name: /data/fast_recovery_area/ORCLDG/backupset/2016_05_01b/o1_mf_nnndf_TAG20160501T173630_cld53gn8_.bkp
File Name: /data/fast_recovery_area/ORCLDG/backupset/2016_05_01b/o1_mf_nnndf_TAG20160501T163622_cld1lpkp_.bkp

Do you really want to catalog the above files (enter YES or NO)? yes
cataloging files...
cataloging done

I’m not really sure what this is all about, but this is what I got so far. I will update if I find anything else on that matter (database version: 11.2.0.4)…

I will also try to post the steps we did in order to setup the backup from the standby and primary (including creating the catalog repository) if anyone interested.