SNAPSHOT CONTROLFILE Isn’t Just a Backup Copy of a Control File Anymore

Posted in: Technical Track

Recently, I have troubleshooted failing RMAN backups in 11.2.0.2 RAC environment. The backups failed with an error message “ORA-00245: control file backup operation failed”. The solution for the issue is simple. You may know it already, but starting from 11.2.0.2 version, a control file copy in RAC environments needs to be located on a shared location.

This fact is clearly indicated in the 11GR2 documentation: “You can specify a cluster file system or a raw device destination for the location of your snapshot control file. This file is shared across all nodes in the cluster and MUST be accessible by all nodes in the cluster.” I am providing error messages that I saw in relation to the issue in the second part of this blog post with the simple way to resolve it.

More than just a copy

But let’s stop for a minute or two and think about why this change took place. Does that means that control file snapshot copies are more than just a copy starting from version 11.2.0.2? While I don’t know all the details yet (I would like to conduct an additional research in this area), the Oracle documentation says: “RMAN creates the snapshot control file so that it has a consistent version of a control file to use when either resynchronizing the recovery catalog or backing up the control file.

During my troubleshooting efforts, I have noticed that backups sometimes failed just in the middle of data file or archive log backups. This could be explained by the fact that at some point during RMAN backups, Oracle needs to synchronize a controlfile and a RMAN catalog database. If there are some concurrent operations on a current controlfile, Oracle tries to use a snapshot copy. If a snapshot copy isn’t shared, the operation may fail. While the triggers for this issue are not 100% clear for me, SNAPSHOT CONTROLFILE is clearly not just an additional control file copy anymore (as many of us used to think). I’m not sure when I will manage to find time for additional research myself, but if you have information to share or time to shed some light here, please let me/us know.

Solution and error messages

The solution is simple. Just re-configure RMAN to place a snapshot controlfile copy to a shared location (e.g. ASM disk group):

configure snapshot controlfile name to '+DISKGROUP/dbname/controlfile/snapcf_dbname.f';

I have observed the following error messages related to the issue:

## alert.log
Thu Jun 28 01:33:57 2012
Control file backup creation failed.
Backup target file size found to be zero.
Errors in file /d01/DB/11.2.0/admin/DB_host/diag/rdbms/DB/DB2/trace/DB2_arc2_13590.trc:
ORA-27037: unable to obtain file status
Linux-x86_64 Error: 2: No such file or directory
Additional information: 3
Thu Jun 28 01:37:34 2012

## trace file
*** 2012-06-28 01:33:57.986
Linux-x86_64 Error: 2: No such file or directory
Additional information: 3
Redo shipping client performing standby login

## RMAN log files
channel t1: finished piece 1 at 2012-06-27:23:21:30
piece handle=DB_1830_1_p6nekdq4_1_1.incr tag=HOST-DB1-INCRDB comment=API Version 2.0,MMS Version 5.0.0.0
channel t1: backup set complete, elapsed time: 00:02:57
released channel: t1
released channel: t2
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03002: failure of backup plus archivelog command at 06/27/2012 23:21:30

RMAN-03009: failure of backup command on t2 channel at 06/27/2012 23:19:15
ORA-00245: control file backup operation failed

archived log file name=+ARCH/DB/archivelog/2012_06_19/thread_2_seq_434.487.786331921 RECID=1129 STAMP=786331922
archived log file name=+ARCH/DB/archivelog/2012_06_19/thread_2_seq_435.484.786339421 RECID=1133 STAMP=786339423
archived log file name=+ARCH/DB/archivelog/2012_06_19/thread_1_seq_398.480.786340595 RECID=1137 STAMP=786340595
archived log file name=+ARCH/DB/archivelog/2012_06_19/thread_2_seq_436.483.786340593 RECID=1135 STAMP=786340593
RMAN-08137: WARNING: archived log not deleted, needed for standby or upstream capture process
archived log file name=+ARCH/DB/archivelog/2012_06_19/thread_2_seq_437.478.786340749 thread=2 sequence=437
archived log file name=+ARCH/DB/archivelog/2012_06_19/thread_1_seq_399.479.786340749 RECID=1139 STAMP=786340748
Finished backup at 2012-06-19:04:41:18
released channel: t1
released channel: t2
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03008: error while performing automatic resync of recovery catalog
ORA-00245: control file backup operation failed

Starting backup at 2012-06-25:23:22:08
current log archived
released channel: t1
released channel: t2
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03002: failure of backup plus archivelog command at 06/25/2012 23:22:17
RMAN-03014: implicit resync of recovery catalog failed
RMAN-03009: failure of full resync command on default channel at 06/25/2012 23:22:17
ORA-00245: control file backup operation failed

archived log file name=+ARCH/DB/archivelog/2012_06_27/thread_2_seq_588.564.787019667 RECID=1721 STAMP=787019669
archived log file name=+ARCH/DB/archivelog/2012_06_27/thread_2_seq_589.571.787023419 RECID=1725 STAMP=787023421
Finished backup at 2012-06-27:04:04:34

Starting backup at 2012-06-27:04:04:41
channel t1: starting full datafile backup set
channel t1: specifying datafile(s) in backup set
released channel: t1
released channel: t2
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03009: failure of backup command on t1 channel at 06/27/2012 04:04:46
ORA-00245: control file backup operation failed

## OERR description
oerr ora 00245
00245, 00000, "control file backup operation failed"
// *Cause:  Failed to create a control file backup because some process
//          signaled an error during backup creation.
// *Action: Check alert files for further information. This usually happens
//          because some process could not access the backup file during
//          backup creation. Any process of any instance that starts a read/write
//          control file transaction must have an access to the backup control file
//          during backup creation.
email

Interested in working with Yury? Schedule a tech call.

About the Author

Yury is a nice person who enjoys meeting and working with people on challenging projects in the Oracle space. He started to work as an Oracle DBA 14 years ago (1997). For the past 10 years (since 2001) his main area of expertise is Oracle e-Business Suite. Yury is an OCP 7,8,9,10g and OCM 9i,10g. He is a frequent presenter at Oracle related conferences such as Hotsos, UKOUG and AUOUG. Yury is a socially active person. Apart from Social Media (Twitter, Bloging, Facebook) he is the primary organizer of Sydney Oracle Meetup group (250 people). So if you happen to be in Sydney (Australia) drop Yury a message and stop by at one of his Meetups.

5 Comments. Leave new

Hey Yury,

Does the snapshot controlfile location AND the backup controlfile destination BOTH have to be shared locations?

Thanks!
-Ken

Reply
Ashish Man Baisyet
July 7, 2014 10:17 pm

Hi Ken,

Yes, starting from Oracle RAC 11g R2 (11.2.0.2) the snapshot and backup control file has to be on shared location.

Best regards,
Ashish Man Baisyet

Reply
Prabhu Seshadri
August 17, 2014 2:24 pm

Ken,
It’s not required… We can place only controlfile snap on the shared location since the synchronization is not going to happen with controlfile backup while backup running.

Thanks.
Prabhu Seshadri

Reply

Ken,
the snapshot control file needs not be on a shared file. The requirement is that the snapshot control file needs to be created in a location that exists on all nodes. This control file does not need to be globally available to all instances in a RAC. it only needs to be available for the nodes that perform RMAN backup.

Reply

The snapshot control fileis a temporary file that is created at a certain location specified over the operating system. This file is used by RMAN to synchronize the information with the current control file. This file, except for synchronizing the information with the current control file, does not do anything else and hence, is not included in the recovery catalog. It does not contain any information about the current backup that you have taken, though it does know about the last backups that were done and are not marked yet as obsolete. So this file can be used to recover a scenario where you have lost everything and also were not using the catalog. You can use the last backup information recorded in this file and can recover the database with that backup.

It was explained very clearly in “http://www.dba-oracle.com/t_rman_19_snapshot_control_file.htm”

Reply

Leave a Reply

Your email address will not be published. Required fields are marked *