ORA-16069? You May Need A New Standby Controlfile

Posted in: Technical Track

On a recent Monday, I had to perform an emergency Oracle standby switchover for a client whose primary instance host had mysteriously rebooted itself over the previous day. Confidence in that host was, understandably, shaken.

The Oracle Data Guard configuration is a 3-instance setup using Data Guard Broker: one primary, we’ll call it OraA, feeding two standby instances, OraB and OraC. In this particular configuration, we perform switchovers between OraA and OraC. Caught in the middle is OraB, which is on a 60-minute standby delay.

After this particular switchover, OraB started complaining with this message in the alert log:

ORA-16069: Archive Log standby database activation identifier mismatch

We had seen this occasionally in prior switchovers, and the problem would fix itself once the standby delay passed and the OraB standby would process the log notifying it of the switchover. This time, however, recovery was stopped and more than enough time had elapsed. OraA and OraC were performing perfectly fine.

Much of the reference searching I saw suggested that the standby instance would have to be completely rebuilt. Not an appetizing option. A search of metalink turned up Bug 4048687, which seemed to demonstrate a similar problem, although on a different OS/Platform. That solution was to recreate the standby controlfile. Trust me, it sounds more drastic than it is!

Here’s how to do it in just 6 easy steps!

  1. Shutdown the misbehaving standby.
  2. Copy one of the current standby controlfiles for safekeeping (just in case).
  3. On the primary instance, create a new standby controlfile: alter database create standby controlfile as '/tmp/stdby.ctl';
  4. Transfer that new standby controlfile to the standby host.
  5. Copy the new controlfile to the controlfile location(s) used by the instance (you have more than one, right?).
  6. STARTUP MOUNT the standby instance. If you use the Data Guard Broker, it should automatically begin recovery for you; otherwise restart managed recovery with alter database recover managed standby database disconnect;

Voila. Standby recovery should resume nicely, assuming logs are there to apply.

Interested in working with Don? Schedule a tech call.

About the Author

Oracle database administrator for The Pythian Group, headquartered in Ottawa, Ontario, Canada. I am located in Manitowoc, Wisconsin, USA.OCP 10gR2 DBA

4 Comments. Leave new

Recently, I had a similar experience with the Standby database which I resolved by re-creating the standby controlfile from production.

http://momendba.blogspot.com/2009/03/strange-data-guard-inssue.html

Reply

Thanks for sharing, Asif!

Reply
ORA-16069? You May Need A New Standby Controlfile « die Seilerwerks
August 19, 2009 2:20 pm

[…] leave a comment » Originally posted on The Pythian Group blog. […]

Reply

Thank you, Don! Even three years later, your solution helped fix a problem where the primary couldn’t contact the standby for a few hours resulting in a gap. I appreciate your post!

Reply

Leave a Reply

Your email address will not be published. Required fields are marked *