How I Finished a Recovery From an Active Duplicate After Facing Errors ORA-19909 and ORA-01110
The other day I was doing a database cloning in 11.2.0.4 from an active duplicate, and got the following error: ORA-19909: datafile 1 belongs to an orphan incarnation and ORA-01110: data file 1.
This post explains how I solved these errors. It's worth mentioning that what I did here also applies to 12c, 18c and 19c databases.
I first verified that I had the password file from the target database, and that I also had the corresponding TNS entries for it. I created a listener for this task only. Note: the client I was working with required that I create the listener; otherwise I could have worked with the SCAN listener.
Next I verified that I could connect to the target and auxiliary databases correctly. I created the script below to use for this refresh:
################################################################## ## Connectivity Test ################################################################## oracle $ ls $ORACLE_HOME/dbs/orapwSTG* /u01/app/oracle/product/11.2.0.4/dbhome_3/dbs/orapwSTG1 oracle $ rman TARGET sys/********@PROD AUXILIARY sys/********@STG_STATIC connected to target database: PROD (DBID=3874882083) connected to auxiliary database: STG (not mounted) ################################################################## ## STG_STATIC TNS Entry ################################################################## STG_STATIC = (DESCRIPTION = (ADDRESS_LIST = (ADDRESS = (PROTOCOL = TCP)(HOST = auxiliarydbadm01)(PORT = 1523)) ) (CONNECT_DATA = (SERVICE_NAME = STG1) (UR=A) ) ) ################################################################## ## PROD TNS Entry ################################################################## PROD = (DESCRIPTION = (ADDRESS_LIST = (ADDRESS = (PROTOCOL = TCP)(HOST = prod_scan)(PORT = 1521)) ) (CONNECT_DATA = (SERVICE_NAME = PROD1) ) ) ################################################################## ## Listener used ################################################################## LISTENER_REFRESH = (DESCRIPTION_LIST = (DESCRIPTION = (ADDRESS = (PROTOCOL = TCP)(HOST = auxiliarydbadm01)(PORT = 1523)) ) ) SID_LIST_LISTENER_REFRESH = (SID_LIST = (SID_DESC = (ORACLE_HOME = /u01/app/oracle/product/11.2.0.4/dbhome_3) (SID_NAME = QA1) ) (SID_DESC = (ORACLE_HOME = /u01/app/oracle/product/11.2.0.4/dbhome_3) (SID_NAME = STG1) ) (SID_DESC = (ORACLE_HOME = /u01/app/oracle/product/11.2.0.4/dbhome_3) (SID_NAME = RFSH1) ) ) ADR_BASE_LISTENER = /u01/app/oracle ################################################################## ## recover_from_prod.sh ################################################################## oracle $ cat recover_from_prod.sh #!/bin/bash export NLS_DATE_FORMAT='DD-MON-YYYY HH24:MI:SS' export ORACLE_SID=STG1 export ORAENV_ASK=NO . /usr/local/bin/oraenv rman TARGET sys/********@PROD AUXILIARY sys/********@STG_STATIC <<endofrman run { allocate channel prmy1 type disk; allocate channel prmy2 type disk; allocate channel prmy3 type disk; allocate channel prmy4 type disk; allocate channel prmy5 type disk; allocate channel prmy6 type disk; allocate auxiliary channel aux1 type disk; allocate auxiliary channel aux2 type disk; allocate auxiliary channel aux3 type disk; allocate auxiliary channel aux4 type disk; allocate auxiliary channel aux5 type disk; allocate auxiliary channel aux6 type disk; DUPLICATE TARGET DATABASE TO 'STG' FROM ACTIVE DATABASE NOFILENAMECHECK; } endofrman
I knew this duplicate was going to take a while as this was a 35 TB database.
SQL> SELECT ( SELECT SUM(bytes)/power(1024,4) data_size FROM dba_data_files ) + ( SELECT NVL(SUM(bytes),0)/power(1024,4) temp_size FROM dba_temp_files ) + ( SELECT SUM(bytes)/power(1024,4) redo_size FROM sys.v_$log ) + (SELECT SUM(BLOCK_SIZE*FILE_SIZE_BLKS)/power(1024,3) controlfile_size FROM v$controlfile ) "Database Size in TB" FROM dual; Database Size in TB ------------------- 35.1021405
Despite the length of time it took, the restore for the duplicate database went well. However, when the recovery process began, I got the following errors:
oracle $ tail -100 recover_from_prod.log ... Starting recover at 09-NOV-2020 08:27:59 starting media recovery media recovery failed Oracle instance started ... RMAN-11003: failure during parse/execution of SQL statement: alter database recover if needed start until change 281698833114 using backup controlfile ORA-00283: recovery session canceled due to errors ORA-19909: datafile 1 belongs to an orphan incarnation ORA-01110: data file 1: '+DATAC2/stg/datafile/system.636.1056008579'
These errors primarily happened because there were other files in the ASM diskgroup with the same DBID as the auxiliary database I was cloning.
if you're using ASM disk groups, RMAN works a particular way for an active duplicate. It catalogs everything under the disk group the db_recovery_file_dest parameter is set to if the log_archive_dest_1. parameter is not set.
This was true in my case. In the same cluster there was a standby running called proddr for the target database I was cloning.
################################################################## ## db_recovery_file_dest and log_archive_dest_1 parameter values ################################################################## oracle $ echo " > show parameter log_archive_dest_1; > " | sqlplus -s -L / as sysdba NAME TYPE VALUE ------------------------------------ ----------- ------------------------------ log_archive_dest_1 string ... oracle $ echo " > show parameter db_recovery_file_dest; > " | sqlplus -s -L / as sysdba NAME TYPE VALUE ------------------------------------ ----------- ------------------------------ db_recovery_file_dest string +RECOC2 db_recovery_file_dest_size big integer 20000G ################################################################## ## From the recover_from_prod.log ################################################################## ... "+RECOC1/prod/archivelog/2020_11_09/thread_2_seq_535909.41715.1056009955" auxiliary format "+RECOC2" ; catalog clone start with "+RECOC2"; switch clone datafile all; } ... List of Cataloged Files ======================= File Name: +RECOC2/proddr/CONTROLFILE/current.7901.1022257521 File Name: +RECOC2/stg/archivelog/2020_11_09/thread_2_seq_535877.5618.1056010749 ... File Name: +RECOC2/proddr/archivelog/2020_11_07/thread_2_seq_535588.2240.1055891307 File Name: +RECOC2/proddr/archivelog/2020_11_07/thread_1_seq_567337.8034.1055892763 .. datafile 1 switched to datafile copy input datafile copy RECID=67 STAMP=1056011218 file name=+DATAC2/stg/datafile/system.636.1056008579 ... datafile 133 switched to datafile copy input datafile copy RECID=198 STAMP=1056011261 file name=+DATAC2/stg/datafile/aud_tbs.451.1056009851 datafile 134 switched to datafile copy input datafile copy RECID=199 STAMP=1056011262 file name=+DATAC2/stg/datafile/revenue_ods2.622.1055987265 contents of Memory Script: { set until scn 281698833114; recover clone database delete archivelog ; }
Once I'd investigated the error, I proceeded to fix it. The first thing I did was try to open STG database in mount mode to see the contents of the control file. From the log above I could see the datafile copy had switched and updated the control file, and this would help me speed up recovery of this process.
SQL> startup mount; ORACLE instance started. Total System Global Area 2.5655E+10 bytes Fixed Size 2265224 bytes Variable Size 1.0402E+10 bytes Database Buffers 1.5099E+10 bytes Redo Buffers 151113728 bytes * ERROR at line 1: ORA-01103: database name 'PROD' in control file is not 'STG'
From the error above, I could see the control file had the incorrect database name. I would have to recreate the control file for this.
I wanted to avoid going to ASM or the recover log to gather all the datafiles and their location. Instead, I created a small parameter file that included only the control file, the block size, the same compatible parameter and the name of the target database as database PROD as shown in the error ORA-01103 above.
Once I had the database PROD up, I backed up that database control file to trace.
oracle $ cat initprod.ora DB_NAME=PROD DB_UNIQUE_NAME=PROD DB_BLOCK_SIZE=8192 *.control_files='/home/oracle/working/antunez/STG/control02.ctl' *.compatible='11.2.0.4.0' SQL> startup mount pfile='/home/oracle/working/antunez/initprod.ora'; ORACLE instance started. Total System Global Area 513642496 bytes Fixed Size 2254624 bytes Variable Size 402655456 bytes Database Buffers 100663296 bytes Redo Buffers 8069120 bytes Database mounted. SQL> alter database backup controlfile to trace as '/home/oracle/working/antunez/stg.sql'; Database altered. SQL> shutdown abort ORACLE instance shut down.
Once I had the control file trace with the proper datafile location, I modified it to leave only the section of RESETLOGS and change the line of CREATE CONTROL FILE to use the SET command.
With that done, I ran the creation of the control file in STG database.
oracle $ cat /home/oracle/working/antunez/stg.sql CREATE CONTROLFILE REUSE SET DATABASE "STG" RESETLOGS FORCE LOGGING ARCHIVELOG MAXLOGFILES 192 MAXLOGMEMBERS 3 MAXDATAFILES 1024 MAXINSTANCES 32 MAXLOGHISTORY 149504 LOGFILE GROUP 11 ( '+DATAC2/stg/onlinelog/group_11.3162.914669717', ... '+DATAC2/stg/datafile/sysaux.454.1056009483', '+DATAC2/stg/datafile/users.448.1056009331', '+DATAC2/stg/datafile/users.455.1056009819' CHARACTER SET AL32UTF8 ; oracle $ . oraenv <<< STG1 SQL> startup nomount; ORACLE instance started. Total System Global Area 2.5655E+10 bytes Fixed Size 2265224 bytes Variable Size 1.0402E+10 bytes Database Buffers 1.5099E+10 bytes Redo Buffers 151113728 bytes SQL> @/home/oracle/working/antunez/stg.sql Control file created. SQL> select open_mode from v$database; OPEN_MODE -------------------- MOUNTED
What I needed next were the archived redo logs that were going to be used for the recovery, as some of the archivelogs were not copied.
RMAN thought they were already in place for the recovery due to them being in the auxiliary host for the proddr standby database. So I followed the steps below to find the range of archived redo logs I was going to need.
If you remember the script I used, I set the NLS environment variable to this export NLS_DATE_FORMAT='DD-MON-YYYY HH24:MI:SS'
, so the log would contain the details of the times down to seconds from the RMAN output. This turned out to be useful.
I extracted the until scn and the Starting Duplicate Db outputs from the log. I then obtained the range of archived redo log sequences I was going to need from the target database called PROD.
oracle $ cat recover_from_prod.log | egrep "Starting Duplicate Db at | scn" Starting Duplicate Db at 08-NOV-2020 17:07:35 set until scn 281698833114; oracle $ sqlplus sys/****@PROD as sysdba ... SQL> select name from v$database; NAME --------- PROD SQL> col scn format 999999999999999999999 SQL> select timestamp_to_scn(to_timestamp('08-11-2020 17:07:34','dd-mm-yyyy hh24:mi:ss')) scn from dual; SCN ---------------------- 281674037748 SQL> select distinct thread#, sequence#, status, first_time, next_time, first_change#, next_change# from v$archived_log where 281674037748 between first_change# and next_change# order by 1,2; THREAD# SEQUENCE# S FIRST_TIME NEXT_TIME FIRST_CHANGE# NEXT_CHANGE# ---------- ---------- - ------------------- ------------------- ------------- ------------ 1 567552 A 2020-11-08 17:04:19 2020-11-08 17:07:37 2.8167E+11 2.8167E+11 2 535731 A 2020-11-08 17:05:00 2020-11-08 17:07:36 2.8167E+11 2.8167E+11 SQL> select distinct thread#, sequence#, status, first_time, next_time, first_change#, next_change# from v$archived_log where 281698833114 between first_change# and next_change# order by 1,2; THREAD# SEQUENCE# S FIRST_TIME NEXT_TIME FIRST_CHANGE# NEXT_CHANGE# ---------- ---------- - ------------------- ------------------- ------------- ------------ 1 567681 A 2020-11-09 08:05:54 2020-11-09 08:06:09 2.8170E+11 2.8170E+11 1 567682 A 2020-11-09 08:06:09 2020-11-09 08:10:27 2.8170E+11 2.8170E+11 2 535910 A 2020-11-09 08:05:54 2020-11-09 08:06:09 2.8170E+11 2.8170E+11
With the data above, I ran the following to backup copy command. This allowed me to obtain the archived redo logs necessary for this recovery.
Please note I did it to a different location than the current db_recovery_file_dest and I had to connect to both PROD and STG databases for this command.
oracle $ rman TARGET sys/********@PROD AUXILIARY sys/********@STG_STATIC RMAN> run { backup as copy archivelog from sequence 567552 until sequence 567682 thread 1 auxiliary format '/u02/pythian/STG/%t_%s_%r.arc'; backup as copy archivelog from sequence 535731 until sequence 535910 thread 2 auxiliary format '/u02/pythian/STG/%t_%s_%r.arc'; }
Once the copy of the archived redo logs were restored, I registered them to the control file of the STG database.
oracle $ . oraenv <<< STG oracle $ rman target / connected to target database: STG (DBID=3874882083, not open) RMAN> catalog start with '/u02/pythian/STG/; cataloging files... cataloging done List of Cataloged Files ======================= ... File Name: /u02/pythian/STG/1056028905_128319_904050853.arc File Name: /u02/pythian/STG/1056030309_128439_904050853.arc
The last thing I needed to do was recover the STG database to the SCN found in the recover log. I made sure not to connect to the target database called PROD, but rather only to the database called STG.
oracle $ . oraenv <<< STG1 oracle $ rman target / connected to target database: STG (DBID=3874882083, not open) RMAN> run { set until scn 281698833114; recover database; } executing command: SET until clause Starting recover at 09-NOV-20 allocated channel: ORA_DISK_1 channel ORA_DISK_1: SID=3142 device type=DISK starting media recovery archived log for thread 1 with sequence 567626 is already on disk as file /u02/pythian/STG/1056029933_128385_904050853.arc archived log for thread 1 with sequence 567627 is already on disk as file /u02/pythian/STG/1056029940_128386_904050853.arc archived log for thread 1 with sequence 567628 is already on disk as file /u02/pythian/STG/1056029941_128387_904050853.arc archived log for thread 1 with sequence 567629 is already on disk as file /u02/pythian/STG/1056029942_128388_904050853.arc archived log for thread 1 with sequence 567630 is already on disk as file /u02/pythian/STG/1056029943_128389_904050853.arc archived log for thread 1 with sequence 567631 is already on disk as file /u02/pythian/STG/1056029945_128390_904050853.arc archived log for thread 1 with sequence 567632 is already on disk as file /u02/pythian/STG/1056029952_128391_904050853.arc ... archived log file name=/u02/pythian/STG/1056028890_128318_904050853.arc thread=2 sequence=535909 archived log file name=/u02/pythian/STG/1056030309_128439_904050853.arc thread=1 sequence=567680 archived log file name=/u02/pythian/STG/1056030310_128440_904050853.arc thread=1 sequence=567681 archived log file name=/u02/pythian/STG/1056028905_128319_904050853.arc thread=2 sequence=535910 media recovery complete, elapsed time: 00:06:35 Finished recover at 09-NOV-20
After this I simply needed to open the database as resetlogs. I also wanted the cloned database to have a new DBID and to come up as a RAC database. I verified the database and instance status using verify_data_dictionary_11g.sql.
RMAN> alter database open resetlogs; using target database control file instead of recovery catalog database opened RMAN> delete archivelog all; ... Deleted 353 objects RMAN> shutdown immediate database shutdown RMAN> STARTUP MOUNT database mounted RMAN> exit oracle $ nid target=sys/******* DBNEWID: Release 11.2.0.4.0 - Production on Mon Nov 9 14:34:22 2020 Copyright (c) 1982, 2011, Oracle and/or its affiliates. All rights reserved. Connected to database STG (DBID=3874882083) Connected to server version 11.2.0 Control Files in database: +DATAC2/stg/controlfile/control01.ctl +RECOC2/stg/controlfile/control02.ctl Change database ID of database STG? (Y/[N]) => Y Proceeding with operation Changing database ID from 3874882083 to 1490626863 Control File +DATAC2/stg/controlfile/control01.ctl - modified Control File +RECOC2/stg/controlfile/control02.ctl - modified Datafile +DATAC2/stg/datafile/system.636.105600857 - dbid changed ... Datafile +DATAC2/stg/datafile/revenue_ods2.622.105598726 - dbid changed Control File +DATAC2/stg/controlfile/control01.ctl - dbid changed Control File +RECOC2/stg/controlfile/control02.ctl - dbid changed Instance shut down Database ID for database STG changed to 1490626863. All previous backups and archived redo logs for this database are unusable. Database is not aware of previous backups and archived logs in Recovery Area. Database has been shutdown, open database with RESETLOGS option. Succesfully changed database ID. DBNEWID - Completed succesfully. oracle $ sqlplus / as sysdba ... SQL> startup mount ORACLE instance started. Total System Global Area 1.6034E+11 bytes Fixed Size 2261848 bytes Variable Size 2.5233E+10 bytes Database Buffers 1.3475E+11 bytes Redo Buffers 352444416 bytes Database mounted. SQL> ALTER DATABASE OPEN RESETLOGS; Database altered. SQL> ALTER SYSTEM SET cluster_database=TRUE SCOPE=spfile; SQL> shutdown immediate Database closed. Database dismounted. ORACLE instance shut down. oracle $ srvctl start database -d STG oracle $ srvctl status database -d STG Instance STG1 is running on node auxiliarydbadm01 Instance STG2 is running on node auxiliarydbadm02 oracle $ sqlplus / as sysdba SQL> @verify_data_dictionary_11g.sql DATABASE NAME ============= DBNAME --------------- STG ================================================================================================ == Oracle Instance Information ================================================================================================ -- IP Address 1**.***.***.30 Cpu_Count 24 | Host_Name auxiliarydbadm01 Instance_Name STG1 | Database_Status ACTIVE Status OPEN | Startup_Time 09-11-2020 14:43 Version 11.2.0.4.0 | Instance_Role PRIMARY_INSTANCE Database log mode ARCHIVELOG Background Dump Dest /u01_diag/app/oracle/diag/diag/rdbms/stg/STG1/trace Spfile +DATAC2/stg/parameterfile/spfilestg.ora Redo size (Gb) 1.953125 Redo size (Gb) .09765625 -- Database Space (Gb) 35367.70 | SGA (Gb) 149.3303 -- -- Nb. Datafiles 133 | Nb. Tempfiles 15 -- DBA_REGISTRY CONTENTS ================================================================ COMP_ID COMP_NAME VERSION Status -------------------- ---------------------------------------- --------------- -------------------------------------------------- OWB OWB 11.2.0.4.0 VALID APEX Oracle Application Express 3.2.1.00.12 VALID EM Oracle Enterprise Manager 11.2.0.4.0 VALID AMD OLAP Catalog 11.2.0.4.0 VALID SDO Spatial 11.2.0.4.0 VALID ORDIM Oracle Multimedia 11.2.0.4.0 VALID XDB Oracle XML Database 11.2.0.4.0 VALID CONTEXT Oracle Text 11.2.0.4.0 VALID EXF Oracle Expression Filter 11.2.0.4.0 VALID RUL Oracle Rules Manager 11.2.0.4.0 VALID OWM Oracle Workspace Manager 11.2.0.4.0 VALID CATALOG Oracle Database Catalog Views 11.2.0.4.0 VALID CATPROC Oracle Database Packages and Types 11.2.0.4.0 VALID JAVAVM JServer JAVA Virtual Machine 11.2.0.4.0 VALID XML Oracle XDK 11.2.0.4.0 VALID CATJAVA Oracle Database Java Packages 11.2.0.4.0 VALID APS OLAP Analytic Workspace 11.2.0.4.0 VALID XOQ Oracle OLAP API 11.2.0.4.0 VALID RAC Oracle Real Application Clusters 11.2.0.4.0 VALID 19 rows selected. LIST APPLIED PATCHES ======================= ACTION_TIME ID ACTION VERSION BUNDLE COMMENTS ------------------------------ ---------- ------------------------------ --------------- ------------------------ -------------- 24-AUG-13 12.03.45.119862 PM 0 APPLY 11.2.0.4 PSU Patchset 11.2.0.2.0 17-FEB-16 01.17.34.459872 PM 0 jvmpsu.sql 11.2.0.4 RAN jvmpsu.sql 17-FEB-16 01.18.26.518633 PM 20 APPLY 11.2.0.4 EXA BP20 24-JUN-17 05.01.00.394556 AM 0 jvmpsu.sql 11.2.0.4 RAN jvmpsu.sql 24-JUN-17 05.01.00.414355 AM 0 APPLY 11.2.0.4 OJVM PSU post-instal 24-JUN-17 05.01.00.416012 AM 25434033 APPLY Patch 25434033 appli 24-JUN-17 05.01.23.110902 AM 170418 APPLY 11.2.0.4 EXA BP170418 24-JAN-19 07.48.56.741239 PM 180717 APPLY 11.2.0.4 EXA BP180717 24-JAN-19 09.00.45.442572 PM 0 jvmpsu.sql 11.2.0.4 RAN jvmpsu.sql 24-JAN-19 09.00.45.707068 PM 0 APPLY 11.2.0.4 OJVM PSU post-instal 24-JAN-19 09.00.45.709711 PM 27923163 APPLY Patch 27923163 appli 11 rows selected. LIST APPLIED SQL PATCHES ======================= COMMENTS ACTION ACTION_DATE VERSION ------------------------------------------------------------ ------------------------------ -------------------- --------------- RAN jvmpsu.sql jvmpsu.sql 17/02/16 13:17:34 11.2.0.4.5OJVMB P BP20 APPLY 17/02/16 13:18:26 11.2.0.4 BP180717 APPLY 24/01/19 19:48:56 11.2.0.4 RAN jvmpsu.sql jvmpsu.sql 24/01/19 21:00:45 11.2.0.4.180717 OJVMPSU OJVM PSU post-install APPLY 24/01/19 21:00:45 11.2.0.4.180717 OJVMPSU Patch 27923163 applied APPLY 24/01/19 21:00:45 OJVM PSU post-install APPLY 24/06/17 05:01:00 11.2.0.4.170418 OJVMPSU RAN jvmpsu.sql jvmpsu.sql 24/06/17 05:01:00 11.2.0.4.170418 OJVMPSU Patch 25434033 applied APPLY 24/06/17 05:01:00 BP170418 APPLY 24/06/17 05:01:23 11.2.0.4 Patchset 11.2.0.2.0 APPLY 24/08/13 12:03:45 11.2.0.4 11 rows selected. COUNT OF INVALID OBJECTS ======================== COUNT(*) ---------- 0 1 row selected. INVALID OBJECTS GROUPED BY OBJECT TYPE AND OWNER ================================================ no rows selected LIST OF SYS INVALID OBJECTS ======================= no rows selected
If you ever face the errors ORA-19909 and ORA-01110, I hope this blog post helps . It took me a while to solve this, and knowing the above would have come in handy.
Note: This was originally posted on rene-ace.com.
Share this
You May Also Like
These Related Stories
No Comments Yet
Let us know what you think