20 mins vs 2 hours

Recently I have noticed that re-imaging process on the second Oracle Database Appliance node took significantly shorter time comparing with the first node. The difference was so significant that I started to suspect that there were something wrong with either particular set of hardware or some of the re-imaging process steps have failed on the second node. On the first node the process has completed  in 120 minutes, but  on the second it took just 20 around minutes.

I spent quite a bit of time to understand that exactly has been happening. But before I tell you, can I ask you what theoretical options would you come with given the behavior I just described? Please share those with me in the comment section below, please :)

Any mystery can be solved

Question is if we are ready to pay for it. Sometimes it takes quite a bit of effort to get to the truth and very often we don’t have time or interest or budget to find it. In this particular case I was so curious that I have spent good part of my my weekend looking for a clue. On the way I had  to learn a bit of “Anaconda (installer)“, SquashFS file system, how to rebuild ISO image and the way ODA re-imaging process works. The purpose of this paragraph is to encourage you to be curious and don’t leave mysteries unresolved. Invest  some time and you will learn a lot on the way :)

NOTE: I will try to share the way I have troubleshot this problem in my future blog posts.

Bug in the “post-install” script

It appears that the problem is in the way the ISO:/Extras/setupodaovm.sh post install script checks if software RAID have completed re-synchronization of 4 internal HDD partitions (md devices) in between 2 physical disks. There are the following check at the very end of the script:
mdadm --wait /dev/md1
mdadm --wait /dev/md2
mdadm --wait /dev/md3
Each of the lines designed to check if the software RAID completed synchronizing an md device (partition). The following is part of man page for mdadm utility

       -W, --wait
              For  each  md  device  given, wait for any resync, recovery, or reshape activity to finish before returning.  mdadm will return with success if it actually waited for
              every device listed, otherwise it will return failure.

During the re-imaging process all 4 volumes got to be rebuild and need to be synchronized by the software RAID. It worth mentioning that software RAID on ODA configured to re-synchronise  one device at the time. Other devices just seating and waiting they turn in the status DELAYED.  The problem is that if a device is in the state resync=DELAYED the “mdadm –wait” check will not stop and wait for it. Therefore just one of the mdadm checks will wait until re-synchronization process finishes others successfully pass even if a device isn’t synchronized yet (resync=DELAYED). Now let’s have a look on the devices’ sizes and associated synchronization times:

Just to make life a bit more interesting the software RAID picks up a device to be re-synchronized next randomly. That means it is just matter of luck what device will get processed next. If it is md1 device (17GB) then the whole re-imaging process takes 20 minutes. However if the software RAID synchronises md2 device (217GB) during the execution of the mdadm check then the re-imaging process takes ~120 minutes.

A way to fix the problem

I am not a great expert in the Linux System Administration area (I am an Oracle DBA after all) and would rather let Oracle folks make the final call, but it seems to me that in order to make sure that all 4 devices got re-synchronized before the re-imaging process finishes the check should look like the following.

mdadm --wait /dev/md0 /dev/md1 /dev/md2 /dev/md3

Conclusion

To conclude until the issue is fixed know that

1. you may face different ODA nodes’ re-imaging times
2. to be on the safe side check if md devices’ re-synchronization process  is finished by running “cat /proc/mdstat” command before running any business critical processes on your ODA.

Yury

PS “Stay Hungry Stay Foolish” - Steve Jobs

The red carpet has been laid down at this Log Buffer Edition, and you can witness and cheer the cat-walking blog posts from Oracle, SQL Server and MySQL. Every one of them is chic, elegant, sensual in its own right. Enjoy.
Oracle:

Create colored heat maps in SQL*Plus with Kyle Hailey.

Here’s a quick and dirty script to create a procedure (in the SYS schema – so be careful) to check the Hakan Factor for an object.

Connor has a good post about default null for collection parameter.

This is yet another blogpost on Oracle’s direct path read feature which was introduced for non-parallel query processes in Oracle version 11.

Owen Allen has seen some questions about provisioning Oracle Solaris 11. They boil down to this.

SQL Server:

Shashank Srivastava tells us as how to Change the SQL Server Instance Name after Renaming the Windows Host.

Daniel Calbimonte shares as how to synchronize two SSAS Servers.

Data Architecture underpins just about everything we do in IT.  Without a clear understanding of how data is structured, there is no reliable way to derive meaning from it.

Orlando Colamatteo is login-less in Seattle.

Lets get started testing database with tSQLt with Robert Sheldon.

MySQL:

After a lot of fuzz, Anders Karlsson is now releasing MyQuery version 3.5.1.

Nothing like reestablishing a tradition and Dave Stokes is doing just that for MySQL.

Mare Alff is spreading the word about the performance schema.

Slava Akhmechet talks about secondary indexes, batched inserts performance improvements, soft durability mode.

It is a central part of the MySQL philosophy to try and help you as much as you can. There are many occasions when it could tell you that what you are asking for is utterly stupid or give you a bad execution plan because “you asked for it”.

Sometime back I was involved in a project which was to collect information for the servers we are supporting – creating inventory of servers. Logging into each server and collecting information is bit tedious when you have hundreds of server in your environment.  I have spent sometime and created a script that does this work for me, however, I still need to connect to the server – but using SSMS/Query Window and execute the script. And then the script will provide me the details I needed.

This inventory should have details like below for cluster and stand alone instances:

Server Name

OS Name

OS Edition

OS Patch Level

SQL Server IP

Is Clustered

Node1_Name

Node1_IP

Node2_Name

Node2_IP

SQL Server Edition

SQL Server Patch Level

Server Time Zone

SQL Server Version

SQL Server Platform

Processor Core

Physical Memory

Service Account Name

Domain

Looks good ?  Below is the version 1 of this script.

/*
IP address portion using : http://www.sqlservercentral.com/Forums/Topic150196-8-1.aspx
REMOVE sp_configure parameters if you are executing this script on SQL Server 2000

Created By : Hemantgiri S. Goswami | http://www.sql-server-citation.com
Date : 24th March 2013
Version : 1.0

Tested ON:
Windows Server: 2003, 2008, 2008 R2
SQL Server:2000, 2005, 2008, 2008 R2, 2012

*/
sp_configure 'show advanced options',1
RECONFIGURE WITH OVERRIDE
GO
sp_configure 'xp_cmdshell',1
RECONFIGURE WITH OVERRIDE
GO
DECLARE @TimeZone NVARCHAR(100)
 ,@ProductVersion SYSNAME
 ,@PlatForm SYSNAME
 ,@Windows_Version SYSNAME
 ,@Processors SYSNAME
 ,@PhysicalMemory SYSNAME
 ,@ServiceAccount SYSNAME
 ,@IPAddress SYSNAME
 ,@DOMAIN SYSNAME
 ,@MachineType SYSNAME
 ,@SQLServerIP VARCHAR(255)
 ,@CMD VARCHAR(100)
 ,@Node1 VARCHAR(100)
 ,@Node2 VARCHAR(100)
 ,@Node1IP VARCHAR(100)
 ,@Node2IP VARCHAR(100)
 ,@OSEdition VARCHAR(100)
 ,@OSVersion VARCHAR(100)
 ,@OSName VARCHAR(100)
 ,@OSPatchLevel VARCHAR(100)

CREATE TABLE #TempTable
 (
 [Index] VARCHAR(2000),
 [Name] VARCHAR(2000),
 [Internal_Value] VARCHAR(2000),
 [Character_Value] VARCHAR(2000)
 ) ;

INSERT INTO #TempTable
EXEC xp_msver;

-- Replace @Value_Name to N'TimeZoneKeyName' when running on Windows 2008
EXEC master.dbo.xp_regread
 @rootkey = N'HKEY_LOCAL_MACHINE',
 @key = N'SYSTEM\CurrentControlSet\Control\TimeZoneInformation',
 @value_name = N'StandardName',
 @value = @TimeZone output

EXEC master.dbo.xp_regread
 @rootkey = N'HKEY_LOCAL_MACHINE',
 @key = N'SYSTEM\CurrentControlSet\Services\MSSQLServer',
 @value_name = N'ObjectName',
 @value = @ServiceAccount output

EXEC master.dbo.xp_regread
 @rootkey = N'HKEY_LOCAL_MACHINE',
 @key = N'SYSTEM\CurrentControlSet\Control\ProductOptions',
 @value_name = N'ProductType',
 @value = @MachineType output

EXEC master.dbo.xp_regread
 @rootkey = N'HKEY_LOCAL_MACHINE',
 @key = N'SYSTEM\CurrentControlSet\Services\Tcpip\Parameters',
 @value_name = N'Domain',
 @value = @DOMAIN output

EXEC master.dbo.xp_regread
 @rootkey = N'HKEY_LOCAL_MACHINE',
 @key = N'CLUSTER\NODES\1',
 @value_name = N'NodeName',
 @value = @Node1 output

EXEC master.dbo.xp_regread
 @rootkey = N'HKEY_LOCAL_MACHINE',
 @key = N'CLUSTER\NODES\2',
 @value_name = N'NodeName',
 @value = @Node2 output

EXEC master.dbo.xp_regread
 @rootkey = N'HKEY_LOCAL_MACHINE',
 @key = N'SOFTWARE\Microsoft\Windows NT\CurrentVersion',
 @value_name = N'ProductName',
 @value = @OSName output

create table #OSEdition (VALUe varchar(255),OSEdition varchar(255), data varchar(100))
insert into #OSEdition
EXEC master.dbo.xp_regread
 @rootkey = N'HKEY_LOCAL_MACHINE',
 @key = N'SYSTEM\CurrentControlSet\Control\ProductOptions',
 @value_name = N'ProductSuite'
SET @OSEdition = (SELECT TOP 1 OSedition FROM #OsEdition)

EXEC master.dbo.xp_regread
 @rootkey = N'HKEY_LOCAL_MACHINE',
 @key = N'SOFTWARE\Microsoft\Windows NT\CurrentVersion',
 @value_name = N'CSDVersion',
 @value = @OSPatchLevel output

set @cmd = 'ping ' + @Node1
create table #Node1IP (grabfield varchar(255))
insert into #Node1IP exec master.dbo.xp_cmdshell @cmd
set @cmd = 'ping ' + @Node2
create table #Node2IP (grabfield varchar(255))
insert into #Node2IP exec master.dbo.xp_cmdshell @cmd

set @cmd = 'ping ' + @@servername
create table #SQLServerIP (grabfield varchar(255))
insert into #SQLServerIP exec master.dbo.xp_cmdshell @cmd

SET @SQLServerIP=(
 SELECT substring(grabfield, charindex('[',grabfield)+1, charindex(']',grabfield)-charindex('[',grabfield)-1)
 from #SQLServerIP where left(grabfield,7) = 'Pinging'
 )
SET @Node1IP =(
 SELECT substring(grabfield, charindex('[',grabfield)+1, charindex(']',grabfield)-charindex('[',grabfield)-1)
 from #Node1IP where left(grabfield,7) = 'Pinging'
 )

SET @Node2IP =(
 SELECT substring(grabfield, charindex('[',grabfield)+1, charindex(']',grabfield)-charindex('[',grabfield)-1)
 from #Node2IP where left(grabfield,7) = 'Pinging'
 )

SET @ProductVersion = (SELECT Character_Value from #TempTable where [INDEX]=2)
SET @Platform = (SELECT Character_Value from #TempTable where [INDEX]=4)
SET @Windows_Version= (SELECT Character_Value from #TempTable where [INDEX]=15)
SET @Processors = (SELECT Character_Value from #TempTable where [INDEX]=16)
SET @PhysicalMemory = (SELECT Character_Value from #TempTable where [INDEX]=19)
SELECT
 ServerName = @@SERVERNAME
 ,OSName = @OSName
 ,OSEdition = @OSEdition
 ,OSPatchLevel = @OSPatchLevel
 ,SQLServerIP = @SQLServerIP
 ,IsClustered = SERVERPROPERTY('IsClustered')
 ,Node1_Name = @Node1
 ,Node1_IP = @Node1IP
 ,Node2_Name = @Node2
 ,Node2_IP = @Node2IP
 ,SQLServerEdition = SERVERPROPERTY('Edition')
 ,SQLServerLevel = SERVERPROPERTY('ProductLevel')
 ,ServerTimeZone = @TimeZone
 ,SQLServerVersion = @ProductVersion
 ,SQLServerPlatform = @PlatForm
 ,ProcessorCore = @Processors
 ,PhysicalMemory = @PhysicalMemory
 ,ServiceAccountName = @ServiceAccount
 ,WKS_Server = @MachineType
 ,Domain = @DOMAIN

GO
DROP TABLE #Node1IP
DROP TABLE #NODE2IP
DROP TABLE #SQLServerIP
DROP TABLE #TempTable
DROP TABLE #OSEdition
GO
sp_configure 'xp_cmdshell',0
RECONFIGURE WITH OVERRIDE
GO
sp_configure 'show advanced options',0
RECONFIGURE WITH OVERRIDE
GO

Please do post back your feed back for this script, I will try my best to update and post back new version for this script.

- Hemantgiri S. Goswami (Cross posting from http://www.sql-server-citation.com/)

photo credit: Skimaniac via photopin cc

When I started my work in IT, I used to be in a very small shop, and even though we had people in several places in the same state, everything used to be very centralized and from 9 to 5, and because we were basically only 2 people , our action plan used to be a talk over the lunch table and that would be it, we would go ahead and execute it after 5 PM, and I won’t lie sometimes before 5 :) .

Over the years I have understood that even if you are a 2 guy shop or a team of 15 separated by oceans and being miles apart, communication is the most important thing to have on your team, and one of the means of communication is having an action plan in place for any major/medium change you do in your organization. First this will generate discussions amongst your teammates and it will reduce the possibility of errors when you are faced with time and pressure constraints when implementing it.

This might sometimes feel like a mundane and boring task, as it will take an effort to come up with it and it will take time to verify it, but when game day comes along you will see the great benefit of having an action plan.

Another great benefit of having an action plan is that you also have a road map if you need to rollback your change, and that is also critical, because normally any major change or rollback is not done only by one person, take for example a change that takes about 7 or 8 hours to be done, and at the end when the UAT (User Application Testing) is done, 1 or 2 more hours ,the application team decides that a rollback is needed, you are probably not in a good state of mind to do the rollback after 8 hours of continuous work, if you have an action plan, one of your teammates can step in and you can have a rest, even if it is to go to the kitchen and have a sandwich and a coke and forget 10 minutes about that pressure.

As with life and with us being human, having an action plan doesn’t mean that everything will go smoothly or you won’t have an error in there, but believe me, it will reduce in a big way the possibility of an error if you execute it by memory or by doing one yourself without revision.

I do hope that you already have an action plan as part of your major/medium changes, but if you don’t, it is time to get FIT-ACER, here is an example of one (Kudos to Cesar Sanchez as it is his Action Plan Template), use it and modify it to your needs, it is a good start.

Action Plan Template

This week again, the Log Buffer Edition covers the exciting and ever-evolving galaxies of Oracle, MySQL and SQL Server.

Oracle:

When does an Oracle process know it’s on Exadata?

MobaXterm 6.3 has been released. You can find the download and changelog in the usual place.

A new major release (version 3.0) of Randolf’s XPLAN_ASH tool is available for download.

Important !! Clustering Factor Calculation Improvement

Grouping Data Sets by Week Number of the Month

SQL Server:

Jason Crider asks to put some Windows update in his basket.

Data cleansing can be done via SSIS as well as Data Quality Services (DQS) and Master Data Services (MDS).  The lines are a bit blurred when talking about data cleansing using SSIS, DQS and MDS.

How To: Migrate Encrypted Procedures To Windows Azure SQL Database (WASD).

It’s a best practice to keep your index keys as narrow as possible, and SQL Server enforces a maximum key length of 900 bytes on most “normal” clustered and non-clustered indexes.

Webcast Presentation Materials: Introduction to SQL Server Security

MySQL:

This is the start of the heavy travel season for the MySQL Community Team. So if you are attending PHPTek, Drupalcon, Texas LinuxFest, SELF, Redhat Summit, or Lonestar PHP please make sure to say ‘hi’ while you are in Chicago, Portland, Austin, Charlotte, Boston, or Addison.

Your databases in a glance with Glimpsee

Continuent Tungsten 2.0 is a major step forward that includes key feature improvements to help you manage very large datasets on MySQL.

How to create a new (or repair a broken) GTID based slave with Percona XtraBackup

Disk I/O is frequently the performance bottleneck with relational databases. With AWS recently releasing 4,000 PIOPs EBS volumes, Laine Campbell wanted to do some benchmarking with pgbench and PostgreSQL 9.2.

Mothers day is coming up fast. May 12th 2013 is just around the corner and If you have not already done so, make sure you go out and get mom a special gift as soon as possible. If you need a hand picking something great, take a look at this mothers day gift guide by the HuffingtonPost

Many this year like they have in previous years shopped online and bought mom that special gift from an online retailer, a simple, easy and fun process. But is it always? Unfortunately for many, the experience did not live up to expectation due to site performance and both customers and merchants end up losing.

Is this a big deal? We put together this infograph on mothers day shopping. Take a look.

News and views are an integral part of our modern daily life. When it comes to the information-hungry roles such as database professionals, such need becomes more pressing. Log Buffer is one way to keep abreast of news and views from the world of Oracle, SQL Server, and MySQL.

Oracle:

There is a significant update SLOB 2, Kevin Closson informs.

Charles Hooper is grouping Data Sets by Week Number of the Month.

Chris Antognini has shared a script that is used to demo ITL deadlocks.

As Exadata Storage Indexes (SI) are purely memory only structures located on the Exadata storage servers, care needs to be taken in how much memory they can potentially consume.

Connon McDonald says that its not about the outage, but….

SQL Server:

Richard Douglas is giving information about SQL Server events in UK.

Thomas LaRock has been involved in a virtualization projects for almost ten years now.

Kendra Little has five things about Fillfactor.

Kevin Kline suggests to use TPC database benchmarks to save money.

Here’s a quick mystery. You have a SQL Server with several jobs that appear to be running properly and on schedule, but many of them show no history.

MySQL:

How does InnoDB behave without a Primary Key? Jeremy Cole asks.

The MariaDB project is pleased to announce the immediate availability of the MariaDB Java Client 1.1.2.

Normal 0 false false false EN-US X-NONE X-NONE Everybody loves free. It is the best marketing term one could use.

MySQL Cluster Manager 1.2.3 is now available to download from My Oracle Support.

When is a Subquery Executed?  Øystein Grøvlen writes.

“What have I gotten myself into” ran through my head during the first week of training at Pythian. The environment was great, the people were great, the work was great, but many doubts ran through my head as I ran from training meeting to training meeting – how am I going to retain all this information (my memory skills are bad as it is), am I going to meet their expectations, am I too junior for this, I can’t seem to catch up on anything, is this the right career choice.
6 months later and I’m still here! I feel more comfortable with the work, I know I am meeting expectations, I have confidence in my skills, my organizational skills have improved, I know I’ve made the right career choice, even my memory has gotten much better! What changed over the past 6 months you ask? Well, here is how it started.

I was happy and content at my previous employer. I knew what I was doing, I had the skills and knowledge, and I was regarded as one of their senior employees. Although it was mainly recruiting work I was performing, it was under a Sales umbrella. I had always wanted to follow the path of an HR professional, may it be recruiting, organizational design, compensation, employee relations, training – all of which fell under the umbrella of HR. I arrived at Pythian because I was contacted by a previous colleague with an opportunity that I could not resist. Finally, I will get a chance to get my hands dirty and work on my true passion – Human Resourcing (HR). Contrary to some beliefs, HR professionals are not “paper pushers”, administrators or strictly recruiters but in fact we are pieces of the strategic puzzle in any organization. The harsh reality though is that much of the world still believes that we are administrators. So, for me I really wanted to join an organization that leverages HR as a strategic partner for the organization.

What Pythian offered me was the “3 Pillars of Growth” – Sales, Service Delivery and HR. These 3 ‘pillars’ are crucial in helping grow a business. This was further solidified for me by the people I met, the new/ existing plans and programs within HR, and the focus on culture Pythian provided for the growth of the business. This was a great opportunity for me to grow as a person and my career, and be part of a strategic team.

So I arrive at Pythian and was excited to learn the things I have always wanted to learn while providing the expertise I already possessed. In fact, I was delegated as the lead on a major, high profile project within the first week. However, shortly after doubt crept in as I found myself feeling like I could barely tread water (ask my wife, I’m not a very strong swimmer). Here I was absorbing so much information about the company, policies, procedures and now I have to figure out how to role out a project while I was still learning about the internal processes (and performing my main responsibility of recruiting). I rapidly felt like I was taking on too much water and started drowning. I started to loose confidence in my abilities and doubted the decision to take on this roll. I had several discussions with my work colleagues and my peers about my concerns. However, the one discussion that really changed my mind set was a conversation with my wife, she told me – “You worked too hard for this, don’t let yourself be your worst enemy.” There it was, clear as day! It was all my own insecurities, my own self-doubts, myself talking me out of this equation. So, I worked hard to observe and utilize the tools I had been provided throughout my career and at Pythian.

Once I was more open, I quickly noticed that Pythian really believes in their employees. They really empower their employees to do well and have a company culture to nurture their growth. They are always to open to new ideas and are always willing to take on risks with you. In the event that your new idea does not work, they still support you and move on to other ideas without pointing fingers or seeing you as a failure. Pythian’s unwritten culture is to make sure their employees don’t fail. Being in HR here, I was able to observe these facts are not just in our department but it is well exemplified in all business units across Pythian, starting from the CEO and Founder, to the Senior Executive team, to the Leadership team, and so on. I also had the opportunity to address my concerns to my own colleagues and manager and they were nothing but supportive. They provided immediate feedback and were able to provide relatable experiences with me.

Today, I feel like I am a much better swimmer. With the support I was provided by my colleagues, my managers, and the company my confidence in my abilities have grown and I know I have and will continue to add value to the organization. I feel I have met the expectations of the people who hired me, and I know I have made the right decision in joining this team.

This was my own experience of the first 6 months at Pythian. I hope that others have had similar or will have similar experiences as I have had (although I hope they are better swimmers). It has been tough but I’ve enjoyed every second of it. Life is a journey after all!

Reminders are an inevitable part of our personal and professional lives. Keep reminding of oneself about goals, priorities, and milestones is key to stay on top of things. This Log Buffer Edition is a reminder about all those not-to-be-missed blogs out there.

Oracle:

One of the developers was performing a delete using a nested subquery, which was throwing out ORA-00600 errors. The ORA-00600 Lookup Tool on MOS suggested it was caused by a bug, Tim Hall reports.

The healthcare sector has a clear opportunity to modernize care delivery by adopting new data interchange standards and EHR systems. Kyle opines.

This might be something very obvious for the reader but Martin had an interesting revelation recently when implementing parallel_degree_limit_p1 in a resource consumer group.

Alberto has a good post on fast refresh of outer-join-only materialized views.

Charles Hooper is discussing Unexpected Timer Resolution, Unexpected Parked CPUs, Unexpected Power Consumption.

SQL Server:

There are five SQL Server white papers which every DBA should read.

James gets into an interesting discussion about consultants.

Kendra Little wonders as why most people don’t follow best practices.

Kendal Van Dyke writes a quick note to let you know that SQL Power Doc version 1.0.1.0 is now live.

Let’s talk about database application benchmarking with Kevin Kline.

MySQL:

Stewart Smith is sharing a few notes on InnoDB in MySQL 5.7.1.

Alexey Kopytov has blogged more on transaction descriptors optimization.

MySQL Cluster gets a Community Award, Tomas tells us.

Oracle’s VP of MySQL Engineering Tomas Ulin delivered on Tuesday a keynote entitled “Driving MySQL Innovation for Next Generation Applications” at the Percona Live Conference.

MariaDB Introduces Atomic Writes, Arel Schwenke reveals.

This week my DBA team satisfied a request to restore 1 month old lost data.  A review of the RMAN backups showed that we indeed had a valid level 0 from the requested day which was 33 days ago.  However, upon recovery, the necessary archive logs to bring the database to a consistent point, were not available.  With a 28 day retention policy, running level 0 backups every 15 days, and level 1s each night, we did have the necessary level 0 backup, but the archive logs had been deleted.  How to retrieve a copy of a user schema from a database that you cannot open?  With the hidden “_allow_resetlogs_corruption” parameter.

To make matters more difficult, we were also doing just a partial database restore – we only required 1 schema, located in its own tablespace but this tablespace was encrypted.

The partial database restore was simple enough.  The team began the restore and recovery on another server.  The server where recovery is to be performed should have the same path/location of files as the old server, if not then update the path of the datafiles using set newname command in rman recovery, update the logfiles using alter database rename commands and update the tempfile by re-creating it.

RMAN> run
{
ALLOCATE CHANNEL C1 DEVICE TYPE DISK;
ALLOCATE CHANNEL C2 DEVICE TYPE DISK;
ALLOCATE CHANNEL C3 DEVICE TYPE DISK;
set until time "to_date('16-03-2013 07:00:00','dd-mm-yyyy hh24:mi:ss')";
restore database skip forever tablespace 'USERS1','USERS2',' USERS3',' USERS4';
recover database skip forever tablespace 'USERS1','USERS2',' USERS3',' USERS4';
RELEASE  CHANNEL C1;
RELEASE  CHANNEL C3;
RELEASE  CHANNEL C3;
}

RMAN completed the restore of the SYSTEM, SYSAUX, UNDO and our USERS tablespace without issue, but sadly during the recovery we received the Oracle Error:

ORA-01547: warning: RECOVER succeeded but OPEN RESETLOGS would get error below
ORA-01194: file 1 needs more recovery to be consistent
RMAN-03002: failure of recover command at 04/19/2013 17:24:34
RMAN-06053: unable to perform media recovery because of missing log

You know at this point that your data is right there – so close to grab and provide for your customer and become the DBA hero you long to be – but Oracle won’t open the door and let you in.  What to do?  All we need is to get the data and get out.  We are not going to allow users into this half-way house we call an Oracle database.  So, we allow for the corruption with the hidden parameter “_allow_resetlogs_corruption”.  This should not be used lightly – your database will be a mess – but it will allow you to open it, and hopefully get the data you need.

SQL> shutdown immediate
ORA-01109: database not open
Database dismounted.
ORACLE instance shut down.
SQL> startup mount
ORACLE instance started.

Total System Global Area  530288640 bytes
Fixed Size                  2131120 bytes
Variable Size             310381392 bytes
Database Buffers          209715200 bytes
Redo Buffers                8060928 bytes
Database mounted.

SQL> ALTER SYSTEM SET "_allow_resetlogs_corruption"= TRUE SCOPE = SPFILE;
SQL> ALTER SYSTEM SET undo_management=MANUAL SCOPE = SPFILE;
SQL> shutdown immediate
ORA-01109: database not open
Database dismounted.
ORACLE instance shut down.
SQL> startup mount
ORACLE instance started.

Total System Global Area  530288640 bytes
Fixed Size                  2131120 bytes
Variable Size             310381392 bytes
Database Buffers          209715200 bytes
Redo Buffers                8060928 bytes
Database mounted.

SQL> alter database open resetlogs;

Database altered.

Halleluiah! As a DBA, these are words you long to see on your screen “Database opened”.  At this point, I am certain my team has super human DBA talent.  The data is there for the taking!  But wait, still not able to grab it.  The database continues to crash with ORA-600 errors.  The team realizes that the errors are related to Oracle jobs that are failing, we set job_queue_processes=0; and restart the instance.  The database seems to be stable now, but the expdp needs a job process in order to run.  If we modify the job_queue_processes parameter, then the instance will crash.  We go back to our roots and try the old export utility “exp”.  This works great, except if you remember our tablespaces are encrypted.  Exp cannot export the data from them.  Painstakingly, the team moves each object into a new non-encrypted tablespace.  Many ORA-00600 errors are hit along the way.  The road is not smooth, but, with perseverance all objects get moved.  We then create a new UNDO tablespace:

SQL> CREATE UNDO TABLESPACE undo1 datafile '<ora_data_path>\undo1_1.dbf' size 200m autoextend on maxsize unlimited;
Tablespace created.
SQL> ALTER SYSTEM SET undo_tablespace = undo1 SCOPE=spfile;
System altered.
SQL> alter system set undo_management=auto scope=spfile;
System altered.

SQL> shutdown immediate
SQL> startup

Finally the team is able to generate a successful export dump using exp:

exp file=<path_to_exportlocation>\tran.dmp indexes=n direct=y feedback=100000 statistics=none owner=('<schema_owner>')

The team members are now experts at both partial recoveries and getting the front door to an Oracle database open when you fail to have the right keys.  The lesson here?  Keep all your keys to the front door, but remember back doors do exist.