update t
  set id=(case id
             when 1 then 2
             when 2 then 1
             end)
  where id in (1,2);

If that was not the case, you would have to DEFER the constraint validation to the commit time every time you ran a command like the one above, or you would get an error like:

ORA-00001: unique constraint T_PK violated

To read more about how updates work in Oracle, look at Tom Kyte’s Write Consistency Part I, Part II, Seeing a Restart, and Part III, Why is a Restart Important to Us. But that’s a tangent to the subject of this post. What I’ll show here is that you can run a DELETE and an INSERT as parts of the same statement.

The test case

I’m not very sure what you can do with this hidden feature. I’m not sure either how Oracle supports it. What I do know is that it allows you to move a row without ENABLE ROW MOVEMENT, and without the need to DEFER constraints. To illustrate this point, you’ll find below a script that creates two tables, MASTER_T and DETAIL_T, that are linked by a foreign key with an ON DELETE CASCADE constraint:

create table master_t(
         id number,
         text varchar2(50),
         constraint master_t_pk
                    primary key(id)
         );

insert into master_t
       values (1,'Text 1');
insert into master_t
       values (2,'Text 2');

create table detail_t(
         master_id number,
         constraint detail_master_fk
                    foreign key(master_id)
                    references master_t(id)
                    on delete cascade);

insert into detail_t
       values (1);

commit;

So the idea is to run the two following statements as if they were a single UPDATE:

delete from master_t
       where id=1;
insert into master_t
       values (1,'Text 3');
commit;

The tricks

As you might guess, there is not just one, but two tricks:

1. To avoid the referential constraint being checked before
   the INSERT is executed, I use a Streams Apply process and feed it the LCRs corresponding to the DELETE and INSERT above. To make Oracle apply the two statements as though they were executed at the same time, I use the same SCN and transaction identifier in the two LCRs.
2. If you use the LCR$_ROW_RECORD construct to create the LCR programatically, the SCN will be ignored. To workaround that issue, I define the LCRs with XML files, and I instantiate them with the DBMS_STREAMS.convert_xml_to_lcr function.

Note:
I’ve tested it with 11.1.0.7 on Linux x86 32 bits; it’s very likely
that the behavior of DBMS_STREAMS.convert_xml_to_lcr is not something wanted by Oracle.

The setup

To set up Streams, I have used the same schema for the tables, the streams queue, the apply process, and to instantiate the table. To make that setup easier, I have granted DBA to the schema owner. There is not much to pay attention to:

• Make sure the parameter apply_capture is
  set to false when you create the apply process so that it dequeues the persistent messages from the queue and not the buffered messages.
• If you don’t add any rule set to the apply (as I don’t), it will apply all the messages it gets from the queue.

Here is the script that creates the queue and the apply, and instantiates
the table:

begin
   dbms_streams_adm.set_up_queue(
       queue_table => 'custom_queue_table',
       queue_name  => 'custom_queue');
end;
/

declare
  v_name varchar2(256);
begin
   select value into v_name
     from v$parameter
    where name='db_unique_name';
 dbms_apply_adm.create_apply(
     queue_name => 'custom_queue',
     apply_name => 'custom_apply',
     apply_captured => false,
     source_database => v_name );
end;
/

declare
   v_scn number;
   v_name varchar2(256);
begin
   select value into v_name
     from v$parameter
    where name='db_unique_name';
   select dbms_flashback.get_system_change_number
          into v_scn
     from dual;
   dbms_apply_adm.set_table_instantiation_scn(
       source_object_name   => 'master_t',
       source_database_name => v_name,
       instantiation_scn    => v_scn);
end;
/
exec dbms_apply_adm.start_apply('CUSTOM_APPLY');

Enqueue the LCRs

Once everything is ready, you can build the LCRs and enqueue them in the queue so that the DELETE and the INSERT are applied to the table before the referential constraint is checked. Look at the ROWID to make sure the row really is deleted and inserted. You can also check from the table that it doesn’t impact the detail_t table (at least if the constraint doesn’t propagate the change to the detail table):

col id format 99
col text format a6
select rowid, id, text
  from master_t;
ROWID		    ID TEXT
------------------ --- ------
AAATK5AAEAAAARuAAA   1 Text 1
AAATK5AAEAAAARuAAB   2 Text 2

declare
   v_name    varchar2(256);
   v_scn     number;
   x         varchar2(4000);
   y         varchar2(4000);
   xm        xmltype;
   v_any     anydata;
   enqopt    dbms_aq.enqueue_options_t;
   mprop     dbms_aq.message_properties_t;
   enq_msgid RAW(16);
begin
   -- Get DB Name and SCN
   select value into v_name
     from v$parameter
    where name='db_unique_name';
   select dbms_flashback.get_system_change_number
          into v_scn
     from dual;

   -- Define the Publisher
   mprop.SENDER_ID := SYS.AQ$_AGENT(user, null, null);

   -- Build the DELETE LCR and enqueue it
x:='<ROW_LCR xmlns="http://xmlns.oracle.com/streams/schemas/lcr"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://xmlns.oracle.com/streams/schemas/lcr
http://xmlns.oracle.com/streams/schemas/lcr/streamslcr.xsd">
  <source_database_name>[db]</source_database_name>
  <command_type>DELETE</command_type>
  <object_owner>[owner]</object_owner>
  <object_name>[name]</object_name>
  <transaction_id>[txid]</transaction_id>
  <scn>[scn]</scn>
  <old_values>
    <old_value>
      <column_name>ID</column_name>
      <data>
	<number>[id]</number>
      </data>
    </old_value>
    <old_value>
      <column_name>TEXT</column_name>
      <data>
	<varchar2></varchar2>
      </data>
    </old_value>
  </old_values>
</ROW_LCR>';
  x:=replace(x,'[db]',v_name);
  x:=replace(x,'[owner]',user);
  x:=replace(x,'[name]','MASTER_T');
  x:=replace(x,'[txid]','1.1.111');
  x:=replace(x,'[scn]',to_char(v_scn));
  x:=replace(x,'[id]',1);
  x:=replace(x,'','Text 1');
  xm:=xmltype.createXML(x);
  v_any:=dbms_streams.CONVERT_XML_TO_LCR(xm);
  DBMS_AQ.ENQUEUE(
    queue_name          =>  'custom_queue',
    enqueue_options     =>  enqopt,
    message_properties  =>  mprop,
    payload             =>  v_any,
    msgid               =>  enq_msgid);

  -- Build the INSERT LCR and enqueue it
y:='<ROW_LCR xmlns="http://xmlns.oracle.com/streams/schemas/lcr"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://xmlns.oracle.com/streams/schemas/lcr
http://xmlns.oracle.com/streams/schemas/lcr/streamslcr.xsd">
  <source_database_name>[db]</source_database_name>
  <command_type>INSERT</command_type>
  <object_owner>[owner]</object_owner>
  <object_name>[name]</object_name>
  <transaction_id>[txid]</transaction_id>
  <scn>[scn]</scn>
  <new_values>
    <new_value>
      <column_name>ID</column_name>
      <data>
	<number>[id]</number>
      </data>
    </new_value>
    <new_value>
      <column_name>TEXT</column_name>
      <data>
	<varchar2></varchar2>
      </data>
    </new_value>
  </new_values>
</ROW_LCR>';
  y:=replace(y,'[db]',v_name);
  y:=replace(y,'[owner]',user);
  y:=replace(y,'[name]','MASTER_T');
  y:=replace(y,'[txid]','1.1.111');
  y:=replace(y,'[scn]',to_char(v_scn));
  y:=replace(y,'[id]',1);
  y:=replace(y,'','Text 3');
  xm:=xmltype.createXML(y);
  v_any:=dbms_streams.CONVERT_XML_TO_LCR(xm);
  DBMS_AQ.ENQUEUE(
    queue_name          =>  'custom_queue',
    enqueue_options     =>  enqopt,
    message_properties  =>  mprop,
    payload             =>  v_any,
    msgid               =>  enq_msgid);
end;
/
commit;

select rowid, id, text
  from master_t;
ROWID		    ID TEXT
------------------ --- ------
AAATK5AAEAAAARuAAB   2 Text 2
AAATK5AAEAAAARvAAA   1 Text 3

Do it again

If you want to be able to replay the previous step, you can change the value of TEXT back to its previous value:

update master_t set text='Text 1'
 where id=1;

commit;

And you will be able to run more tests.

Conclusion

To conclude this post, there are a few things worth mentioning. First of all, you can check that the detail table is not impacted by the DELETE and INSERT:

select master_id from detail_t;

MASTER_ID
---------
        1

Second, I must confess the “As parts of the same statement” is purely an invention of my criminal mind. What happens is that the referential constraint is checked after the insert. This only prevents an error on the foreign key. If you change the order of the two statements and run the INSERT before the DELETE, Streams will fail with:

ORA-00001: unique constraint (SCOTT.MASTER_T_PK) violated

My final remark is more, I think, a funny way to finish this post, provided you’re sure you’ve understood everything. I’ve imitated the Streams behavior with a MERGE command (i.e. kind of an UPDATE and an INSERT in the same statement). The result is very nice:

select rowid, id, text
  from master_t;

ROWID		    ID TEXT
------------------ --- ------
AAATLQAAEAAABY+AAA   1 Text 1
AAATLQAAEAAABY+AAB   2 Text 2

merge into master_t d
      using (select 'AAATLQAAEAAABY+AAA' rid from dual
             union all
             select 'AAATLQAAEAAABY+AAZ' from dual
             order by rid) s
   on (d.rowid=s.rid)
   when matched then
     update set d.id=3
   when not matched then
     insert (id,text) values (1,'Text 1');

ORA-02292: integrity constraint (SCOTT.DETAIL_MASTER_FK) violated - child

merge into master_t d
      using (select 'AAATLQAAEAAABY+AAA' rid from dual
             union all
             select 'AAATLQAAEAAABY+AAZ' from dual
             order by rid desc) s
   on (d.rowid=s.rid)
   when matched then
     update set d.id=3
   when not matched then
     insert (id,text) values (1,'Text 1');

2 rows merged.

select rowid, id, text
  from master_t;
ROWID		    ID TEXT
------------------ --- ------
AAATLQAAEAAABY+AAA   3 Text 1
AAATLQAAEAAABY+AAB   2 Text 2
AAATLQAAEAAABY+AAC   1 Text 1

commit;

select * from detail_t;

MASTER_ID
---------
        1

• select o.col1, m.col1 from oracle_tab
o, mysql_tab@mysql m where o.col1=m.col1;
• insert into oracle_tab (select * from mysql_tab@mysql);

This post is intended to share, the same way Karun did it for SQL Server last year, some tips related to the setup of the Oracle Gateway for ODBC with MySQL Connector ODBC on Linux.

Prerequisites

I’ve installed all the configuration on my laptop to test it. It’s running Ubuntu Intrepid Ibex 32bits, but I won’t dig into the challenge of installing the MySQL Connector ODBC 5.1 on it. All I’ll tell you is that, if I understand correctly, the version of iodbc that comes with Intrepid doesn’t support MySQL Connector ODBC 5.1 too well, and the messages that it returns are not quite explicit. To be frank, what I did is put that monkey on Augusto’s back. He sorted out everything in a few minutes. I guess I have to thank Augusto twice, just for this post! I’ve also followed his “Installing Oracle 11gR1 on Ubuntu 8.10 Intrepid Ibex” post to install the Oracle part.

But let’s talk about the prerequisites! You need to have installed and configured the following components:

• Oracle database SE1, SE or EE; I installed 11.1.0.7 but it should work with 10g too. You can check in ORACLE_HOME/bin, it has the dg4odbc executable.
• MySQL 4.1, 5.0, 5.1 or 6.0. According to the documentation, those are the MySQL versions supported by the Connector ODBC 5.1.
• MySQL
  Connector ODBC 5.1. The Oracle Gateway for ODBC checks/relies on some features, such as the ODBC descriptor, that are not available in 3.51. You can check the associated documentation and bug 32692 for some details aboutSQLSetDescRec

Creating a MySQLDEMO database, user and table

For the purpose of the demonstration, I’ve created a database, a user and a table named DEMO with the mysql client. You’ll find the script below.

Important Note:
The gateway for ODBC doesn’t look to work correctly when data are stored in utf8 in MySQL, whether or not the Connector/ODBC does the transformation into a non-utf8 character set. For this reason, I set the MySQL database default character set to latin1. I suspect somehow the issue is related to the ODBC driver: if I use latin1 on the client side, Oracle should not see any difference, whatever the storing character set is. Anyway, there is also a limitation on the Oracle side, and it doesn’t handle utf8 correctly with the Connector (see MySupport note 756186.1)

$ mysql -uroot -p

create database demo character set latin1;
grant all privileges on demo.* to 'demo'@'localhost'
   identified by 'demo' with grant option;
flush privileges;
exit;

$ mysql -udemo -pdemo -Ddemo

create table demo (
   col1 integer,
   col2 date,
   col3 varchar(10),
   col4 varchar(10) character set utf8,
   col5 varbinary(10)) engine innodb;

insert into demo(col1, col2, col3, col4, col5)
   values(1, cast(now() as date), '0123456789', '0123456789', '0123456789');

select * from demo \G

*********** 1. row ************
col1: 1
col2: 2009-03-11
col3: 0123456789
col4: 0123456789
col5: 0123456789

exit;

Creating an ODBC DSN to access the DEMO database

Once the database was created, I created a user DSN in the Oracle owner, so that the the listener can get it via the dg4odbc program. By default the file that store the user DSN is $HOME/.odbc.ini, but you can change it to any file/location that fits your needs. This is how the file looks like on my server:

$ cat ~oracle/.odbc.ini
[ODBC Data Sources]
demo = MySQL ODBC Driver 5.1

[demo]
Driver      = /home/oracle/mysql515/lib/libmyodbc5.so
DATABASE    = demo
DESCRIPTION = MySQL ODBC 5.1.5 Connector Sample
PORT        = 3306
SERVER      = 127.0.0.1
# UID         = demo
# PWD         = demo
CHARSET     = latin1
TRACEFILE   = /tmp/myodbc-demodsn.trc
TRACE       = OFF

Make sure the CHARSET parameter is set so that it doesn’t use utf8.

Configuring dg4odbc to use the DSN

dg4odbc gets its settings from a file named init[SID].ora located in $ORACLE_HOME/hs/admin. In this case, SID is an arbitrary parameter we’ll configure in the listener.orafile (see next section). I’ve used SID=mysql for this demo, and the initmysql.ora file looks like the one below:

$ cat $ORACLE_HOME/hs/admin/initmysql.ora
#
# HS init parameters
#
HS_FDS_CONNECT_INFO=demo
HS_FDS_TRACE_LEVEL=0
HS_FDS_SHAREABLE_NAME=/home/oracle/mysql515/lib/libmyodbc5.so
HS_LANGUAGE=AMERICAN_AMERICA.WE8ISO8859P15
# HS_NLS_NCHAR=AL32UTF8
#
# ODBC specific environment variables
#
set ODBCINI=/home/oracle/.odbc.ini
set LD_LIBRARY_PATH=/home/oracle/mysql515/lib

#
# Environment variables required for the non-Oracle system
#
set HOME=/home/oracle

As you can see above:

• ODBCINI is the location odbc.ini the file.
• HS_FDS_CONNECT_INFO points to the right DSN.
• HS_FDS_SHAREABLE_NAME points to ODBC driver shared library.
• HS_LANGUAGE is set to avoid the problem described in Oracle MySupport “756186.1: Error Ora-28500 and Sqlstate I Issuing Selects From a Unicode Oracle RDBMS With Dg4odbc To Mysql”.

Configuring the listener

To configure the listener, I had to change the listener.ora file to add the SID defined in the previous section and associate it with the gateway for ODBC; here is a copy of my setup used; I’ve kept all the settings (host, port, dynamic registration) default:

$ cat $ORACLE_HOME/network/admin/listener.ora

SID_LIST_LISTENER =
  (SID_LIST =
    (SID_DESC =
      (ORACLE_HOME = /u01/app/oracle/product/11.1.0/db_1)
      (SID_NAME = mysql)
      (PROGRAM = dg4odbc)
      (ENVS ="LD_LIBRARY_PATH=/home/oracle/mysql515/lib:/usr/lib:$ORACLE_HOME/lib")
    )
  )

Once the listener is setup, you can bounce or reload it; if it’s not started, just start it:

$ lsnrctl start

And add an entry in the listener.ora file like the one below; make sure you’ve added HS=OK and that it’s not in the CONNECT_DATA clause:

MYSQL =
  (DESCRIPTION =
    (ADDRESS_LIST =
      (ADDRESS = (PROTOCOL = TCP)
                 (HOST = localhost)
                 (PORT = 1521)
      )
    )
    (CONNECT_DATA =
      (SID = mysql)
    )
    (HS = OK)
  )

Creating a database link to connect to MySQL from Oracle

At this point, and after a little debugging of the different layers, I was able to create a database link from Oracle to MySQL. To do so, I connected to Oracle and used the CREATE DATABASE LINK command:

$ sqlplus / as sysdba

create database link mysql
  connect to "demo"
  identified by "demo"
  using 'mysql';

select "col3" from "demo"@mysql;

col3
---------------------------------
0123456789

Note:
The case policy differs between Oracle and MySQL, and you must
always surround the table and columns name with double quotes.

Using the Gateway for ODBC

If you remember correctly, I stored the string “0123456789″ in col3, col4, and col5. You’ll see some of the issues with the character set by querying a datum stored in utf8:

select "col4" from "demo"@mysql;

col4
----------------------------------------
0 1 2 3 4

and one stored in a varbinary:

select "col5" from "demo"@mysql;

col5
--------------------
30313233343536373839

But you’ll also be able to enjoy some of the features of your new Oracle/MySQL integrated environment, and be able to create a table in Oracle using data from MySQL:

create table demo as
  select "col1" col1, "col2" col2, "col3" col3
    from "demo"@mysql;

select * from demo;

COL1 COL2      COL3
---- --------- ----------
   1 11-MAR-09 0123456789

Unfortunately, you cannot insert data directly from Oracle into MySQL with an insert as select:

insert into "demo"@mysql("col1").
  select 2 from dual;

ERROR at line 2:
ORA-02025: all tables in the SQL statement must be at the remote database

But you can workaround that issue with some PL/SQL (I’m not saying it’s efficient):

begin
  for i in (select col1, col2, col3 from demo) loop
     insert into "demo"@mysql("col1","col2", "col3")
       values (2,i.col2, i.col3);
  end loop;
end;
/

select "col1","col2", "col3"
  from "demo"@mysql;

col1 col2      col3
---- --------- ----------
   1 11-MAR-09 0123456789
   2 11-MAR-09 0123456789

To prevent the access to MySQL from Oracle, you can drop the database link:

drop database link mysql;

That is it. It works pretty well so far and, despite the limits of such an approach, it can be quite useful for those that want to migrate from MySQL to Oracle.

I performed this test with Oracle 11.1.0.7 on Linux x86. It consists in using dbms_backup_restore instead of nid to rename the database. You’ll find below the few steps require to get to it.

Step 1. Open the database in read-only mode

First, stop the instance with an immediate shutdown. If we were to use nid, we would mount the instance, but with dbms_backup_restore, we need to access the package. For this reason, we have to open the database in read-only mode. Here are the commands I ran:

sqlplus / as sysdba

shutdown immediate;

startup open read only;

Step 2. Get the old values; set the new ones…

Once we can access the database, we can check its NAME and DBID. The script below does these checks and prompts the user for the new NAME and DBID. To that result, it queries V$DATABASE:

var old_name varchar2(20)
var old_dbid number
var new_name varchar2(20)
var new_dbid number

exec select name, dbid -
       into :old_name,:old_dbid -
       from v$database

print old_name

accept new_name prompt "Enter the new Database Name:"
Enter the new Database Name:FRANCE

accept new_dbid prompt "Enter the new Database ID:"
Enter the new Database ID:1

exec :new_name:='&&new_name'
exec :new_dbid:=&&new_dbid

Step 3. Make the changes

I won’t go into all the details of the package. Instead, here is PL/SQL block you can run to make the change:

set serveroutput on
exec dbms_output.put_line('Convert '||:old_name||  -
     '('||to_char(:old_dbid)||') to '||:new_name|| -
     '('||to_char(:new_dbid)||')')

Convert BLACK(361377223) to FRANCE(1)

declare
  v_chgdbid   binary_integer;
  v_chgdbname binary_integer;
  v_skipped   binary_integer;
begin
  dbms_backup_restore.nidbegin(:new_name,
       :old_name,:new_dbid,:old_dbid,0,0,10);
  dbms_backup_restore.nidprocesscf(
       v_chgdbid,v_chgdbname);
  dbms_output.put_line('ControlFile: ');
  dbms_output.put_line('  => Change Name:'
       ||to_char(v_chgdbname));
  dbms_output.put_line('  => Change DBID:'
       ||to_char(v_chgdbid));
  for i in (select file#,name from v$datafile)
     loop
     dbms_backup_restore.nidprocessdf(i.file#,0,
       v_skipped,v_chgdbid,v_chgdbname);
     dbms_output.put_line('DataFile: '||i.name);
     dbms_output.put_line('  => Skipped:'
       ||to_char(v_skipped));
     dbms_output.put_line('  => Change Name:'
       ||to_char(v_chgdbname));
     dbms_output.put_line('  => Change DBID:'
       ||to_char(v_chgdbid));
     end loop;
  for i in (select file#,name from v$tempfile)
     loop
     dbms_backup_restore.nidprocessdf(i.file#,1,
       v_skipped,v_chgdbid,v_chgdbname);
     dbms_output.put_line('DataFile: '||i.name);
     dbms_output.put_line('  => Skipped:'
       ||to_char(v_skipped));
     dbms_output.put_line('  => Change Name:'
       ||to_char(v_chgdbname));
     dbms_output.put_line('  => Change DBID:'
       ||to_char(v_chgdbid));
     end loop;
  dbms_backup_restore.nidend;
end;
/

ControlFile:
=> Change Name:1
=> Change DBID:1
DataFile: /u01/app/oracle/oradata/BLACK/system01.dbf
=> Skipped:0
=> Change Name:1
=> Change DBID:1
DataFile: /u01/app/oracle/oradata/BLACK/sysaux01.dbf
=> Skipped:0
=> Change Name:1
=> Change DBID:1
DataFile: /u01/app/oracle/oradata/BLACK/undotbs01.dbf
=> Skipped:0
=> Change Name:1
=> Change DBID:1
DataFile: /u01/app/oracle/oradata/BLACK/users01.dbf
=> Skipped:0
=> Change Name:1
=> Change DBID:1
DataFile: /u01/app/oracle/oradata/BLACK/streams_tbs.dbf
=> Skipped:0
=> Change Name:1
=> Change DBID:1
DataFile: /u01/app/oracle/oradata/BLACK/temp01.dbf
=> Skipped:0
=> Change Name:1
=> Change DBID:1

Step 4. Change db_name and open the database

Before you can open the database, you have to change the db_name parameter in the spfile. Once you’ve done so, you should be able to open it with resetlogs. That’s the script I ran to get to that result:

create pfile from spfile;

!cat initBLACK.ora | \
   sed "s/db_name='BLACK'/db_name='FRANCE'/" \
   > initFRANCE.ora

shutdown immediate;

startup mount pfile=initFRANCE.ora

alter database open resetlogs;

create spfile from pfile='initFRANCE.ora';

startup force;

Conclusion

I probably have the coolest Database Name and ID in the world now:

select name, dbid from v$database;

NAME		DBID
--------- ----------
FRANCE		   1

But if you think about it, I also changed some information in the datafile, even though the database was opened in read-only mode. Interesting?

You don’t need any complicated schemas to check this out for yourself; just one table as below:

create table x (
         a number,
         b number,
         c number);

begin
  for i in 1..1000 loop
    for j in 1..10 loop
      insert into x values (j,j,j);
    end loop;
  end loop;
end;
/

commit;

exec dbms_stats.gather_table_stats(-
                 user,-
                 'X');

Execute a query with a WHERE clause containing two correlated columns:

explain plan for select c from x where a=1 and b=1;

select * from table(dbms_xplan.display);

--------------------------------------------------------------------------
| Id  | Operation	  | Name | Rows  | Bytes | Cost (%CPU)| Time	 |
--------------------------------------------------------------------------
|   0 | SELECT STATEMENT  |	 |   100 |   900 |     7   (0)| 00:00:01 |
|*  1 |  TABLE ACCESS FULL| X	 |   100 |   900 |     7   (0)| 00:00:01 |
--------------------------------------------------------------------------

The optimizer estimates the number of rows returned as 100, which is equivalent to considering that each one of the conditions in the WHERE clause returns 1000 rows, and that those columns are not correlated. (In that case that’s obviously wrong, because a equals b). So, no change with 10g? If you now create a multi-column index on (a,b), it estimates changes:

alter session set events '10053 trace name context level 1';

create index xidx on x(a,b);

explain plan for select /*+ no_index */ c from x where a=1 and b=1;

select * from table(dbms_xplan.display);

--------------------------------------------------------------------------
| Id  | Operation	  | Name | Rows  | Bytes | Cost (%CPU)| Time	 |
--------------------------------------------------------------------------
|   0 | SELECT STATEMENT  |	 |  1000 |  9000 |     7   (0)| 00:00:01 |
|*  1 |  TABLE ACCESS FULL| X	 |  1000 |  9000 |     7   (0)| 00:00:01 |
--------------------------------------------------------------------------

alter session set events '10053 trace name context off';

You can see from the 10053 trace that, as is the case with extended statistics, there is a correction factor for the number of matching row estimate:

Access path analysis for X
***************************************
SINGLE TABLE ACCESS PATH
  Single Table Cardinality Estimation for X[X]
  ColGroup (#1, Index) XIDX
    Col#: 1 2    CorStregth: 10.00
  ColGroup Usage:: PredCnt: 2  Matches Full: #1  Partial:  Sel: 0.1000
  Table: X  Alias: X
    Card: Original: 10000.000000  Rounded: 1000  Computed: 1000.00  Non Adjusted: 1000.00
  Access Path: TableScan
    Cost:  7.16  Resp: 7.16  Degree: 0
      Cost_io: 7.00  Cost_cpu: 2412429
      Resp_io: 7.00  Resp_cpu: 2412429
  ColGroup Usage:: PredCnt: 2  Matches Full: #1  Partial:  Sel: 0.1000
  ColGroup Usage:: PredCnt: 2  Matches Full: #1  Partial:  Sel: 0.1000
  Access Path: index (AllEqRange)
    Index: XIDX
    resc_io: 23.00  resc_cpu: 554643
    ix_sel: 0.100000  ix_sel_with_filters: 0.100000
    Cost: 23.04  Resp: 23.04  Degree: 1
  Best:: AccessPath: TableScan
         Cost: 7.16  Degree: 1  Resp: 7.16  Card: 1000.00  Bytes: 0

And, obviously, that was not the case in 10g:

alter session set optimizer_features_enable='10.2.0.4';

explain plan for select /*+ no_index 10204 */ c from x where a=1 and b=1;

select * from table(dbms_xplan.display);

--------------------------------------------------------------------------
| Id  | Operation	  | Name | Rows  | Bytes | Cost (%CPU)| Time	 |
--------------------------------------------------------------------------
|   0 | SELECT STATEMENT  |	 |   100 |   900 |     7   (0)| 00:00:01 |
|*  1 |  TABLE ACCESS FULL| X	 |   100 |   900 |     7   (0)| 00:00:01 |
--------------------------------------------------------------------------

alter session set optimizer_features_enable='11.1.0.7';

Every change can have some drawbacks:

explain plan for select /*+ no_index */ c from x where a=1 and b=2;

SQL> select * from table(dbms_xplan.display);

--------------------------------------------------------------------------
| Id  | Operation	  | Name | Rows  | Bytes | Cost (%CPU)| Time	 |
--------------------------------------------------------------------------
|   0 | SELECT STATEMENT  |	 |  1000 |  9000 |     7   (0)| 00:00:01 |
|*  1 |  TABLE ACCESS FULL| X	 |  1000 |  9000 |     7   (0)| 00:00:01 |
--------------------------------------------------------------------------

SQL> select count(*)
  2    from dba_stat_extensions
  3   where table_name='X';

  COUNT(*)
----------
	 0

In my case, the move to 11g has been really beneficial and without any change other than the upgrade itself. This quirk, however, does give me the feeling that once again, the move to a new release could yet have a few surprises.

1. Installation of 10.2 And 11.1 Databases
2. Patches of 10.2 And 11.1 databases
3. Cloning Software and databases
4. Install a 10.2 RAC Database
5. Add a Node to a 10.2 RAC database
6. Remove a Node from a 10.2 RAC database
7. Install a 11.1 RAC Database
8. Add a Node to a 11.1 RAC database
9. Remove a Node from a 11.1 RAC database (this post!)
10. A ton of other stuff you should know

In what follows, we’ll remove rac-server5 from the configuration. We assume that rac-server5 is not accessible anymore. However, to cover the case when the node is still available, I’ll mark specific sections with a (*), meaning that the step should be done only if you still have access to that node; it can just be ignored otherwise.

Backup the Voting Disk and the OCR

Backup the voting disk with dd and the OCR with ocrconfig:

rac-server1$ su -
rac-server1# cd $ORA_CRS_HOME/bin
rac-server1# ./crsctl query css votedisk
rac-server1# mkdir -p /home/oracle/backup
rac-server1# chown oracle:dba /home/oracle/backup
rac-server1# dd if=/dev/sdb5                        \
                of=/home/oracle/backup/votedisk.bak \
                bs=4k
rac-server1# ./ocrconfig -manualbackup
rac-server1# cp /u01/app/crs/cdata/Xluster/backup_20080707_225600.ocr \
             /home/oracle/backup/.

Remove the services from the instance you plan to delete

Update the services so that they don’t contain the instance you plan to remove, with srvctl modify service:

rac-server1$ # The command below lists the services
rac-server1$ srvctl config service -d ORCL
rac-server1$ # The command below modify the OLTP service
rac-server1$ srvctl modify service -d ORCL \
                -s OLTP -n                 \
                -i "ORCL1,ORCL2,ORCL3,ORCL4"

Remove the instance from the node

With DBCA

DBCA is probably the easiest and fastest way to delete an instance from a RAC database (you can also do it manually, of course). DBCA doesn’t actually require the node-to-be-removed to be available. Obviously, if that’s not the case, the associated files—oratab, background_dump_dest—won’t be deleted. Run the command below from any of the servers:

rac-server1$ . oraenv
             ORCL
rac-server1$ dbca -silent -deleteInstance \
         -gdbName ORCL                    \
         -instanceName ORCL5              \
         -sysDBAPassword xxx

You need to use the SYS password or at least to know the one from a SYSDBA user.

Manually

In some situations, it may be useful to know how to delete an instance from a database manually. The steps to perform this operation are similar to the ones for 10.2 described in the 6^th post of this series.

Remove ASM from the node

You cannot delete ASM with DBCA. Deleting an ASM instance, however, is straightforward. It consists in: (1) stopping the ASM instance (*); (2) removing the ASM instance from the OCR; and (3) deleting the ASM init.ora file (*). To proceed, execute the commands below:

rac-server1$ srvctl stop asm -n rac-server5

rac-server1$ srvctl remove asm -n rac-server5

rac-server1$ ssh rac-server5
rac-server5$ rm $ORACLE_HOME/dbs/init+ASM5.ora

Remove the Listener from the node

With 11.1, you don’t need to use NETCA to delete a listener. The good news is that you can now use srvctl instead of crs_unregister and thus use a supported way to remove a listener from a configuration when the node is not available anymore. To proceed run:

rac-server1$ srvctl stop listener -n rac-server5
rac-server1$ srvctl remove listener -n rac-server5

Once this is done, update all the network files from the remaining nodes and from the node you are removing(*), and remove all references to that node.

Remove the Database Software

By removing the software, there are two separate things to do: (1) Update the Oracle Inventory on all the nodes that will remain. That way you will remove all the software installer links to the server; (2) remove the software from that node (*). This second operation is required only if the node is to be reused.

Update the inventory of the remaining nodes

Oracle Universal Installer can do this from any of the remaining nodes. What you’ll declare in that specific case is that only rac-server1, rac-server2, rac-server3, and rac-server4 will still be part of the clustered installation. To update the inventories of those nodes, run:

rac-server1$ export ORACLE_HOME=/u01/app/oracle/product/11.1.0/db_1
rac-server1$ cd $ORACLE_HOME/oui/bin
rac-server1$ ./runInstaller -silent -updateNodeList \
             ORACLE_HOME=$ORACLE_HOME               \
             "CLUSTER_NODES={rac-server1,rac-server2,rac-server3,rac-server4}"

Once you’ve updated the inventory, Oracle Universal Installer will never again prompt you for rac-server5 when used from any of those four nodes.

Delete the Database Software from the node to be removed (*)

One of the limits of the Oracle Universal Installer with RAC is that you can not use it for one node only. It’s probably a good thing in a way as you cannot apply a Patch Set on one node only and mess up everything. Unfortunately when it comes time that you want to remove the software from one server only, you have to work around that limitation. The way to do it is to update the inventory of that node only and let it think it is the only node of the cluster. Once you’ve done this, you’ll be able to execute the Universal Installer with various syntaxes like detachHome or deinstall. To change the inventory on the node to be removed, connect to it and run:

rac-server5$ export ORACLE_HOME=/u01/app/oracle/product/11.1.0/db_1
rac-server5$ cd $ORACLE_HOME/oui/bin
rac-server5$ ./runInstaller -silent -updateNodeList \
             ORACLE_HOME=$ORACLE_HOME               \
             CLUSTER_NODES="" -local

Once the inventory updated, all the runInstaller command you’ll run will touch only the local ORACLE_HOME (assuming it is not shared). Then, as described in the documentation, you can remove the ORACLE_HOME you want and withdraw it from the cluster:

rac-server5$ cat /etc/oraInst.loc
rac-server5$ cd /u01/app/oraInventory/ContentsXML
rac-server5$ grep NAME inventory.xml
rac-server5$ export ORACLE_HOME=/u01/app/oracle/product/11.1.0/db_1
rac-server5$ cd $ORACLE_HOME/oui/bin
rac-server5$ ./runInstaller -silent -deinstall -removeallfiles \
              "REMOVE_HOMES={/u01/app/oracle/product/11.1.0/db_1}"

You can also detach the ORACLE_HOME from the inventory with the -detachHome syntax as below:

rac-server5$ cat /etc/oraInst.loc
rac-server5$ cd /u01/app/oraInventory/ContentsXML
rac-server5$ grep NAME inventory.xml
rac-server5$ export ORACLE_HOME=/u01/app/oracle/product/11.1.0/db_1
rac-server5$ cd $ORACLE_HOME/oui/bin
rac-server5$ ./runInstaller -silent -detachHome                  \
              ORACLE_HOME="/u01/app/oracle/product/11.1.0/db_1" \
              ORACLE_HOME_NAME="OraDB111Home1"

This second approach allow you to keep the ORACLE_HOME and delete its content only if or when you want:

rac-server5$ rm -rf /u01/app/oracle/product/11.1.0/db_1

If that’s the last database software installed on the server, you can delete the oratab file too:

rac-server5# rm /etc/oratab

Remove the ONS Configuration

To remove the ONS subscription from the server, you first have to identify the port that is used. With 11g, that port is stored in the OCR. Dump the OCR and locate the port in the dump file:

rac-server1$ cd $ORA_CRS_HOME/bin
rac-server1$ ./ocrdump /home/oracle/backup/ocr.dmp
rac-server1$ grep -A1 "DATABASE.ONS_HOSTS.rac-server5.PORT" \
                      /home/oracle/backup/ocr.dmp

Note: If you’ve installed the 11.1.0.6 clusterware without adding the ONS configuration, the OCR may not store the remote port. If that’s the case, just skip the next step.

Then, you can remove the server and its port from the ONS remote configuration:

rac-server1$ cd $ORA_CRS_HOME/bin
rac-server1$ ./racgons remove_config rac-server5:6250

Remove the NodeApps

The nodeapps includes the GSD, the ONS, and the VIP. You can simply remove them from any of the node with the srvctl remove nodeapps command:

rac-server1$ srvctl stop nodeapps -n rac-server5
rac-server1$ su -
rac-server1# ./srvctl remove nodeapps -n rac-server5

You can check that the nodeapps have been removed by querying its status with srvctl, or by checking the resources named ora.rac-server5 as below:

rac-server1$ cd $ORA_CRS_HOME/bin
rac-server1$ ./crs_stat |grep "ora.rac-server5"

Remove the Clusterware Software

Removing the clusterware software is very similar from removing the Database Software—there are two separate things to do. (1) Update the Oracle Inventory on all the nodes that remain; and (2) remove the clusterware from that node (*). That second operation is required only if the node is to be reused.

Update the inventory of the remaining nodes

The syntax is identical to that of database removal, just add CRS=TRUE:

rac-server1$ export ORA_CRS_HOME=/u01/app/crs
rac-server1$ cd $ORA_CRS_HOME/oui/bin
rac-server1$ ./runInstaller -silent -updateNodeList  \
     ORACLE_HOME=$ORA_CRS_HOME                       \
     "CLUSTER_NODES={rac-server1,rac-server2,rac-server3,rac-server4}" \
     CRS=TRUE

Once you’ve updated the inventory, Oracle Universal Install will never again prompt you for rac-server5 when used from any of those four nodes.

Delete the Clusterware Software from the node to be removed (*)

To delete the clusterware from the node you want to delete, you must first stop it (if you haven’t already):

rac-server5$ su -
rac-server5# cd /u01/app/crs/bin
rac-server5# ./crsctl stop crs
rac-server5# ./crsctl disable crs

Then, update the inventory as below:

rac-server5$ export ORA_CRS_HOME=/u01/app/crs
rac-server5$ cd $ORA_CRS_HOME/oui/bin
rac-server5$ ./runInstaller -silent -updateNodeList \
             ORACLE_HOME=$ORA_CRS_HOME              \
             CLUSTER_NODES=""                       \
             CRS=TRUE                               \
             -local

Finally, run the Universal Installer with the deinstall and CRS=TRUE directives as below:

rac-server5$ cat /etc/oraInst.loc
rac-server5$ cd /u01/app/oraInventory/ContentsXML
rac-server5$ grep NAME inventory.xml
rac-server5$ export ORA_CRS_HOME=/u01/app/crs
rac-server5$ cd $ORA_CRS_HOME/oui/bin
rac-server5$ ./runInstaller -silent -deinstall -removeallfiles \
              "REMOVE_HOMES={/u01/app/crs}"                    \
              CRS=TRUE

Additional cleanup (*)

If the node you are removing is still accessible and you plan to reuse it (say, for another cluster) there is some additional cleanup to do:

• Delete any file that remains in the Clusterware Home. For some reason, you may have to remove all the files if the -removeallfiles didn’t do its job.
• Delete any specific entries in the oracle .profile file.
• Delete any specific entries in the crontab.
• Delete the oraInv.loc file and the inventory.
• Replace the inittab with the one backed up from before the clusterware install named inittab.no_crs.
• Delete the /var/tmp/.oracle directory.
• Delete the Startup/Shutdown services, i.e., all the /etc/init.d/init* and /etc/rc?.d/*init.crs files with Oracle or Redhat Enterprise Linux.
• Delete the clusterware and ocr.loc files, i.e., the /etc/oracle directory with Oracle or Redhat Enterprise Linux
• Delete the storage-specific configuration files to prevent altering the shared storage from that node (/etc/fstab for NFS, udev for ASM, or the raw devices)

Remove the Node from the Cluster Configuration

Everything has been removed but if you connect to any of the remaining nodes and run olsnodes, you’ll see the server is always registered in the OCR:

rac-server1$ cd /u01/app/crs/bin
rac-server1$ ./olsnodes -n -i
rac-server1    1    rac-server1-priv rac-server1-vip
rac-server2    2    rac-server2-priv rac-server2-vip
rac-server3    3    rac-server3-priv rac-server3-vip
rac-server4    4    rac-server4-priv rac-server4-vip
rac-server5    5    rac-server5-priv

To remove that server from the OCR, connect as root on any of the remaining nodes and use its name and number with the rootdeletenode.sh script as below:

rac-server1$ su -
rac-server1# cd /u01/app/crs/install
rac-server1# ./rootdeletenode.sh rac-server5,5
rac-server1# exit
rac-server1$ cd /u01/app/crs/bin
rac-server1$ olsnodes -n -i

More to come

This 9^th post is the last one about the fundamentals of the 10g and 11g database and silent syntaxes, though there are many more syntaxes you may want to look at. The 10^th and last post will present some of the additional syntaxes. But you’ll have to wait again!

(*) If you cannot access the server you are removing, don’t run this step

First you need to install an 11.1.0.7 database! Then you must load your data and capture your queries in an SQL Tuning Set. Below is a very simple and short example:

connect / as sysdba

create table demo (x number,
                   y varchar2(4000));

begin
    for i in 1..10000 loop
        insert into demo (x,y) values (i,rpad('X',4000,'X'));
    end loop;
end;
/

commit;

begin
dbms_sqltune.create_sqlset(
           sqlset_name=>'DEMOSTS',
           description=>'SQL Tuning Set to evaluate Exadata');
end;
/

exec dbms_application_info.set_module('TUNING', null);

select x from demo  where x=1;

X
-
1

exec dbms_application_info.set_module(null,null);

DECLARE
   l_cursor DBMS_SQLTUNE.sqlset_cursor;
BEGIN
   OPEN l_cursor FOR
   SELECT VALUE(p)
      FROM TABLE (DBMS_SQLTUNE.select_cursor_cache (
                   'module = ''TUNING''', -- basic_filter
                   NULL, -- object_filter
                   NULL, -- ranking_measure1
                   NULL, -- ranking_measure2
                   NULL, -- ranking_measure3
                   NULL, -- result_percentage
                   1) -- result_limit
                 ) p;  

DBMS_SQLTUNE.load_sqlset (
               sqlset_name => 'DEMOSTS',
               populate_cursor => l_cursor);
END;
/ 

col sql_text format a50
SELECT sql_id, sql_text
    FROM TABLE(DBMS_SQLTUNE.select_sqlset ('DEMOSTS')); 

SQL_ID	      SQL_TEXT
------------- --------------------------------------------------
4ynqdxdhu08p1 select x from demo  where x=1

Once that is done, all you have to do is to run the tcellsim.sql script located in the $ORACLE_HOME/rdbms/admin directory of the 11.1.0.7 software and let it guide you through. Here’s an example of the use of that script:

(If you cannot see the output in an iframe, here it is: exadata.html.)

I guess it’s a bit optimistic. All I need now is 140k USD to make sure that’s the case. Can anyone send me the money?

1. Installation of 10.2 And 11.1 Databases
2. Patches of 10.2 And 11.1 databases
3. Cloning Software and databases
4. Install a 10.2 RAC Database
5. Add a Node to a 10.2 RAC database
6. Remove a Node from a 10.2 RAC database
7. Install a 11.1 RAC Database
8. Add a Node to a 11.1 RAC database (this post!)
9. Remove a Node from a 11.1 RAC database
10. A ton of other stuff you should know

Before you start

Make sure you’ve kept a copy of the voting disk and that you have a backup of the OCR. Check that the locations for all the components to be installed, i.e.: Inventory, Clusterware, ASM, database software, OCR, Voting Disks, and data files, are writable. Confirm that all the prerequisites are met for the node and for the whole cluster with the node to be added.

Refer to the Oracle Clusterware Administration and Deployment Guide
11g Release 1 (11.1) – 4 Adding and Deleting Oracle Clusterware Homes
and the Oracle Real Application Clusters Administration and Deployment Guide
11g Release 1 (11.1) – 9 Adding and Deleting Oracle RAC from Nodes on Linux and UNIX Systems for the complete reference of how to perform these steps.

Adding the new node to the Clusterware

We’ll assume we want to add a new node, rac-server5, to the cluster we’ve build in the previous post. Connect as the Clusterware owner on any of the existing nodes and run the command below:

rac-server1$ cd /u01/app/crs/oui/bin
rac-server1$ ./addNode.sh -silent CLUSTER_NEW_NODES={rac-server5} \
                CLUSTER_NEW_PRIVATE_NODE_NAMES={rac-server5-priv} \
                CLUSTER_NEW_VIRTUAL_HOSTNAMES={rac-server5-vip}

Once the software is on the new node, you have to execute the three following scripts as root:

1. orainsRoot.sh from the new node:

   rac-server5# /u01/oraInventory/orainstRoot.sh
   

2. rootaddnode.sh from the node you’ve pushed the clusterware from:

   rac-server1# cd /u01/appcrs/install/rootaddnode.sh

3. root.sh from the new node:

   rac-server5# /u01/app/crs/root.sh

Adding the new node to the ONS configuration

With 11.1, the remote port of the ONS is now stored only in the OCR. To add a remote port for the ONS of the new node you’ve added, connect as oracle on any of the nodes and run the command below:

rac-server5# export ORACLE_HOME=/u01/app/crs
rac-server5# cd $ORCLE_HOME/bin
rac-server5# ./racgons add_config rac-server5:6250

Adding the database software to the new node

Once the Clusterware is installed and started on the new node, add the database software to the new node. Make sure the software owner has the correct privileges on the directories to create the ORACLE_HOME:

rac-server5# mkdir -p /usr/app/oracle
rac-server5# chown -R oracle:oinstall /usr/app/oracle

Connect as the software owner on any of the other nodes and run and run the addNode.sh command as below:

rac-server1$ export ORACLE_HOME=/u01/app/oracle/product/11.1.0/db_1
rac-server1$ cd $ORACLE_HOME/oui/bin
rac-server1$ ./addNode.sh -silent CLUSTER_NEW_NODES={rac-server5}

Execute the root.sh script as root on the new node:

rac-server5# cd /usr/app/oracle/product/11.1.0/db_1
rac-server5# ./root.sh

Repeat the above set of operations as many times as necessary to install all the ORACLE_HOMEs you want on the new node. That’s especially true if you use a separate ORACLE_HOME for ASM. With 11.1, you can also use Cloning to Extend Oracle RAC to Nodes in the Same Cluster.

Adding the listener to the new node

With 11.1, you can now use the srvctl add listener command to add the listener resource to the OCR. However, it’s simpler to use NETCA as below:

rac-server1$ export ORACLE_HOME=/u01/app/oracle/product/11.1.0/db_1
rac-server1$ export PATH=$ORACLE_HOME/bin:$PATH
rac-server1$ netca /silent /responsefile \
             $ORACLE_HOME/network/install/netca_typ.rsp \
             /nodeinfo rac-server5

With 11.1 there is no need to set the DISPLAY variable.

Adding ASM to the new node

You can use DBCA to configure the new ASM instance. To do so, run it with -configureASM and the list of nodes. The nodes that don’t exist will be created:

rac-server1$ dbca -silent -configureASM        \
 -nodeList rac-server1,rac-server2,rac-server3,\
rac-server4,rac-server5                        \
 -asmSysPassword xxxx                          \
 -emConfiguration NONE

Adding an instance to the New Node with DBCA

You can add the new instance with DBCA too:

rac-server1$ dbca -silent -addInstance  \
  -gdbName ORCL                         \
  -sysDBAPassword xxx                   \
  -nodelist rac-server5

Manually adding an instance to the new node

You can also manually execute all the steps DBCA takes behind the scene. These steps are identical to the ones from 10.2. Refer to the 10.2 post of the series for more details.

Adding the services

You should configure the services on the new instance. Here is an example with the OLTP service declared as preferred on all the instances:

rac-server5$ srvctl modify service -d ORCL -s OLTP \
    -n -i "ORCL1,ORCL2,ORCL3,ORCL4,ORCL5"
rac-server5$ srvctl start service -d ORCL -s OLTP \
    -i ORCL5
rac-server5$ srvctl config service -d ORCL -s OLTP
rac-server5$ srvctl status service -d ORCL -s OLTP

More to come

We are almost done with RAC silent installations. The next post will explore the removal of a node from an 11.1 RAC. Then, we’ll finish the series with a look at many more features of the Oracle Installer.

1. Installation of 10.2 And 11.1 Databases
2. Patches of 10.2 And 11.1 databases
3. Cloning Software and databases
4. Install a 10.2 RAC Database
5. Add a Node to a 10.2 RAC database
6. Remove a Node from a 10.2 RAC database
7. Install a 11.1 RAC Database (this post!)
8. Add a Node to a 11.1 RAC database
9. Remove a Node from a 11.1 RAC database
10. A ton of other stuff you should know

As for the Installation of a 10.2 RAC Database, this post shows how to (1) install the 11.1 clusterware, (2) install the 11.1 database, and (3) create a RAC database. It doesn’t explore any Patch Set upgrade since 11.1.0.7 is not out for now. Another interesting question, however, is how to upgrade the 10.2 clusterware to 11.1, since it has to be done in place.

So let’s get into it.

Checking the prerequisites

As with 10.2, the best way to start is definitely to check, double- and triple-check that all the prerequisites are met. You can refer to the 10g post to find more about how to use RDA and the CVU for this purpose. Check also the installation documentation for your platform and Metalink Note 169706.1.

Install Oracle 11.1 Clusterware

Once you’ve made sure all the prerequisites are met, you can install or upgrade the 11.1 clusterware.

Install the 11.1 Clusterware from scratch

The steps and commands to install 11.1 clusterware are exactly the same as the ones to install 10.2 clusterware. The only difference is due to a typo in the crs.rsp response file that comes with the 11.1.0.6 distribution — namely, the FROM_LOCATION parameter doesn’t point the correct location. To overcome this issue, just add the parameter in the runInstaller command line. Below is the syntax that matches that from the 10.2 post; refer to it for more detail about the meaning of the parameters.

cd clusterware
export DISTRIB=`pwd`
echo $DISTRIB

./runInstaller -silent                                           \
  -responseFile $DISTRIB/response/crs.rsp                        \
  FROM_LOCATION=$DISTRIB/stage/products.xml                      \
  ORACLE_HOME="/u01/app/crs"                                     \
  ORACLE_HOME_NAME="OraCrsHome"                                  \
  s_clustername="rac-cluster"                                    \
sl_tableList={"rac-server1:rac-server1-priv:rac-server1-vip:N:Y",\
"rac-server2:rac-server2-priv:rac-server2-vip:N:Y",\
"rac-server3:rac-server3-priv:rac-server3-vip:N:Y",\
"rac-server4:rac-server4-priv:rac-server4-vip:N:Y"}\
  ret_PrivIntrList={"bond0:10.0.0.0:1","bond1:192.168.1.0:2",\
"bond2:10.1.0.0:3"}                                              \
  n_storageTypeOCR=1                                             \
  s_ocrpartitionlocation="/dev/sdb1"                             \
  s_ocrMirrorLocation="/dev/sdc1"                                \
  n_storageTypeVDSK=1                                            \
  s_votingdisklocation="/dev/sdb2"                               \
  s_OcrVdskMirror1RetVal="/dev/sdc2"                             \
  s_VdskMirror2RetVal="/dev/sdd1"

Once the clusterware is installed, you only have to connect as root on each of the servers and run the orainstRoot.sh and root.sh scripts:

rac-server1# /u01/app/oraInventory/orainstRoot.sh
rac-server2# /u01/app/oraInventory/orainstRoot.sh
rac-server3# /u01/app/oraInventory/orainstRoot.sh
rac-server4# /u01/app/oraInventory/orainstRoot.sh
rac-server1# /u01/app/crs/root.sh
rac-server2# /u01/app/crs/root.sh
rac-server1# /u01/app/crs/root.sh
rac-server1# /u01/app/crs/root.sh

Note that, unlike what use to happen with 10.2, if you use a private network (i.e. 192.168.x.x, 10.x.x.x or 172.[16-31].x.x), it should not affect the installation.

Upgrade your 10.2 Clusterware to 11.1

If the install didn’t change between 10.2 and 11.1, the clusterware upgrade differs slightly from applyingthe patch set on top of the clusterware. The principle stays the same, however: (1) The clusterware has to be applied in-place; and (2) on a rolling upgrade fashion. However, the way you do it now is:

1. Stop the clusterware and its managed resources on one or several node so that you can apply the patch on top of them.
2. Apply the 11.1 release from one of the stopped nodes to all the stopped nodes.
3. Apply the rootupgrade script to all the nodes that were upgraded.
4. For another set of servers, repeat (1), (2), and (3), and do it until you’ve upgraded all the servers.

In the case of the RAC we installed in this series’s article on the 10.2 RAC Intall, one way you could do the upgrade would be to upgrade rac-server1 as a first step and as a second step upgrade rac-server2, rac-server3, and rac-server4 all together. Obviously, you could also upgrade them all together or one-by-one depending on your requirements. Let’s have a look at the syntaxes for the first scenario.

Important Note: The Clusterware has to be 10.2.0.3 or higher to upgrade to 11.1.0.6.

Step 1: Prepare rac-server1 for the upgrade

You first have to push the clusterware distribution to the server and unzip it. Once this is done, you should run the preupdate.sh script as root with the clusterware ORACLE_HOME and owner as parameters. Below is an example of the associated syntax:

rac-server1$ cd clusterware
rac-server1$ export DISTRIB=`pwd`
rac-server1$ su root
rac-server1# cd $DISTRIB/upgrade
rac-server1#./preupdate.sh -crshome /u01/app/crs -crsuser oracle

This step will stop all the managed resources and the clusterware.

Step 2: Install the 11.1 clusterware in rac-server1

Connect as oracle (or, if it’s not oracle, as the clusterware owner) and run a command like the one below:

$ cd clusterware
$ export DISTRIB=`pwd`
$ ./runInstaller -silent                      \
    -responseFile $DISTRIB/response/crs.rsp   \
    FROM_LOCATION=$DISTRIB/stage/products.xml \
    REMOTE_NODES={}                           \
    ORACLE_HOME=/u01/app/crs                  \
    ORACLE_HOME_NAME="OraCrsHome"

If you’ve been reading this series, -silent, -responseFile, ORACLE_HOME and ORACLE_HOME_NAME will be familiar to you. FROM_LOCATION is in the command thanks to a typo in the crs.rsp file that come with 11.1.0.6 for Linux x86 — it doesn’t point to the right location. REMOTE_NODES is the list of nodes in addition of the local node onto which you’ll install clusterware 11.1. In this case, because we apply the upgrade on rac-server1 only, which is the local node, the list has to be an empty list.

Step 3: Apply the rootupgrade script

Once you’ve applied the 11.1 release on top of the 11.1 clusterware for that first note, just run the rootupgrade script. It will complete the upgrade of that node and restart all the resources. As root:

rac-server1# cd /u01/crs/install
rac-server1# ./rootupgrade

Step 4: Prepare the other servers for the upgrade

We will then apply the patch set from rac-server2 on the three remaining servers, To proceed, push the clusterware distribution to rac-server2 and the preupdate.sh script to the three servers. Run that script as root on those three servers as you did on the first one:

rac-server2# /preupdate.sh -crshome /u01/app/crs -crsuser oracle
rac-server3# /tmp/preupdate.sh -crshome /u01/app/crs -crsuser oracle
rac-server4# /tmp/preupdate.sh -crshome /u01/app/crs -crsuser oracle

This step will stop all the managed resources and the clusterware.

Step 5: Install the 11.1 clusterware on rac-server2, rac-server2 and rac-server4 all together

Connect as oracle (or, if it’s not oracle, as the clusterware owner) on the server you’ll use to do the install, and run a command like the one below:

rac-server2$ cd clusterware
rac-server2$ export DISTRIB=`pwd`
rac-server2$ ./runInstaller -silent           \
    -responseFile $DISTRIB/response/crs.rsp   \
    FROM_LOCATION=$DISTRIB/stage/products.xml \
    REMOTE_NODES={rac-server3,rac-server4}    \
    ORACLE_HOME=/u01/app/crs                  \
    ORACLE_HOME_NAME="OraCrsHome"

REMOTE_NODES is used to list all the nodes onto which you’ll install the clusterware. The local node will also be installed.

Step 6: Apply the rootupgrade script

Once you’ve applied the 11.1 release on top of the 10.2 clusterware for that first note, just run the rootupgrade script as root on each one of the nodes:

rac-server2# cd /u01/crs/install/rootupgrade
rac-server3# cd /u01/crs/install/rootupgrade
rac-server4# cd /u01/crs/install/rootupgrade

Once all the nodes are installed, you should be able to see that the 11.1 release is active by running on any of the nodes:

/u01/crs/install/bin/crsctl query crs activeversion

Install Oracle 11.1 RAC Database Software

Install the Oracle RAC Database Base Release

Once the clusterware has been installed, installing RAC Database software is very similar to installing a non-RAC database software, you just need to specify which servers you want the software to be installed on. The first step is downloading the software and unzipping it:

$ unzip linux.x64_11gR1_database.zip
$ cd database
$ export DISTRIB=`pwd`

To install the database software, you don’t need to modify the response files. You only have to run a command like the one below, in the case of an Enterprise Edition:

export DISTRIB=`pwd`

runInstaller -silent                                   \
      -responseFile $DISTRIB/response/enterprise.rsp   \
       FROM_LOCATION=$DISTRIB/stage/products.xml       \
       ORACLE_BASE=/u01/app/oracle                     \
       ORACLE_HOME=/u01/app/oracle/product/11.1.0/db_1 \
       ORACLE_HOME_NAME=ORADB111_Home1                 \
       CLUSTER_NODES={"rac-server1","rac-server2",\
"rac-server3","rac-server4"}                           \
       n_configurationOption=3                         \
       s_nameForDBAGrp="dba"                           \
       s_nameForASMGrp="dba"

Or in the case of a Standard Edition:

runInstaller -silent                                   \
      -responseFile $DISTRIB/response/standard.rsp     \
       FROM_LOCATION=$DISTRIB/stage/products.xml       \
       ORACLE_BASE=/u01/app/oracle                     \
       ORACLE_HOME=/u01/app/oracle/product/11.1.0/db_1 \
       ORACLE_HOME_NAME=ORADB111_Home1                 \
       CLUSTER_NODES={"rac-server1","rac-server2"}     \
       n_configurationOption=3

As you can see, only a few parameters differ from the non-RAC database installation described in Part 1 of this series:

• CLUSTER_NODES contains the list of cluster nodes you want to install the database software on.
• FROM_LOCATION is used when the response file doesn’t point to the location of the products.xml file.
• s_nameForDBAGrp, s_nameForASMGrp, and s_nameForOPERGrp are used to specify non-default groups for SYSDBA, SYSASM, and SYSOPER.

Once the software is installed, you have to execute the root.sh script from the ORACLE_HOME. Connect as root on every server and run:

rac-server1# /u01/app/oracle/product/11.1.0/db_1/root.sh
rac-server2# /u01/app/oracle/product/11.1.0/db_1/root.sh
rac-server3# /u01/app/oracle/product/11.1.0/db_1/root.sh
rac-server4# /u01/app/oracle/product/11.1.0/db_1/root.sh

Install the Oracle RAC database Patch Set

The first 11.1 Patch Set is not available at the time of this writing.

Configure the Listeners

The fastest way to create and configure the listeners is to use NETCA as below:

$ export ORACLE_HOME=/u01/app/oracle/product/11.1.0/db_1
$ export PATH=$ORACLE_HOME/bin:$PATH
$ netca /silent \
 /responsefile $ORACLE_HOME/network/install/netca_typ.rsp \
 /nodeinfo rac-server1,rac-server2,rac-server3,rac-server4

Unlike other tools, NETCA uses the “/” character instead of “-” for its flags. With 11.1, the DISPLAY environment variable can stay empty.

Configure Automatic Storage Management

If you plan to use ASM from the newly installed ORACLE_HOME or another one you’ve installed earlier, you can use DBCA to configure it in silent mode. The syntax is the same as the 10.2 asm configuration syntax:

$ dbca -silent                   \
    -nodelist rac-server1,rac-server2,\
rac-server3,rac-server4               \
    -configureASM                     \
    -asmSysPassword change_on_install \
    -diskString "/dev/sd*"            \
    -diskList "/dev/sde,/dev/sdf"     \
    -diskGroupName DGDATA             \
    -redundancy EXTERNAL              \
    -emConfiguration NONE

Create a RAC Database with DBCA

You can use DBCA again to create the RAC database. The syntax is the same as that explained in the 10.2 RAC post:

$ dbca -silent                             \
       -nodelist rac-server1,rac-server2,\
rac-server3,rac-server4                    \
       -createDatabase                     \
       -templateName General_Purpose.dbc   \
       -gdbName ORCL                       \
       -sid ORCL                           \
       -SysPassword change_on_install      \
       -SystemPassword manager             \
       -emConfiguration NONE               \
       -storageType ASM                    \
         -asmSysPassword change_on_install \
         -diskGroupName DGDATA             \
       -characterSet WE8ISO8859P15         \
       -totalMemory 500

From my tests, I have found that -nodelist has to come before the -createDatabase flag. 11.1 also allows you to specify the size of the memory instead of a percentage.

More to come

If you’re used to installing 10.2 RAC databases in silent mode, doing it with 11.1 is very similar. The next two posts will explore the addition and removal of new cluster nodes. Expect them very soon.

I’ve run the queries that follow on top of 10.2.0.3 on Linux X86_64.

Sample Table

Create and Fill up a table to run your queries. You’ll find the script you need below:

create table X1(a number,b number);

begin
   for i in 1..1000000 loop
      insert into X1 values (i,mod(i,100000));
   end loop;
end;
/

commit;

exec dbms_stats.gather_table_stats(user, 'X1');

Case 1: 4164 Consistent Gets for 0.14 seconds

First, let’s assume that a few Consistent Gets means good performance. Look at the following query:

set timing on
set autotrace on

select count(*)
  from (select distinct X2.a
          from X1,X1 X2
         where X1.b=X2.b
           and X2.a=1);

Elapsed: 00:00:00.14

Execution Plan
----------------------------------------------------------
Plan hash value: 4182727558

-------------------------------------------------------------------
| Id  | Operation             | Name | Rows  | Bytes | Cost (%CPU)|
-------------------------------------------------------------------
|   0 | SELECT STATEMENT      |      |     1 |    2  |  998   (7) |
|   1 |  SORT AGGREGATE       |      |     1 |    2  |            |
|   2 |   VIEW                |      |     1 |    2  |  998   (7) |
|   3 |    SORT UNIQUE NOSORT |      |     1 |   14  |  998   (7) |
|*  4 |     HASH JOIN         |      |    10 |  140  |  997   (6) |
|*  5 |      TABLE ACCESS FULL| X1   |     1 |    9  |  494   (6) |
|   6 |      TABLE ACCESS FULL| X1   |  1002K| 4895K |  491   (5) |
-------------------------------------------------------------------

Predicate Information (identified by operation id):
---------------------------------------------------
   4 - access("X1"."B"="X2"."B")
   5 - filter("X2"."A"=1)

Statistics
----------------------------------------------------------
        1  recursive calls
        0  db block gets
     4164  consistent gets
        0  physical reads
        0  redo size
      515  bytes sent via SQL*Net to client
      492  bytes received via SQL*Net from client
        2  SQL*Net roundtrips to/from client
        0  sorts (memory)
        0  sorts (disk)
        1  rows processed

Case 2: 4164 Consistent Gets for 4.17 seconds

The beauty of this case is that the execution plan is almost the same and the autotrace statistics are exactly the same:

set timing on
set autotrace on

select count(*)
  from (select distinct X2.a
          from X1,X1 X2
         where X1.b=X2.b);

COUNT(*)
----------
   1000000

Elapsed: 00:00:04.17

Execution Plan
----------------------------------------------------------
Plan hash value: 920584761

-----------------------------------------------------------------
| Id  | Operation             | Name | Rows  | Bytes |Cost (%CPU)|
-------------------------------------------------------------------
|   0 | SELECT STATEMENT      |      |    1 |        |24821   (6)|
|   1 |  SORT AGGREGATE       |      |    1 |        |           |
|   2 |   VIEW                |      | 1002K|        |24821   (6)|
|   3 |    HASH UNIQUE        |      | 1002K|    13M |24821   (6)|
|*  4 |     HASH JOIN         |      |   10M|   134M | 2932   (7)|
|   5 |      TABLE ACCESS FULL| X1   | 1002K|  4895K |  491   (5)|
|   6 |      TABLE ACCESS FULL| X1   | 1002K|  8811K |  491   (5)|
------------------------------------------------------------------

Predicate Information (identified by operation id):
---------------------------------------------------
    4 - access("X1"."B"="X2"."B")

Statistics
----------------------------------------------------------
        0  recursive calls
        0  db block gets
     4164  consistent gets
        0  physical reads
        0  redo size
      515  bytes sent via SQL*Net to client
      492  bytes received via SQL*Net from client
        2  SQL*Net roundtrips to/from client
        0  sorts (memory)
        0  sorts (disk)
        1  rows processes

Conclusion

Don’t trust “the common wisdom”, even if you’ve experienced it dozens of times. Oh, and don’t forget to drop the table:

drop table X1 purge;

P.S.: Thank you Riyaj for opening my eyes!

Here is the complete series agenda:

1. Installation of 10.2 And 11.1 Databases
2. Patches of 10.2 And 11.1 databases
3. Cloning Software and databases
4. Install a 10.2 RAC Database
5. Add a Node to a 10.2 RAC database
6. Remove a Node from a 10.2 RAC database (this post!)
7. Install a 11.1 RAC Database
8. Add a Node to a 11.1 RAC database
9. Remove a Node from a 11.1 RAC database
10. A ton of other stuff you should know

Now for the substance of this part.

Backup the Voting Disk

A good way to start is usually to think about the worse thing that could happen and how to step back if you mess something up. There is probably less risk when you remove a node than when you add a new node. However, make sure you have a backup of the voting disk and the OCR. To proceed, run the dd and ocrsconfig commands as below:

rac-server5$ cd $ORA_CRS_HOME/bin
rac-server5$ ./crsctl query css votedisk
rac-server5$ mkdir -p /home/oracle/backup
rac-server5$ dd if=/dev/sdb5                        \
                of=/home/oracle/backup/votedisk.bak \
                bs=4k
rac-server5$ ./ocrconfig -showbackup

Remove the Services from the Instance You Plan to Delete

Using srvctl modify, update the services so that they don’t contain the instance you plan to remove:

rac-server5$ # The command below lists the services
rac-server5$ srvctl config service -d ORCL
rac-server5$ # The command below modify the OLTP service
rac-server5$ srvctl modify service -d ORCL \
                -s OLTP -n                 \
                -i "ORCL1,ORCL2,ORCL3,ORCL4"

Remove the Instance from the Node with DBCA

DBCA is probably the easiest and fastest way to delete an instance from a RAC database. If we assume you want to remove rac-server5 from the RAC configuration we’ve built in post 4 and post 5, you’ll start by running the command below from any of the servers:

rac-server5$ . oraenv
             ORCL
rac-server5$ dbca -silent -deleteInstance \
         -gdbName ORCL                    \
         -instanceName ORCL5              \
         -sysDBAPassword xxx

You need to use the SYS password or at least to know the one from a SYSDBA user.

Manually Remove the Instance from the Node

In some situations, it’s useful to know how to manually delete an instance from a database. In that case, the steps to follow are these.

Step 1: Stop the instance you want to remove *

Use srvctl as below. In the case the server is gone, it’s very likely you don’t have to stop the instance.

srvctl stop instance -d ORCL -i ORCL5

Step 2: Remove the instance from the clusterware

Once the instance is stopped, you can delete it from the database’s list of instances by running the following command below from any one of the cluster nodes:

rac-server1$ srvctl remove instance -d ORCL -i ORCL5

Step 3: Remove the init.ora and password file *

Connect on the server you are removing and delete those two files:

rac-server5$ cd $ORACLE_HOME/dbs
rac-server5$ rm initORCL.ora
rac-server5$ rm orapwORCL

Step 4: Remove the parameters from the spfile

For the instance you’ve stopped, display the parameters that are set at the instance level with the query below on any of the remaining instances:

SQL> col name format a30
SQL> col value format a78
SQL> set lines 120
SQL> select name, value
       from v$spparameter
      where sid='ORCL5';

Reset all those parameters from the spfile as below:

SQL> alter system reset thread
         scope=spfile sid='ORCL5';
SQL> alter system reset instance_number
         scope=spfile sid='ORCL5';
SQL> alter system local_listener
         scope=spfile sid='ORCL5';
SQL> alter system undo_tablespace
         scope=spfile sid='ORCL5';

Step 5: Remove the UNDO tablespace

From Step 4, you should have figure out what the UNDO tablespace from the instance was. You can check the tablespace and drop it:

SQL> drop tablespace UNDOTBS5
          including contents and datafiles;

Step 6: Drop the redo log thread

From Step 4, you should have figured out what the instance thread was. You can disable it and drop the associated redo log groups:

SQL> alter database disable thread 5;
SQL> select group#
      from v$log
     where thread#=5;
SQL> alter database drop logfile group 13;
SQL> alter database drop logfile group 14;
SQL> alter database drop logfile group 15;

Step 7: Change the TNS aliases

The easiest way to manage the network files is to have the exact same files on each one of the servers. The entries you’d want to have in the tnsnames.ora file are:

• LISTENER_<server_name> for each one of the servers. These aliases point to the VIP end of the listener from each one of the servers and are used in the local_listener parameter of each instance.
• LISTENERS_<gDbName> is an alias that points to all the listener VIPs, and is used by the remote_listener parameter.
• <gDbName> is an alias that points to all the listener VIPs to connect to the database.
• <Instance_Name> are aliases that point to the local listener and specify the instance_name parameter to force the connection to a specific instance.

Edit the tnsnames.ora files on all the nodes and remove the VIP alias from the ORCL and LISTENERS_ORCL aliases. Also remove the ORCL5 and LISTENER_ORCL5 aliases.

Step 8: Delete the administration directories *

Locate the various administration directories for the instance you are removing, and remove them from the server:

rac-server5$ cd /u01/app/oracle/admin
rac-server5$ rm -rf ORCL

Step 9: Remove the instance prefix *

Edit the oratab file and delete the entry for the RAC database on the node you are removing.

Remove ASM from the Node

None of the assistants will give you a hand in this, but deleting an ASM instance is straightforward. It consists in (1) stopping the ASM instance*; (2) removing the ASM instance from the OCR; and (3) deleting the ASM init.ora file*. Execute the commands below:

rac-server1$ srvctl stop asm -n rac-server5
rac-server1$ su -

rac-server1# srvctl remove asm -n rac-server5
rac-server1# exit
rac-server1$ ssh rac-server5
rac-server5$ rm $ORACLE_HOME/dba/init+ASM5.ora

Remove the Listener from the Node

The only supported way to remove the listener with 10g is to use NETCA. If the server is still part of the cluster, up and running, you can just run the command below:

export DISPLAY=:1
rac-server5$ netca /silent /deinst /nodeinfo rac-server5

If you don’t have all the nodes up and running, this command will fail, and the only way to remove the listener with 10.2, even if not supported, will be to run crs_unregister as below from one of the remaining nodes. (Does anybody want to comment that practice?):

rac-server1$ cd $ORA_CRS_HOME/bin
rac-server1$ ./crs_stat |grep lsnr
rac-server1$ ./crs_unregister \
       ora.rac-server5.LISTENER_RAC-SERVER5.lsnr

Be careful! It works with the listeners but it won’t with any other Oracle Resource. If you run that command and it fails for any reason, you’ll have to restore the OCR.

Remove the Database Software

By removing the software, there are two separate things to do: (1) update the Oracle Inventory on all the nodes that remain so that you remove all the software installer links to the server; (2) remove the software from that node*. This second operation is required only if the node is to be reused.

Update the inventory of the remaining nodes

Oracle Universal Installer (OUI) can do this from any of the remaining nodes. What you’ll declare in that specific case is that only rac-server1, rac-server2, rac-server3, and rac-server4 will still be part of the clustered installation. In order to update the inventories of these nodes, run:

rac-server1$ export ORACLE_HOME=/u01/app/oracle/product/10.2.0/db_1
rac-server1$ cd $ORACLE_HOME/oui/bin
rac-server1$ ./runInstaller -silent -updateNodeList \
             ORACLE_HOME=$ORACLE_HOME               \
             "CLUSTER_NODES={rac-server1,rac-server2,rac-server3,rac-server4}"

Once you’ve updated the inventory, OUI will never again prompt you for rac-server5 when used from any of these four nodes.

Delete the Database Software from the node to be removed *

One of the limits of the OUI with RAC is that you can not use it for one node only. It’s probably a good thing in a way, as you cannot apply a Patch Set on one node only and mess up everything. Unfortunately, when it happens that you want to remove the software from one server only, you have to work around that limit. The way to do it is to update the inventory of that node only and let it think it is the only node of the cluster. Once you’ve done that, you’ll be able to execute the OUI with various syntaxes such as detachHome or deinstall. To change the inventory on the node to be removed, connect to it and run:

rac-server5$ export ORACLE_HOME=/u01/app/oracle/product/10.2.0/db_1
rac-server5$ cd $ORACLE_HOME/oui/bin
rac-server5$ ./runInstaller -silent -updateNodeList \
             ORACLE_HOME=$ORACLE_HOME               \
             "CLUSTER_NODES={rac-server5}"          \
             -local

Once the inventory updated, all the runInstaller commands you give will touch only the local ORACLE_HOME (assuming it is not shared). You can then, as described in the documentation, remove the ORACLE_HOME you want and withdraw it from the cluster:

rac-server5$ cat /etc/oraInst.loc
rac-server5$ cd /u01/app/oraInventory/ContentsXML
rac-server5$ grep NAME inventory.xml
rac-server5$ export ORACLE_HOME=/u01/app/oracle/product/10.2.0/db_1
rac-server5$ cd $ORACLE_HOME/oui/bin
rac-server5$ .runInstaller -silent -deinstall -removeallfiles \
              "REMOVE_HOMES={/u01/app/oracle/product/10.2.0/db_1}"

You can also detach the ORACLE_HOME from the inventory with the -detachHome syntax as below:

rac-server5$ cat /etc/oraInst.loc
rac-server5$ cd /u01/app/oraInventory/ContentsXML
rac-server5$ grep NAME inventory.xml
rac-server5$ export ORACLE_HOME=/u01/app/oracle/product/10.2.0/db_1
rac-server5$ cd $ORACLE_HOME/oui/bin
rac-server5$ .runInstaller -silent -detachHome                  \
              ORACLE_HOME="/u01/app/oracle/product/10.2.0/db_1" \
              ORACLE_HOME_NAME="OraDB102Home1"

This second approach allow you to keep the ORACLE_HOME and delete its content only if or when you want:

rac-server5$ rm -rf /u01/app/oracle/product/10.2.0/db_1

If that’s the last database software installed on the server, you can delete the oratab file too:

rac-server5$ rm /etc/oratab

Remove the ONS Configuration

In order to remove the ONS subscription from the server, you can first query the ons.config file as below:

rac-server5$ cd $ORA_CRS_HOME/opmn/conf
rac-server5$ grep remoteport ons.config

If you cannot access the server because it is not available anymore, you can also dump the OCR with ocrdump and look at the content of the ONS config from that file. Once you know what port has to be deleted from the configuration, remove it from the cluster registry from any of the nodes:

rac-server1$ cd $ORA_CRS_HOME/bin
rac-server1$ ./racgons remove_config rac-server5:6200

Remove the NodeApps

The nodeapps includes the GSD, the ONS, and the VIP. You can simply remove them from any of the nodes with the srvctl remove nodeapps command:

rac-server1$ srvctl stop nodeapps -n rac-server5
rac-server1$ su -
rac-server1# ./srvctl remove nodeapps -n rac-server5

You can check that the nodeapps has been removed by querying their status with srvctl or by checking the resources named ora.rac-server5 as below:

rac-server1$ cd $ORA_CRS_HOME/bin
rac-server1$ ./crs_stat |grep "ora.rac-server5"

Remove the Clusterware Software

Removing the clusterware software is very similar to removing the database software–there are two separate things to do: (1) update the Oracle Inventory on all the nodes that remain; and (2) remove the clusterware from that node*. This second operation is required only if the node is to be reused.

Update the inventory of the remaining nodes

The syntax is identical to that of the database removal, except you have to add the CRS=TRUE directive as below:

rac-server1$ export ORA_CRS_HOME=/u01/app/crs
rac-server1$ cd $ORA_CRS_HOME/oui/bin
rac-server1$ ./runInstaller -silent -updateNodeList  \
     ORACLE_HOME=$ORA_CRS_HOME                       \
     "CLUSTER_NODES={rac-server1,rac-server2,rac-server3,rac-server4}" \
     CRS=TRUE

Once you’ve updated the inventory, OUI will never again prompt you for rac-server5 when used from any of those four nodes.

Delete the Clusterware Software from the node to be removed *

To delete the clusterware from the node you want to delete, you must first stop it, if you haven’t already:

rac-server5$ su -
rac-server5# cd /u01/app/crs/bin
rac-server5# ./crsctl stop crs
rac-server5# ./crsctl disable crs

Then, update the inventory as below:

rac-server5$ export ORA_CRS_HOME=/u01/app/crs
rac-server5$ cd $ORA_CRS_HOME/oui/bin
rac-server5$ ./runInstaller -silent -updateNodeList \
             ORACLE_HOME=$ORA_CRS_HOME              \
             "CLUSTER_NODES={rac-server5}"          \
             CRS=TRUE                               \
             -local

Finally, run the OUI with the deinstall and CRS=TRUE directives as below:

rac-server5$ cat /etc/oraInst.loc
rac-server5$ cd /u01/app/oraInventory/ContentsXML
rac-server5$ grep NAME inventory.xml
rac-server5$ export ORA_CRS_HOME=/u01/app/crs
rac-server5$ cd $ORA_CRS_HOME/oui/bin
rac-server5$ .runInstaller -silent -deinstall -removeallfiles \
              "REMOVE_HOMES={/u01/app/crs}"                   \
              CRS=TRUE

Additional cleanup *

If the node you are removing is still accessible, and you plan to reuse it (say, for another cluster) there is some additional cleanup to do:

• delete any file that would remain in the Clusterware Home
• delete any specific entries in the oracle .profile file
• delete any specific entries in the crontab
• delete the oraInv.loc file and the inventory
• replace the inittab with the one backed up before the clusterware install: inittab.no_crs
• delete the /var/tmp/.oracle directory
• delete the Startup/Shutdown services, i.e., with Oracle or Redhat Enterprise Linux, all the /etc/init.d/init* and /etc/rc?.d/*init.crs files
• delete the clusterware and ocr.loc files, i.e., with Oracle or Redhat Enterprise Linux, the /etc/oracle directory
• delete the storage-specific configuration files to prevent altering the shared storage from that node (/etc/fstab for NFS, udev for ASM or the raw devices)

Remove the Node from the Cluster Configuration

Everything has been removed, but if you connect to any of the remaining nodes and run olsnodes, you’ll see the server is always registered in the OCR:

rac-server1$ cd /u01/app/crs/bin
rac-server1$ ./olsnodes -n -i
rac-server1    1    rac-server1-priv rac-server1-vip
rac-server2    2    rac-server2-priv rac-server2-vip
rac-server3    3    rac-server3-priv rac-server3-vip
rac-server4    4    rac-server4-priv rac-server4-vip
rac-server5    5    rac-server5-priv

In order to remove that server from the OCR, connect as root on any of the remaining nodes and use its name and number with the rootdeletenode.sh script as below:

rac-server1$ su -
rac-server1# cd /u01/app/crs/install
rac-server1# ./rootdeletenode.sh rac-server5,5
rac-server1# exit
rac-server1$ cd /u01/app/crs/bin
rac-server1$ olsnodes -n -i

More to come

This is it! In these three last posts, you’ve installed a 10.2 RAC, added one node, and removed one node without any X Display. Any comments for now? I hope you’ll agree that it’s pretty easy once you’re use to it. You’ll probably start (if it wasn’t the case before) to leverage RAC’s ability to scale up and down accordingly to your needs.

In the parts to follow, we’ll do the same with an 11.1 RAC.

* If you cannot access the server you are removing, don’t run this step. BACK