Running RAC and ASM on Linux (Finally!)

Sep 19, 2007 / By Grégory Guillou

Tags:

After weeks of not having the time I wanted for this, it’s finally done! Today, I installed RAC on Oracle Enterprise Linux 5 (OEL5), and I can tell you that there’s nothing exceptional about the process.

The only trouble I encountered had nothing directly to do with the installation. In order for the device rights to be assigned when RHEL5 or OEL5 start up, you have to create a file in /etc/udev/rules.d, but I have already spoken about that in my last post on the subject of raw devices.

Besides that, it’s all quite simple once 10g’s prerequisites are met. After an hour and a half, it was all wrapped up, at least for two nodes. I didn’t see anything revolutionary differences, but nonetheless there are some points worth mentioning.

1. There’s no need to launch VIPCA manually if your public network uses reserved IP addresses. Nonetheless, it’s quite alright to do so, because beneath 80% of examples, Oracle networks are in the 192.168.x.x or 10.x.x.x ranges. Oracle has also fixed the Cluster Verify Utility, which now simply returns a warning instead of an error when this happens:

Interfaces found on subnet "192.168.245.0" that are likely
candidates for a private interconnect:
sacha eth1:192.168.245.2

WARNING:
Could not find a suitable set of interfaces for VIPs.

On the other hand, note that you must have a ping-able GATEWAY at your public address and that the CVU doesn’t seem to validate it. If you don’t have one, the penalty is that your VIP won’t start.

2. No more RAW DEVICES are needed. As I have mentioned before, OCR and Voting Disks can be directly in the partitions (in fact, that’s even the recommended configuration now). It’s the same for ASM disks — for example, you can just change the parameter indicating the path under which ASM will discover disks (asm_diskstring) to /dev/sdb*, and with DBCA, you can directly adjust your block devices without ASM.

3. There is a command for verifying the state of all the cluster’s nodes.

./crsctl check cluster
sacha  ONLINE
kilian ONLINE

4. There is a command for making a manual backup of OCR. It’s practical to do so before adding a node to the cluster.

./ocrconfig -manualbackup

sacha   2007/09/16 08:09:13
/u01/crs/cdata/arkz/backup_20070916_080913.ocr

./ocrconfig -showbackup manual

sacha   2007/09/16 08:09:13
/u01/crs/cdata/arkz/backup_20070916_080913.ocr

5. OPROCD replaces hangcheck-timer. I’ve written about this before, and verified it.

ps -ef |grep oproc
root 5580 4625  0 07:38 ? 00:00:00 /bin/sh/etc/init.d/init.cssd oprocd
root 6142 5580  0 07:38 ? 00:00:00 /u01/crs/bin/oprocd run -t 1000 -m 500

lsmod |grep hangcheck

6. ASMCMD now has a cp command. This makes it possible to do things such as the working examples below.

. oraenv
ORACLE_SID = [oracle] ? +ASM1
The Oracle base for ORACLE_HOME=/u01/app/oracle/product/11.1.0/db_1 is /u01/app/oracle

asmcmd
ASMCMD> cd DATA/ORCL/CONTROLFILE

ASMCMD> pwd
+DATA/ORCL/CONTROLFILE

ASMCMD> cp Current.260.633428899 +DATA/ORCL/CONTROLFILE/gark.ctl
source +DATA/ORCL/CONTROLFILE/Current.260.633428899
target +DATA/ORCL/CONTROLFILE/Gark.ctl
copying file(s)...
file, +DATA/orcl/controlfile/gark.ctl, copy committed.

ASMCMD> ls
Current.260.633428899
gark.ctl

7. The lsdsk command allows you to see disks associated with ASM. For example, with the -d option, you can see which disks belong to a DiskGroup.

 oraenv
ORACLE_SID = [oracle] ? +ASM1
The Oracle base for ORACLE_HOME=/u01/app/oracle/product/11.1.0/db_1 is /u01/app/oracle

asmcmd
ASMCMD> lsdsk -k -d DATA
Total_MB  Free_MB  OS_MB  Name       Failgroup  Path
   1529      864   1529  DATA_0000  DATA_0000  /dev/sdb5
   2337     1306   2337  DATA_0001  DATA_0001  /dev/sdb7

A few remarks:

  • It’s possible to launch ASMCMD when ASM is stopped. The message that follows, ASMCMD-08103: failed to connect to ASM; ASMCMD running in non-connected mode, indicates that you will work directly on the heads of the disks.
  • For reasons I imagine are linked to the fact that in my case, the asm_diskstring parameter does not have the default value ("/dev/sdb*"), lsdsk seems not to work when in non-connected mode.
  • ASMCMD’s new commands look very interesting, particularly md_backup, md_restore and remap. Okay, enough ASM stuff for now.

8. You can both add and remove Voting Disks in a working cluster. Let’s start with OCR, which seems to work already. As the root user, you can add and remove an OCR Mirror while the clusterware is working as follows:

cd /u01/crs/bin

./ocrcheck
Status of Oracle Cluster Registry is as follows :
  Version                  :          2
  Total space (kbytes)     :     248724
  Used space (kbytes)      :       2252
  Available space (kbytes) :     246472
  ID                       :  433771587
  Device/File Name         :  /dev/sdb8
                             Device/File integrity check succeeded

                             Device/File not configured

  Cluster registry integrity check succeeded

./ocrconfig -replace ocrmirror /dev/sdb9

./ocrcheck
Status of Oracle Cluster Registry is as follows :
  Version                  :          2
  Total space (kbytes)     :     248724
  Used space (kbytes)      :       2252
  Available space (kbytes) :     246472
  ID                       :  433771587
  Device/File Name         :  /dev/sdb8
                             Device/File integrity check succeeded
  Device/File Name         :  /dev/sdb9
                             Device/File integrity check succeeded

  Cluster registry integrity check succeeded

./ocrconfig -replace ocrmirror

./ocrcheck
Status of Oracle Cluster Registry is as follows :
     Version                  :          2
     Total space (kbytes)     :     248724
     Used space (kbytes)      :       2252
     Available space (kbytes) :     246472
     ID                       :  433771587
     Device/File Name         :  /dev/sdb8
                                Device/File integrity check succeeded

                                Device/File not configured

     Cluster registry integrity check succeeded

You can do the same thing with Voting Disk while the clusterware is as:

dd if=/dev/sdb10 of=/u01/crs/cdata/arkz/voting.copy bs=4k
62244+0 records in
62244+0 records out
254951424 bytes (255 MB) copied, 10.0618 seconds, 24.2 MB/s

./crsctl query css votedisk
0.     0    /dev/sdb10
Located 1 voting disk(s).

./crsctl add css votedisk /dev/sdb11
Successful addition of voting disk /dev/sdb11.

./crsctl add css votedisk /dev/sdb12
Successful addition of voting disk /dev/sdb12.

./crsctl query css votedisk
0.     0    /dev/sdb10
1.     0    /dev/sdb11
2.     0    /dev/sdb12
Located 3 voting disk(s).

# At first glance, it's looks like it's not possible to remove Voting Disk 1:
./crsctl delete css votedisk /dev/sdb10
Failure 8 with Cluster Synchronization Services while deleting voting disk.

# Too bad! But we can easily remove the other Voting Disks 
# and logically, a problem with Voting Disk 1 would not
# affect the cluster.
# Too bad again!  I can't test this with my configuration.
./crsctl delete css votedisk /dev/sdb11
Successful deletion of voting disk /dev/sdb11.

./crsctl delete css votedisk /dev/sdb12
Successful deletion of voting disk /dev/sdb12.

./crsctl query css votedisk
0.     0    /dev/sdb10
Located 1 voting disk(s).

9. You can kill sessions cluster-wide.

# Session 1
sqlplus / as sysdba

grant select
on gv_$session
to scott;

# Session 2
sqlplus scott/tiger

select sid, serial#, inst_id
from gv$session
where audsid=sys_context('USERENV','SESSIONID');

SID    SERIAL#    INST_ID
--- ---------- ----------
67        610          1

# Session 1
alter system
kill session '67, 610, @1'
immediate;

# Session 2
/
select sid, serial#, inst_id
*
ERROR at line 1:
ORA-03135: connection lost contact
Process ID: 29700
Session ID: 67 Serial number: 610

10. You can launch an AWR script for the whole RAC instead of instance-by-instance. Use spawrrac.sql as follows:

@?/rdbms/admin/spawrrac

Instances in this AWR schema
~~~~~~~~~~~~~~~~~~~~~~~~~~~~

                        Instance
  DB Id    DB Name         Count
----------- ------------ --------
1161209635 ORCL                2

Enter value for dbid: 1161209635
Using 1161209635 for database Id

Specify the number of days of snapshots to choose from
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Entering the number of days (n) will result in the most recent
(n) days of snapshots being listed.  Pressing  without
specifying a number lists all completed snapshots.

Listing the last 31 days of Completed Snapshots

                                         Snap Instance
DB Name        Snap Id End Interval Time Level    Count
------------ --------- ----------------- ----- --------
ORCL                 1 16 Sep 2007 10:00     1        2
                    2 16 Sep 2007 11:00     1        2
                    3 16 Sep 2007 12:00     1        2

Specify the Begin and End Snapshot Ids
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Enter value for begin_snap: 1
Begin Snapshot Id specified: 1

Enter value for end_snap: 2
End   Snapshot Id specified: 2

Specify the Report Name
~~~~~~~~~~~~~~~~~~~~~~~
The default report file name is spawrrac_1_2.  To use this name,
press  to continue, otherwise enter an alternative.

Enter value for report_name:

[...]

End of Report ( spawrrac_1_2.lst )

10 Responses to “Running RAC and ASM on Linux (Finally!)”

  • Gawie says:

    Hi Gregory,

    Could you point me to the previous posts regarding etc/udev/rules.d and the fact that no RAW devices are needed anymore.

    Thank you for the information in this post
    Gawie

  • Grégory says:

    Gawie,

    Lets start with the bad news : My original post about raw devices is in French !

    What it says is that with RHEL5/OEL5, the /etc/udev/permissions.d directory doesn’t exist anymore and you have to create a file under /etc/udev/rules.d to set the permissions on boot ; you should be able to get the content of the file from the post even if it’s in French !

    Regarding Oracle recommendation about “raw vs block” devices you’ll find a couple of informations in different 11g docs :

    1- (About 10g first) This section explains that starting with clusterware 10.2 and RHEL4, you can use block (instead of raw) devices to store the OCR and the voting disk.

    2- (11g) In this section, the doc says that Oracle recommends using block rather that raw devices on Linux. BTW, this section explains how to configure block devices while storing one file per block device (As you use to do with raw devices).

    3- (10g and 11g) This section explains that Oracle recommends using block devices with ASMLib to create ASM disks. This OTN page contains the supported release of ASMLib, including the one for RHEL4 and RHEL5. Well (Shame on me !) this is not what I did ! I’ve changed the asm_diskstring parameter and I’ve bound the ASM DISKS to the block devices. I probably shouldn’t ! I’m pretty sure the problem I’ve encountered with lsdsk will be solved by doing the setup this way : I’ll check soon !

    Anyway 1 and 3 show that you don’t need the raw devices anymore (even with 10g and RHEL4). But it doesn’t prevent you to set the permission via the “udev” dynamic device naming.

  • [...] no permissions.d directory on RHEL5 and OEL5. Last September, Grégory Guillou from Pythian blogged about installing on Red Hat 5 and referenced a post on his French blog that shows how to setup a RULES file for SCSI block [...]

  • sorry, i don’t understand one thing: can i use directly /dev/sdb* for asm disks with 10gR2? it seems to me that also in 11g documentation Oracle says to use ASMLib but i can’t find ASMLib very useful. Morehover i’ve met problem described in Metalink Note 457369.1 that suggest to use asm_diskstring = /dev/oracleasm/disks/* but devices files under /dev/oracleasm/disks/* have same major e minor numbers of original block devices (/dev/sdb*) so i think that is the same think. So my question is: does oracle support the use of asm_diskstring=/dev/sdb* ?

    Compliments for the post.

    Thanks,
    Cristian

  • Grégory says:

    the answer with Linux is :
    a- prefer using ASMLib. In particular is enables to discover the disks on all nodes without partprobe and it is optimized for ASM.
    b- If you cannot, use the block device, i.e /dev/sdb* and NOT raw devices

  • sdquirra says:

    hi,
    using /dev/sd* devices for ocr and voting disk, i’m getting a “the location /dev/sdb1, entered for the oracle cluster registry (ocr) is not shared across all the nodes in the cluster….”.
    This post also http://forums.oracle.com/forums/thread.jspa?threadID=641837&tstart=0&messageID=2474588 says that, for ocr and voting disk, 10g keep using raw dev or cluster fs.
    It is correct?
    thnx

    • kamlesh says:

      you will get the error “the location /dev/sdb1, entered for the oracle cluster registry (ocr) is not shared across all the nodes in the cluster….”. when you have not shared the device /dev/sdb1 for both nodes. Please check its physical location it should be shared in shared path.

  • Ritesh Singh says:

    Great Article .. Keep it up

  • goran says:

    related to your statement that oprocd replaces hangcheck-timer … afaik and according to oracle doc this is not true both for 10g and 11g.

    Can you pointed me to the link where this is stated?

    thanks,
    goran

  • balaji says:

    Hi, i was getting an error “ocr location is inaccessible” while running root.sh. This is what i did
    a. created raw devices, and formated it with fdisk(as specified in one rac installation doc’ but actually i am not sure why they are asking us to partition the disk as it will become ext3 partitioned and not raw devices)
    b. granted the required permisssions ocr root:oinstall and voting disk oracle:oinstall
    c. but not able to access ocr, should i use block devices else format the ocr and voting disk using ocfs. your advice is greatly appreciated

Leave a Reply

  • (will not be published)

XHTML: You can use these tags: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>