"ORA-600 [kfrValAcd30] signaled when the expected change sequence doesn't match with the sequence we find, during the recovery of the diskgroup. This is some kind of inconsistency in the ACD block which is usually caused by a platform specific install/operational issue. It may also happen if there is any IO or storage issue . This should be further investigated by the OS/Storage vendor. Solution The only way to resolve this is by recreating the respective diskgroup. How To Drop and Recreate ASM Diskgroup DOC ID - 563048.1"The proposed solution in MOS Note 563048.1 starts off with: Erase ASM metadata information for disk using OS command: !!! YOU HAVE TO BE CAREFUL WHEN TYPING DISK NAME !!! For Unix platforms using 'dd' Unix command. Example:
dd if=/dev/zero of=/dev/raw/raw13 bs=1048576 count=50Okay, any client following this recommendation, has now officially nuked the first 50MB of data on the disk without backing up anything. If possible, it is always a good idea to backup as much of the damaged environment as possible, so if what you try fails, you can restart at the last known good point.
To be fair, earlier in the MOS note (in the Goals section, this course of action is qualified with: "Erasing the header using "dd" command is very dangerous operation and need to be done under support supervision and when it [is] confirmed by support that fixing the header is impossible. ... Backup disk header for all member disks in diskgroup"This MOS note has a purpose. It tackles the issue of dropping a disk group that is not being dropped with SQL commands. So the note is good, the advice less so for two reasons. Firstly, no attempt had been made restore data availability either with regular restore procedures or by attempting to extract any data from the disks. Dropping the disk groups as a first step will cause all data in the diskgroup to be lost. Secondly, I was not convinced that the disk groups could not be dropped without resorting to wiping the headers when the time came to drop the disk group. I ignored the "dd" advice and tested the several other options in a 12.1.0.2 two-node RAC lab environment. ASM disk groups may sometimes have problems due to bugs and more commonly due to I/O subsystem related failures. The software is generally resilient enough to fix itself, but if there is an issue with a disk group that refuses to mount, here are two options to consider.
[code language="sql"] SQL> select group_number, name, compatibility, database_compatibility from v$asm_diskgroup where name='TEST'; GROUP_NUMBER NAME COMPATIBILITY DATABASE_COMPATIBILITY ------------ ------------ ------------- ---------------------- 2 TEST 12.1.0.0.0 10.1.0.0.0 SQL> select path, redundancy from v$asm_disk where group_number=2; PATH REDUNDANCY ------------------------------------------------------------ ---------- /dev/asm-disk4 UNKNOWN [/code]
[code language="sql"] SQL> select name from v$datafile where name like '+TEST%'; NAME -------------------------------------------------------------------------------- +TEST/CDBRAC/DATAFILE/test_tbs.256.912785223[/code]
View the report produced for corruptions or any issues.
[code language="sql"]$ amdu -diskstring '/dev/asm-disk4' -dump TEST amdu_2016_05_25_14_25_22/ $ ls -lrt ./amdu_2016_05_25_14_25_22/ total 52244 -rw-r--r--. 1 oracle oinstall 4240 May 25 14:25 TEST.map -rw-r--r--. 1 oracle oinstall 53485568 May 25 14:25 TEST_0001.img -rw-r--r--. 1 oracle oinstall 2981 May 25 14:25 report.txt $ cat report.txt -*-amdu-*- ******************************* AMDU Settings ******************************** ORACLE_HOME = /u01/app/12.1.0.2/grid System name: Linux Machine: x86_64 amdu run: 25-MAY-16 14:25:22 Endianess: 1 ... ************************** SCANNING DISKGROUP TEST *************************** ---------------------------- SCANNING DISK N0001 ----------------------------- Disk N0001: '/dev/asm-disk4' ------------------------- SUMMARY FOR DISKGROUP TEST ------------------------- Allocated AU's: 53 Free AU's: 5061 AU's read for dump: 53 Block images saved: 13058 Map lines written: 53 Heartbeats seen: 0 Corrupt metadata blocks: 0 Corrupt AT blocks: 0 [/code]
In this note, the assumption is that attempts to repair and mount the diskgroup have failed. The ASM disk headers were damaged but the actual datafiles stored in the diskgroup were intact. When there are usable RMAN backups it is a simple matter to restore the missing files, recover them and go about dropping and recreating the damaged diskgroup. When backups are not available, utilities like AMDU can be used to extract datafiles without requiring diskgroups to be mounted. Notice that focus was placed on data availability and the damaged diskgroups are the last items to be considered.
Dropping and recreating faulty diskgroups should not involve concerns about data loss because by the time you get to this task, the data involved should already be available elsewhere. Backing up metadata is good practice. Consider using md_backup or query the v$asm views from the backup scripts to ensure the output is always included in the backup log file. The advice provided by the support engineer had to be weighed carefully and balanced with common sense. Why risk data loss by wiping headers and forcibly dropping disk groups without attempting to restore data availability first?
Ready to optimize your Oracle Database for the future?