From: Sue-the-sysadamin To:the-dba Date: 2011-10-28 17:54:34 Dear DBA, We will be patching the linux systems this Friday night, October 28th at 21:00. The list of patches is attached. Let us know if there are any conflicts with Oracle. BTW, we need to know by 14:00 tomorrow. SueYikes! The SysAdmin wants to apply these patches this week, and needs to know if there are any conflicts this afternoon. So you open up the list of patches. When you see the lengthy list of patches you suddenly realize the long lunch you had planned with your sweetheart is going to be rescheduled. I recently received several lists of patches that were to be applied to a number of different linux servers that were all running oracle, which led to this blog entry.
[oracle@oravm01 patch_check]$ ldd $ORACLE_HOME/bin/oracle libodm11.so => /u01/app/oracle/product/11.2.0/vmdb01/lib/libodm11.so (0x00002ab54b700000) libcell11.so => /u01/app/oracle/product/11.2.0/vmdb01/lib/libcell11.so (0x00002ab54b802000) libskgxp11.so => /u01/app/oracle/product/11.2.0/vmdb01/lib/libskgxp11.so (0x00002ab54b958000) librt.so.1 => /lib64/librt.so.1 (0x0000003ed8c00000) libnnz11.so => /u01/app/oracle/product/11.2.0/vmdb01/lib/libnnz11.so (0x00002ab54bb2c000) libclsra11.so => /u01/app/oracle/product/11.2.0/vmdb01/lib/libclsra11.so (0x00002ab54bef4000) libdbcfg11.so => /u01/app/oracle/product/11.2.0/vmdb01/lib/libdbcfg11.so (0x00002ab54c00d000) libhasgen11.so => /u01/app/oracle/product/11.2.0/vmdb01/lib/libhasgen11.so (0x00002ab54c130000) libskgxn2.so => /u01/app/oracle/product/11.2.0/vmdb01/lib/libskgxn2.so (0x00002ab54c66e000) libocr11.so => /u01/app/oracle/product/11.2.0/vmdb01/lib/libocr11.so (0x00002ab54c771000) libocrb11.so => /u01/app/oracle/product/11.2.0/vmdb01/lib/libocrb11.so (0x00002ab54c90b000) libocrutl11.so => /u01/app/oracle/product/11.2.0/vmdb01/lib/libocrutl11.so (0x00002ab54caa5000) libasmclnt11.so => /u01/app/oracle/product/11.2.0/vmdb01/lib/libasmclnt11.so (0x00002ab54cbb1000) libaio.so.1 => /usr/lib64/libaio.so.1 (0x00002ab54cd33000) libdl.so.2 => /lib64/libdl.so.2 (0x0000003ed8000000) libm.so.6 => /lib64/libm.so.6 (0x0000003ed7c00000) libpthread.so.0 => /lib64/libpthread.so.0 (0x0000003ed8400000) libnsl.so.1 => /lib64/libnsl.so.1 (0x0000003edb400000) libc.so.6 => /lib64/libc.so.6 (0x0000003ed7800000) /lib64/ld-linux-x86-64.so.2 (0x0000003ed7400000)Here we can see that the oracle binary is dependent on several library files. Many of them are owned by oracle, but some of them are files that are part of the Linux system.
[oracle@oravm01 patch_check]$ rpm -qf /lib64/libm.so.6 glibc-2.5-49What happens if rpm -qf is run against one of the oracle owned dependency shared library files?
[oracle@oravm01 patch_check]$ rpm -qf /u01/app/oracle/product/11.2.0/vmdb01/lib/libnnz11.so file /u01/app/oracle/product/11.2.0/vmdb01/lib/libnnz11.so is not owned by any packageAs you might expect, rpm has no knowledge of oracle owned files, and reports that fact.
[oracle@oravm01 patch_check]$ head -5 patch_list.txt amtu-1.0.6-2.el5.x86_64 audit-1.7.18-2.el5.x86_64 audit-libs-1.7.18-2.el5.i386 audit-libs-1.7.18-2.el5.x86_64 audit-libs-python-1.7.18-2.el5.x86_64
[oracle@oravm01 patch_check]$ set | grep -E 'ORA|LD_LIB' LD_LIBRARY_PATH=/u01/app/oracle/product/11.2.0/vmdb01/lib ORACLE_BASE=/u01/app/oracle ORACLE_HOME=/u01/app/oracle/product/11.2.0/vmdb01 ORACLE_HOSTNAME=oraexp01.jks.com ORACLE_SID=vmdb01 ORACLE_TERM=xterm ORAENV_ASK=NO ORAHOME=/u01/app/oracle/product/11.2.0/vmdb01 ORA_NLS33=/u01/app/oracle/product/11.2.0/vmdb01/ocommon/nls/admin/dataNow run the script. Notice that STDERR has been redirected to a file - more on that later
[oracle@oravm01 patch_check]$ ./pc.pl -verbosity 0 -oracle_home $ORACLE_HOME -linux_patch_list patch_list.txt 2> pc.err #### Possible Conflicts Found ##### -------------------- --- gdbm-1.8.0-26.2.1.el5_6.1.x86_64 old: 1.8.020 26.220 new: 1.8.020 26.2.1.el5_6.120 -------------------- --- glibc-2.5-58.el5_6.4.x86_64 old: 2.520 4920 new: 2.520 58.el5_6.420 -------------------- --- libgcc-4.1.2-50.el5.x86_64 old: 4.1.220 4820 new: 4.1.220 50.el520 -------------------- --- libstdc++-4.1.2-50.el5.x86_64 old: 4.1.220 4820 new: 4.1.220 50.el520pc.pl has found 4 RPM packages that are scheduled to be updated and that Oracle is dependent on. This new information may cause the patch schedule to be pushed back until it is determined that these updates will have no adverse affects on Oracle. Now let's consider the file pc.err, where all error messages were redirected to. We see there 336 lines in the error file.
[oracle@oravm01 patch_check]$ wc -l pc.err 336 pc.errThe majority of the errors are from rpm -qf - as seen earlier the RPM database knows nothing of files owned by Oracle. As such, these are errors we are not really concerned with. There may however be errors hiding in the file that we are concerned about.
[oracle@oravm01 patch_check]$ head -7 pc.err ########################### working on /u01/app/oracle/product/11.2.0/vmdb01/jdk/jre/lib/amd64/libfontmanager_g.so dependencies libawt_g.so => not found ########################### working on /u01/app/oracle/product/11.2.0/vmdb01/jdk/jre/lib/amd64/libfontmanager_g.so dependencies libjava_g.so => not found ###########################Following is another error - this one is not an rpm error, so it may be something to be concerned about.
##### ldd error ############ error encountered with /usr/bin/ldd /u01/app/oracle/product/11.2.0/vmdb01/jdk/jre/lib/amd64/libmanagement_g.so run the command manually to see the error ###########################Running the command manually results in a segmentation fault.
[oracle@oravm01 patch_check]$ ldd /u01/app/oracle/product/11.2.0/vmdb01/jdk/jre/lib/amd64/libmanagement_g.so /usr/bin/ldd: line 116: 5216 Segmentation fault LD_TRACE_LOADED_OBJECTS=1 LD_WARN= LD_BIND_NOW= LD_LIBRARY_VERSION=$verify_out LD_VERBOSE= "$@"A quick google search on 'ldd segmentation fault' suggests this is a linux bug. We will not be investigating this further, as we aren't too concerned with that at this time. With the large number of errors that we do not really care about being reported by the rpm -qf command, how can we reduce the number of errors?
#!/bin/sh # set the environment to call pc.pl. (patch check) # the LD_LIBRARY_PATH OHOMES_FILE=oracle_homes.txt PATCH_LIST_FILE=patch_list.txt [ -r "$OHOMES_FILE" ] || { echo The file $OHOMES_FILE is missing exit 1 } while read ohome do sid=$(basename $ohome) LOGFILE=patch_check_${sid}.log ERRFILE=patch_check_${sid}.err echo $ohome # you may have to experiment a bit to get the needed paths in LD_LIBRARY_PATH # in 11g there will probably still be a few files that need to be checked manually export LD_LIBRARY_PATH=$ohome/lib:$ohome/jdk/jre/lib/amd64:$ohome/jdk/jre/lib/amd64/server ./pc.pl -verbosity 0 -oracle_home $ohome -linux_patch_list $PATCH_LIST_FILE >$LOGFILE 2>$ERRFILE done < $OHOMES_FILE Lets run the script and then check the error logs. [oracle@oravm01 patch_check]$ ./pc.sh /u01/app/oracle/product/11.2.0/grid /u01/app/oracle/product/11.2.0/vmdb01 Checking on the output files shows that the error logs are much smaller than what was seen previously. [oracle@oravm01 patch_check]$ ls -ltar *.log *.err -rw-r--r-- 1 oracle oinstall 455 Nov 14 13:18 patch_check_grid.err -rw-r--r-- 1 oracle oinstall 34384 Nov 14 13:18 patch_check_grid.log -rw-r--r-- 1 oracle oinstall 296 Nov 14 13:18 patch_check_vmdb01.err -rw-r--r-- 1 oracle oinstall 34384 Nov 14 13:18 patch_check_vmdb01.log [oracle@oravm01 patch_check]$ wc -l patch*.err 9 patch_check_grid.err 6 patch_check_vmdb01.err 15 total By including some more library directories in LD_LIBRARY_PATH, rpm -qf was able to resolve many more dependencies, greatly reducing the false positives that were included in the error files. Here are the total contents of the error files:
[oracle@oravm01 patch_check]$ cat *.err Use of uninitialized value in hash element at ./pc.pl line 393. Use of uninitialized value in hash element at ./pc.pl line 393. ########################### working on /u01/app/oracle/product/11.2.0/grid/jdk/jre/lib/amd64/libjawt_g.so dependencies libmawt_g.so => not found ########################### working on /u01/app/oracle/product/11.2.0/grid/jdk/jre/lib/amd64/libjawt.so dependencies libmawt.so => not found rpmq: no arguments given for query ########################### working on /u01/app/oracle/product/11.2.0/vmdb01/jdk/jre/lib/amd64/libjawt_g.so dependencies libmawt_g.so => not found ########################### working on /u01/app/oracle/product/11.2.0/vmdb01/jdk/jre/lib/amd64/libjawt.so dependencies libmawt.so => not found
The 'uninitialized value' errors are from Perl - this is likely a bug in the script. Looking at the script I can see that while it is a script in the bug, it is not causing any incorrect reporting.[oracle@oravm01 patch_check]$ grep -n . pc.pl | grep -3 393
389: ($dummy1,$dummy2,$file) = split(/\s+/,$line);
390: }
392: #print " ###### FILE: $file\n";
393: push @{$self->{olibDeps}->{$file}{dependents}}, $exe;
394: }
395: }
397: # all library dependencies now in %olibDepend
Line 393 should be executed only if there is a value defined for $file. Changing it as follows this fixed the bug:
push @{$self->{olibDeps}->{$file}{dependents}}, $exe if $file;
The small number of lines now found in the .err files makes it simple to determine if these errors are an issue to be concerned with.
The zip file containing all scripts can be downloaded here - Patch Check Repo on Github
Ready to optimize your Oracle Database for the future?