A common scenario in the life of a DBA on a linux server looks something like this:
From: Sue-the-sysadamin To:the-dba Date: 2011-10-28 17:54:34 Dear DBA, We will be patching the linux systems this Friday night, October 28th at 21:00. The list of patches is attached. Let us know if there are any conflicts with Oracle. BTW, we need to know by 14:00 tomorrow. Sue
Yikes! The SysAdmin wants to apply these patches this week, and needs to know if there are any conflicts this afternoon. So you open up the list of patches. When you see the lengthy list of patches you suddenly realize the long lunch you had planned with your sweetheart is going to be rescheduled. I recently received several lists of patches that were to be applied to a number of different linux servers that were all running oracle, which led to this blog entry.
Matching Oracle shared lib dependencies to RPM files
Being faced with the task of determining which, if any, components of Oracle would possibly be affected by the list of RPMs to be upgraded across several servers with is a daunting task. One quick method to deal with this is to look for glibc in the list of RPMs. The list I received contained the following:
- glibc-2.5-58.el5_6.4.i686
- glibc-2.5-58.el5_6.4.x86_64
- glibc-common-2.5-58.el5_6.4.x86_64
- glibc-devel-2.5-58.el5_6.4.i386
- glibc-devel-2.5-58.el5_6.4.x86_64
- glibc-headers-2.5-58.el5_6.4.x86_64
Since nearly everything will be affected by an update to glibc, it would be a good idea to test these new RPMs out on the test server for that purpose before deploying them in productions. What if however you need a more exacting list of which Oracle components are affected by which packages? This information is a little harder to come by. While Oracle does supply a list of RPMs that must be installed on a Linux system, that information does not help you in determining which Oracle files are dependent on the RPMs in question. If the list of RPMs to be updated is small, it may even be that that few, if any of the Oracle files on a server may be affected. Here's how that can be determined. This may not be complete, but I think its a pretty good start.
Find the dependency files
Stepping through this manually, we can check an oracle executable file for the list of other files that is dependent on with ldd, which will display shared library dependencies.
[oracle@oravm01 patch_check]$ ldd $ORACLE_HOME/bin/oracle
libodm11.so => /u01/app/oracle/product/11.2.0/vmdb01/lib/libodm11.so (0x00002ab54b700000)
libcell11.so => /u01/app/oracle/product/11.2.0/vmdb01/lib/libcell11.so (0x00002ab54b802000)
libskgxp11.so => /u01/app/oracle/product/11.2.0/vmdb01/lib/libskgxp11.so (0x00002ab54b958000)
librt.so.1 => /lib64/librt.so.1 (0x0000003ed8c00000)
libnnz11.so => /u01/app/oracle/product/11.2.0/vmdb01/lib/libnnz11.so (0x00002ab54bb2c000)
libclsra11.so => /u01/app/oracle/product/11.2.0/vmdb01/lib/libclsra11.so (0x00002ab54bef4000)
libdbcfg11.so => /u01/app/oracle/product/11.2.0/vmdb01/lib/libdbcfg11.so (0x00002ab54c00d000)
libhasgen11.so => /u01/app/oracle/product/11.2.0/vmdb01/lib/libhasgen11.so (0x00002ab54c130000)
libskgxn2.so => /u01/app/oracle/product/11.2.0/vmdb01/lib/libskgxn2.so (0x00002ab54c66e000)
libocr11.so => /u01/app/oracle/product/11.2.0/vmdb01/lib/libocr11.so (0x00002ab54c771000)
libocrb11.so => /u01/app/oracle/product/11.2.0/vmdb01/lib/libocrb11.so (0x00002ab54c90b000)
libocrutl11.so => /u01/app/oracle/product/11.2.0/vmdb01/lib/libocrutl11.so (0x00002ab54caa5000)
libasmclnt11.so => /u01/app/oracle/product/11.2.0/vmdb01/lib/libasmclnt11.so (0x00002ab54cbb1000)
libaio.so.1 => /usr/lib64/libaio.so.1 (0x00002ab54cd33000)
libdl.so.2 => /lib64/libdl.so.2 (0x0000003ed8000000)
libm.so.6 => /lib64/libm.so.6 (0x0000003ed7c00000)
libpthread.so.0 => /lib64/libpthread.so.0 (0x0000003ed8400000)
libnsl.so.1 => /lib64/libnsl.so.1 (0x0000003edb400000)
libc.so.6 => /lib64/libc.so.6 (0x0000003ed7800000)
/lib64/ld-linux-x86-64.so.2 (0x0000003ed7400000)
Here we can see that the oracle binary is dependent on several library files. Many of them are owned by oracle, but some of them are files that are part of the Linux system.
Find the RPM
The next step is find which RPM the files are part of. We can use the rpm command for this. The -q option is for 'query', while the '-f' causes rpm to report which package a file is associated with. Choosing one of the linux owned files as an example:
[oracle@oravm01 patch_check]$ rpm -qf /lib64/libm.so.6
glibc-2.5-49
What happens if rpm -qf is run against one of the oracle owned dependency shared library files?
[oracle@oravm01 patch_check]$ rpm -qf /u01/app/oracle/product/11.2.0/vmdb01/lib/libnnz11.so
file /u01/app/oracle/product/11.2.0/vmdb01/lib/libnnz11.so is not owned by any package
As you might expect, rpm has no knowledge of oracle owned files, and reports that fact.
Does the patch affect Oracle?
The final step is to compare the current package with the patch list. In this case there is a match - glibc-2.5-49 is to be replaced with glibc-2.5.58. Now all that you need to do is repeat this process for all oracle binary files. This obviously does not work well as a manual process. Fortunately this is not too difficult a task to automate. Those readers already familiar with me will not be surprised to learn that this automation involves the use of Perl.
The Patch Check Script
Or rather, scripts. The Perl script pc.pl is complemented by the shell script pc.sh. The reason for using a driver shell script is that there may be more than one Oracle home directory on the server that you need to check. One of the few limitations of using Perl with Oracle involves the LD_LIBRARY_PATH variable. This variable cannot be set from within a Perl script, as the Perl compiler loads in the value of LD_LIBRARY_PATH before running your script, so subsequent attempts to change it are futile. There are some hacks that will work on some platforms that allow LD_LIBRARY_PATH to be changed at runtime, but I would rather not go that route. The limitations of using LD_LIBARY_PATH are pretty well known, so it is best just to work around that particular issue.
The RPM patch list
The list of RPM should bin a file, one RPM per line. The file names should not contain the .rpm extension. If the extension is included, you will need to remove it. Either that, or you will need to change the script to work with the .rpm extension. Why does the list of RPMs not include the .rpm extension? Simple, that's how I received the list. Here is a sample of files from a list of RPMs to be updated:
[oracle@oravm01 patch_check]$ head -5 patch_list.txt
amtu-1.0.6-2.el5.x86_64
audit-1.7.18-2.el5.x86_64
audit-libs-1.7.18-2.el5.i386
audit-libs-1.7.18-2.el5.x86_64
audit-libs-python-1.7.18-2.el5.x86_64
Which Files are searched?
Not all files in an Oracle home are processed. The find command is used with the -perm /u+x option to find all files that have the executable bit set for the file owner. These files are then processed to determine RPM dependencies.
The pc.pl Perl script
Let's invoke pc.pl to see what it does when run against a single Oracle home with the standard setting for LD_LIBRARY_PATH and other oracle environment variables. Here's the current environment:
[oracle@oravm01 patch_check]$ set | grep -E 'ORA|LD_LIB'
LD_LIBRARY_PATH=/u01/app/oracle/product/11.2.0/vmdb01/lib
ORACLE_BASE=/u01/app/oracle
ORACLE_HOME=/u01/app/oracle/product/11.2.0/vmdb01
ORACLE_HOSTNAME=oraexp01.jks.com
ORACLE_SID=vmdb01
ORACLE_TERM=xterm
ORAENV_ASK=NO
ORAHOME=/u01/app/oracle/product/11.2.0/vmdb01
ORA_NLS33=/u01/app/oracle/product/11.2.0/vmdb01/ocommon/nls/admin/data
Now run the script. Notice that STDERR has been redirected to a file - more on that later
[oracle@oravm01 patch_check]$ ./pc.pl -verbosity 0 -oracle_home $ORACLE_HOME -linux_patch_list patch_list.txt 2> pc.err
#### Possible Conflicts Found #####
--------------------
--- gdbm-1.8.0-26.2.1.el5_6.1.x86_64
old: 1.8.020 26.220
new: 1.8.020 26.2.1.el5_6.120
--------------------
--- glibc-2.5-58.el5_6.4.x86_64
old: 2.520 4920
new: 2.520 58.el5_6.420
--------------------
--- libgcc-4.1.2-50.el5.x86_64
old: 4.1.220 4820
new: 4.1.220 50.el520
--------------------
--- libstdc++-4.1.2-50.el5.x86_64
old: 4.1.220 4820
new: 4.1.220 50.el520
pc.pl has found 4 RPM packages that are scheduled to be updated and that Oracle is dependent on. This new information may cause the patch schedule to be pushed back until it is determined that these updates will have no adverse affects on Oracle. Now let's consider the file pc.err, where all error messages were redirected to. We see there 336 lines in the error file.
[oracle@oravm01 patch_check]$ wc -l pc.err
336 pc.err
The majority of the errors are from rpm -qf - as seen earlier the RPM database knows nothing of files owned by Oracle. As such, these are errors we are not really concerned with. There may however be errors hiding in the file that we are concerned about.
[oracle@oravm01 patch_check]$ head -7 pc.err
###########################
working on /u01/app/oracle/product/11.2.0/vmdb01/jdk/jre/lib/amd64/libfontmanager_g.so dependencies
libawt_g.so => not found
###########################
working on /u01/app/oracle/product/11.2.0/vmdb01/jdk/jre/lib/amd64/libfontmanager_g.so dependencies
libjava_g.so => not found
###########################
Following is another error - this one is not an rpm error, so it may be something to be concerned about.
##### ldd error ############
error encountered with /usr/bin/ldd /u01/app/oracle/product/11.2.0/vmdb01/jdk/jre/lib/amd64/libmanagement_g.so
run the command manually to see the error
###########################
Running the command manually results in a segmentation fault.
[oracle@oravm01 patch_check]$ ldd /u01/app/oracle/product/11.2.0/vmdb01/jdk/jre/lib/amd64/libmanagement_g.so
/usr/bin/ldd: line 116: 5216 Segmentation fault LD_TRACE_LOADED_OBJECTS=1 LD_WARN= LD_BIND_NOW= LD_LIBRARY_VERSION=$verify_out LD_VERBOSE= "$@"
A quick google search on 'ldd segmentation fault' suggests this is a linux bug. We will not be investigating this further, as we aren't too concerned with that at this time. With the large number of errors that we do not really care about being reported by the rpm -qf command, how can we reduce the number of errors?
The pc.sh shell script
The pc.sh script serves two purposes. It reads the file oracle_homes.txt and runs pc.pl for each ORACLE_HOME. The pc.sh script also sets the LD_LIBARY_PATH variable to eliminate many of the false positives seen in pc.err previously. Following is the entire pc.sh script. You may want to adjust the values used for LD_LIBARY_PATH to suit your environment.
#!/bin/sh
# set the environment to call pc.pl. (patch check) # the LD_LIBRARY_PATH
OHOMES_FILE=oracle_homes.txt PATCH_LIST_FILE=patch_list.txt
[ -r "$OHOMES_FILE" ] || { echo The file $OHOMES_FILE is missing exit 1 }
while read ohome do sid=$(basename $ohome) LOGFILE=patch_check_${sid}.log ERRFILE=patch_check_${sid}.err
echo $ohome
# you may have to experiment a bit to get the needed paths in LD_LIBRARY_PATH # in 11g there will probably still be a few files that need to be checked manually export LD_LIBRARY_PATH=$ohome/lib:$ohome/jdk/jre/lib/amd64:$ohome/jdk/jre/lib/amd64/server
./pc.pl -verbosity 0 -oracle_home $ohome -linux_patch_list $PATCH_LIST_FILE >$LOGFILE 2>$ERRFILE
done < $OHOMES_FILE Lets run the script and then check the error logs. [oracle@oravm01 patch_check]$ ./pc.sh /u01/app/oracle/product/11.2.0/grid /u01/app/oracle/product/11.2.0/vmdb01 Checking on the output files shows that the error logs are much smaller than what was seen previously. [oracle@oravm01 patch_check]$ ls -ltar *.log *.err -rw-r--r-- 1 oracle oinstall 455 Nov 14 13:18 patch_check_grid.err -rw-r--r-- 1 oracle oinstall 34384 Nov 14 13:18 patch_check_grid.log -rw-r--r-- 1 oracle oinstall 296 Nov 14 13:18 patch_check_vmdb01.err -rw-r--r-- 1 oracle oinstall 34384 Nov 14 13:18 patch_check_vmdb01.log [oracle@oravm01 patch_check]$ wc -l patch*.err 9 patch_check_grid.err 6 patch_check_vmdb01.err 15 total By including some more library directories in LD_LIBRARY_PATH, rpm -qf was able to resolve many more dependencies, greatly reducing the false positives that were included in the error files. Here are the total contents of the error files:
[oracle@oravm01 patch_check]$ cat *.err Use of uninitialized value in hash element at ./pc.pl line 393. Use of uninitialized value in hash element at ./pc.pl line 393. ########################### working on /u01/app/oracle/product/11.2.0/grid/jdk/jre/lib/amd64/libjawt_g.so dependencies libmawt_g.so => not found ########################### working on /u01/app/oracle/product/11.2.0/grid/jdk/jre/lib/amd64/libjawt.so dependencies libmawt.so => not found rpmq: no arguments given for query ########################### working on /u01/app/oracle/product/11.2.0/vmdb01/jdk/jre/lib/amd64/libjawt_g.so dependencies libmawt_g.so => not found ########################### working on /u01/app/oracle/product/11.2.0/vmdb01/jdk/jre/lib/amd64/libjawt.so dependencies libmawt.so => not found
The 'uninitialized value' errors are from Perl - this is likely a bug in the script. Looking at the script I can see that while it is a script in the bug, it is not causing any incorrect reporting.
[oracle@oravm01 patch_check]$ grep -n . pc.pl | grep -3 393
389: ($dummy1,$dummy2,$file) = split(/\s+/,$line);
390: }
392: #print " ###### FILE: $file\n";
393: push @{$self->{olibDeps}->{$file}{dependents}}, $exe;
394: }
395: }
397: # all library dependencies now in %olibDepend Line 393 should be executed only if there is a value defined for $file. Changing it as follows this fixed the bug:
push @{$self->{olibDeps}->{$file}{dependents}}, $exe if $file; The small number of lines now found in the .err files makes it simple to determine if these errors are an issue to be concerned with.
Download the scripts
The zip file containing all scripts can be downloaded here -
Patch Check Repo on Github
No Comments Yet
Let us know what you think