The best times to apply Oracle quarterly patches
# crsctl config has exec(): 0509-036 Cannot load program crsctl.bin because of the following errors: rtld: 0712-001 Symbol ztca_Shutdown was referenced from module crsctl.bin(), but a runtime definition of the symbol was not found. rtld: 0712-001 Symbol ztpk_SetKeyInfo was referenced from module /u01/11.2.0/grid/lib/libhasgen11.so(), but a runtime definition of the symbol was not found. rtld: 0712-001 Symbol ztpk_DestroyKey was referenced from module /u01/11.2.0/grid/lib/libhasgen11.so(), but a runtime definition of the symbol was not found. rtld: 0712-001 Symbol ztpk_Sign was referenced from module /u01/11.2.0/grid/lib/libhasgen11.so(), but a runtime definition of the symbol was not found. rtld: 0712-001 Symbol ztpk_Verify was referenced from module /u01/11.2.0/grid/lib/libhasgen11.so(), but a runtime definition of the symbol was not found. rtld: 0712-002 fatal error: exiting.To make matters worse, there was a breakdown in communication among Oracle's support engineers. I recall one engineer mentioned there were issues with the current patch release and, unfortunately, this was not documented in the SR. Another contribution to the failure is that the patch download was corrupted since the checksum did not match and was not checked. Maybe there are no issues after all. Fast forward. We tried patching the environment again, only to have it fail for the second time. Finally, the same Oracle support engineer who did not think there were issues with patch release has now confirmed there are issues. Lesson learned. 1. Don't be too hasty to patch. 2. Always get the name of the engineer you are speaking with and ensure the conversation is documented in SR. 3. Trust, but verify, as Oracle engineers are human too. 4. Lastly, have backup. How many backup GI and DB homes before patching? This was the saving grace. Happy patching and stayed tuned! In the next post, I will share how we managed to recover from the disaster.
On this page
Share this
Share this
More resources
Learn more about Pythian by reading the following blogs and articles.
Recovering from quarterly PSU patching disaster
Recovering from quarterly PSU patching disaster
Nov 16, 2018 12:00:00 AM
2
min read
Linux Patching and Oracle: Detect RPM conflicts before they happen
Linux Patching and Oracle: Detect RPM conflicts before they happen
Nov 15, 2011 12:00:00 AM
9
min read
Cleaning up PID files when Oracle GRID/RAC upgrades and patches fail
Cleaning up PID files when Oracle GRID/RAC upgrades and patches fail
Jan 16, 2018 12:00:00 AM
2
min read
Ready to unlock value from your data?
With Pythian, you can accomplish your data transformation goals and more.