The other day we had a situation with an OEM 12cR2. We performed maintenance in a 2-node cluster and both instances came up fine, but one of the instances in OEM's Cluster Database target was showing a pending status. It goes to say that before the maintenance, both showed as up.
So the first thing we did was to bounce the agent and clear its state.
oracle@testhost [oemagent] /u01/app/oracle/agent/agent12c/bin
oracle $ ./emctl stop agent;./emctl clearstate agent;./emctl start agent;./emctl upload agent
But with our luck, that neither helped nor changed anything. The next thing we did was to check if the agent was communicating with the OMS. We checked from the server where the agent resided to see if it was responding.
oracle@testhost [oemagent] /u01/app/oracle/agent/agent12c/bin
oracle $ emctl verifykey
Oracle Enterprise Manager Cloud Control 12c Release 2
Copyright (c) 1996, 2012 Oracle Corporation. All rights reserved.
-----------------------------------------------------
verifykey: Successfully Completed communication with agent
EMD pingOMS completed successfully
-----------------------------------------------------
We concluded that they were communicating, so that was another theory to discard...
Then, we tried something that used to work in 11g: move the upload, state directories, and then bounce the agent, clearing the state.
oracle@testhost[oemagent] /u01/app/oracle/agent/agent12c/sysman/emd
oracle $ mv upload upload.27Aug2013
oracle@testhost [oemagent] /u01/app/oracle/agent/agent12c/sysman/emd
oracle $ mkdir upload
oracle@testhost [oemagent] /u01/app/oracle/agent/agent12c/sysman/emd
oracle $ mv state state.27Aug2013
oracle@testhost [oemagent] /u01/app/oracle/agent/agent12c/sysman/emd
oracle $ mkdir state
oracle@testhost [oemagent] /u01/app/oracle/agent/agent12c/bin
oracle $ cd /u01/app/oracle/agent/agent12c/bin
oracle@testhost [oemagent] /u01/app/oracle/agent/agent12c/bin
oracle $ ./emctl stop agent;./emctl clearstate agent;./emctl start agent;./emctl upload agent
But as you might have guessed by now, this didn't help at all. So the next thing we verified — and this is where we hit the jackpot — were the targets that the agent had registered. Here we saw that the Cluster Database wasn't listed.
oracle@test[oemagent] /u01/app/oracle/agent/agent12c/bin
oracle $ emctl config agent listtargets | grep "CLUSTER_TEST" | wc -l
0
OEM12c has a great feature that allows you to resynch the agent via OEM. Here are the steps:
Once you do that, you will see an output like below:
resyncState: IN executeCommand
resyncState: validated parameters
Starting resync RESYNC_20130827141300 for agent testhost:9999
Getting list of all the Targets to remove from the the agent - testhost:9999
Removing 27 targets from the agent
Removing list of plugins from the agent
Getting list of all the Targets from the repository for the agent - testhost:9999
Pushing list of plugins to the agent
Promoting list of targets to the agent
Re-deploying Metric Extensions to the targets
Saving target collections to the agent
Cleaning state on the agent
size of repos blackout list is 0
Retrieving java callbacks from em_gcha_callbacks table
Pdp settings syncronized successfully
resyncState: resync of agent succeeded!
And voilà!!! Once we did this, we could see the CLUSTER_TEST in our target agent.
oracle@testhost [oemagent] /u01/app/oracle/agent/agent12c/bin
oracle $ emctl config agent listtargets | grep "CLUSTER_TEST"
[CLUSTER_TEST_test1, oracle_database]
[CLUSTER_TEST, rac_database]
When we had confirmed that the target that we wanted was present, we did an upload and verified the status of the agent. To our great pleasure, both targets were visible as up and running in OEM! :)
oracle@testhost [oemagent] /u01/app/oracle/agent/agent12c/bin
oracle $ ./emctl upload agent; ./emctl status agent
I hope this can help you out when you face something like this! I can guarantee that the next time I'm faced with a similar situation, the first thing that I will do is grab a target list from the agent.
Note: The names and ports were changed to maintain anonymity.
Ready to optimize your Oracle Database for the future?