How to Fix a Target with a Pending Status in OEM 12cR2

Posted in: Technical Track

The other day we had a situation with an OEM 12cR2. We performed maintenance in a 2-node cluster and both instances came up fine, but one of the instances in OEM’s Cluster Database target was showing a pending status. It goes to say that before the maintenance, both showed as up.

So the first thing we did was to bounce the agent and clear its state.

oracle@testhost [oemagent] /u01/app/oracle/agent/agent12c/bin
 oracle $ ./emctl stop agent;./emctl clearstate agent;./emctl start agent;./emctl upload agent

But with our luck, that neither helped nor changed anything. The next thing we did was to check if the agent was communicating with the OMS. We checked from the server where the agent resided to see if it was responding.

oracle@testhost [oemagent] /u01/app/oracle/agent/agent12c/bin
oracle $ emctl verifykey
Oracle Enterprise Manager Cloud Control 12c Release 2
Copyright (c) 1996, 2012 Oracle Corporation. All rights reserved.

-----------------------------------------------------
verifykey: Successfully Completed communication with agent
EMD pingOMS completed successfully

-----------------------------------------------------

We concluded that they were communicating, so that was another theory to discard… Then, we tried something that used to work in 11g: move the upload, state directories, and then bounce the agent, clearing the state.

oracle@testhost[oemagent] /u01/app/oracle/agent/agent12c/sysman/emd
oracle $ mv upload upload.27Aug2013

oracle@testhost [oemagent] /u01/app/oracle/agent/agent12c/sysman/emd
oracle $ mkdir upload

oracle@testhost [oemagent] /u01/app/oracle/agent/agent12c/sysman/emd
oracle $ mv state state.27Aug2013

oracle@testhost [oemagent] /u01/app/oracle/agent/agent12c/sysman/emd
oracle $ mkdir state

oracle@testhost [oemagent] /u01/app/oracle/agent/agent12c/bin
oracle $ cd /u01/app/oracle/agent/agent12c/bin
oracle@testhost [oemagent] /u01/app/oracle/agent/agent12c/bin
oracle $ ./emctl stop agent;./emctl clearstate agent;./emctl start agent;./emctl upload agent

But as you might have guessed by now, this didn’t help at all.

So the next thing we verified — and this is where we hit the jackpot — were the targets that the agent had registered. Here we saw that the Cluster Database wasn’t listed.

oracle@test[oemagent] /u01/app/oracle/agent/agent12c/bin
oracle $ emctl config agent listtargets | grep "CLUSTER_TEST" | wc -l

0

OEM12c has a great feature that allows you to resynch the agent via OEM. Here are the steps:

  • Go to Setup –> Manage Cloud Control –> Agents;
  • Click on the testdrv01 agent;
  • On the drop down menu from Agent, choose Resynchronization;
  • Be sure to select “Unblock agent on successful completion of agent resynchronization”.

Once you do that, you will see an output like below:

resyncState: IN executeCommand
 resyncState: validated parameters
 Starting resync RESYNC_20130827141300 for agent testhost:9999
 Getting list of all the Targets to remove from the the agent - testhost:9999
 Removing 27 targets from the agent
 Removing list of plugins from the agent
 Getting list of all the Targets from the repository for the agent - testhost:9999
 Pushing list of plugins to the agent
 Promoting list of targets to the agent
 Re-deploying Metric Extensions to the targets
 Saving target collections to the agent
 Cleaning state on the agent
 size of repos blackout list is 0
 Retrieving java callbacks from em_gcha_callbacks table
 Pdp settings syncronized successfully
 resyncState: resync of agent succeeded!

And voilà!!! Once we did this, we could see the CLUSTER_TEST in our target agent.

oracle@testhost [oemagent] /u01/app/oracle/agent/agent12c/bin
oracle $ emctl config agent listtargets | grep "CLUSTER_TEST"
[CLUSTER_TEST_test1, oracle_database]
[CLUSTER_TEST, rac_database]

When we had confirmed that the target that we wanted was present, we did an upload and verified the status of the agent. To our great pleasure, both targets were visible as up and running in OEM! :)

oracle@testhost [oemagent] /u01/app/oracle/agent/agent12c/bin
 oracle $ ./emctl upload agent; ./emctl status agent

I hope this can help you out when you face something like this! I can guarantee that the next time I’m faced with a similar situation, the first thing that I will do is grab a target list from the agent.

Note: The names and ports were changed to maintain anonymity.

email

Interested in working with Rene? Schedule a tech call.

About the Author

Currently I am an Oracle ACE ; Speaker at Oracle Open World, Oracle Developers Day, OTN Tour Latin America and APAC region and IOUG Collaborate ; Co-President of ORAMEX (Mexico Oracle User Group) and Web Events Chair for IOUG Cloud Computing Special Interest Group (SIG); At the moment I am an Oracle DBA at Pythian.In my free time I like to say that I'm Movie Fanatic, Music Lover and bringing the best from México (Mexihtli) to the rest of the world and in the process photographing it ;)

20 Comments. Leave new

1)Agent is currently Blocked. It will not be able to upload any alerts or metric data to the Oracle Management Service(OMS) until it is unblocked manually. Please ensure that all issues related to Agent misconfiguration have been resolved before unblocking the Agent.
2)System has detected that this agent never uploaded to the repository successfully. Repository does not have enough information to restore this agent.
At time of resycronisation below error arise please suggest

Reply

Hi Meghna

Sorry I didn’t see this earlier, if you go to Setup –> Manage Cloud Control –> Agents, you can select the agent and from there you can block it or unblock it. From there I think you can try an upload

Reply
Joseph Crawford
December 13, 2013 5:18 pm

I was very pleased to find this thread. We have a similar experience with a twist. We have jobs scheduled out of Cloud Control that have the execution of “. oraenv” which work right up to the point of a node failover (we have an active/passive configuration). After failover, although the agent communicates with the targets, jobs that have “. oraenv” in the script execution fail with “oraenv not found.” Through trial and tribulation we likewise determined that we have to resyncronize the agent (including unblock), and like you, things fall back into place. Short of editing every script to fully path the “oraenv” we have no choice but to resyncronize after node failover.

It would be nice if failover was flawless, but it’s good to know we can get things back in order.

Reply

Nice to hear that this post helped you you Joseph

Reply
Wallace Roberto
March 13, 2014 9:05 am

Thanks!! this post helped me a lot!!!

Reply

No problem Wallace :)

Reply

I have installed an agent on Solaris 11, SPARC 64 bit on oem12c 12.1.0.3 OMS. Host and agent appears in console but with status pending. targets.xml file has only one line, can you comment what could be the reason of be at this state.

Reply
Stellios Kes
May 3, 2014 2:56 pm

You can also manually unblock using emcli:
emcli resyncAgent -agent=”:3872″

Reply

we happened to come across similar situation in our environment, where the aagent was not showing any targets registered to it. We tried to re-sync the agent but the re-sync job failed with the following log

resyncState: IN executeCommand
resyncState: validated parameters
Starting resync RESYNC_20141128072008 for agent nvscrubp13:3872
Getting list of all the Targets to remove from the the agent - nvscrubp13:3872
Removing list of plugins from the agent
Getting list of all the Targets from the repository for the agent - nvscrubp13:3872
Pushing list of plugins to the agent
Promoting list of targets to the agent
Re-deploying Metric Extensions to the targets
Saving target collections to the agent
Cleaning state on the agent
size of repos blackout list is 0
Retrieving java callbacks from em_gcha_callbacks table
Pdp settings syncronized successfully
resyncState: resync of agent finished with errors.

What further steps can be taken to know the actual errors in re-sync as mentioned in the logs and how can we resolve this situation?

Reply

Hi Subhan

That actually doesn’t say much, I would suggest you take a look at the agent logs, you can find more information there as to what is happening, the logs you can find them here
https://docs.oracle.com/cd/E24628_01/doc.121/e24473/logging.htm#EMADM11772

Reply
Subhan Singh karki
February 17, 2015 12:03 am

Hi Rene,

Thank you for the reply. There was not much issue. We did not follow proper way of resyncing the agent. This is how we solved the issue. We stopped the agent, clearstate agent, secure agent and then restarted the agent. After this we tried to upload the agent, but it was blocked and at this point we started a resync of this agent, which resolved the issue.

Reply
Subhan Singh karki
February 17, 2015 12:03 am

Hi Rene,

Thank you for the reply. There was not much issue. We did not follow proper way of resyncing the agent. This is how we solved the issue. We stopped the agent, clearstate agent, secure agent and then restarted the agent. After this we tried to upload the agent, but it was blocked and at this point we started a resync of this agent, which resolved the issue.

Reply

Hi Subhan

Good to hear it all worked out :)

Regards

René

Reply
Igor Gorbatovsky
April 23, 2015 10:26 pm

Hi Rene,

Actually, the first step solved my issue ))

Thanks and have a happy day!

Reply

Hi Igor

I am glad that the first step solved your issue :)

Regards

Reply
Joseph Amalraj
April 29, 2015 11:30 am

Great post. Will definitely carry out these steps next time, I come across an agent with pending status.

Reply
Gilbert Standen
January 4, 2016 1:31 pm

Very helpful! Thank you! Had a similar problem in OEM 13c and these instructions helped.

Reply

Thanks!! this post helped me a lot!!!

Reply

My Target status is showing down. I tried with bouncing the agent, but was not of any help. Agent is up and running though. Any suggestion will be really helpful

Reply

My Target status is showing down. I tried with bouncing the agent, but was not of any help. Agent is up and running though. Any suggestion will be really helpful

Reply

Leave a Reply

Your email address will not be published. Required fields are marked *