A simple command that saved me

So there I was, reviewing the environment to create an action plan for patching. To my surprise, I noticed the following when checking DBFS: [code] Oracle Instance not alive for sid "DBFS2" [/code] I will demonstrate the simple command I used to ensure that all services that are supposed to be running are running. What happened? It turns out there was a network outage and when starting resources, DBFS was missing.

It's much harder to know what are all the resources that need to be started for a complex environment with RAC, GoldenGate, DataGuard, ACFS, DBFS. crsctl stat res -t -w '((TARGET != ONLINE) or (STATE != ONLINE)'

-------------------------------------------------------------------------------- 
Name Target State Server State details 
-------------------------------------------------------------------------------- 
Cluster Resources 
-------------------------------------------------------------------------------- 
ora.xxxxxxxx.yy_users_dr.svc 1 OFFLINE OFFLINE STABLE 2 OFFLINE OFFLINE STABLE 
-------------------------------------------------------------------------------- 

crsctl stat res -t -w '((TARGET != ONLINE) or (STATE != ONLINE))' 
-------------------------------------------------------------------------------- 
Name Target State Server State details 
-------------------------------------------------------------------------------- 
Local Resources 
-------------------------------------------------------------------------------- 
dbfs_mount OFFLINE ONLINE hostxxxx01 STABLE OFFLINE OFFLINE hostxxxx02 STABLE 
-------------------------------------------------------------------------------- 
Cluster Resources 
-------------------------------------------------------------------------------- 
ora.dbfs.db 2 OFFLINE OFFLINE Instance Shutdown,STABLE ora.xxxxxxxx.yy_users_dr.svc 1 OFFLINE OFFLINE STABLE 2 OFFLINE OFFLINE STABLE xag.ggue_src.goldengate 1 OFFLINE ONLINE hostxxxx01 STABLE 
--------------------------------------------------------------------------------

The above is not good because State=ONLINE but Target=OFFLINE for dbfs_mount at hostxxxx02.

It seems Oracle is seldom intuitive, as checking the condition (TARGET != STATE) does not return any results.

crsctl stat res -t -w '((TARGET != STATE))' 

In conclusion, always check TARGET=STATE for CRS to avoid having services down that are supposed to be running. Alternatively, I suppose just restarting CRS versus starting each service individually might be better, provided resources are integrated with CRS.