Troubleshooting Oracle Cloud Infrastructure Automatic Backups

I could start this blog the same way I started this one because everything is supposed to be easy and at a 1 click distance in any cloud, so it should be the same for the database backups in Oracle Cloud! If is it indeed truly easy to set up automatic databases backups in a few clicks: (if you haven't manually recreated the database, don't do it, I did it for you already :)) . . . then troubleshooting is another story . . . Indeed, you may find this kind of output in your console one day: And here you are "at 1 click distance" to . . . the documentation. You have no way of opening the backup logfiles in your browser like I would have expected, and the documentation shows how to manually troubleshoot the backups (like on any non-cloud database **but** with the cloud software system - let's look at how this works!). I can now confirm that vi, grep and friends have not been killed by the cloud :) First of all, you need to check the databases which are running there. You are supposed to already know, but double-checking with dbcli ensures that your system is healthy from a cloud configuration perspective and that you could go further:

[root@oci_server ~]# dbcli list-databases
 
 ID DB Name DB Type DB Version CDB Class Shape Storage Status DbHomeID 
 ---------------------------------------- ---------- -------- -------------------- ---------- -------- -------- ---------- ------------ ----------------------------------------
 d5b748a3-c9ca-4ad0-87b4-a6228b4a3e57 MYDB122 Si 12.2.0.1.190115 true Oltp ASM Configured 72371d3f-74ee-4925-b0c1-232b5a4c9145 
 [root@oci_server ~]#
 

Okay, now we can have the list of jobs which occurred on this database (I tailed it as this command will list all the jobs, feel free to "grep Failure" if you wish):

[root@oci_server ~]# dbcli list-jobs | grep -i MYDB122 | tail -20
 4c3860ac-8cf1-4987-9ef1-fc2e2442ce7a Create Regular-L0 Backup with TAG-DBTRegular-L01569481521878j3k for Db:MYDB122 in OSS:bYmzpVkMq1BLgxHxZSwc September 26, 2019 7:06:38 AM UTC Failure 
 6f604571-d73c-4c0f-b172-836628fc386d Create Regular-L0 Backup with TAG-DBTRegular-L01569567793191oB6 for Db:MYDB122 in OSS:bYmzpVkMq1BLgxHxZSwc September 27, 2019 7:05:12 AM UTC Failure 
 be730740-b599-4c9b-8bfe-50d1c3ec50ab Create Regular-L0 Backup with TAG-DBTRegular-L01569654323853e8o for Db:MYDB122 in OSS:bYmzpVkMq1BLgxHxZSwc September 28, 2019 7:06:54 AM UTC Failure 
 ca020c2f-12d3-435d-aa78-22b3e8718d1d Create Regular-L0 Backup with TAG-DBTRegular-L0156974066756851B for Db:MYDB122 in OSS:bYmzpVkMq1BLgxHxZSwc September 29, 2019 7:20:54 AM UTC Failure 
 2d00f5a5-5ede-4fd6-b196-b460750e367b Create Regular-L0 Backup with TAG-DBTRegular-L01569827110982sy7 for Db:MYDB122 in OSS:bYmzpVkMq1BLgxHxZSwc September 30, 2019 7:07:07 AM UTC Failure 
 c8d910ef-1cd3-4310-9636-5bb6ba15cbb1 Create Regular-L0 Backup with TAG-DBTRegular-L01569913536096G7c for Db:MYDB122 in OSS:bYmzpVkMq1BLgxHxZSwc October 1, 2019 7:07:18 AM UTC Failure 
 5b9c15b8-8938-43ba-8e16-db25e8b94407 Create Regular-L0 Backup with TAG-DBTRegular-L01569999888129pmL for Db:MYDB122 in OSS:bYmzpVkMq1BLgxHxZSwc October 2, 2019 7:05:54 AM UTC Failure 
 010909de-bae0-4511-bda9-81d2955cfd75 Create Regular-L0 Backup with TAG-DBTRegular-L01570086276758tz0 for Db:MYDB122 in OSS:bYmzpVkMq1BLgxHxZSwc October 3, 2019 7:21:21 AM UTC Failure 
 5aae3578-7451-4c7c-9071-c643e7f01417 Create Regular-L0 Backup with TAG-DBTRegular-L01570172676836JSm for Db:MYDB122 in OSS:bYmzpVkMq1BLgxHxZSwc October 4, 2019 7:05:40 AM UTC Success 
 ea9c92b4-299e-4a15-93aa-c3f40f290a9b Delete Backup for Database name: MYDB122_phx1bf October 4, 2019 7:18:26 AM UTC Success 
 76d0531a-3599-41ba-b440-4261db78c292 Create Regular-L1 Backup with TAG-DBTRegular-L11570259048404mCf for Db:MYDB122 in OSS:bYmzpVkMq1BLgxHxZSwc October 5, 2019 7:05:49 AM UTC Success 
 4055c4bc-9594-491f-b18c-82c15081cf5c Delete Backup for Database name: MYDB122_phx1bf October 5, 2019 7:54:06 AM UTC Success 
 18424327-f775-40e5-bbdb-f73e03453c0a Create Regular-L0 Backup with TAG-DBTRegular-L01570345462290fs3 for Db:MYDB122 in OSS:bYmzpVkMq1BLgxHxZSwc October 6, 2019 7:05:28 AM UTC Success 
 5a05e4c7-d857-46f0-9030-5f33f6722860 Delete Backup for Database name: MYDB122_phx1bf October 6, 2019 7:49:22 AM UTC Success 
 a7a51004-2da4-452f-a02e-1b6f11afc87a Create Regular-L1 Backup with TAG-DBTRegular-L11570431880603Lmh for Db:MYDB122 in OSS:bYmzpVkMq1BLgxHxZSwc October 7, 2019 7:15:23 AM UTC Success 
 f0404baf-84cd-4c58-a4cd-5ccbf3bd3ccd Delete Backup for Database name: MYDB122_phx1bf October 7, 2019 8:18:46 AM UTC Success 
 249b81d9-1efd-4494-bf6f-0bfae8acaf76 Create Regular-L1 Backup with TAG-DBTRegular-L11570518241132W91 for Db:MYDB122 in OSS:bYmzpVkMq1BLgxHxZSwc October 8, 2019 7:05:23 AM UTC Success 
 2b019e8a-a466-4084-87d3-03984c037754 Delete Backup for Database name: MYDB122_phx1bf October 8, 2019 8:19:44 AM UTC Success 
 93303906-4af4-499e-98b9-067010e05134 Create Regular-L1 Backup with TAG-DBTRegular-L11570604660275VZv for Db:MYDB122 in OSS:bYmzpVkMq1BLgxHxZSwc October 9, 2019 7:05:56 AM UTC Success 
 2f04d01b-1912-43cb-91b0-2dd231013030 Delete Backup for Database name: MYDB122_phx1bf October 9, 2019 8:00:09 AM UTC Success 
 [root@oci_server ~]# 
 

We can see here that some backups are successful and some are not. Let's check the latest one that failed:

[root@oci_server ~]# dbcli describe-job -i 010909de-bae0-4511-bda9-81d2955cfd75 -j
 {
  "jobId" : "010909de-bae0-4511-bda9-81d2955cfd75",
  "status" : "Failure",
  "message" : "DCS-10001:Internal error encountered: Failed to run RMAN command. Please refer log at location : oci_server: /opt/oracle/dcs/log/oci_server/rman/bkup/MYDB122_phx1bf/rman_backup_DBTRegular-L01570086276758tz0_2019-10-03_07-22-07-6121233765598704321.log.",
  "reports" : [ {
  "taskId" : "TaskZLockWrapper_7698",
  "taskName" : "task:TaskZLockWrapper_7698",
  "taskResult" : "DCS-10001:Internal error encountered: Failed to run RMAN command. Please refer log at location : oci_server: /opt/oracle/dcs/log/oci_server/rman/bkup/MYDB122_phx1bf/rman_backup_DBTRegular-L01570086276758tz0_2019-10-03_07-22-07-6121233765598704321.log.",
  "startTime" : "October 03, 2019 07:21:24 AM UTC",
  "endTime" : "October 03, 2019 07:22:21 AM UTC",
  "status" : "Failure",
  "taskDescription" : null,
  "parentTaskId" : "TaskServiceRequest_7697",
  "jobId" : "010909de-bae0-4511-bda9-81d2955cfd75",
  "tags" : [ ],
  "reportLevel" : "Error",
  "updatedTime" : "October 03, 2019 07:22:21 AM UTC"
  }, {
  "taskId" : "TaskSequential_7699",
  "taskName" : "task:TaskSequential_7699",
  "taskResult" : "DCS-10001:Internal error encountered: Failed to run RMAN command. Please refer log at location : oci_server: /opt/oracle/dcs/log/oci_server/rman/bkup/MYDB122_phx1bf/rman_backup_DBTRegular-L01570086276758tz0_2019-10-03_07-22-07-6121233765598704321.log.",
  "startTime" : "October 03, 2019 07:21:24 AM UTC",
  "endTime" : "October 03, 2019 07:22:21 AM UTC",
  "status" : "Failure",
  "taskDescription" : null,
  "parentTaskId" : "TaskZLockWrapper_7698",
  "jobId" : "010909de-bae0-4511-bda9-81d2955cfd75",
  "tags" : [ ],
  "reportLevel" : "Error",
  "updatedTime" : "October 03, 2019 07:22:21 AM UTC"
  }, {
  "taskId" : "TaskParallel_7700",
  "taskName" : "Validate backup config",
  "taskResult" : "",
  "startTime" : "October 03, 2019 07:21:24 AM UTC",
  "endTime" : "October 03, 2019 07:21:24 AM UTC",
  "status" : "Success",
  "taskDescription" : null,
  "parentTaskId" : "TaskSequential_7699",
  "jobId" : "010909de-bae0-4511-bda9-81d2955cfd75",
  "tags" : [ ],
  "reportLevel" : "Info",
  "updatedTime" : "October 03, 2019 07:21:24 AM UTC"
  }, {
  "taskId" : "TaskZJsonRpcExt_7703",
  "taskName" : "Validate opc_pfile configuration",
  "taskResult" : "successfully validated the opc_config",
  "startTime" : "October 03, 2019 07:21:24 AM UTC",
  "endTime" : "October 03, 2019 07:21:24 AM UTC",
  "status" : "Success",
  "taskDescription" : null,
  "parentTaskId" : "TaskSequential_7699",
  "jobId" : "010909de-bae0-4511-bda9-81d2955cfd75",
  "tags" : [ ],
  "reportLevel" : "Info",
  "updatedTime" : "October 03, 2019 07:21:24 AM UTC"
  }, {
  "taskId" : "TaskParallel_7705",
  "taskName" : "Database container validation",
  "taskResult" : "",
  "startTime" : "October 03, 2019 07:21:24 AM UTC",
  "endTime" : "October 03, 2019 07:21:25 AM UTC",
  "status" : "Success",
  "taskDescription" : null,
  "parentTaskId" : "TaskSequential_7699",
  "jobId" : "010909de-bae0-4511-bda9-81d2955cfd75",
  "tags" : [ ],
  "reportLevel" : "Info",
  "updatedTime" : "October 03, 2019 07:21:25 AM UTC"
  }, {
  "taskId" : "TaskParallel_7708",
  "taskName" : "libopc existence check",
  "taskResult" : "",
  "startTime" : "October 03, 2019 07:21:25 AM UTC",
  "endTime" : "October 03, 2019 07:21:25 AM UTC",
  "status" : "Success",
  "taskDescription" : null,
  "parentTaskId" : "TaskSequential_7699",
  "jobId" : "010909de-bae0-4511-bda9-81d2955cfd75",
  "tags" : [ ],
  "reportLevel" : "Info",
  "updatedTime" : "October 03, 2019 07:21:25 AM UTC"
  }, {
  "taskId" : "TaskZJsonRpcExt_7711",
  "taskName" : "Backup Validations",
  "taskResult" : "Successfully validated for database backup",
  "startTime" : "October 03, 2019 07:21:25 AM UTC",
  "endTime" : "October 03, 2019 07:21:46 AM UTC",
  "status" : "Success",
  "taskDescription" : null,
  "parentTaskId" : "TaskSequential_7699",
  "jobId" : "010909de-bae0-4511-bda9-81d2955cfd75",
  "tags" : [ ],
  "reportLevel" : "Info",
  "updatedTime" : "October 03, 2019 07:21:46 AM UTC"
  }, {
  "taskId" : "TaskZJsonRpcExt_7713",
  "taskName" : "Recovery Window validation",
  "taskResult" : "successfully validated recovery window",
  "startTime" : "October 03, 2019 07:21:46 AM UTC",
  "endTime" : "October 03, 2019 07:21:53 AM UTC",
  "status" : "Success",
  "taskDescription" : null,
  "parentTaskId" : "TaskSequential_7699",
  "jobId" : "010909de-bae0-4511-bda9-81d2955cfd75",
  "tags" : [ ],
  "reportLevel" : "Info",
  "updatedTime" : "October 03, 2019 07:21:53 AM UTC"
  }, {
  "taskId" : "TaskZJsonRpcExt_7716",
  "taskName" : "Archivelog deletion policy configuration",
  "taskResult" : "Skipping the Archivelog deletion policy configuration",
  "startTime" : "October 03, 2019 07:21:53 AM UTC",
  "endTime" : "October 03, 2019 07:22:01 AM UTC",
  "status" : "Success",
  "taskDescription" : null,
  "parentTaskId" : "TaskParallel_7715",
  "jobId" : "010909de-bae0-4511-bda9-81d2955cfd75",
  "tags" : [ ],
  "reportLevel" : "Info",
  "updatedTime" : "October 03, 2019 07:22:01 AM UTC"
  }, {
  "taskId" : "TaskZJsonRpcExt_7718",
  "taskName" : "Database backup",
  "taskResult" : "DCS-10001:Internal error encountered: Failed to run RMAN command. Please refer log at location : oci_server: /opt/oracle/dcs/log/oci_server/rman/bkup/MYDB122_phx1bf/rman_backup_DBTRegular-L01570086276758tz0_2019-10-03_07-22-07-6121233765598704321.log.",
  "startTime" : "October 03, 2019 07:22:01 AM UTC",
  "endTime" : "October 03, 2019 07:22:21 AM UTC",
  "status" : "Failure",
  "taskDescription" : null,
  "parentTaskId" : "TaskSequential_7699",
  "jobId" : "010909de-bae0-4511-bda9-81d2955cfd75",
  "tags" : [ ],
  "reportLevel" : "Error",
  "updatedTime" : "October 03, 2019 07:22:21 AM UTC"
  } ],
  "createTimestamp" : "October 03, 2019 07:21:21 AM UTC",
  "resourceList" : [ ],
  "description" : "Create Regular-L0 Backup with TAG-DBTRegular-L01570086276758tz0 for Db:MYDB122 in OSS:bYmzpVkMq1BLgxHxZSwc",
  "updatedTime" : "October 03, 2019 07:22:21 AM UTC"
 }
 [root@oci_server ~]# 
 

I am not sure why the Oracle documentation points to use the "-j" option which is to have a JSON output. Indeed, a regular TEXT output looks clearer to me. Maybe because the JSON outputs usually have more information than the TEXT outputs. But for this particular case, the TEXT output has the log file I am looking for which is enough for now:

[root@oci_server ~]# dbcli describe-job -i 010909de-bae0-4511-bda9-81d2955cfd75M 
 
 Job details 
 ----------------------------------------------------------------
  ID: 010909de-bae0-4511-bda9-81d2955cfd75
  Description: Create Regular-L0 Backup with TAG-DBTRegular-L01570086276758tz0 for Db:MYDB122 in OSS:bYmzpVkMq1BLgxHxZSwc
  Status: Failure
  Created: October 3, 2019 7:21:21 AM UTC
  Message: DCS-10001:Internal error encountered: Failed to run RMAN command. Please refer log at location : oci_server: /opt/oracle/dcs/log/oci_server/rman/bkup/MYDB122_phx1bf/rman_backup_DBTRegular-L01570086276758tz0_2019-10-03_07-22-07-6121233765598704321.log.
 
 Task Name Start Time End Time Status 
 ---------------------------------------- ----------------------------------- ----------------------------------- ----------
 task:TaskZLockWrapper_7698 October 3, 2019 7:21:24 AM UTC October 3, 2019 7:22:21 AM UTC Failure 
 task:TaskSequential_7699 October 3, 2019 7:21:24 AM UTC October 3, 2019 7:22:21 AM UTC Failure 
 Validate backup config October 3, 2019 7:21:24 AM UTC October 3, 2019 7:21:24 AM UTC Success 
 Validate opc_pfile configuration October 3, 2019 7:21:24 AM UTC October 3, 2019 7:21:24 AM UTC Success 
 Database container validation October 3, 2019 7:21:24 AM UTC October 3, 2019 7:21:25 AM UTC Success 
 libopc existence check October 3, 2019 7:21:25 AM UTC October 3, 2019 7:21:25 AM UTC Success 
 Backup Validations October 3, 2019 7:21:25 AM UTC October 3, 2019 7:21:46 AM UTC Success 
 Recovery Window validation October 3, 2019 7:21:46 AM UTC October 3, 2019 7:21:53 AM UTC Success 
 Archivelog deletion policy configuration October 3, 2019 7:21:53 AM UTC October 3, 2019 7:22:01 AM UTC Success 
 Database backup October 3, 2019 7:22:01 AM UTC October 3, 2019 7:22:21 AM UTC Failure 
 
 [root@oci_server ~]# 
 

The above output makes it clear that we'll find the problem in the /opt/oracle/dcs/log/oci_server/rman/bkup/MYDB122_phx1bf/rman_backup_DBTRegular-L01570086276758tz0_2019-10-03_07-22-07-6121233765598704321.log file. In this log file, you'll find all the RMAN logs of the backup including the RMAN code which is used by the automatic backups:

RMAN> set echo on;
 2> set command id to "010909de-bae0-4511-bda9-81d2955c";
 3> report schema;
 4> show all;
 5> list incarnation of database;
 6> set echo on;
 7> set encryption on;
 8> backup force device type sbt as compressed backupset incremental level 0 database tag 'DBTRegular-L01570086276758tz0' format 'DBTRegular-L01570086276758tz0_df_%d_%I_%U_%T_%t_set%s' plus archivelog not backed up tag 'DBTRegular-L01570086276758tz0' format 'DBTRegular-L01570086276758tz0_arc_%d_%I_%U_%T_%t_set%s' ;
 9> backup device type sbt as compressed backupset current controlfile tag 'DBTRegular-L01570086276758tz0' format 'DBTRegular-L01570086276758tz0_cf_%d_%I_%U_%T_%t_set%s' spfile tag 'DBTRegular-L01570086276758tz0' format 'DBTRegular-L01570086276758tz0_spf_%d_%I_%U_%T_%t_set%s' ;
 10> delete force noprompt obsolete;
 11> set encryption off;
 12>
 

And the error which was, in this case, due to an FRA full:

Starting backup at 2019/10/03 07:22:16
 ORACLE error from target database:
 ORA-16038: log 3 sequence# 2319 cannot be archived
 ORA-19809: limit exceeded for recovery files
 ORA-00312: online log 3 thread 1: '+RECO/MYDB122_PHX1BF/ONLINELOG/group_3.259.1008210757'
 

You can now fix your FRA full issue and the next backup should be successful! Let's explore a bit more and have a look at a successful backup:

[root@oci_server ~]# dbcli describe-job -i 93303906-4af4-499e-98b9-067010e05134 
 
 Job details 
 ----------------------------------------------------------------
  ID: 93303906-4af4-499e-98b9-067010e05134
  Description: Create Regular-L1 Backup with TAG-DBTRegular-L11570604660275VZv for Db:MYDB122 in OSS:bYmzpVkMq1BLgxHxZSwc
  Status: Success
  Created: October 9, 2019 7:05:56 AM UTC
  Message: 
 
 Task Name Start Time End Time Status 
 ---------------------------------------- ----------------------------------- ----------------------------------- ----------
 Validate backup config October 9, 2019 7:06:00 AM UTC October 9, 2019 7:06:00 AM UTC Success 
 Validate opc_pfile configuration October 9, 2019 7:06:00 AM UTC October 9, 2019 7:06:00 AM UTC Success 
 Database container validation October 9, 2019 7:06:00 AM UTC October 9, 2019 7:06:00 AM UTC Success 
 libopc existence check October 9, 2019 7:06:00 AM UTC October 9, 2019 7:06:00 AM UTC Success 
 Backup Validations October 9, 2019 7:06:00 AM UTC October 9, 2019 7:06:22 AM UTC Success 
 Recovery Window validation October 9, 2019 7:06:22 AM UTC October 9, 2019 7:06:28 AM UTC Success 
 Archivelog deletion policy configuration October 9, 2019 7:06:28 AM UTC October 9, 2019 7:06:36 AM UTC Success 
 crosscheck database backup October 9, 2019 7:06:36 AM UTC October 9, 2019 7:06:57 AM UTC Success 
 Database backup October 9, 2019 7:06:57 AM UTC October 9, 2019 7:59:14 AM UTC Success 
 Restore preview validate database October 9, 2019 7:59:14 AM UTC October 9, 2019 7:59:29 AM UTC Success 
 Backup TDE Wallet files to ObjectStore October 9, 2019 7:59:29 AM UTC October 9, 2019 7:59:29 AM UTC Success 
 
 [root@oci_server ~]# 
 

Unfortunately, there's no log file information here. The JSON output neither has the log file name when a backup is successful (I won't paste it here as it is a very long output to show ... nothing interesting at this point). So where can I find the log files of the backups (successful or not)?

• • There's no log file for the successful backups (this is not a bug, this is a feature as confirmed by Oracle Support):

• • The unsuccessful backup logs can be found here:

/opt/oracle/dcs/log/`hostname -s`/rman/bkup/DB_UNIQUE_NAME/rman_backup_UNIQUE_BACKUP_TAG_A_TIMESTAMP.log
 

Note that you can also directly check / grep the /opt/oracle/dcs/log/dcs-agent.log file which contains some JSON outputs of some backups as well as a few different kinds of DEBUG information. I say some JSON outputs because we can see in the below example information about the October 3rd failed backup and about the October 9th successful backup, but nothing in between but DEBUG information:

 "parentTaskId" : "TaskSequential_7699",
  "jobId" : "010909de-bae0-4511-bda9-81d2955cfd75",
  "tags" : [ ],
  "reportLevel" : "Error"
  } ],
  "createTimestamp" : "Thu Oct 03, 2019 07:21:21.018 (UTC) [1570087281018]",
  "resourceList" : [ ],
  "description" : "Create Regular-L0 Backup with TAG-DBTRegular-L01570086276758tz0 for Db:MYDB122 in OSS:bYmzpVkMq1BLgxHxZSwc"
 }
 2019-10-09 23:39:19,456 INFO [dw-494] [] c.o.d.c.t.CustomSslContextFactory: entering SSLEngine
 2019-10-09 23:39:20,571 INFO [dw-53] [] c.o.d.c.t.CustomSslContextFactory: entering SSLEngine
 2019-10-09 23:39:20,602 INFO [dw-513 - GET /jobs?count=100] [] c.o.d.a.r.JobsApi: Received GET request to getJobsList on JobsApi
 2019-10-09 23:39:20,602 INFO [dw-513 - GET /jobs?count=100] [] c.o.d.a.r.JobsApi: where clause:
 2019-10-09 23:39:48,764 INFO [dw-72] [] c.o.d.c.t.CustomSslContextFactory: entering SSLEngine
 2019-10-09 23:39:49,742 INFO [dw-53] [] c.o.d.c.t.CustomSslContextFactory: entering SSLEngine
 2019-10-09 23:39:49,814 INFO [dw-41 - GET /jobs/93303906-4af4-499e-98b9-067010e05134] [] c.o.d.a.r.JobsApi: Received GET request to getJobDetail on JobsApi with jobid = 93303906-4af4-499e-98b9-067010e05134
 2019-10-09 23:39:49,818 DEBUG [dw-41 - GET /jobs/93303906-4af4-499e-98b9-067010e05134] [] c.o.d.c.t.r.ServiceJobReport: Add TaskReport r.id=93303906-4af4-499e-98b9-067010e05134 jid=93303906-4af4-499e-98b9-067010e05134 status=Success
 2019-10-09 23:39:49,818 DEBUG [dw-41 - GET /jobs/93303906-4af4-499e-98b9-067010e05134] [] c.o.d.c.t.r.ServiceJobReport: Add TaskReport r.id=TaskParallel_10120 jid=93303906-4af4-499e-98b9-067010e05134 status=Success
 2019-10-09 23:39:49,818 DEBUG [dw-41 - GET /jobs/93303906-4af4-499e-98b9-067010e05134] [] c.o.d.c.t.r.ServiceJobReport: Add TaskReport r.id=TaskZJsonRpcExt_10123 jid=93303906-4af4-499e-98b9-067010e05134 status=Success
 2019-10-09 23:39:49,818 DEBUG [dw-41 - GET /jobs/93303906-4af4-499e-98b9-067010e05134] [] c.o.d.c.t.r.ServiceJobReport: Add TaskReport r.id=TaskParallel_10125 jid=93303906-4af4-499e-98b9-067010e05134 status=Success
 2019-10-09 23:39:49,818 DEBUG [dw-41 - GET /jobs/93303906-4af4-499e-98b9-067010e05134] [] c.o.d.c.t.r.ServiceJobReport: Add TaskReport r.id=TaskParallel_10128 jid=93303906-4af4-499e-98b9-067010e05134 status=Success
 2019-10-09 23:39:49,818 DEBUG [dw-41 - GET /jobs/93303906-4af4-499e-98b9-067010e05134] [] c.o.d.c.t.r.ServiceJobReport: Add TaskReport r.id=TaskZJsonRpcExt_10131 jid=93303906-4af4-499e-98b9-067010e05134 status=Success
 2019-10-09 23:39:49,818 DEBUG [dw-41 - GET /jobs/93303906-4af4-499e-98b9-067010e05134] [] c.o.d.c.t.r.ServiceJobReport: Add TaskReport r.id=TaskZJsonRpcExt_10133 jid=93303906-4af4-499e-98b9-067010e05134 status=Success
 2019-10-09 23:39:49,818 DEBUG [dw-41 - GET /jobs/93303906-4af4-499e-98b9-067010e05134] [] c.o.d.c.t.r.ServiceJobReport: Add TaskReport r.id=TaskZJsonRpcExt_10136 jid=93303906-4af4-499e-98b9-067010e05134 status=Success
 2019-10-09 23:39:49,818 DEBUG [dw-41 - GET /jobs/93303906-4af4-499e-98b9-067010e05134] [] c.o.d.c.t.r.ServiceJobReport: Add TaskReport r.id=TaskZJsonRpcExt_10138 jid=93303906-4af4-499e-98b9-067010e05134 status=Success
 2019-10-09 23:39:49,818 DEBUG [dw-41 - GET /jobs/93303906-4af4-499e-98b9-067010e05134] [] c.o.d.c.t.r.ServiceJobReport: Add TaskReport r.id=TaskZJsonRpcExt_10140 jid=93303906-4af4-499e-98b9-067010e05134 status=Success
 2019-10-09 23:39:49,818 DEBUG [dw-41 - GET /jobs/93303906-4af4-499e-98b9-067010e05134] [] c.o.d.c.t.r.ServiceJobReport: Add TaskReport r.id=TaskZJsonRpcExt_10142 jid=93303906-4af4-499e-98b9-067010e05134 status=Success
 2019-10-09 23:39:49,818 DEBUG [dw-41 - GET /jobs/93303906-4af4-499e-98b9-067010e05134] [] c.o.d.c.t.r.ServiceJobReport: Add TaskReport r.id=TaskZJsonRpcExt_10144 jid=93303906-4af4-499e-98b9-067010e05134 status=Success
 2019-10-09 23:39:49,820 DEBUG [dw-41 - GET /jobs/93303906-4af4-499e-98b9-067010e05134] [] c.o.d.c.t.r.ReportApi: Job Report:
 {
  "updatedTime" : "Wed Oct 09, 2019 07:59:29.518 (UTC) [1570607969518]",
  "jobId" : "93303906-4af4-499e-98b9-067010e05134",
  "status" : "Success",
  "message" : null,
  "reports" : [ {
 

Indeed:

[root@oci_server ~]# grep 010909de-bae0-4511-bda9-81d2955cfd75 /opt/oracle/dcs/log/dcs-agent.log | wc -l # October 3rd failed backup
 48
 [root@oci_server ~]# grep 76d0531a-3599-41ba-b440-4261db78c292 /opt/oracle/dcs/log/dcs-agent.log | wc -l # October 5th successful backup
 0
 [root@oci_server ~]# grep a7a51004-2da4-452f-a02e-1b6f11afc87a /opt/oracle/dcs/log/dcs-agent.log | wc -l # October 7th successful backup
 0
 [root@oci_server ~]# grep 93303906-4af4-499e-98b9-067010e05134 /opt/oracle/dcs/log/dcs-agent.log | wc -l # October 9th successful backup
 5505
 [root@oci_server ~]# 
 

Oracle says " If you require more information, review the /opt/oracle/dcs/log/dcs-agent.log file" so let's take it like this, as it seems that we don't have all the information we expect in this file. By the way, it is interesting to note that this "DCS agent" was already here with Oracle Database Appliance (ODA) and is used to "allow you to manage your database through the cloud platform". It then obviously needs to be improved as we are unable to see a log file through the console (which may come later on). Thanks for reading and enjoy the Oracle Cloud (OCI) Automatic Backups Troubleshooting!