RMAN-06059: expected archived log not found, loss of archived log compromises recoverability

Issue -

RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03002: failure of backup plus archivelog command at 07/17/2018 22:04:37
RMAN-06059: expected archived log not found, loss of archived log compromises recoverability
ORA-19625: error identifying file /Ora/APEX/APEX_1_13_981655091.arc
ORA-27037: unable to obtain file status
Linux-x86_64 Error: 2: No such file or directory
Additional information: 3

Solution -

The archive backup looking for old archive file which is not available on the server.

commands -
RMAN> list backup;
RMAN>crosscheck backup;
RMAN>crosscheck archivelog all;
RMAN>delete expired backup;

These commands delete the expired  backups and archives.

RMAN restore and recovery script in Oracle !12c

run
{
allocate channel c1 type disk;
allocate channel c2 type disk;
allocate channel c3 type disk;
allocate channel c4 type disk;
allocate channel c5 type disk;
allocate channel c6 type disk;
allocate channel c7 type disk;
set until scn 5761438110;
restore database;
recover database;
alter database open resetlogs;
release channel c1;
release channel c2;
release channel c3;
release channel c4;
release channel c5;
release channel c6;
release channel c7;
}

RMAN-06556: datafile 1 must be restored from backup older than SCN

Issue - When recovering database until SCN, recovery failed with error  RMAN-03002 and RMAN-06556

using channel ORA_DISK_8
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03002: failure of recover command at 07/17/2018 20:53:35
RMAN-06556: datafile 1 must be restored from backup older than SCN 5761437810

Solution -
Below scripts useful to identify the SCN for the data files can be recovered - 
1.
select  min(FHSCN) "LOW FILEHDR SCN", max(FHSCN) "MAX FILEHDR SCN", max(FHAFS) "Min PITR ABSSCN"from X$KCVFH ;

LOW FILEHDR SCN  MAX FILEHDR SCN  Min PITR ABSSCN
---------------- ---------------- ----------------
5761438110       5761438110       0

2.

SQL> select file#, status, checkpoint_change# from v$datafile where STATUS like '%SYS%';

     FILE# STATUS  CHECKPOINT_CHANGE#
---------- ------- ------------------
         1 SYSTEM          5761438110



RMAN>  recover database until scn 5761438110;

Starting recover at 17-JUL-18
using channel ORA_DISK_1
using channel ORA_DISK_2
using channel ORA_DISK_3
using channel ORA_DISK_4
using channel ORA_DISK_5
using channel ORA_DISK_6
using channel ORA_DISK_7
using channel ORA_DISK_8

starting media recovery
media recovery complete, elapsed time: 00:00:00

Finished recover at 17-JUL-18

FNDLOAD - Uploading from the data file XXB_CONC.ldt The file XXB_CONC.ldt could not be opened for reading

FNDLOAD - Uploading from the data file XXB_CONC.ldt The file XXB_CONC.ldt could not be opened for reading

Issue -

[applmgr@xxora-app UPLOAD]$ FNDLOAD apps/xxxx 0 Y UPLOAD $FND_TOP/patch/115/import/afcpprog.lct _CONC.ldt UPLOAD_MODE=REPLACE CUSTOM_MODE=FORCE
Log filename : L9021287.log
Report filename : O9021287.out

[applmgr@xxora-app UPLOAD]$ cat L9021287.log
+---------------------------------------------------------------------------+
Application Object Library: Version : 12.0.0
Copyright (c) 1979, 1999, Oracle Corporation. All rights reserved.
 module:
+---------------------------------------------------------------------------+
Current system time is 19-SEP-2017 11:44:09
+---------------------------------------------------------------------------+
Uploading from the data file XXB_CONC.ldt
The file XXB_CONC.ldt could not be opened for reading
+---------------------------------------------------------------------------+
Concurrent request completed
Current system time is 19-SEP-2017 11:44:09
+---------------------------------------------------------------------------+

Action plan -
Verified the file permissions, it has full permission
[applmgr@xxora-app UPLOAD]$ ls -ltr
total 92
-rwxrwxrwx 1 applmgr dba 24914 Sep 19 09:43 XXB_CONC.ldt

Observed the file in DOS format and converted to UNIX

[applmgr@xxora-app UPLOAD]$ dos2unix XXB_CONC.ldt
[applmgr@xxora-app UPLOAD]$ file XXB_CONC.ldt
XXB_CONC.ldt: ASCII text
dos2unix: converting file XXB_CONC.ldt to UNIX format ...
[applmgr@xxora-app UPLOAD]$ FNDLOAD apps/apps 0 Y UPLOAD $FND_TOP/patch/115/import/afcpprog.lct XXB_CONC.ldt UPLOAD_MODE=REPLACE CUSTOM_MODE=FORCE
Log filename : L9021288.log
[applmgr@xxora-app UPLOAD]$ cat L9021288.log
+---------------------------------------------------------------------------+
Application Object Library: Version : 12.0.0
Copyright (c) 1979, 1999, Oracle Corporation. All rights reserved.
 module:
+---------------------------------------------------------------------------+
Current system time is 19-SEP-2017 11:46:16
+---------------------------------------------------------------------------+
Uploading from the data file XXB_CONC.ldt
Altering database NLS_LANGUAGE environment to AMERICAN
Dump from LCT/LDT files (/applmgr/apps/apps_st/appl/fnd/12.0.0/patch/115/import/afcpprog.lct(120.2.12010000.2), XXB_CONC.ldt) to stage tables
Dump LCT file /applmgr/apps/apps_st/appl/fnd/12.0.0/patch/115/import/afcpprog.lct(120.2.12010000.2) into FND_SEED_STAGE_CONFIG
Dump LDT file XXB_CONC.ldt into FND_SEED_STAGE_ENTITY
Dumped the batch (EXECUTABLE _  , PROGRAM _  ) into FND_SEED_STAGE_ENTITY
Upload from stage tables
+---------------------------------------------------------------------------+
Concurrent request completed successfully
Current system time is 19-SEP-2017 11:46:18
+---------------------------------------------------------------------------+
Report filename : O9021288.out

BEA-090518 Could not decrypt the password attribute value of fro m the file ./servers/AdminServer/security/boot.properties

Issue -

<Critical> <Security> <BEA-090518> <Could not decrypt the password attribute value of {3DES}WGwHDRyv6synigUqT6lSGQ== fro
m the file ./servers/AdminServer/security/boot.properties. If you have copied an encrypted attribute from boot.properties from another domain into ./s
ervers/AdminServer/security/boot.properties, change the encrypted attribute to its cleartext value then reboot the server. The attribute will be re-en
crypted. Otherwise, change all encrypted attributes to their cleartext values, then reboot the server. All encryptable attributes will be re-encrypted
. The decryption failed with the exception weblogic.security.internal.encryption.EncryptionServiceException: com.rsa.jsafe.JSAFE_PaddingException: Cou
ld not perform unpadding: invalid pad byte..>
Enter username to boot WebLogic server: <Info> <WebLogicServer> <BEA-000215> <Loaded License : /apps/bea92/license.bea>
 <Notice> <WebLogicServer> <BEA-000365> <Server state changed to STARTING>
 <Info> <WorkManager> <BEA-002900> <Initializing self-tuning thread pool>

Server failed. Reason:

There are 1 nested errors:

java.lang.ClassCastException: com.octetstring.vde.backend.BackendRoot
        at weblogic.ldap.EmbeddedLDAP.start(EmbeddedLDAP.java:273)
        at weblogic.t3.srvr.SubsystemRequest.run(SubsystemRequest.java:64)
        at weblogic.work.ExecuteThread.execute(ExecuteThread.java:209)
        at weblogic.work.ExecuteThread.run(ExecuteThread.java:181)



Solution -
Took back up of /bea92/domains/base_domain/servers/mx1/data/nodemanager/boot.properties

removed the boot.properties

Edited the username and password intries in

managed server domain security folder
./bea92/domains/mx_domain/servers/AdminServer/security/boot.properties

EmbeddedLDAP BEA-171522 An error occurred while initializing the Embedded LDAP Server

Issue -

 <EmbeddedLDAP> <BEA-171522> <An error occurred while initializing the Embedded LDAP Server. The exception tho
wn is java.lang.ClassCastException: com.octetstring.vde.backend.BackendRoot. This may indicate a problem with the data files for the Embedded LDAP Ser
ver. If the problem is with the data files and it can not be corrected, backups of previous versions of the data files exist in ./servers/AdminServer/
data/ldap/backup.>
 <Critical> <WebLogicServer> <BEA-000362> <Server failed. Reason:

There are 1 nested errors:

java.lang.ClassCastException: com.octetstring.vde.backend.BackendRoot
        at weblogic.ldap.EmbeddedLDAP.start(EmbeddedLDAP.java:273)
        at weblogic.t3.srvr.SubsystemRequest.run(SubsystemRequest.java:64)
        at weblogic.work.ExecuteThread.execute(ExecuteThread.java:209)
        at weblogic.work.ExecuteThread.run(ExecuteThread.java:181)

>
 <Notice> <WebLogicServer> <BEA-000365> <Server state changed to FAILED>
 <Error> <WebLogicServer> <BEA-000383> <A critical service failed. The server will shut itself down>
 <Notice> <WebLogicServer> <BEA-000365> <Server state changed to FORCE_SHUTTING_DOWN>

Solution -

Took back up of ldap directory
Removed the ldap directory from managed domain server  /apps/bea92/domains/mx_domain/servers/AdminServer/data/

Restarted the services

Message icon - Warning For server Server-1, the Node Manager associated with machine Machine-0 is not reachable

Issue -
Message icon - Warning    For server Server-1, the Node Manager associated with machine Machine-0 is not reachable

Message icon - Warning    All of the servers selected are currently in a state which is incompatible with this operation or are not associated with a running Node Manager. No action will be performed. 

<Warning> <Uncaught exception in server handler: javax.net.ssl.SSLKeyException: [Security:090482]BAD_CERTIFICATE alert was r
eceived from abc-xyz.com - xxx.xx.xx.xx. Check the peer to determine why it rejected the certificate chain (trusted CA configuration,
hostname verification). SSL debug tracing may be required to determine the exact reason the certificate was rejected.>
javax.net.ssl.SSLKeyException: [Security:090482]BAD_CERTIFICATE alert was received from . Check the peer to
determine why it rejected the certificate chain (trusted CA configuration, hostname verification). SSL debug tracing may be required to determine the
exact reason the certificate was rejected.
        at com.certicom.tls.interfaceimpl.TLSConnectionImpl.fireException(Unknown Source)
        at com.certicom.tls.interfaceimpl.TLSConnectionImpl.fireAlertReceived(Unknown Source)
        at com.certicom.tls.record.alert.AlertHandler.handle(Unknown Source)
        at com.certicom.tls.record.alert.AlertHandler.handleAlertMessages(Unknown Source)
        at com.certicom.tls.record.MessageInterpreter.interpretContent(Unknown Source)
        at com.certicom.tls.record.MessageInterpreter.decryptMessage(Unknown Source)
        at com.certicom.tls.record.ReadHandler.processRecord(Unknown Source)
        at com.certicom.tls.record.ReadHandler.readRecord(Unknown Source)
        at com.certicom.tls.record.ReadHandler.readUntilHandshakeComplete(Unknown Source)
        at com.certicom.tls.interfaceimpl.TLSConnectionImpl.completeHandshake(Unknown Source)
        at com.certicom.tls.record.ReadHandler.read(Unknown Source)
        at com.certicom.io.InputSSLIOStreamWrapper.read(Unknown Source)
        at sun.nio.cs.StreamDecoder$CharsetSD.readBytes(StreamDecoder.java:411)
        at sun.nio.cs.StreamDecoder$CharsetSD.implRead(StreamDecoder.java:453)
        at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:183)
        at java.io.InputStreamReader.read(InputStreamReader.java:167)
        at java.io.BufferedReader.fill(BufferedReader.java:136)
        at java.io.BufferedReader.readLine(BufferedReader.java:299)
        at java.io.BufferedReader.readLine(BufferedReader.java:362)
        at weblogic.nodemanager.server.Handler.run(Handler.java:66)
        at java.lang.Thread.run(Thread.java:595)


Resolution -

Modified the correct host name at node manager "Listen Address" and bounced the weblogic admin server and node manager.