Search This Blog

Wednesday, October 9, 2013

Discoverer upgrade steps (Oracle Enterprise Manager 10g Application Server Control Release 10.1.2.0.2 to Oracle Enterprise Manager 10g Application Server Control Release 10.1.2.3.0)

Steps:
1)      Upgrade the infra – database to 10.1.0.5 from 10.1.0.4.2
2)      10g 10.1.2.3 patch set 3 application
3)      Apply latest cumulative patch 11674847


---1) Upgrade the infra – database to 10.1.0.5 from 10.1.0.4.2

a)      Bring down Discoverer , Infra and Database services

-rw-r--r--  1 oradisco dba       210 Jul 16 10:58 disco.env
-rw-r--r--  1 oradisco dba       233 Jul 16 11:04 infra.env
 [oradisco@disco ~]$ pwd
/home/oradisco
n  AS Home

[oradisco@disco ].
[oradisco@disco bin]$ ./opmnctl stopall
opmnctl: stopping opmn and all managed processes...
================================================================================
opmn id=disco.world.com:6201
    6 of 7 processes stopped.

ias-instance id=oraas1.disco.world.com
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
--------------------------------------------------------------------------------
ias-component/process-type/process-set:
    OC4J/OC4J_BI_Forms/default_island

Error
--> Process (pid=1999)
    time out while waiting for a managed process to stop
    Log:
    /data/oradisco/OraHome_AS1/opmn/logs/OC4J~OC4J_BI_Forms~default_island~1

opmnctl: graceful stop of processes failed, trying forceful shutdown...

[oradisco@disco bin]$

[oradisco@disco bin]$ ./emctl status iasconsole
TZ set to US/Pacific
Oracle Enterprise Manager 10g Application Server Control Release 10.1.2.0.2
Copyright (c) 1996, 2005 Oracle Corporation.  All rights reserved.
http://disco.world.com:1156/emd/console/aboutApplication
Oracle Enterprise Manager 10g Application Server Control is running.
------------------------------------------------------------------
Logs are generated in directory /data/oradisco/OraHome_Infra/sysman/log
[oradisco@disco bin]$ ./emctl stop iasconsole
TZ set to US/Pacific
Oracle Enterprise Manager 10g Application Server Control Release 10.1.2.0.2
Copyright (c) 1996, 2005 Oracle Corporation.  All rights reserved.
http://disco.world.com:1156/emd/console/aboutApplication

Stopping Oracle Enterprise Manager 10g Application Server Control ...  ...  Stopped.

n  Infra Home
[oradisco@disco bin]$ ./opmnctl stopall
opmnctl: stopping opmn and all managed processes...
================================================================================


[oradisco@disco bin]$ ./emctl stop iasconsole
TZ set to US/Pacific
Oracle Enterprise Manager 10g Application Server Control Release 10.1.2.0.2
Copyright (c) 1996, 2005 Oracle Corporation.  All rights reserved.
http://disco.world.com:1156/emd/console/aboutApplication

Stopping Oracle Enterprise Manager 10g Application Server Control ...  ...  Stopped.

n  Database
SQL> select name from v$database;

NAME
---------
INFRA

SQL> shut immediate;
Database closed.
Database dismounted.
ORACLE instance shut down.           

[oradisco@disco bdump]$ lsnrctl stop LISTENER

LSNRCTL for Linux: Version 10.1.0.5.0 - Production on 12-OCT-2011 23:17:54

Copyright (c) 1991, 2004, Oracle.  All rights reserved.

Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=disco.world.com)(PORT=1521)))
The command completed successfully

n  Take the existed Homes backup
[oradisco@disco oradisco]$ pwd
/data/oradisco
[oradisco@disco oradisco]$ vi /home/oradisco/nohup.out

[oradisco@disco oradisco]$ jobs
[1]   Running                 nohup tar czf oradata_12OCT11.tar.gz oradata &
[2]-  Running                 nohup tar czf OraHome_Infra_12OCT11.tar.gz OraHome_Infra &
[3]+  Running                 nohup tar czf OraHome_AS1_12OCT11.tar.gz OraHome_AS1 &

n  install 10.1.0.5 DB patch set

[oradisco@disco Disk1]$ ./runInstaller -ignoreSysPreReqs

Starting Oracle Universal Installer...





(/data/oradisco/OraHome_Infra/root.sh)and following is the output of the same.

-------------Output------------------
Running Oracle 10g root.sh script...

The following environment variables are set as:
    ORACLE_OWNER= oradisco
    ORACLE_HOME=  /data/oradisco/OraHome_Infra

Enter the full pathname of the local bin directory: [/usr/local/bin]:
The file "dbhome" already exists in /usr/local/bin.  Overwrite it? (y/n)
[n]: y
   Copying dbhome to /usr/local/bin ...
The file "oraenv" already exists in /usr/local/bin.  Overwrite it? (y/n)
[n]: y
   Copying oraenv to /usr/local/bin ...
The file "coraenv" already exists in /usr/local/bin.  Overwrite it? (y/n)
[n]: y
   Copying coraenv to /usr/local/bin ...

Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root.sh script.
Now product-specific root actions will be performed.
[root@disco ~]#


[oradisco@disco bin]$ sqlplus '/as sysdba'

SQL*Plus: Release 10.1.0.5.0 - Production on Wed Oct 12 23:09:41 2011

Copyright (c) 1982, 2005, Oracle.  All rights reserved.


Connected to:
Oracle Database 10g Enterprise Edition Release 10.1.0.5.0 - Production
With the Partitioning, OLAP and Data Mining options

SQL> shutdown
Database closed.
Database dismounted.
ORACLE instance shut down.
SQL> STARTUP UPGRADE
ORACLE instance started.

Total System Global Area  281018368 bytes
Fixed Size                   779000 bytes
Variable Size             227810568 bytes
Database Buffers           50331648 bytes
Redo Buffers                2097152 bytes
Database mounted.
Database opened.
SQL> SPOOL patch.log
SQL> @/ORACLE_HOME/rdbms/admin/catpatch.sql
SP2-0310: unable to open file "/ORACLE_HOME/rdbms/admin/catpatch.sql"
SQL> @$ORACLE_HOME/rdbms/admin/catpatch.sql
DOC>######################################################################
DOC>######################################################################
DOC>    The following PL/SQL block will cause an ORA-20000 error and
DOC>    terminate the current SQLPLUS session if the user is not SYS.
DOC>    Disconnect and reconnect with AS SYSDBA.
DOC>######################################################################
DOC>######################################################################
DOC>#
|
|
|
V
DOC>#######################################################################
DOC>#######################################################################
DOC>
DOC>   The above query lists the SERVER components in the database,
DOC>   along with their current patch set version and status.
DOC>
DOC>   Please review the status and version columns and look for
DOC>   any errors in the spool log file.   If there are errors in the spool
DOC>   file, or any components are not VALID or not the correct version,
DOC>   consult the Patch Set Notes.
DOC>
DOC>   Next shutdown immediate, restart for normal operation, and then
DOC>   run utlrp.sql to recompile any invalid application objects.
DOC>
DOC>#######################################################################
DOC>#######################################################################
DOC>#
SQL>
SQL> spool off
SQL> shutdown
Database closed.
Database dismounted.
ORACLE instance shut down.

SQL> startup
ORACLE instance started.

Total System Global Area  281018368 bytes
Fixed Size                   779000 bytes
Variable Size             227810568 bytes
Database Buffers           50331648 bytes
Redo Buffers                2097152 bytes
Database mounted.
Database opened.

SQL> select name from v$database;

NAME
---------
INFRA

1 row selected.

SQL> @$ORACLE_HOME/rdbms/admin/utlrp.sql

TIMESTAMP
--------------------------------------------------------------------------------
COMP_TIMESTAMP UTLRP_BGN  2011-10-12 23:33:33

1 row selected.


PL/SQL procedure successfully completed.


TIMESTAMP
--------------------------------------------------------------------------------
COMP_TIMESTAMP UTLRP_END  2011-10-12 23:33:59

1 row selected.


PL/SQL procedure successfully completed.

Invoking Ultra Search Install/Upgrade validation procedure VALIDATE_WK
Ultra Search VALIDATE_WK done with no error

PL/SQL procedure successfully completed.

SQL> select count(*) from dba_objects where status = 'INVALID';

  COUNT(*)
----------
         1

1 row selected.


n  Application Server 10g 10.1.2.3 patch set 3 Patch 5983622 on AS Home (Discoverer)

[oradisco@disco patchset_10123]$ cd Disk1/
[oradisco@disco Disk1]$ ls -ltr
total 620
-rwxr-xr-x   1 oradisco dba   1280 Dec 18  2007 runInstaller
drwxr-xr-x   5 oradisco dba   4096 Jan  2  2008 install
drwxr-xr-x   2 oradisco dba   4096 Mar 31  2008 doc
drwxr-xr-x  10 oradisco dba   4096 Apr  3  2008 stage
-rwxr-xr-x   1 oradisco dba 612054 Apr  4  2008 README.html
[oradisco@disco Disk1]$ ./runInstaller
Starting Oracle Universal Installer...

Checking installer requirements...





--- Note as we don’t know the user “ias_admin” credentials, had reset the password as follows in both Infra and AS Homes

 Change ias_admin password directly in configuration file
–Backup $ORACLE_HOME/sysman/j2ee/config/jazn-data.xml
–Search for entry like below

  <user>
    <name>ias_admin</name>
      <credentials>{903}8QkQ/crno3lX0f3+67dj6WxW9KJMXaCu</credentials>
  </user>

and Update new password (welcome2 like )
  <user>
    <name>ias_admin</name>
      <credentials>
!welcome2</credentials>
  </user>

Note ! (Exclamation Mark in front of password. This signifies that password is stored in clear text)


[oradisco@disco config]$ vi jazn-data.xml
[oradisco@disco config]$ pwd
/data/oradisco/OraHome_Infra/sysman/j2ee/config
[oradisco@disco config]$ cd /data/oradisco/OraHome_Infra/bin
[oradisco@disco bin]$ emctl set password welcome2 welcome123
TZ set to US/Pacific
-------à AS Home
[oradisco@disco ~]$ cd /data/oradisco/OraHome_AS1/sysman/j2ee/config/jazn-data.xml
[oradisco@disco ~]$ cp /data/oradisco/OraHome_AS1/sysman/j2ee/config/jazn-data.xml /data/oradisco/OraHome_AS1/sysman/j2ee/config/jazn-data.xml_bkp
[oradisco@disco ~]$ vi /data/oradisco/OraHome_AS1/sysman/j2ee/config/jazn-data.xml
[oradisco@disco ~]$ cd $ORACLE_HOME/bin
[oradisco@disco bin]$ emctl set password welcome2 welcome123
TZ set to US/Pacific
Oracle Enterprise Manager 10g Application Server Control Release 10.1.2.0.2

Copyright (c) 1996, 2005 Oracle Corporation.  All rights reserved.




----Note as we don’t know the user “cn=orcladmin” password credentials , had reset the password as follows on both Infra and AS nodes


SQL> select USERNAME,ACCOUNT_STATUS from dba_users where  USERNAME like '%ODS%';

USERNAME                       ACCOUNT_STATUS
------------------------------ --------------------------------
ODS                            OPEN

SQL> alter user ODS identified by welcome2;

User altered.

[oradisco@disco ~]$ cd /data/oradisco/OraHome_Infra/opmn/bin/
[oradisco@disco bin]$ ./opmnctl stopproc ias-component=OID
Internal error message 684 could not be found in the msb file
opmnctl: stopping opmn managed processes...
Internal error message 684 could not be found in the msb file
[oradisco@disco bin]$ ps -ef | grep -i oidmon
oradisco  3182 15070  0 02:49 pts/1    00:00:00 grep -i oidmon
[oradisco@disco bin]$ ps -ef | grep -i oidldapd
oradisco  3207 15070  0 02:50 pts/1    00:00:00 grep -i oidldapd
[oradisco@disco bin]$ ps -ef | grep -i odisrv
oradisco  3361 15070  0 02:50 pts/1    00:00:00 grep -i odisrv
[oradisco@disco bin]$ cd $ORACLE_HOME/ldap/admin
[oradisco@disco admin]$ ls -ltr oidpwdlldap1
-rw-r-x---  1 oradisco dba 207 Oct 13 01:46 oidpwdlldap1
[oradisco@disco admin]$ ls -ltr oidpwdrxxxx
ls: oidpwdrxxxx: No such file or directory
[oradisco@disco admin]$ ls -ltr oidpwdr*
-rw-r-x---  1 oradisco dba 331 May 19  2009 oidpwdrinfra
[oradisco@disco admin]$ mv oidpwdlldap1 oidpwdlldap1_bkp1
[oradisco@disco admin]$ mv oidpwdrinfra oidpwdrinfra_bkp2

[oradisco@disco admin]$ sqlplus ods/welcome1@INFRA

SQL*Plus: Release 10.1.0.5.0 - Production on Thu Oct 13 02:52:01 2011

Copyright (c) 1982, 2005, Oracle.  All rights reserved.


Connected to:
Oracle Database 10g Enterprise Edition Release 10.1.0.5.0 - Production
With the Partitioning, OLAP and Data Mining options

SQL> Disconnected from Oracle Database 10g Enterprise Edition Release 10.1.0.5.0 - Production
With the Partitioning, OLAP and Data Mining options
[oradisco@disco admin]$ oidpasswd connect=INFRA create_wallet=true
        password:
confirm password:
password set

[oradisco@disco admin]$ ls -ltr oidpwdr*
-rw-r-x---  1 oradisco dba 331 May 19  2009 oidpwdrinfra_bkp2
-rw-r--r--  1 oradisco dba 215 Oct 13 02:52 oidpwdrinfra
[oradisco@disco admin]$ ls -ltr oidpwdlldap1
-rw-r--r--  1 oradisco dba 207 Oct 13 02:52 oidpwdlldap1


[oradisco@disco admin]$ oidpasswd connect=INFRA unlock_su_acct=true
OID DB user password:
OID super user account unlocked successfully.
[oradisco@disco admin]$ oidpasswd connect=INFRA reset_su_password=true
OID DB user password:
    new password:
confirm password:
password set

[oradisco@disco bin]$ $ORACLE_HOME/bin/ldapbind -p 389 -D cn=orcladmin -w welcome2
bind successful
[oradisco@disco bin]$ pwd

/data/oradisco/OraHome_Infra/opmn/bin



n  Application Server 10g 10.1.2.3 patch set 3 Patch 5983622 on Infra Home
As I ran into below issue and Infra home seems not upgraded then I applied the same patch set on Infra Home too

[oradisco@disco bin]$ ./emctl stop iasconsole
TZ set to US/Pacific
Oracle Enterprise Manager 10g Application Server Control Release 10.1.2.0.2
Copyright (c) 1996, 2005 Oracle Corporation.  All rights reserved.
http://disco.ntc.com:1156/emd/console/aboutApplication

Stopping Oracle Enterprise Manager 10g Application Server Control ...
 Cannot determine Oracle Enterprise Manager 10g Application Server Control process; /data/oradisco/OraHome_Infra/bin/emctl.pid does not exist.
Oracle Enterprise Manager 10g Application Server Control may not be running.
[oradisco@disco bin]$ pwd
/data/oradisco/OraHome_Infra/bin
















n  apply the latest Cumulative patch 11674847 if not the following error will thrown out while connecting Discoverer
"Missing Applications Security Library: fnd115"


[oradisco@disco 11674847]$ which opatch
/usr/bin/which: no opatch in (/usr/kerberos/bin:/usr/local/bin:/bin:/usr/bin:/usr/X11R6/bin:/home/oradisco/bin:/data/oradisco/OraHome_AS1/bin:/data/oradisco/OraHome_AS1/opmn/bin)
[oradisco@disco 11674847]$ /data/oradisco/OraHome_AS1/OPatch/opatch apply

Oracle Interim Patch Installer version 1.0.0.0.58
Copyright (c) 2007 Oracle Corporation. All Rights Reserved..

We recommend you refer to the OPatch documentation under
OPatch/docs for usage reference. We also recommend using
the latest OPatch version. For the latest OPatch version
and other support related issues, please refer to document
293369.1 which is viewable from metalink.oracle.com

Oracle Home           : /data/oradisco/OraHome_AS1
Oracle Home Inventory : /data/oradisco/OraHome_AS1/inventory
Central Inventory     : /home/oradisco/oraInventory
   from               : /etc/oraInst.loc
OUI location          : /data/oradisco/OraHome_AS1/oui
OUI shared library    : /data/oradisco/OraHome_AS1/oui/lib/linux/liboraInstaller.so
Java location         : /data/oradisco/OraHome_AS1/jre/1.4.2/bin/java
Log file location     : /data/oradisco/OraHome_AS1/.patch_storage/<patch ID>/*.log

Creating log file "/data/oradisco/OraHome_AS1/.patch_storage/11674847/Apply_11674847_10-13-2011_04-15-00.log"

Looking for /home/oradisco/patchset_10123/11674847/files/ocm.zip...
Invoking fuser to check for active processes.

Backing up comps.xml ...

OPatch detected non-cluster Oracle Home from the inventory and will patch the local system only.


Please shut down Oracle instances running out of this ORACLE_HOME
(Oracle Home = /data/oradisco/OraHome_AS1)
Is this system ready for updating?
Please respond Y|N >
Y
Executing the Apply pre-patch script (/home/oradisco/patchset_10123/11674847/custom/scripts/pre)...
Applying patch 11674847...

Patching copy files...



Inventory is good and does not have any dangling patches.


Updating inventory...

Verifying patch...
Backing up comps.xml ...

Executing the Apply post-patch script (/home/oradisco/patchset_10123/11674847/custom/scripts/post)...

OPatch succeeded.

Reference doc : How to Unlock/Reset Super User cn=orcladmin When the ODS's Password Has Been Forgotten or is Unknown? [ID 472752.1]
Oracle Application Server Patch Set Notes Addendum 10g Release 2 (10.1.2) Patch Set 3 (10.1.2.3.0) [ID 555681.1]

Steps/How to download latest patchsets.sh using sftp

Here are the steps to download, to access this, the port 2021 needs to be opened from your environment.

oracle-ven1-/home/oradev $sftp -o PORT=2021 dba@xyz.com@sftp.oracle.com
Connecting to sftp.oracle.com...
The authenticity of host 'sftp.oracle.com (141.146.1.165)' can't be established.
RSA key fingerprint is ab:8f:0e:ef:57:ae:3f:49:b9:ab:d6:2d:b5:47:42:25.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'sftp.oracle.com,141.146.1.165' (RSA) to the list of known hosts.
*******************************************************************************
 Oracle SFTP Server

 The use of this SFTP server is governed by United States Export Administration
 Regulations. Details of the U.S. Commercial Encryption Export Controls can be
 found at the Bureau of Industry and Security web site.
 All Oracle products are subject to U.S. Export Laws.
 Diversion contrary to U.S. law is prohibited


 Oracle Employees:
 Under no circumstances are Oracle Employees authorized to download software for
 the purpose of distributing it to customers. Oracle products are available to
 employees for internal use or demonstration purposes only. Customers may
 download Oracle software through the Oracle Technology Network or through
 Oracle's e-Delivery site. Failure to comply with these requirements may result
 in disciplinary action.
 ****************************************************************************
password
Enter password for dba@xyz.com
Password:
sftp> pwd
Remote working directory: /
sftp> cd support /outgoing/PATCHSET_COMPARE_TOOL
sftp> get patchsets.sh
Fetching //support/outgoing/PATCHSET_COMPARE_TOOL/patchsets.sh to patchsets.sh
//support/outgoing/PATCHSET_COMPARE_TOOL/patchsets.sh                                                              100%  816KB 815.9KB/s   00:01

sftp> bye

Ref: Oracle Applications Current Patchset Comparison Utility - patchsets.sh (Doc ID 139684.1)

Thursday, October 3, 2013

Inbound mails not delivering to oracle apps after user approved - workflow

Issue:Inbound mails not delivering to oracle apps after user approved in workflow and are failing with time out error


Solution:
I cleared the unwanted INBOX mails for wf@ account and bounced the workflow services , now I see all INBOX mails got processed.

Ref: 11i-12 A Guide For Troubleshooting Workflow Notification Emails - Inbound and Outbound (Doc ID 831982.1)

Transportable tablespace refresh

  1.check tablespace for the user which need to refresh -------------------------------------------------------------------  SQL> select ...