DB Console EMAGENT process is still running

Issue:

I see that the EMAGENT process is still running , even after i shutdown the DB Console. It is writing XML logs in the "upload" directory. I tried to shutdown the agent using command "emctl stop agent", but the command is not working.

 Solution:

Due to various reasons, such as crashing, incorrect configuration, corrupted memory or process termination, the DB Console processes may be left over in an undetermined state making start/stop operation failing.

 Below are 3 main processes for DB Console:
1. "emwd.pl dbconsole"
    Called EM WatchDog process, this one is controlling all the other ones.
2. "emagent"
    This is the EM agent process, parent PID is the WatchDog one above.
3. "java"
    This is the DB Console process it self (the UI console, OMS, etc...). Parent PID is also the emwd one (WatchDog)

Here I manually cleared the agent process

Ref: How To Cleanup All DBConsole left-over Processes? [ID 1171814.1]

DB Console re-installation steps - Issues

-bash-4.2$ ./emca -deconfig dbcontrol db

STARTED EMCA at Apr 8, 2012 9:25:17 AM
EM Configuration Assistant, Version 11.2.0.0.2 Production
Copyright (c) 2003, 2005, Oracle.  All rights reserved.

Enter the following information:
Database SID: TEST

Do you wish to continue? [yes(Y)/no(N)]: Y
Apr 8, 2012 9:25:36 AM oracle.sysman.emcp.EMConfig perform
INFO: This operation is being logged at /db1/oracle/TEST/cfgtoollogs/emca/TEST/emca_2012_04_08_09_25_16.log.
Apr 8, 2012 9:25:36 AM oracle.sysman.emcp.util.DBControlUtil stopOMS
INFO: Stopping Database Control (this may take a while) ...
Enterprise Manager configuration completed successfully
FINISHED EMCA at Apr 8, 2012 9:25:46 AM

-bash-4.2$ emca -deconfig dbcontrol db -repos drop

STARTED EMCA at Apr 8, 2012 9:37:27 AM
EM Configuration Assistant, Version 11.2.0.0.2 Production
Copyright (c) 2003, 2005, Oracle.  All rights reserved.

Enter the following information:
Database SID: TEST
Listener port number: 1571
Password for SYS user:
Password for SYSMAN user:

----------------------------------------------------------------------
WARNING : While repository is dropped the database will be put in quiesce mode.
----------------------------------------------------------------------
Do you wish to continue? [yes(Y)/no(N)]: Y
Apr 8, 2012 9:39:03 AM oracle.sysman.emcp.EMConfig perform
INFO: This operation is being logged at /db1/oracle/TEST/cfgtoollogs/emca/TEST/emca_2012_04_08_09_37_27.log.
Apr 8, 2012 9:39:04 AM oracle.sysman.emcp.util.GeneralUtil initSQLEngineLoacly
WARNING: ORA-01017: invalid username/password; logon denied

Apr 8, 2012 9:39:04 AM oracle.sysman.emcp.ParamsManager checkListenerStatusForDBControl
WARNING: Error initializing SQL connection. SQL operations cannot be performed
Apr 8, 2012 9:39:04 AM oracle.sysman.emcp.EMDBPreConfig performDeconfiguration
WARNING: EM is not configured for this database. No EM-specific actions can be performed. Some of the possible reasons may be:
 1) EM is configured with different hostname then physical host. Set environment variable ORACLE_HOSTNAME=<hostname> and re-run EMCA script
 2) ORACLE_HOSTNAME is set. Unset it and re-run EMCA script
Apr 8, 2012 9:39:04 AM oracle.sysman.emcp.EMReposConfig invoke
INFO: Dropping the EM repository (this may take a while) ...
Apr 8, 2012 9:39:05 AM oracle.sysman.emcp.EMReposConfig invoke
INFO: Repository successfully dropped
Enterprise Manager configuration completed successfully
FINISHED EMCA at Apr 8, 2012 9:39:05 AM




SQL> shut immediate;
Database closed.
Database dismounted.
ORACLE instance shut down.
SQL> STARTUP RESTRICT;
ORACLE instance started.

Total System Global Area 6415597568 bytes
Fixed Size                  2170304 bytes
Variable Size            1409286720 bytes
Database Buffers         4865392640 bytes
Redo Buffers              138747904 bytes
Database mounted.
Database opened.
SQL> EXEC sysman.emd_maintenance.remove_em_dbms_jobs;

PL/SQL procedure successfully completed.

SQL> EXEC sysman.setEMUserContext('',5);

PL/SQL procedure successfully completed.

SQL> REVOKE dba FROM sysman;
REVOKE dba FROM sysman
*
ERROR at line 1:
ORA-01951: ROLE 'DBA' not granted to 'SYSMAN'


SQL> DECLARE
CURSOR c1 IS
  2    3  SELECT owner, synonym_name name
  4  FROM dba_synonyms
  5  WHERE table_owner = 'SYSMAN';
  6  BEGIN
  7  FOR r1 IN c1 LOOP
  8  IF r1.owner = 'PUBLIC' THEN
  9  EXECUTE IMMEDIATE 'DROP PUBLIC SYNONYM '||r1.name;
 10  ELSE
 11  EXECUTE IMMEDIATE 'DROP SYNONYM '||r1.owner||'.'||r1.name;
 12  END IF;
 13  END LOOP;
 14  END;
 15  /

PL/SQL procedure successfully completed.

SQL> DROP USER mgmt_view CASCADE;

User dropped.

SQL> DROP ROLE mgmt_user;

Role dropped.

SQL> DROP USER sysman CASCADE;

User dropped.

SQL> ALTER SYSTEM DISABLE RESTRICTED SESSION;

System altered.



-bash-4.2$ emca -config dbcontrol db -repos create

STARTED EMCA at Apr 8, 2012 9:42:12 AM
EM Configuration Assistant, Version 11.2.0.0.2 Production
Copyright (c) 2003, 2005, Oracle.  All rights reserved.

Enter the following information:
Database SID: TEST
Listener port number: 1571
Listener ORACLE_HOME [ /db1/oracle/TEST/product/11.2.0 ]:
Password for SYS user:
Password for DBSNMP user:
Password for SYSMAN user:
Email address for notifications (optional):
Outgoing Mail (SMTP) server for notifications (optional):
-----------------------------------------------------------------

You have specified the following settings

Database ORACLE_HOME ................ /db1/oracle/TEST/product/11.2.0

Local hostname ................ localhost.my
Listener ORACLE_HOME ................ /db1/oracle/STEST/product/11.2.0
Listener port number ................ 1571
Database SID ................ TEST
Email address for notifications ...............
Outgoing Mail (SMTP) server for notifications ...............

-----------------------------------------------------------------
Do you wish to continue? [yes(Y)/no(N)]: Y
Apr 8, 2012 9:43:08 AM oracle.sysman.emcp.EMConfig perform
INFO: This operation is being logged at /db1/oracle/TEST/cfgtoollogs/emca/TEST/emca_2012_04_08_09_42_12.log.
Apr 8, 2012 9:43:10 AM oracle.sysman.emcp.EMReposConfig createRepository
INFO: Creating the EM repository (this may take a while) ...
Apr 8, 2012 9:48:49 AM oracle.sysman.emcp.EMReposConfig invoke
INFO: Repository successfully created
Apr 8, 2012 9:48:53 AM oracle.sysman.emcp.EMReposConfig uploadConfigDataToRepository
INFO: Uploading configuration data to EM repository (this may take a while) ...
Apr 8, 2012 9:50:13 AM oracle.sysman.emcp.EMReposConfig invoke
INFO: Uploaded configuration data successfully
Apr 8, 2012 9:50:47 AM oracle.sysman.emcp.util.DBControlUtil secureDBConsole
INFO: Securing Database Control (this may take a while) ...
Apr 8, 2012 9:50:53 AM oracle.sysman.emcp.util.DBControlUtil secureDBConsole
INFO: Database Control secured successfully.
Apr 8, 2012 9:50:54 AM oracle.sysman.emcp.util.DBControlUtil startOMS
INFO: Starting Database Control (this may take a while) ...
Apr 8, 2012 9:51:25 AM oracle.sysman.emcp.EMDBPostConfig performConfiguration
INFO: Database Control started successfully
Apr 8, 2012 9:51:25 AM oracle.sysman.emcp.EMDBPostConfig performConfiguration
INFO: >>>>>>>>>>> The Database Control URL is https://localhost.my:5500/em <<<<<<<<<<<
Apr 8, 2012 9:51:28 AM oracle.sysman.emcp.EMDBPostConfig invoke
WARNING:
************************  WARNING  ************************

Management Repository has been placed in secure mode wherein Enterprise Manager data will be encrypted.  The encryption key has been placed in the file: /db1/oracle/TEST/product/11.2.0/localhost.me_TEST/sysman/config/emkey.ora. Ensure this file is backed up as the encrypted data will become unusable if this file is lost.

***********************************************************
Enterprise Manager configuration completed successfully


Ref: When to Use the Create/Drop Dbconsole Commands which are Given in Note:278100.1 [ID 1377211.1]
How To Drop, Create And Recreate the Database Control (DB Control) Release 10g and 11g [ID 278100.1]

RMAN-06091: no channel allocated for maintenance (of an appropriate type)

Got below error when I try to delete obsolete backups using RMAN> DELETE NOPROMPT OBSOLETE;
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03002: failure of delete command at 04/06/2012 14:23:27
RMAN-06091: no channel allocated for maintenance (of an appropriate type)

Even I tried with below option

RMAN> delete obsolete;

RMAN> delete noprompt obsolete;

RMAN> delete force obsolete;

Solution:

Run the following to specify the device type , in my case device type is disk

RMAN> delete noprompt obsolete device type disk;

If not worked use below:

ALLOCATE CHANNEL FOR MAINTENANCE DEVICE TYPE DISK;

run

{

report obsolete;

CROSSCHECK BACKUP;

CROSSCHECK COPY;

DELETE EXPIRED BACKUP;

DELETE EXPIRED COPY;

delete obsolete;

}

release channel;

Ref: Encountered RMAN-03002 and RMAN-06091 when Deleting Obsolete Backups [ID 567555.1]

Rman Could Not Delete Obsolete Backupsets Due To Change From Sbt_tape To Disk and To New Server [ID 1287755.1]

Advanced Filters

The Linux commands "grep" , "sed","awk"

1) grep:
The grep program searches a file or files for lines that have a certain pattern
syntax : $grep "pattern" files

The command g/re/p, which means "globally search for a regular expression and print all lines containing it

usage:
$grep "Unix" *
$ls -ltr | grep April

grep options:

-v --> print all lines that do not match pattern
-n --> print the matched line with line number
-l --> print only the names of files with matching lines
-c --> print only count of matching lines
-i --> print that match either upper or lower

Examples with above options one by one:

$ grep -v bash /etc/passwd | grep -v nologin
sync:x:5:0:sync:/sbin:/bin/sync
shutdown:x:6:0:shutdown:/sbin:/sbin/shutdown
halt:x:7:0:halt:/sbin:/sbin/halt
news:x:9:13:news:/etc/news:
ldap:x:55:55:LDAP User:/var/lib/ldap:/bin/false
radiusd:x:95:95:radiusd user:/:/bin/false

$ grep -n root /etc/passwd
1:root:x:0:0:root:/root:/bin/bash
12:operator:x:11:0:operator:/root:/sbin/nologin

$ grep -l siri /etc/passwd
/etc/passwd

$ grep -c root /etc/passwd
2

$ grep -i root /etc/passwd
root:x:0:0:root:/root:/bin/bash
operator:x:11:0:operator:/root:/sbin/nologin

$ grep -i  env ~/.bash* | grep -v apps
/home/appluat/.bash_history:ls -lrt *.env
/home/appluat/.bash_history:env | grep CONFIG
/home/appluat/.bash_history:env | grep - i secure
/home/appluat/.bash_history:env | grep -i secure
/home/appluat/.bash_profile:# User specific environment and startup programs

$ grep -i vnc ~/.bash*
/home/appluat/.bash_history:ps -fuappluat|grep vnc
/home/appluat/.bash_history:ps -ef|grep vnc
/home/appluat/.bash_history:vncserver


--grep and regular expressions:
1) line and word anchors
Here is the example of which exclusively display the lines start with "vnc"
$ grep ^vnc ~/.bash*
/home/appluat/.bash_history:vncserver

Example of the word ending with "secure"
$ grep secure$ ~/.bash*
/home/appluat/.bash_history:cd secure
/home/appluat/.bash_history:env | grep - i secure
/home/appluat/.bash_history:env | grep -i secure



 2) Character classes:
Here it lists the entries/containing  either of u,e words
$ grep [ue] ~/.bash*
/home/appluat/.bash_history:cat backup_apps_tier.sh
/home/appluat/.bash_history:sh backup_apps_tier.sh
/home/appluat/.bash_history:cat backup_apps_tier.sh

3) wild cards
Here it lists all words starting with "s" and ending in "e" from the system's dictionary

$ grep '\<s.*e\>' ~/.bash*
/home/appluat/.bash_history:cd ./appl/fnd/12.0.0/secure/
/home/appluat/.bash_history:cd secure
/home/appluat/.bash_history:env | grep - i secure
/home/appluat/.bash_history:env | grep -i secure