Search This Blog

Sunday, September 11, 2011

Issue: “Active Users” concurrent request is completed with error

Issue: “Active Users” concurrent request is completed with error

Error: ar60run: relocation error: /d03/ivdev1/oracle/ivdev1ora/8.0.6/network/jre11/lib/i686/native_threads/libjava.so: symbol __libc_wait, version GLIBC_2.0 not defined in file libc.so.6 with link time reference

Fix: add the following entry in adovars.env
LD_PRELOAD=/lib/libcwait.so
export LD_PRELOAD
Bounce Concurrent manager


Tuesday, March 22, 2011

Corrupt blocks by NOLOGGING - Error explanation and solution


This document is intended for Customers and Oracle Support.

ORA-1578 / ORA-26040 Corrupt blocks by NOLOGGING - Error explanation and solution

When a segment is defined with the NOLOGGING attribute and if a NOLOGGING/UNRECOVERABLE operation updates the segment, the online redo log file is updated with minimal information to invalidate the affected blocks when a RECOVERY is later performed.

If the associated redo/archived log file is used to RECOVER the data files, Oracle invalidates such blocks and the error ORA-26040 along with error ORA-1578 are reported by SQL statements in the next block reads.

Errors Example:
SQL> select * from test_nologging;

ORA-01578: ORACLE data block corrupted (file # 11, block # 84)
ORA-01110: data file 4: '/oradata/users.dbf'
ORA-26040: Data block was loaded using the NOLOGGING option

The NOLOGGING attribute is stored in column LOGGING in data dictionary views like:

DBA_TABLES, DBA_INDEXES, DBA_LOBS, DBA_TAB_PARTITIONS, DBA_LOB_PARTITIONS, DBA_TAB_SUBPARTITIONS, etc.

LOGGING='NO' indicates NOLOGGING.

The way for Oracle to identify that the block was previously invalidated due to NOLOGGING is by updating most of the bytes in that block with 0xff only if that "invalidate" redo is applied in a Recovery.

The block is then marked as Soft Corrupt meaning that the next block read will report the ORA-1578/ORA-26040 errors.


The SCN in the block corresponds to the SCN in the REDO RECORD for when the "INVALIDATE" change was applied in a recovery. This is useful to know the timestamp for when the block was marked as soft corrupt due to NOLOGGING.

RMAN/DBV and Corrupted Blocks by NOLOGGING

DBV prints the generic message DBV-200 in rdbms versions lower than 10.2.0.4 and error DBV-201 in RDBMS versions greater or equal to 10.2.0.4  ( Note  5031712.8 ):
DBV-00200: Block, dba 46137428, already marked corrupted
DBV-00201: Block, DBA 46137428, marked corrupt for invalid redo application

In rdbms versions lower than 10.2.0.5 RMAN reports is with a generic message like:
RMAN reports it in v$database_block_corruption with CORRUPTION_TYPE=LOGICAL
When there is a generic message besides the error ORA-26040, a block dump might be taken and see if the byte 0xff is along the block or if the block is associated to a segment, try to read it with a SQL statement for which errors ORA-1578/ORA-26040 will be produced if the block is corrupt due to a recovery with a NOLOGGING operation.

For RMAN to identify if the block is corrupt by NOLOGGING, an enhancement has been provided in Bug 7396077.  See Note 7396077.8

RMAN backups don't fail due to NOLOGGING corrupt blocks. In general RMAN does not fails with soft corrupt blocks so the MAXCORRUPT clause is not necessary in such cases.

Important change in 11g


FORCE LOGGING is irrelevant in NOARCHIVELOG mode; this was a change introduced in 11g.

Reference Note 1071869.1

SOLUTION


Note that the data inside the affected blocks is not salvageable. Methods like "Media Recovery" or "RMAN blockrecover" will not fix the problem unless the data file was backed up after the NOLOGGING operation was registered in the Redo Log.

In order to resolve the errors and if it is not an INDEX the segment can be recovered from a backup like an export dump or from another source. If backups are not available the segment might be recreated following the next steps:

  • If it is an INDEX, drop/create the index.
  • If it is a TABLE then procedure DBMS_REPAIR.SKIP_CORRUPT_BLOCKS can be used to skip the corrupt block in SQL statements and decide to re-create the table. Note 556733.1 has an example of DBMS_REPAIR. 
  • If it is a LOB segment associated to a LOB column in a Table, use Note 293515.1
  • If the error is produced in a Physical STANDBY database, the option is to restore the affected file from the PRIMARY database (only if the problem is not present in the PRIMARY).
Run script provided in Note 472231.1 to identify any additional corrupted objects.

 

References

NOTE:1071869.1 - ORA-1578 ORA-26040 in 11g for DIRECT PATH with NOARCHIVELOG even if LOGGING is enabled
NOTE:290161.1 - The Gains and Pains of Nologging Operations
NOTE:293515.1 - ORA-1578 ORA-26040 in a LOB segment - Script to solve the errors
NOTE:472231.1 - How to identify all the Corrupted Objects in the Database reported by RMAN
NOTE:556733.1 - DBMS_REPAIR SCRIPT
NOTE:7396077.8 - Bug 7396077 - RMAN does not differentiate NOLOGGING corrupt blocks that produce ORA-1578/ORA-26040
NOTE:819533.1 - How to identify the corrupt Object reported by ORA-1578 / RMAN / DBVERIFY
 ORA-1578 / ORA-26040 Corrupt blocks by NOLOGGING - Error explanation and solution [ID 794505.1]

Wednesday, March 16, 2011

Creating a Single Instance Physical Standby for a RAC Primary(Dual Node)

PREAMBLE :-

This Document gives basic information about the steps carried out to setup an Standby Database. The primary database (production DB) is a dual node RAC database. The Physical standby database is a Single Node/Single instance. The DB version on both the Primary and Target is 10.2.0.3.

The Environment:-

Production
(2 Linux Servers) With DB version 10.2.0.3.0 Real Application Cluster.
Standby Server(1 Linux Server) with the same version of DB as the Primary.
The Physical layout of the mounts/DB files/Log file etc in the standby are exact replica of the the primary instance.

1) Setup the primary Database.

a) Enabled the Force logging in the Primary Database using the below command as sysdba
SQL> ALTER DATABASE FORCE LOGGING;
Database altered.
b) Created the password file on both the RAC nodes in Primary as the Redolog files from both the servers will be transferred to the Standby.
/d02/oraprod/product/10.2.0/dbs
cd $ORACLE_HOME/dbs
orapwd file=orapwPROD1 password=PASSWORD force=y
(Repeat the same in the other Node)
c) The Standby Log is not created now as it is only during the testing of the failover.

d) Setup the Primary database Initialization parameters need for Standby.(These parameters should be added in both the nodes)

Note: the Parameter Remote_login_passwordfile should be set to exclusive.
*.workarea_size_policy='AUTO'# Required 11i setting
#Archiving
.log_archive_config='send'
*.log_archive_dest_1='location=/oracle04/oraprod/arch01'
*.log_archive_dest_2='location=/oracle03/oraprod/arch02'
.log_archive_dest_3='SERVICE=PRDN_STANDBY LGWR ASYNC REOPEN=100'
*.log_archive_dest_state_1='ENABLE'
*.log_archive_dest_state_2='ENABLE'
*.log_archive_dest_state_3='ENABLE'
*.log_archive_format='prod_%s_%t_%r.arc'
*.log_archive_max_processes=8
*.log_archive_start=TRUE
*.standby_archive_dest='/d01/oraprod/prodarc'

e) Once the new parameters needed for Standby are added bounce the instance for the parameters to take effect.

2) Standby Database steps

a) Create control file in the primary database which will be used by the standby database.

SQL> ALTER DATABASE CREATE STANDBY CONTROLFILE AS
'/data01/home/oraprod/PRODSTBY.ctl';
Database altered.
3) Backup the primary Database and copy the data files to the target server to the required mounts.

In our case i had shutdown the database and took a Tar backup in the local server and copied to the Standby server. We should copy all the archive files to the standby server which will be used for recovery.

4) Copy the initialization parameter file from the primary to the standby node and make necessary changes.

[oraprod@standby dbs]$ more initPROD.ora
PROD.__db_cache_size=2097152000
PROD.__java_pool_size=33554432
PROD.__large_pool_size=16777216
PROD.__shared_pool_size=2097152000
PROD.__streams_pool_size=33554432
*._b_tree_bitmap_plans=FALSE# Required 11i setting
*._fast_full_scan_enabled=FALSE
*._immediate_commit_propagation=TRUE
*._like_with_bind_as_equality=TRUE
*._lm_global_posts=TRUE
*._sort_elimination_cost_ratio=5
*._sqlexec_progression_cost=2147483647
*._system_trig_enabled=true
*._trace_files_public=TRUE
*.aq_tm_processes=1
*.background_dump_dest='/d02/oraprod/product/10.2.0/admin/PROD_standby/bdump'
#*.cluster_database_instances=2
#*.cluster_database=TRUE
*.compatible='10.2.0'
*.control_files='/oracle03/oraprod/data01/PRODSTBY.ctl','/oracle03/oraprod/data01/PRODSTBY1.ctl'
*.core_dump_dest='/d02/oraprod/product/10.2.0/admin/PROD1_PROD_standby/cdump'
*.cursor_sharing='EXACT'# Required 11i settting
*.db_block_checking='FALSE'
*.db_block_checksum='TRUE'
*.db_block_size=8192
*.db_file_multiblock_read_count=8# Required 11i setting
*.db_files=1024# Max. no. of database files
*.db_name='PROD'
*.dml_locks=10000
*.instance_name='PROD'
*.instance_number=1
*.job_queue_processes=2
*.log_buffer=10485760
*.log_checkpoint_interval=100000
*.log_checkpoint_timeout=1200# Checkpoint at least every 20 mins.
*.log_checkpoints_to_alert=TRUE
*.max_dump_file_size='20480'# trace file size
*.nls_comp='binary'# Required 11i setting
*.nls_date_format='DD-MON-RR'
*.nls_language='american'
*.nls_length_semantics='BYTE'# Required 11i setting
*.nls_numeric_characters='.,'
*.nls_sort='binary'# Required 11i setting
*.nls_territory='america'
*.olap_page_pool_size=4194304
*.open_cursors=600# Consumes process memory, unless using MTS.
*.optimizer_secure_view_merging=false
*.parallel_max_servers=8
*.parallel_min_servers=0
*.pga_aggregate_target=1G
*.plsql_code_type='INTERPRETED'# Default 11i setting
*.plsql_native_library_dir='/d02/oraprod/product/10.2.0/plsql/nativelib'
*.plsql_native_library_subdir_count=149
*.plsql_optimize_level=2# Required 11i setting
*.processes=800# Max. no. of users x 2
*.service_names='PROD'
*.session_cached_cursors=500
*.sessions=400# 2 X processes
*.sga_target=4G
*.shared_pool_reserved_size=200M
*.shared_pool_size=2000M
*.thread=1
*.timed_statistics=true
*.undo_management='AUTO'# Required 11i setting
*.undo_tablespace='APPS_UNDOTS1'# Required 11i setting
*.user_dump_dest='/d02/oraprod/product/10.2.0/admin/PROD_standby/udump'
*.utl_file_dir='/usr/tmp','/usr/tmp','/d02/oraprod/product/10.2.0/appsutil/outbound/PROD1_standby','/usr/tmp','/d02/pr
od/oracle/prodappl/xxdn/11.5.0/data','/d02/prod/oracle/prodappl/xxdn/11.5.0/data/inbound','/d02/prod/oracle/prodappl/xxdn/11.
5.0/data/outbound','/usr/tmp'
*.workarea_size_policy='AUTO'# Required 11i setting
#Archiving
.log_archive_config='send'
*.log_archive_dest_1='location=/oracle04/oraprod/prodarc'
*.log_archive_dest_2='SERVICE=PROD_PROD LGWR ASYNC REOPEN=100'
#.log_archive_dest_3='SERVICE=PRDN_STANDBY LGWR ASYNC REOPEN=100'
*.log_archive_dest_state_1='ENABLE'
*.log_archive_dest_state_2='defer'
#*.log_archive_dest_state_3='ENABLE'
*.log_archive_format='prod_%s_%t_%r.arc'
*.log_archive_max_processes=8
*.log_archive_start=TRUE
*.standby_archive_dest='/oracle04/oraprod/prodarc'
*.standby_file_management='AUTO'

5) Create all the required dump directories and archive log destinations mentioned in the parameter file.

6) Copy and place the standby control file created in the Primary database to the Standby server.


7) Configure the listener ,tnsnames in the standby DB to reflect the entries of the both the RAC instances.

Tnsnames.Ora entry:

PROD_PROD=
(DESCRIPTION =
(ADDRESS = (PROTOCOL = TCP)(HOST =prod1)(PORT = 1581))
(ADDRESS = (PROTOCOL = TCP)(HOST =prod2)(PORT = 1581))
(LOAD_BALANCE = yes)
(CONNECT_DATA =
(SERVICE_NAME = PROD)
(FAILOVER_MODE =
(TYPE = SELECT)
(METHOD = BASIC)
(RETRIES = 200)
(DELAY = 5)
)
)
)

Also Add the entry of the Standby Instance in the tnsnames of both the RAC nodes and test is by issuing tnsping



8) Setup the Standby instance and bring up the instance .

a) Setup the initialization parameters to point to the standby database, and bringup the instance with mount option and place it in the standby mode.


SQL> Startup mount;
SQL> ALTER DATABASE RECOVER MANAGED STANDBY DATABASE DISCONNECT FROM SESSION;
Database altered.

9) The Standby database will automatically apply all the archives and the current logs will applied. Check the Alert logs and confirm that the standby is in Sync.

Basic Daily DBA Checklist

These are the some basic DBA daily check list.
1) Daily Procedures:
Many of these procedures can be automated using shell scripts and a scheduler like cron or even using tools such as Oracle Enterprise Manager (OEM).
2) Verify all instances availability:
Make sure that all databases are available. Log into each instance and run daily reports or test scripts.
3 ) Look for any new alert log entries:
Each Oracle Instance has a background dump destination, usually in $ORACLE_BASE/ORACLE_SID/bdump. The alert log file (alert_ORACLE_SID.log) is in this location and should be checked for errors each day.
4) Verify success of database backup:
Keep a schedule of database backups and review logs to assure backups occurred successfully.
5) Verify enough resources for acceptable performance:
Keep a record of acceptable minimum amount / percentage free in tablespaces and compare it daily to reports.
6)Review contention for CPU, memory, network or disk resources:
Work with system administrators to develop methods to review contention for these resources.
7)verify the DR sync (if applicable):
Check to determine if latest archive logs have been applied to the database by looking at the alert log file for the standby(DR) database.
8)Maintan the monitoring scripts:
DBA can maintain the above listed monitoring scripts scheduled in crontab

Saturday, March 12, 2011

approach to get analysis on high CPU loads

1. Since the CPU usage was high, this query will show up in the “top session” field ---please check it immediately otherwise the you may lose the DDL.
2. If you lose the DDL, run AWR snapshot during the interval this “high CPU” was happening and look for the “offending query”.
3. From that line, get the “hash value” and run the following query:
a. select child_number,
b. io_cost,
c. lpad(' ',2 * (depth),'| ') || operation
d. || decode(options,
e. null, null, ' (' || initcap(options) || ')')
f. || ' ' || object_owner ||'.'|| object_name op
g. from v$sql_plan
h. where hash_value = &hash_value
i. order by child_number, id;
4. You will get the plan based on this.
5. If the previous two options doesn’t help, in this case, since you know the concurrent program name, run it in a development environment with trace on (from the concurrent program form) and extract the top DDL based on tkprof option (sort='(prsela,exeela,fchela)')
6. Use this DDL to generate explain plan in DEV and compare it in PROD.

Usually, whenever there is a high CPU usage, there is a “enqueue” contention or a bad index (fragmented or needs an updated stats or index rebuild).

TNS-12518: TNS:listener could not hand off client connection

The listener was went down at when load was high and it didn't turn on automatically (my environment is RAC)

TNS-12518: TNS:listener could not hand off client connection
TNS-12571: TNS:packet writer failure
TNS-12560: TNS:protocol adapter error
TNS-00530: Protocol adapter error
Linux Error: 104: Connection reset by peer

[oracle@myoracle admin]$ cat /var/log/messages
cat: /var/log/messages: Permission denied

as per “Oracle” issue seems to be due to “Extensive paging/swapping activity is a clear indication that the sytem is running out of the physical memory” and solutions offered as below

1. Increase the physical memory of the system or
[oracle@myoracle OPatch]$ free
total used free shared buffers cached
Mem: 32960108 31415136 1544972 0 870980 21807352
-/+ buffers/cache: 8736804 24223304
Swap: 16777200 366380 16410820

2. Apply the Patch 6139856 for unpublished Bug 6139856 if available for your platform or
[oracle@myoracle OPatch]$ ./opatch lsinventory | grep 6139856
[oracle@myoracle OPatch]$

3. Configure Hugepages on the OS. Ref : Note 361323.1
[oracle@myoracle OPatch]$ grep Hugepagesize /proc/meminfo
Hugepagesize: 2048 kB

Ref: TNS Listener Crash with Core dump [ID 549932.1]
HugePages on Linux: What It Is... and What It Is Not... [ID 361323.1]

Friday, March 11, 2011

how to check flashback enabled or not

SQL> select flashback_on from v$database;
FLASHBACK_ON
------------------
YES

To make flashback off
sql>ALTER DATABASE FLASHBACK OFF;
To make flashback on
sql>shut immediate
sql>startup mount exclusive
sql> alter database flashback on;


Flashback description:
Space issue in Flash Recovery Area( FRA )

"Flash Recovery Area" is used to set up a disk area where the database can create and manage a variety of files related to backup and recovery. Use of the flash recovery area is strongly recommended. Consider configuring a flash recovery area as one of the first steps in implementing a backup strategy. The flash recovery area is an Oracle-managed directory, file system, or Automatic Storage Management disk group.

* Oracle creates archived logs and flashback logs in the flash recovery area.
* RMAN can store its backup sets and image copies in the flash recovery area
Planning Space Usage and Location for the Flash Recovery Area

The amount of disk space to allocate for the flash recovery area depends on the size and activity levels on the database, which determine the size of datafiles and redo logs files in addition to the recovery objectives. The flash recovery area should be large enough for copies of the datafiles, control files, online redo logs, and archived logs needed to recover the database, and also the copies of these backup files that are kept based on the retention policy.

Flash recovery area should be on a separate disk .
How Oracle Manages Disk Space in the Flash Recovery Area ?

Space in the flash recovery area is balanced among backups and archived logs that must be kept according to the retention policy, and other files which may be subject to deletion. Oracle Database does not delete eligible files from the flash recovery area until the space must be reclaimed for some other purpose. Thus, files recently moved to tape are often still available on disk for use in recovery. The recovery area can thus serve as a cache for tape. When the flash recovery area is full, Oracle Database automatically deletes eligible files to reclaim space in the recovery area as needed.

If the RMAN retention policy requires keeping a set of backups larger than the flash recovery area disk quota, or if the retention policy is set to NONE, then the flash recovery area can fill completely with no reclaimable space.

How Oracle Notify space crunch in FRA ?

The database issues a warning alert when reclaimable space is less than 15% and a critical alert when reclaimable space is less than 3%. To warn the DBA of this condition, an entry is added to the alert log and to the DBA_OUTSTANDING_ALERTS table (used by Enterprise Manager). Nevertheless, the database continues to consume space in the flash recovery area until there is no reclaimable space left.

When the recovery area is completely full, the error you will receive is as follows, where nnnnn is the number of bytes required and mmmm is the disk quota:

ORA-19809: limit exceeded for recovery files
ORA-19804: cannot reclaim nnnnn bytes disk space from mmmmm limit
Deletion Rules for the Flash Recovery Area

The following rules govern when files become eligible for deletion from the recovery area:

- Permanent files are never eligible for deletion.

- Files that are obsolete under the retention policy are eligible for deletion.

- Transient files that have been copied to tape are eligible for deletion.

- Archived redo logs are not eligible for deletion until all the consumers of the logs have satisfied their requirements ( If Archived Redo Log Deletion Policy is set ) .

- Foreign archived logs that have been mined by a LogMiner session on a logical standby database are eligible for deletion. Unlike an ordinary archived redo log, a foreign archived redo log has a different DBID.

The safe and reliable way to control deletion of files from the flash recovery area is to configure your retention policy ("Configuring the Backup Retention Policy") and archived log deletion policy ("Configuring an Archived Redo Log Deletion Policy"). To increase the likelihood that files moved to tape are retained on disk, increase the flash recovery area quota.

Requirement :

Configure RMAN retention policy
Configure archived log deletion policy

If retention policy is set to none then no files will be eligible for deletion. If files are never considered obsolete, then a file can only be deleted from the flash recovery area if it has been backed up to some other disk location or to a tertiary storage device such as tape.

How to Monitor Flash Recovery Area Space Usage ?

V$RECOVERY_FILE_DEST : To find out the current location, disk quota, space in use, space reclaimable by deleting files, and total number of files in the flash recovery area.

V$FLASH_RECOVERY_AREA_USAGE : To find out the percentage of the total disk quota used by different types of files. Also, you can determine how much space for each type of file can be reclaimed by deleting files that are obsolete, redundant, or already backed up to tape.

For example
SQL>Select file_type, percent_space_used as used,percent_space_reclaimable as reclaimable,
number_of_files as "number" from v$flash_recovery_area_usage;

FILE_TYPE USED RECLAIMABLE number
------------ ---------- ----------- ----------
CONTROLFILE 0 0 0
ONLINELOG 0 0 0
ARCHIVELOG 4.77 0 2
BACKUPPIECE 56.80 0 10
IMAGECOPY 0 0 0
FLASHBACKLOG 11.68 11.49 63

SQL> select name, space_limit as Total_size ,space_used as Used,
SPACE_RECLAIMABLE as reclaimable ,NUMBER_OF_FILES as "number"
from V$RECOVERY_FILE_DEST;

NAME TOTAL_SIZE USED RECLAIMABLE Number
------------------------ ---------- ---------- ----------- --------
E:\oracle\flash_recovery_area 2147483648 353280000 246841344 75


How to resolve a full flash recovery area when no files are eligible for deletion ?

Files in Flash Recovery Area are auto managed . Whenever there is a space pressure oracle automatically delete files which are out of retention policy. But if no files are eligible for deletion then manual intervention required .

You have a number of choices on how to resolve a full flash recovery area when no files are eligible for deletion:

1. Make more disk space available and increase DB_RECOVERY_FILE_DEST_SIZE
SQL> alter system set db_recovery_file_dest_size=xG SCOPE=BOTH; -- (larger amount)


2. Move backups from the flash recovery area to tertiary storage such as tape.
RMAN>BACKUP RECOVERY AREA;

Note : Flashback logs cannot be backed up outside the recovery area and so are not backed up by BACKUP RECOVERY AREA.

3. Run DELETE for any files that have been removed with an operating system utility. If you use host operating system commands to delete files, then the database will not be aware of the resulting free space.
RMAN>CROSSCHECK BACKUP;
RMAN>CROSSCHECK ARCHIVELOG ALL;

RMAN>Delete expired backup;
RMAN>Delete expired archivelog all;
RMAN>Delete force obsolete;

4. Make sure that your guaranteed restore points are necessary. If not, delete them
SQL>Drop restore point ;

5. If flashback logs are enable then make sure you have enough space for all the flashback logs. If its not required then you can turn off flashback.
SQL>Alter database FLASHBACK OFF;

6. Review your backup retention policy and if required change the RMAN RETENTION POLICY
RMAN>CONFIGURE RETENTION POLICY TO RECOVERY WINDOW OF 7 DAYS;
Exceptions :

- If RMAN is not part of backup strategy and archivelogs are going to FRA then manual intervention required for deletion of archivelogs. Periodically purse old archivelogs

for example
RMAN>Delete archivelog all completed before 'SYSDATE-7';

- By default RMAN backup goes to FRA. While taking RMAN backup if backup location explictly specified to flash recovery area location then those backup pieces are not considered as part of FRA for auto managment.

- For Archivelogs backup to FRA use USE_DB_RECOVERY_FILE_DEST rather than giving explict path of FRA
SQL> alter system set log_archive_dest_10='LOCATION=USE_DB_RECOVERY_FILE_DEST' scope=both;

Transportable tablespace refresh

  1.check tablespace for the user which need to refresh -------------------------------------------------------------------  SQL> select ...