11g New Features - Incident packaging using "adrci" utility

As an Oracle DBA, you interface with Oracle support many a times; specially when you are faced with some critical errors such ORA-600 or ORA-7445 etc. And each time you interact with Oracle support, you have to provide quite few logs (alert log, trace files etc.) so that support can assist you in resolving the problem. At times the SR goes back and forth just to additional information - such as providing additional trace files etc. since you may have missed it earlier.

But starting 11g, your life will become a little easy thanks to a new utility called "ADRCI" - ADR Command Interpreter. Now you don't have to dig around trace files and other files to determine the files that Support would require for analysis. You can simply use "adrci" to package all the files that would be required for analysis with a few simple commands.

Apart from the packaging ability, one can use adrci to :-

a. view alert log (show alert)
b. view other trace files
c. view health check reports

Viewing alert log and trace files is easy but as I said, the best use of this utility is to package incident / problems encountered so that can the packaged file can be easily be uploaded to Oracle support when needed. You do not have to search through trace files and other files to determine the files that are required for analysis.

Here is a quick demonstration of how useful this utility can be when you have to upload files ( required to analyze a specific problem) to Oracle support :-

1. Launch adrci

$ adrci

ADRCI: Release 11.1.0.6.0 - Beta on Mon Apr 7 16:11:06 2008

Copyright (c) 1982, 2007, Oracle. All rights reserved.

ADR base = "/app/oracle"

2. Check the incidents reported in alert log

adrci> show incident

ADR Home = /app/oracle/diag/rdbms/11GLAB:
*************************************************************************
INCIDENT_ID PROBLEM_KEY CREATE_TIME
-------------------- ----------------------------------------------------------- ----------------------------------------
21746 ORA 4031 2008-04-07 16:57:11.039525 +05:30
21745 ORA 4031 2008-04-07 16:57:00.356082 +05:30
21715 ORA 4031 2008-04-07 16:57:16.796655 +05:30
21714 ORA 4031 2008-04-07 16:57:07.883365 +05:30
21713 ORA 4031 2008-04-07 16:57:00.694116 +05:30
5 rows fetched

3. Identify the specific incident for which you want to create a package so that you can upload it to Oracle support

adrci> IPS CREATE PACKAGE INCIDENT 21713
Created package 2 based on incident id 21713, correlation level typical

This creates a pacakge of the incident 21713 in "incpkg" directory; you can then add diagnostic data to the package.

4. Finally generate the package for the incident, which then can be uploaded to metalink while seeking support from Oracle.

adrci> ips generate package 3
Generated package 3 in file /app/oracle/product/11.1.0/db_1/dbs/ORA4031_20080407170431_COM_1.zip, mode complete

5. As reported above a zip file has been created with all relevant logs. Now you can upload this zip file to Oracle support and seek their help in resolving your problem.

You can also view the information generated in "incpkg" directory

$cd /app/oracle/diag/rdbms/11GLAB/incpkg/pkg_3/seq_1
$ ls -ltr
-rw-r----- 1 oracle oinstall 499 Apr 7 17:14 metadata.xml
-rw-r----- 1 oracle oinstall 21968 Apr 7 17:14 manifest_3_1.xml
-rw-r----- 1 oracle oinstall 26270 Apr 7 17:14 manifest_3_1.txt
-rw-r----- 1 oracle oinstall 20064 Apr 7 17:14 manifest_3_1.html
drwxr-xr-x 2 oracle oinstall 4096 Apr 7 17:14 export
drwxr-xr-x 2 oracle oinstall 4096 Apr 7 17:14 crs
-rw-r----- 1 oracle oinstall 62789 Apr 7 17:14 config.xml

We used to achieve the same earlier by running some scripts or collecting the logs/trace files manually. But with adrci, this task is pretty simplified; I think this is surely going to reduce the time to diagnose and resolve any problem.

References :- Oracle® Database Administrator's Guide 11g Release 1 (11.1) Part Number B28310-04
Oracle® Database Utilities 11g Release 1 (11.1) Part Number B28319-02 - ADRCI: ADR Command Interpreter

11g New Features - ADR

This is a big change in 11g. Now you will not find "alertSID.log" in it's familiar location i.e admin/SID/bdump. It has now got a new location; In fact, Oracle has standardized to store all logs/trace at a single base directory instead of scattering them all over. It's called "Automatic Diagnostic Repository" or "ADR". The parameters "background_dump_dest" and "user_dump_dest" are now deprecated and being replaced by a new parameter called "DIAGNOSTIC_DEST". What more - the alert.log is also available in an xml; Don't worry the text version will still be available.

So from now onwards, you should use "diagnostic_dest" parameter to specify the destination for logs/traces. Oracle creates a sub-directory called "diag" beneath "diagnostic_dest" directory where it will keep all logs/traces. When I say all logs, it is all logs/trace files for all its products/tools including ASM, CRS, Listener etc. This is very great move specifically when CRS stack etc are used; the CRS logs are scattered all over the place.

Here is the new directory structure where you can find all log and trace files :-

"diagnostic_dest\diag\product_name\instance_name\"

Note: I have just expanded and explained key folders in rdbms sub-directory which stores all log and trace files for Oracle database.

diag
 rdbms
     SID
         sweep -->
         stage -->
         incpkg -->
         incident --> incident dump files
         hm --> health monitor files
         cdump --> core_dump_dest
         metadata --> incident metadata files
         lck -->
         ir -->
         alert --> location for the alert log in xml format
         trace --> location for all trace files and text version of alert.log 
 ofm
 netcman
 lsnrctl
 diagtool
 crs
 asm
 tnslsnr
 clients

Some other major improvements are :-

• Now each critical errors such as ORA-600, ORA-7445 etc. are treated as incidents. Tags the logs with incident id's; Incident alerts are sent to EM and makes entry into alert.log as usual.
• Incident Packaging service :- IPS enables you to automatically and easily gather the diagnostic data—traces, dumps, health check reports, and more—pertaining to a critical error and package the data into a zip file for transmission to Oracle Support. Awesome indeed.
• Automatic capture of diagnostic data upon failure
• Health checks on detecting a critical error
• V$DIAG_INFO is a new view which lists all ADR locations for the database instance
• There is a new column called "TRACEFILE" in V$PROCESS which specifies the trace file name for each process.
  

Above all, 11g comes with a new command line utility called "adrci" to view alert/trace files, package incident trace files so that uploading to oracle support is easy, view health check reports etc. I will explain about this tool in my next post. So all said, the ADR is a great improvement, a great step forward to standardize diagnostic information.

References :- Oracle® Database Administrator's Guide 11g Release 1 (11.1) Part Number B28310-04