Friday, December 19, 2014

Resolving Gaps in DataGuard Physical Standby DB Using RMAN Incremental Backup



In case of an archive log gone missing or corrupt, We have to bring back the standby to sync with the primary.

When the logs missing or corrupt is less in number (say below 15),

 we can ship the logs which were missing in the standby site from the primary site (scp/sftp/ftp)
and then we can register the log file in the standby so that the gap can be resolved.

Find the archives which are missing by issueing the following command.

SQL> select * from v$archive_gap;

This would give the gap sequences. Or you can use the v$managed_standby view to find where the log apply stuck.
SQL> select sequence#,process,status from v$managed_standby;

Here u can see status as wait for log for say sequence# 100 but your primary would've proceeded to sequence# 110
At primary
SQL> select max(sequence#) from v$archived_log;      ---> This would show you 110

Copy the logs to the standby site from the primary site

$ scp log_file_name_n.arc oracle@standby:/log/file/location/log_file_name_n.arc

At standby site
SQL> alter database register logfile '/log/file/location/log_file_name_n.arc';
logfile registered

Do the log file registration at the standby site until all the missing log files are registered.
 Now apply would take place and your standby will become sync with the primary.

This is easy process if you have missing or corrupt logs in lesser number.
 But when the difference is huge (say around 500 logs) this method is very time consuming and not a proper approach.  Else you have to rebuild the standby database from scratch.


As an enhancement from 10g, an incremental backup created with BACKUP INCREMENTAL... FROM SCN can be used to refresh the standby database with changes at the primary database since the last SCN at Standby  and then managed recovery can resume i.e. Compensate for the missing archive logs.


 Step 1: On the primary:
SQL> select current_scn from v$database;
CURRENT_SCN
-----------
144710998

On the standby:
SQL> select current_scn from v$database;
CURRENT_SCN
-----------
130158742

Clearly there is a difference. But this by itself does not indicate a problem;
since the standby is expected to lag behind the primary (this is an asynchronous non-real time apply setup).
 The real question is how much it is lagging in the terms of wall clock.
To know that use the scn_to_timestamp function to translate the SCN to a timestamp:

SQL> select scn_to_timestamp(144710998) from dual;
SCN_TO_TIMESTAMP(1447102)
-------------------------------
18-AUG-11 08.54.28.000000000 AM

Run the same query to know the timestamp associated with the SCN of the standby database as well
SQL> select scn_to_timestamp(130158742) from dual;
SCN_TO_TIMESTAMP(1301571)
-------------------------------
13-AUG-11 07.19.27.000000000 PM

Note: Run it on the primary database, since it will fail in the standby in a mounted mode
This shows that the standby is four and half days lagging!

Step 2: [Standby] Stop the managed standby apply process:
SQL> alter database recover managed standby database cancel;
Database altered.

Step 3: [Standby] Shutdown the standby database
SQL> shutdown immediate

Step 4: [Primary] On the primary, take an incremental backup from the SCN number where the standby has been stuck:
RMAN> run {
 allocate channel c1 type disk format '/u01/app/oracle/rmanbackup1/bkup/%U.bkp';
 backup incremental from scn 10574600 database;
 }

Step 5: [Primary] On the primary, create a new standby controlfile:
SQL> alter database create standby controlfile as '/u01/app/oracle/rmanbackup1/bkup/for_standby.ctl';
Database altered.

(NB: First Step is tested)
or

RMAN> backup current controlfile for standby format '/backups/tempfol/_%U';

Step 6: [Primary] Copy these files to standby host:
oracle@dba1 /u01/backup]$ scp * oracle@dba2:/u01/backup

Step 7: [Standby]
Bring up the instance in nomount mode:
SQL> startup nomount

Step 8: [Standby] Check the location of the controlfile:
SQL> show parameter control_files
NAME TYPE VALUE
------------------------------------ ----------- ------------------------------
control_files string /u01/oradata/standby_cntfile.ctl

Step 9: [Standby] Replace the controlfile with the one you just created in primary.
 $ cp /u01/backup/for_standby.ctl /u01/oradata/standby_cntfile.ctl

or

RMAN> RESTORE STANDBY CONTROLFILE FROM '/u01/app/oracle/oradata/backup/for_standby.ctl';
(NB: this step is tested)
Step 10: [Standby] Mount the standby database:
SQL> alter database mount standby database;

Step 11: [Standby] Connect to RMAN. RMAN does not know about these files yet; so you must let it know – by a process called cataloging. Catalog these files:
$ rman target=/
RMAN> catalog start with '/u01/app/oracle/oradata/backup';

Step 12: Recover these files:
RMAN> recover database noredo;

Step 13: After some time, the recovery fails with the message:

archive log filename=/u01/oradata/1_18108_697108460.dbf thread=1 sequence=18109
ORA-00310: archived log contains sequence 18108; sequence 18109 required

This happens because we have come to the last of the archived logs. The expected archived log with sequence# 18108 has not been generated yet.

Step 14: At this point exit RMAN and start managed recovery process:

SQL> alter database open;
Database altered.
SQL> alter database recover managed standby database disconnect from session;
Database altered.

Step 15: Check the SCN’s in primary and standby:
[Standby] SQL> select current_scn from v$database;
CURRENT_SCN
-----------
144747125
[Primary] SQL> select current_scn from v$database;
CURRENT_SCN
-----------
144747111

Step 16:
Initiate the transfer of archived redo log files on the primary database
SQL>alter system switch logfile;

Wednesday, December 10, 2014

Managing Automatic Workload Repository (AWR)

To Generate the AWR report run the following query:
1)
SELECT
   output
FROM  
   TABLE
   (dbms_workload_repository.awr_report_text
      (1226709381,1,15276,15277 )
   );


2)

Save the data into a flat file


Another way to generate the Report

1)
BEGIN
DBMS_WORKLOAD_REPOSITORY.CREATE_SNAPSHOT ();
END;
/

2)
SQL> sqlplus / as sysdba

SQL> @$ORACLE_HOME/rdbms/admin/awrrpt.sql

First, you need to specify whether you want an HTML or a text report.

the

Enter value for report_type:

Specify the number of days for which you want to list snapshot Ids.

Enter value for num_days: 1

After the list displays, you are prompted for the beginning and ending snapshot Id for the workload repository report.

Enter value for begin_snap: 95
Enter value for end_snap: 97

Next, accept the default report name or enter a report name. The default name is accepted in the following example:

Enter value for report_name:
Using the report name awrrpt_1_95_97

The workload repository report is generated.

Oracle 11g Dataguard_Broker Setup Guide

Last login: Mon Oct 28 12:31:04 2013 from 192.168.0.47
 
[oracle@report ~]$ sqlplus / as sysdba

SQL*Plus: Release 11.2.0.3.0 Production on Mon Oct 28 15:49:10 2013

Copyright (c) 1982, 2011, Oracle.  All rights reserved.


Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 - 64bit Production
With the Partitioning, OLAP, Data Mining and Real Application Testing options

SQL> show parameter dg_broker_start

NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
dg_broker_start                      boolean     FALSE
SQL> ALTER SYSTEM SET DG_BROKER_START=TRUE SCOPE=BOTH;

System altered.

SQL> show parameter dg_broker_start

NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
dg_broker_start                      boolean     TRUE
SQL> !dgmgrl
DGMGRL for Linux: Version 11.2.0.3.0 - 64bit Production

Copyright (c) 2000, 2009, Oracle. All rights reserved.

Welcome to DGMGRL, type "help" for information.
DGMGRL> connect sys/sys123
Connected.
DGMGRL> show configuration
ORA-16532: Data Guard broker configuration does not exist

Configuration details cannot be determined by DGMGRL
DGMGRL> DGMGRL>
SQL> !dgmgrl
DGMGRL for Linux: Version 11.2.0.3.0 - 64bit Production

Copyright (c) 2000, 2009, Oracle. All rights reserved.

Welcome to DGMGRL, type "help" for information.
DGMGRL> !dgmgrl
Unrecognized command "!dgmgrl", try "help"
DGMGRL>
DGMGRL>
DGMGRL>
DGMGRL>
DGMGRL>
DGMGRL> create configuration 'primDR' as primary database is 'prim' connect identified is 'prim';
create configuration 'primDR' as primary database is 'prim' connect identified is 'prim';
                                                                    ^
Syntax error before or at "identified"
DGMGRL> DGMGRL>
SQL>  create configuration 'primDR' as primary database is 'prim' connect identifier is 'prim';
 create configuration 'primDR' as primary database is 'prim' connect identifier is 'prim'
        *
ERROR at line 1:
ORA-00901: invalid CREATE command


SQL> dgmgrl
SP2-0042: unknown command "dgmgrl" - rest of line ignored.
SQL> !dgmgrl
DGMGRL for Linux: Version 11.2.0.3.0 - 64bit Production

Copyright (c) 2000, 2009, Oracle. All rights reserved.

Welcome to DGMGRL, type "help" for information.
DGMGRL>  create configuration 'primDR' as primary database is 'prim' connect identifier is 'prim';
not logged on
DGMGRL> connect sys/sys123
Connected.
DGMGRL>  create configuration 'primDR' as primary database is 'prim' connect identifier is 'prim';
Configuration "primDR" created with primary database "prim"
DGMGRL>
DGMGRL> add database 'stan' as connect identifier is 'stan';
Database "stan" added
DGMGRL> show configuration

Configuration - primDR

  Protection Mode: MaxPerformance
  Databases:
    prim - Primary database
    stan - Physical standby database

Fast-Start Failover: DISABLED

Configuration Status:
DISABLED

DGMGRL> enable configuration
Enabled.
DGMGRL> show configuration

Configuration - primDR

  Protection Mode: MaxPerformance
  Databases:
    prim - Primary database
    stan - Physical standby database
      Error: ORA-16525: the Data Guard broker is not yet available

Fast-Start Failover: DISABLED

Configuration Status:
ERROR

DGMGRL> show database prim

Database - prim

  Role:            PRIMARY
  Intended State:  TRANSPORT-ON
  Instance(s):
    prim

Database Status:
SUCCESS

DGMGRL> show database stan

Database - stan

  Role:            PHYSICAL STANDBY
  Intended State:  APPLY-ON
  Transport Lag:   (unknown)
  Apply Lag:       (unknown)
  Real Time Query: OFF
  Instance(s):
    stan

Database Status:
DGM-17016: failed to retrieve status for database "stan"
ORA-16525: the Data Guard broker is not yet available
ORA-16625: cannot reach database "stan"

DGMGRL> SHOW CONFIGURATION

Configuration - primDR

  Protection Mode: MaxPerformance
  Databases:
    prim - Primary database
    stan - Physical standby database
      Error: ORA-16532: Data Guard broker configuration does not exist

Fast-Start Failover: DISABLED

Configuration Status:
ERROR

DGMGRL> DGMGRL>
SQL> dgmgrl
SP2-0042: unknown command "dgmgrl" - rest of line ignored.
SQL> !dgmgrl
DGMGRL for Linux: Version 11.2.0.3.0 - 64bit Production

Copyright (c) 2000, 2009, Oracle. All rights reserved.

Welcome to DGMGRL, type "help" for information.
DGMGRL> SHOW CONFIGURATION
not logged on
DGMGRL> connect sys/sys123
Connected.
DGMGRL> SHOW CONFIGURATION

Configuration - primDR

  Protection Mode: MaxPerformance
  Databases:
    prim - Primary database
    stan - Physical standby database

Fast-Start Failover: DISABLED

Configuration Status:
SUCCESS

DGMGRL> show database prim

Database - prim

  Role:            PRIMARY
  Intended State:  TRANSPORT-ON
  Instance(s):
    prim

Database Status:
SUCCESS

DGMGRL> show database stan

Database - stan

  Role:            PHYSICAL STANDBY
  Intended State:  APPLY-ON
  Transport Lag:   0 seconds
  Apply Lag:       0 seconds
  Real Time Query: ON
  Instance(s):
    stan

Database Status:
SUCCESS

DGMGRL> show database verbose stan

Database - stan

  Role:            PHYSICAL STANDBY
  Intended State:  APPLY-ON
  Transport Lag:   0 seconds
  Apply Lag:       0 seconds
  Real Time Query: ON
  Instance(s):
    stan

  Properties:
    DGConnectIdentifier             = 'stan'
    ObserverConnectIdentifier       = ''
    LogXptMode                      = 'ASYNC'
    DelayMins                       = '0'
    Binding                         = 'OPTIONAL'
    MaxFailure                      = '0'
    MaxConnections                  = '1'
    ReopenSecs                      = '300'
    NetTimeout                      = '30'
    RedoCompression                 = 'DISABLE'
    LogShipping                     = 'ON'
    PreferredApplyInstance          = ''
    ApplyInstanceTimeout            = '0'
    ApplyParallel                   = 'AUTO'
    StandbyFileManagement           = 'AUTO'
    ArchiveLagTarget                = '0'
    LogArchiveMaxProcesses          = '5'
    LogArchiveMinSucceedDest        = '1'
    DbFileNameConvert               = '/u01/app/oracle/oradata/prim/, /u01/app/oracle/oradata/stan/, /u01/app/oracle/oradata/prim/, /u01/app/oracle/oradata/stan/'
    LogFileNameConvert              = '/u01/app/oracle/oradata/prim/, /u01/app/oracle/oradata/stan/'
    FastStartFailoverTarget         = ''
    InconsistentProperties          = '(monitor)'
    InconsistentLogXptProps         = '(monitor)'
    SendQEntries                    = '(monitor)'
    LogXptStatus                    = '(monitor)'
    RecvQEntries                    = '(monitor)'
    SidName                         = 'stan'
    StaticConnectIdentifier         = '(DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=mobile)(PORT=1521))(CONNECT_DATA=(SERVICE_NAME=stan_DGMGRL)(INSTANCE_NAME=stan)(SERVER=DEDICATED)))'
    StandbyArchiveLocation          = '/u01/app/oracle/oradata/stan/arch'
    AlternateLocation               = ''
    LogArchiveTrace                 = '0'
    LogArchiveFormat                = '%t_%s_%r.dbf'
    TopWaitEvents                   = '(monitor)'

Database Status:
SUCCESS

DGMGRL> show database prim statusreport
STATUS REPORT
       INSTANCE_NAME   SEVERITY ERROR_TEXT

DGMGRL> show database prim logxptstatus
LOG TRANSPORT STATUS
PRIMARY_INSTANCE_NAME STANDBY_DATABASE_NAME               STATUS
                prim                 stan                     

DGMGRL> show database prim InconsistentProperties
INCONSISTENT PROPERTIES
   INSTANCE_NAME        PROPERTY_NAME         MEMORY_VALUE         SPFILE_VALUE         BROKER_VALUE

DGMGRL> help

The following commands are available:

add            Adds a standby database to the broker configuration
connect        Connects to an Oracle database instance
convert        Converts a database from one type to another
create         Creates a broker configuration
disable        Disables a configuration, a database, or fast-start failover
edit           Edits a configuration, database, or instance
enable         Enables a configuration, a database, or fast-start failover
exit           Exits the program
failover       Changes a standby database to be the primary database
help           Displays description and syntax for a command
quit           Exits the program
reinstate      Changes a database marked for reinstatement into a viable standby
rem            Comment to be ignored by DGMGRL
remove         Removes a configuration, database, or instance
show           Displays information about a configuration, database, or instance
shutdown       Shuts down a currently running Oracle database instance
sql            Executes a SQL statement
start          Starts the fast-start failover observer
startup        Starts an Oracle database instance
stop           Stops the fast-start failover observer
switchover     Switches roles between a primary and standby database

Use "help <command>" to see syntax for individual commands

DGMGRL> help switchover

Switches roles between a primary and standby database

Syntax:

  SWITCHOVER TO <standby database name>;

DGMGRL> add database 'stan2' connect identifier is 'stan2' MAINTAINED AS PHYSICAL;
add database 'stan2' connect identifier is 'stan2' MAINTAINED AS PHYSICAL;
                     ^
Syntax error before or at "connect"
DGMGRL> add database as 'stan2' connect identifier is 'stan2' MAINTAINED AS PHYSICAL;
add database as 'stan2' connect identifier is 'stan2' MAINTAINED AS PHYSICAL;
                ^
Syntax error before or at "stan2"
DGMGRL> add database as 'stan2' connect identifier is 'stan2';^H^H^H
> ;
add database as 'stan2' connect identifier is 'stan2';
                ^
Syntax error before or at "stan2"
DGMGRL>
DGMGRL> add database 'stan2' as connect identifier is 'stan2' maintained as physical;
Database "stan2" added
DGMGRL> show configuration

Configuration - primDR

  Protection Mode: MaxPerformance
  Databases:
    prim  - Primary database
    stan  - Physical standby database
    stan2 - Physical standby database (disabled)

Fast-Start Failover: DISABLED

Configuration Status:
SUCCESS

DGMGRL> show database stan2

Database - stan2

  Role:            PHYSICAL STANDBY
  Intended State:  OFFLINE
  Transport Lag:   (unknown)
  Apply Lag:       (unknown)
  Real Time Query: OFF
  Instance(s):
    stan2

Database Status:
DISABLED

DGMGRL> show database stan

Database - stan

  Role:            PHYSICAL STANDBY
  Intended State:  APPLY-ON
  Transport Lag:   0 seconds
  Apply Lag:       0 seconds
  Real Time Query: ON
  Instance(s):
    stan

Database Status:
SUCCESS

DGMGRL> enable configuration
Enabled.
DGMGRL> show configuration

Configuration - primDR

  Protection Mode: MaxPerformance
  Databases:
    prim  - Primary database
    stan  - Physical standby database
    stan2 - Physical standby database

Fast-Start Failover: DISABLED

Configuration Status:
SUCCESS

DGMGRL> show database stan2

Database - stan2

  Role:            PHYSICAL STANDBY
  Intended State:  APPLY-ON
  Transport Lag:   0 seconds
  Apply Lag:       0 seconds
  Real Time Query: OFF
  Instance(s):
    stan2

Database Status:
SUCCESS

DGMGRL> show configuration

Configuration - primDR

  Protection Mode: MaxPerformance
  Databases:
    prim  - Primary database
    stan  - Physical standby database
    stan2 - Physical standby database

Fast-Start Failover: DISABLED

Configuration Status:
SUCCESS

DGMGRL> show database stan2

Database - stan2

  Role:            PHYSICAL STANDBY
  Intended State:  APPLY-ON
  Transport Lag:   0 seconds
  Apply Lag:       0 seconds
  Real Time Query: OFF
  Instance(s):
    stan2

Database Status:
SUCCESS

DGMGRL> show database stan

Database - stan

  Role:            PHYSICAL STANDBY
  Intended State:  APPLY-ON
  Transport Lag:   0 seconds
  Apply Lag:       0 seconds
  Real Time Query: ON
  Instance(s):
    stan

Database Status:
SUCCESS

DGMGRL> show database prim

Database - prim

  Role:            PRIMARY
  Intended State:  TRANSPORT-ON
  Instance(s):
    prim

Database Status:
SUCCESS

DGMGRL> help switchover

Switches roles between a primary and standby database

Syntax:

  SWITCHOVER TO <standby database name>;